LL'13 E-commerce Challenge

As part of the Living Labs for Information Retrieval Evaluation (LL'13) workshop, we are considering organising a challenge in the e-commerce domain with the involvement of a medium-sized online retailer. The goal of this challenge would be to (i) allow academics to work with real users and data (esp. those who otherwise would have no access to such data) and (ii) to provide a starting point for the discussions at the workshop. Keep in mind that the focus should be on the living labs methodology; while we would be comparing different systems on some product search or recommendation task, the ultimate goal is to learn about practical and technical challenges associated with living labs architecture and design.

Although we are given a lot of freedom to figure out what we really want to do, there are some constraints given that this experiment would involve an actual company. First and foremost, data cannot leave the walls of the organisation. One possible solution would be to ship the code to the organisation, i.e., participants would need to submit their code. Second, this experiment can under no circumstances hinder the normal daily operations of this e-commerce site. We won't have the resources and capacity to rigorously check all submitted code so that it could be integrated into a live system. It would be possible, however, to run participants' code on previously collected data. Also, for head queries, it would be possible to compute results offline which would then be shown on actual search result pages (using some sort of interleaving) that are displayed to users. There is likely to be some limitations on the platform and tools that can be used.

As you probably have understood by now, at this point everything is still in flux and subject to discussion. Our aim with this questionnaire is to get an initial assessment regarding whether there is sufficient interest to warrant the effort of organising the challenge and to get an understanding of what the community thinks our focus should be. Should we decide to go forward with the challenge, we will most likely be needing several iterations to have all the details finalised. In any case, your initial feedback would be extremely valuable to us.

    This is a required question
    This is a required question

    Participation

    This is a required question
    This is a required question
    This is a required question
    This is a required question
    This is a required question

    Data sources

    Rate the different (possible) data sources based on how "interesting" you would find them to work with as part of the challenge. We certainly won't be able to include all of them, so please try to aim for a strictly monotonic ranking.
    This is a required question
    This is a required question
    This is a required question
    This is a required question
    This is a required question

    Practical issues

    This is a required question
    This is a required question

    Outcomes

    This is a required question
    This is a required question