Although we are given a lot of freedom to figure out what we really want to do, there are some constraints given that this experiment would involve an actual company. First and foremost, data cannot leave the walls of the organisation. One possible solution would be to ship the code to the organisation, i.e., participants would need to submit their code. Second, this experiment can under no circumstances hinder the normal daily operations of this e-commerce site. We won't have the resources and capacity to rigorously check all submitted code so that it could be integrated into a live system. It would be possible, however, to run participants' code on previously collected data. Also, for head queries, it would be possible to compute results offline which would then be shown on actual search result pages (using some sort of interleaving) that are displayed to users. There is likely to be some limitations on the platform and tools that can be used.
As you probably have understood by now, at this point everything is still in flux and subject to discussion. Our aim with this questionnaire is to get an initial assessment regarding whether there is sufficient interest to warrant the effort of organising the challenge and to get an understanding of what the community thinks our focus should be. Should we decide to go forward with the challenge, we will most likely be needing several iterations to have all the details finalised. In any case, your initial feedback would be extremely valuable to us.