ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
If the paper reports on multiple studies involving the crowd as subjects, fill out the checklist per experiment. Besides, if an experiment uses different (potentially interconnected) micro-tasks, then report the Task and Quality control sections for each task.
2
ItemItem No.RecommendationPage No.
3
4
Experimental design
5
Input dataset1Describe how the input dataset for the experiment was obtained and if it is publicly available. Also, touch on its reputation and difficulty (if applicable)
6
Allocation to experimental conditions2State how the participants were assigned to the experimental conditions or treatments, and how this step was implemented in the crowdsourcing platform
7
Experimental design to task mapping3Describe what research design was used in the experiment and how were the experimental conditions mapped to crowdsourcing tasks
8
Execution of experimental conditions4Report how the crowdsourcing tasks, representing the experimental conditions, were executed (e.g., in parallel, sequentially, or mixed)
9
Execution timeframe5State over what timeframe the experiment was executed
10
Pilots6Describe if pilot studies were performed before the main experiment
11
Returning workers7Report the strategies used to prevent returning workers, i.e., workers who finish the experiment and then reenter it later because the study was still running
12
13
14
15
Crowd
16
Target population8Describe the criteria used to determine the workers who are allowed to participate (e.g., acceptance rate, tasks completed, demographics, working environment). And also include the strategy used to identify such workers.
17
Sampling mechanism9Report what strategies were used to recruit a diverse or representative set of workers from the target population
18
19
20
21
Task
22
Task interface10Report and show the task interface as seen by workers
23
Task interface source11Provide a link to an online repository with the source code of the task interface (typically a combination of HTML, CSS, and JavaScript)
24
Instructions12Describe and show the instructions of the task as seen by workers
25
Reward strategy13State the mechanisms used to reward and motivate workers (e.g., payments)
26
Time allotted14Report if a time constraint was defined for workers to complete the task (if so, describe also how much)
27
28
29
30
Quality control
31
Rejection criteria15State the criteria used to accept or reject a contribution from a worker (e.g., workers can be allowed to submit the task and reject it afterward, submissions can be blocked based on prior rejections or on time spent on the task)
32
Number of votes per item16Describe, if applicable, how many workers solved the same item or data unit
33
Aggregation method17Report, if applicable, how the contributions from workers were aggregated (e.g., majority voting)
34
Training18State if workers performed a training session or pre-task qualification test. If so, describe 1) the training, 2) the items used as the training set, and 3) if it was performed before or as part of the task
35
In-task checks19Report the mechanisms embedded in the task to guard the quality of the results. Also, state if and how workers were allowed to revise their answers.
In case gold items or attention checks were used, describe how these items were selected, how frequently they appear, and the threshold used to filter out workers underperforming on these items.
36
Post-task checks20Report the steps performed upon task completion to safeguard the quality of the results (e.g., post hoc analysis)
37
Dropouts prevention mechanisms21Indicate the strategies used to deal with worker dropouts (i.e., workers who leave the task unfinished)
38
39
40
41
Outcome
42
Number of participants22Indicate how many workers participated in the experiment (in total and per condition)
43
Number of contributions23Report the number of contributions (e.g., votes) in total and per condition
44
Excluded participants24Indicate the number of participants not considered for the data analysis, including the reason for exclusion.
45
Discarded data25State the number of contributions excluded before the data analysis
46
Dropout rate26Describe the dropout rate of the participants in the experimental conditions. If applicable, also show breakdowns per milestone of progress within the task (e.g., after 2, 3, and 5 questions).
47
Participant Demographics27Report the demographics of the participants (e.g., age, country, language)
48
Data processing28Report any data transformation, augmentation, and/or filtering step performed on the raw dataset obtained from the crowdsourcing platform.
49
Output dataset29Provide a link to the dataset resulting from the experiment. Also, indicate if the dataset contains the aggregated or individual contributions from workers
50
51
52
53
Requester
54
Platform(s) used30Indicate the crowdsourcing platform(s) selected for the experiment
55
Implemented features31Report any additional feature implemented to support the experiment, covering missing functionality from the selected platform(s)
56
Fair compensation32State whether workers were compensated fairly and according to legal minimum wage
57
Requester-Worker interactions33Describe concrete requester-worker interactions taking place as part of the experiment
58
Privacy & Data Treatment34Report any relevant privacy regulations and methods used to comply, especially if the output is put online (e.g., the data could be anonymized to meet privacy policies).
59
Informed consent35Indicate if an informed consent was used
60
Participation awareness36State if workers were informed they took part in an experiment
61
Ethical approvals37Report if the study received ethical approval from the corresponding institutional authority
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100