The Labor Economics of �Paid Crowdsourcing

Name of Venue - 1 March 2010

John Horton

john.joseph.horton@gmail.com

Lydia Chilton

hmslydia@cs.washington.edu

Work in conjunction with:

​

Crowdsourcing

• A form of “peer production” in which work traditionally performed by an employee is outsourced to an “undefined, generally large group of people in the form of an open call” [cite]
• Examples: Wikipedia, Google Image Labeler, Yahoo Answers.
• Crowdsourcing platforms need to attract a crowd.
• Entertainment.
• Altruism, desire to help and gain recognition and reputation for one’s helpfulness.
• Money.
• Using money as a inducement has advantages.
• Non-monetary incentives depend on the nature of the task or the indentity of the proposer, which limit their usefulness.
• Non-monetary incentives are hard to adjust. If people play a game because it’s fun, it’s hard to make it 10% more fun and thus get 10% more output.

​

Amazon’s Mechanical Turk (AMT)

• A popular crowdsourcing platform for very short tasks and very small amounts of money (2 cents to 20 cents)
• Workers are almost entire US citizens, acedotally, they tend to be college educated, potentially stay-at-home parents.

Goal: Run Experiments on AMT to test Labor Models of Crowdsourcing

• Previous research on AMT found workers respond to prices at least consist with rational behavior. However, the relationship between incentives and output is complex.
• Higher pay did not improve quality of work
• Piece rate or quota payment affected output
• Some, but very few previous economics data on worker’s reservation wages
• Studies of other crowdsourcing platforms (Wikipedia, the ESP Game) have been done, but those systems do not involve money and thus the analyses are not approached from the perspective of labor economics.

​

Outline

• Motivate a simple rational model of labor supply.
• Identify key parameters to measure
• Introduce our experimental setup for measuring reservation wage
• Present results
• Discuss how the results fit the model and propose a guideline for crowdsourcing.

​

A Simple Rational Model of Crowdsourcing Labor Supply

• We want a model that will predict a worker’s output
• y = output
• y* = the last unit of output from the worker
• P(y) = payment as a function of output
• p(y) = P’(y). In the case that P(y) = wage_rate*y, p(y) = wage_rate.
• A ration user will supply output, y, to maximize Payment – Cost

​

​

• Workers will rationally set y* (the last unit of work) where p(y*) = c(y*)
• If p(y*) = ωt then a worker’s reservation wage is

​

​

• We can experimentally determine p(y*) on AMT by offering users a task where they are offered less and less money to do small amounts of work until they elect not to work anymore. That payment is p(y*)

Measuring Reservation Wage

Instructions before starting

Message between

sets of 10 clicks

Payment

• In order to find reservation wage, the price for each set of 10 clicks decreased such that the wages approach P bar asymptotically:

​

​

• Example:

​

​

​

​

​

• Fractional payment problem: pay probabilistically

​

​

Two Experiments to Determine Reservation Wage

• Δ Difficulty
• Fitt’s law says that moving between targets with a pointing device (e.g. mouse) on targets becomes logrhymically more difficult with distance (where difficulty is measured in the time it takes to move from one target to another)
• Is time per click, total output and reservation wage affected by the distance between the targets. (300px apart and 600px apart)
• Δ Price
• Is time per click, total output and reservation wage affected by offering different baseline price? (P bar)

​

Δ Difficulty Results

92 participants

42 randomly assigned to “Easy”

72 self-reported females

​

Δ Difficulty Discussion

• Despite being more time consuming, workers completed the same average output (number of blocks)
• Unexpectedly, reservation wage was higher for the easy task. (Mean reservation wages: $0.89/hour for Hard and $1.50/hour for Easy)

​

Δ Price Results

198 participants

42 randomly assigned to “Easy”

72 self-reported females

​

Δ Price Discussion

• As expected, lower pay reduces output (fewer blocks completed)
• Surprisingly, reservation wage not the same for the two pay groups. Low wages had a lower reservation wage. (Mean reservation wages: $0.71/hour for Low and $1.56/hour for High)

​

Why might the lower wage group have lower reservation wage?

​

• Worker error: misinterpreting the per block payment leading to marginal payments “bleeding over”

​

• Non-time based marginal costs

​

• Target earning behavior: workers try to reach a self-imposed earnings goal rather than respond to the current offered wage

Evidence for Target Earning

• At least some subjects were clearly trying to pursue the maximum earnings possible, despite low wages associated with this strategy. (Game-like mentality)
• Further evidence is the distribution of worker output over wages divisible and not divisible by 5.
• Non-target earnings quit very early.

​

​

Contributions

Our goal was to measure key labor economic properties (reservation wage) of AMT in order to better model output of crowdsourcing productions.

1. We present two versions of an experiment that can extract detailed information on worker output at various price points. In future work we can run similar experiments: for example varying the number of clicks per set to further investigate the phenomenon of target earning.

2. Demonstrate some results that support the traditional model of labor economics and others that point to alternative models

3. Further research is needed to confirm the phenomenon of target earnings, but based on this data is a prevalent mechanism which workers use to determine their output. Crowdsourced projects should take this into consideration when designing their interfaces.

​