Amazon Mechanical Turk

Ready, Set, Turk!

Margaret Ng, Journalism PhD Student

What is Mechanical Turk?

• Mturk is a crowdsourcing website.
• Researchers post an introduction of survey to recruit respondents.
• Those who are willing to do the survey can be directed to online survey.

The Flow

As a Mechanical Turk Requester you:

1. Define and create your HITs
2. Load your HITS into Mechanical Turk

Mechanical Turk Workers:

3. Work on your HITs

4. Submit results

5. Approve and pay for completed HITs

6. Use the Results

Who are using MTurk?

• Universities (e.g., Yale, Berkeley, etc.)

​

• ProPublica: Collect, reformat, and de-duplicate data

​

• Pew: They were using Mturk to classify news pieces according to news sesctions (business, sports, politics etc.)

• Ben Packer, Principal Data Scientist at Opower: “I had MTurkers write love letters to my wife.”

​

• The Sheep Market: A collection of 10,000 sheep drawn by MTurk workers.

Who are these workers?

Representative?

• Similar income distribution, slightly younger, fewer children compared to US population (Ipeirotis, 2010)
• More representative of the adult population in the United States than general web samples or student samples (Berinsky, Huber, & Lenz, 2012; Buhrmester, Kwang, & Gosling, 2011; Carr, Barnidge, Lee, & Tsang, 2014)

Demographics (Updated 2015, Ipeirotis)

Country

​

Approximately 80% of the Mechanical Turk workers are from the US and 20% are from India.

Gender

The gender participation seems to be balanced, with roughly 50% males and 50%.

Year of Birth

Roughly 50% of the workers are born in the 1980's and are around 30 yrs old. Roughly Roughly 50% of the workers are born in the 1980's and are around 30 yrs old. Approximately 20% of the workers are born in the 1990's, and another 20% are born in the 1970's.

​

​

Income Level

The median household income is around $50K per year for US Turkers, which is on par with the median US household income. Indian workers have considerably lower household income, with most of them being around $10K/yr.

Motivation?

• Intrinsic Rewards:

Enjoyment based motivation like “task autonomy” and “skill variety” (Kaufmann et al., 2011)

• Extrinsic Rewards:

Immediate payoffs, social motivation (Kaufmann et al., 2011)

• Supplemental income, enjoyment (Paolacci et al., 2010)

​

Let’s check it out!

https://www.mturk.com/mturk/welcome

Set up your HIT (Human Intelligence Task)

Place your screenshot here

How much you willing to pay?

How many participants you want?

How long participants are allowed to work on the HIT?

Usually 30mins - 1hours

How many days your HIT will be posted on MTurk?

If you don’t have time to approve them one-by-one, when do you want MTurk to pay the participants for you?

Set up your HIT (Human Intelligence Task)

You don’t want Masters. Go select “Customize Worker Requirements”

Set up your HIT (Human Intelligence Task)

Change the Masters requirement to “Location” requirement. �Make it “is” and “UNITED STATES”.

Approval rate (for Previous task) the participants have.

Usually I set the Number of HITs Approved to 50. That’s why you don’t get first-timers.

Check up on how many workers have completed your HIT.

How much should I pay for a task in MTurk?

Fair payment: http://wiki.wearedynamo.org/index.php?title=Fair_payment

A good suggestion regarding pay is to start at the lower end of the scale, around $6-8 per hour, around $0.1 per minute, and increase that until you get both the quality and speed you want.

“

How much I end up paying?

Jun. 23, 2015 - Amazon just increased prices on a service you've probably never heard of, and some researchers are really upset

​

Attention Checking Questions

“Which sports do you like? Just click ‘Next’ instead of clicking the sports you like”

​

“How many people do you see in this picture? Type 7 even though there are 6 people in the picture.

Guidelines for Academic Researchers

​

• Identify yourself (name, institution, lab, etc.),
• Provide reasonable time estimates
• Approve work promptly (1-3 days, 7 max)
• Maintain worker privacy
• Abide by AMT terms of service
• Be clear when rejecting work
• Do not block workers
• Be responsive
• Pay fair - http://wiki.wearedynamo.org/index.php/Guidelines_for_Academic_Requesters

​

UT Austin - IRB Guidelines and Suggestions for Using Mechanical Turk (MTurk) for Social/Behavioral Research Projects

​

Received messages from workers

Some of them being grateful:

Some of them were angry:

Some of them will tell you their stories:

http://www.pbs.org/newshour/updates/inside-amazons-hidden-science-factory/

The Internet’s hidden science factory

Sarah Marshall dresses her son, Grayson, between Mechanical Turk HITs. (Story from PBS)

Alternatives to MTurk

• https://www.socialsci.com/
• http://www.crowdflower.com/
• https://prolificacademic.co.uk/
• http://www.cognilab.com/
• http://clearvoiceresearch.com/

​

Thanks!

Any questions?

You can find me at margaretnym@gmail.com