EECS 598 - Human Computation and Crowdsourcing Systems


Welcome to Human Computation and Crowdsourcing Systems! This course is designed to be an introduction to crowdsourcing and human computation from a technical, system-building point of view. The course will consist of in-class lectures and assignments as well as a large project component that will serve as a hands-on intro to crowdsourcing for human computation.

Presentation Sign-Ups

Course Overview / Schedule

Week 1  (9/8-9/10)

Introduction, goals, and overview - Human Computation: A Survey and Taxonomy of a Growing Field

Crowd types, platforms, and hiring - Demographics of Mechanical Turk

Talks: Sign-up for talk slots. URL: 

Week 2  (9/15-9/17)

Quality control challenges - Financial Incentives and the “Performance of Crowds”

Quality control methods pt. 2 - Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers

 Mini-project: “Be an worker”. Choose two platforms (e.g., Mechanical Turk and Upwork) and work the equivalent of 2-3 hours on each. Hand-in: Describe outcomes and issues faced.

Week 3  (9/22-9/24)

Quality control methods pt.2, algorithms *and* Games with a Purpose - [No reading]

Machine learning for response evaluation - Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

 Mini-project: Post a task and collect ~100 data points. Create a dataset of your choice. Something interesting to you. Ideally this will be an external HIT using the MTurk API, but collecting a dataset you can use later is most important. Hand-in: Dataset + summary and/or visualization.  [[ see next project ]]

Week 4  (9/29-10/1)

Task decomposition - Turkit: Human Computation Algorithms on Mechanical Turk

Crowdsourcing workflows - Soylent: A Word Processor with a Crowd Inside

Week 5  (10/6-10/8)

Crowd task decomposition - Collaboratively Crowdsourcing Workflows with Turkomatic

Real-time crowdsourcing - Crowds in Two Seconds: Enabling Real-time Crowd-Powered Systems

—> Bonus paper recommendation: The Jabberwocky Programming Environment for Structured Social Computing

Week 6  (10/13-10/15)

Continuous crowdsourcing and crowd agents - Chorus: A Crowd-Powered Conversational Assistant

        <Fall Break>

 Project phase 0: Meet with groups. Schedule meetings with me about possible project ideas.

Week 7  (10/20-10/22)

Automated support of workflows - Human Intelligence Needs Artificial Intelligence

+ Expert Crowdsourcing -  Expert crowdsourcing with flash teams

--> (ALT paper: The Anatomy of a Large-Scale Social Search Engine)

< Iterate on project ideas + groups >

 Mini-project: Plan a synchronous crowdsourcing task and responses.

Week 8  (10/27-10/29)

Improving over time - Shepherding the Crowd Yields Better Work

Self-coordination - Apparition: Crowdsourced User Interfaces That Come To Life As You Sketch Them

+ Open-ended tasks - Ensemble: Exploring Complementary Strengths of Leaders and Crowds

 Project phase 1: Initial proposal for final project due. Hand-in: Written proposal ideas.

Week 9  (11/3-11/5)

Supporting users Pt.1: Human-Powered Access Technology - VizWiz: Nearly Real-time Answers to Visual Questions

Supporting users Pt.2: Productivity support - EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing

 Project phase 2: Final project proposals and review. Hand-in: Written final proposal.

Week 10  (11/10-11/12)

        [[ NO CLASS 11/10 ]] —  Spend this time designing an initial prototype and experimental plan with your teammates. The ‘plan’ for each can just be a sketch and a few notes, but must be clear in your minds =)

        Privacy and Threats - Information Extraction and Manipulation Threats in Crowd-Powered Systems

                --> (ALT paper: Instance-Privacy Preserving Crowdsourcing)

Week 11  (11/17-11/19)

Applications: Vision / Natural language interaction  - Real-Time Captioning by Groups of Non-Experts

        --> (ALT paper: [[LabelMe or Jia Deng’s bird annotation work]])

Applications: Translation - Crowdsourcing Translation: Professional Quality from Non-Professionals

Week 12  (11/24)

Theory: Voting theory and agent modeling - [[TBD]]

--> (ALT paper: Interactive Crowds: Real-time Crowdsourcing and Crowd Agents)

Related work double-header: Citizen Science and Learning @ Scale - [[TBD]]

Week 13  (12/1-12/3)

Related work: Management science / operations research - [[TBD]]

Open challenges in crowdsourcing / discussion - [[TBD]]

Week 14  (12/8)

Final project presentations

< No class after 12/8 >


In-class participation and assignments: 15%

Individual projects: 20%

Group project: 45%

Paper presentations: 20%


Each project will have a data component (something which must be collected and logged in a structured way), and a written component (describing the results and any issues that may have been encountered). A system description will also be required for each project, and will help students practice writing system descriptions in research papers.

Paper Presentations

Each week, we will cover 1-2 papers from the literature in detail. For each, one team of students will be assigned to present the paper, and another team will be assigned to be official ‘discussants’ for the paper. The discussants’ job is to deeply evaluate the work and have conversation-provoking questions ready in advance of the presentation itself. While there will be assigned discussants for each paper, everyone is expected to have read the paper being presented and participate in the discussion.

Contact Information and Logistics

By far the best way to contact me is by email. I’m happy to setup a meeting if more detailed discussion is needed.


Office: BBB 2636