EECS 598 - Human Computation and Crowdsourcing Systems
Introduction
Welcome to Human Computation and Crowdsourcing Systems! This course is designed to be an introduction to crowdsourcing and human computation from a technical, system-building point of view. The course will consist of in-class lectures and assignments as well as a large project component that will serve as a hands-on intro to crowdsourcing for human computation.
Course Overview / Schedule
Week 1 (9/8-9/10)
Introduction, goals, and overview - Human Computation: A Survey and Taxonomy of a Growing Field
Crowd types, platforms, and hiring - Demographics of Mechanical Turk
⇒ Talks: Sign-up for talk slots. URL: https://docs.google.com/spreadsheets/d/1_vhc_rQVAZ3rt2_AFMfXzOX1xJ9UZXXauWI9cyRJKmQ/edit#gid=0
Week 2 (9/15-9/17)
Quality control challenges - Financial Incentives and the “Performance of Crowds”
Quality control methods pt. 2 - Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers
⇒ Mini-project: “Be an worker”. Choose two platforms (e.g., Mechanical Turk and Upwork) and work the equivalent of 2-3 hours on each. Hand-in: Describe outcomes and issues faced.
Week 3 (9/22-9/24)
Quality control methods pt.2, algorithms *and* Games with a Purpose - [No reading]
Machine learning for response evaluation - Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks
⇒ Mini-project: Post a task and collect ~100 data points. Create a dataset of your choice. Something interesting to you. Ideally this will be an external HIT using the MTurk API, but collecting a dataset you can use later is most important. Hand-in: Dataset + summary and/or visualization. [[ see next project ]]
Week 4 (9/29-10/1)
Task decomposition - Turkit: Human Computation Algorithms on Mechanical Turk
Crowdsourcing workflows - Soylent: A Word Processor with a Crowd Inside
Week 5 (10/6-10/8)
Crowd task decomposition - Collaboratively Crowdsourcing Workflows with Turkomatic
Real-time crowdsourcing - Crowds in Two Seconds: Enabling Real-time Crowd-Powered Systems
—> Bonus paper recommendation: The Jabberwocky Programming Environment for Structured Social Computing
Week 6 (10/13-10/15)
Continuous crowdsourcing and crowd agents - Chorus: A Crowd-Powered Conversational Assistant
<Fall Break>
⇒ Project phase 0: Meet with groups. Schedule meetings with me about possible project ideas.
Week 7 (10/20-10/22)
Automated support of workflows - Human Intelligence Needs Artificial Intelligence
+ Expert Crowdsourcing - Expert crowdsourcing with flash teams
--> (ALT paper: The Anatomy of a Large-Scale Social Search Engine)
< Iterate on project ideas + groups >
⇒ Mini-project: Plan a synchronous crowdsourcing task and responses.
Week 8 (10/27-10/29)
Improving over time - Shepherding the Crowd Yields Better Work
Self-coordination - Apparition: Crowdsourced User Interfaces That Come To Life As You Sketch Them
+ Open-ended tasks - Ensemble: Exploring Complementary Strengths of Leaders and Crowds
⇒ Project phase 1: Initial proposal for final project due. Hand-in: Written proposal ideas.
Week 9 (11/3-11/5)
Supporting users Pt.1: Human-Powered Access Technology - VizWiz: Nearly Real-time Answers to Visual Questions
Supporting users Pt.2: Productivity support - EmailValet: Managing Email Overload through Private, Accountable Crowdsourcing
⇒ Project phase 2: Final project proposals and review. Hand-in: Written final proposal.
Week 10 (11/10-11/12)
[[ NO CLASS 11/10 ]] — Spend this time designing an initial prototype and experimental plan with your teammates. The ‘plan’ for each can just be a sketch and a few notes, but must be clear in your minds =)
Privacy and Threats - Information Extraction and Manipulation Threats in Crowd-Powered Systems
--> (ALT paper: Instance-Privacy Preserving Crowdsourcing)
Week 11 (11/17-11/19)
Applications: Vision / Natural language interaction - Real-Time Captioning by Groups of Non-Experts
--> (ALT paper: [[LabelMe or Jia Deng’s bird annotation work]])
Applications: Translation - Crowdsourcing Translation: Professional Quality from Non-Professionals
Week 12 (11/24)
Theory: Voting theory and agent modeling - [[TBD]]
--> (ALT paper: Interactive Crowds: Real-time Crowdsourcing and Crowd Agents)
Related work double-header: Citizen Science and Learning @ Scale - [[TBD]]
Week 13 (12/1-12/3)
Related work: Management science / operations research - [[TBD]]
Open challenges in crowdsourcing / discussion - [[TBD]]
Week 14 (12/8)
Final project presentations
< No class after 12/8 >
Grading
In-class participation and assignments: 15%
Individual projects: 20%
Group project: 45%
Paper presentations: 20%
Projects
Each project will have a data component (something which must be collected and logged in a structured way), and a written component (describing the results and any issues that may have been encountered). A system description will also be required for each project, and will help students practice writing system descriptions in research papers.
Paper Presentations
Each week, we will cover 1-2 papers from the literature in detail. For each, one team of students will be assigned to present the paper, and another team will be assigned to be official ‘discussants’ for the paper. The discussants’ job is to deeply evaluate the work and have conversation-provoking questions ready in advance of the presentation itself. While there will be assigned discussants for each paper, everyone is expected to have read the paper being presented and participate in the discussion.
Contact Information and Logistics
By far the best way to contact me is by email. I’m happy to setup a meeting if more detailed discussion is needed.
Email: wlasecki@umich.edu
Office: BBB 2636