1 of 34

Intervention Design

SP’23 — Critical Analysis in Computing

2 of 34

@1m — Intro / Intervention Definition

3 of 34

What is “An Intervention”, “A ‘good’ intervention”?

Consider—

  • A TA verbally abuses a student during OH

What is the harm?

How can it remedied?

How can it be prevented?

4 of 34

@5m — Real-World Example

5 of 34

Example of an iterative design process

As we work through this example…

  • Why did they design this way?
  • How did they collect feedback?
  • How did they act on the feedback?

​

​

6 of 34

Timeline-as-outline?

  • Problem Identification
  • Art-as-outreach; Need Finding??
  • Initial Platform?
    • How designed, why? Target goals
    • Actual usage
  • Design Iteration 1
    • Academic offshoot version
    • Platform deployment
    • Usage, Pros/Cons, Failure to Thrive [why?]
  • Design Iteration 2
    • Align with organic usage of initial platform → community formation
    • Community-driven needs → smaller, iterative effective interventions?

7 of 34

Turkopticon

Irani and Silberman

  • Irani, Lilly C., and M. Six Silberman. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2013, pp. 611–20.
  • Salehi, Niloufar, et al. “We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers.” Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15, ACM Press, 2015, pp. 1621–30.
  • Irani, Lilly C., and M. Six Silberman. “Stories We Tell About Labor: Turkopticon and the Trouble with ‘Design.’” Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2016, pp. 4573–86.
  • Daemo:

8 of 34

Amazon Mechanical Turk

9 of 34

Turkopticon

Needfinding

  • Survey through AMT on a “Workers’ Bill of Rights”
    • 67 Respondents
  • Open-ended interviews with workers and employers.
  • Attending crowdwork conferences.

Irani, Lilly C., and M. Six Silberman. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2013, pp. 611–20.

10 of 34

Turkopticon

Intervention

  • Communicativity: How responsive has this requester been to communications or concerns you have raised?
  • Generosity: How well has this requester paid for the amount of time their HITs take?
  • Fairness: How fair has this requester been in approving or rejecting your work?
  • Promptness: How promptly has this requester approved your work and paid?

Irani, Lilly C., and M. Six Silberman. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2013, pp. 611–20.

11 of 34

Turkopticon

Intervention Challenges

  • Bootstrapping

​

  • Balancing anonymity and reputation

​

  • Moderation

​

Irani, Lilly C., and M. Six Silberman. “Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2013, pp. 611–20.

12 of 34

“Staying with the trouble” - Donna Haraway

In a thesis statement, Haraway writes: "Staying with the trouble means making oddkin; that is, we require each other in unexpected collaborations and combinations, in hot compost piles. We become - with each other or not at all."

13 of 34

Dynamo

Intervention II

  • Tackling the problems of “stalling” and “friction”.
  • Turkers led letter writing campaigns
    • 2 responses from AMT.
    • 100 Researchers committed to fair task practices.

Salehi, Niloufar, et al. “We Are Dynamo: Overcoming Stalling and Friction in Collective Action for Crowd Workers.” Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI ’15, ACM Press, 2015, pp. 1621–30.

14 of 34

“Stories we tell about labor”

Critical Reflection

  • Centering turkers’ incisive critiques about development and research in crowdwork.

​

  • “Designers, programmers, and creatives appeared to have all the agency”

Irani, Lilly C., and M. Six Silberman. “Stories We Tell About Labor: Turkopticon and the Trouble with ‘Design.’” Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, 2016, pp. 4573–86.

15 of 34

Daemo

Is an alternative AMT possible? Competing perspectives.

  • What did the researchers claim?
    • “Bake a better deal into the DNA of the whole platform.” [1]
    • It has transparency built in through discussions.
  • How did the workers respond?
    • Academics maintain all the control.
    • Academics trivialized workers’ expertise.
    • “In this process, workers should be treated as equal, deeply invested, expert partners in building a better future of crowd work — not slotted into a process managed by a small team of researchers, designers, and programmers.” [2]

​

[1] https://www.wired.com/story/amazons-turker-crowd-has-had-enough/

​

[2] https://medium.com/@SocialContractCR/daemos-crowd-has-had-enough-5b62c1fceb30

16 of 34

Turkopticon

Where is it today?

  • A non-profit organization advocating for turkers rights.
  • Turkopticon is maintained by Alex, Ashley, Brook, Krystal, Clearblue, Phil, Sherry, with other dedicated Turk workers.

  • “Yeah it was train this AI or eat today, I choose to eat... I knew eventually it will go **** up.”� - [deleted] user on r/mturk

17 of 34

@40m —

Walkthrough of this term

18 of 34

Your Project for this Class

  • Apr 23: Group Formation & Project Directions
  • Today (Apr 25): Stakeholder Mapping
    • Homework: Initial Proposal (due Thursday!)
  • Thursday (Apr 27): Workshop Day
    • Homework: Stakeholder Outreach
    • Homework: Revised Project Proposal
  • Week 6 (May 7): Revised Project Proposal Due
  • Week 7 (May 16, 18): Project Workshops
    • Stakeholder shareout
    • Project proposal critiques
  • Week 9 (May 30): Revised Project Proposal Due
  • Week 10~11 (June 8, 12): Final Presentation & Report
    • Oral and written summary of design process and outcome

19 of 34

Apr 23: Group Formation & Project Directions

​

​

  • Goal: Find folks with similar interests
  • Goal: Identify problems you think are important or interesting
    • Not yet about solutions
    • Be able to articulate why this is an important problem
      • Can you express specific harms?

20 of 34

Today (Apr 25): Stakeholder Mapping

​

​

Goal: Answer who are all the folks around your space and how they’re connected

Homework: Initial Intervention Proposal

First draft consideration of what kind of intervention your group may consider designing that could address a real-world harm. This is not a contract. Your project will evolve through the design process. At this stage, you will produce a 1-2 page (500-1000 words) extended abstract, which should include:

  • Overall problem focus (a technology/a company/a tool etc)
  • List a few potential or real harms (read and cite a few news articles)
  • One or more possible approaches/interventions to address the problem.

Your overall objective should come from your own motivation—what are you interested in improving (and some why here too if it is relevant)? The needs assessment, however, should demonstrate with external information—not just your own perspective—where there are actual pain points and limitations.

21 of 34

Thursday (Apr 27): Workshop Day

  • Proposal Presentations
    • Give the “elevator speech”
      • One slide max, if any
      • Truly just 30s — hear back from group what they understood
    • Give the “VC Pitch”
      • 5m — Give the full story
  • Proposal Workshop
    • Summarize feedback
    • Iterate on design
    • Create your stakeholder outreach plan

22 of 34

May 7: Revised Project Proposal Due

Aka: What you’re doing between Apr 27 and May 7

Revise your initial proposal and produce another 1-2 page proposal for the group.

Your submission should include:

  • Overall problem focus (a technology/a company/a tool etc)
  • List a few potential or real harms (read and cite a few news articles)
  • A stakeholder map.
  • One or more possible approaches/interventions to address the problem.

​

​

23 of 34

May 7 – May 16

Course staff reviews proposals, prepares feedback

24 of 34

Week 7 (May 16, 18): Project Workshops

​

​

Intervention Design Workshops

Stakeholder share-out

​

Proposal presentations and critiques

​

Workshop time for project design

25 of 34

May 16: Stakeholder Shareout!

Aka: What you’re doing between Apr 27 and May 16 — Part 2

  • Stakeholder Outreach
    • Option 1: Participant Observation (Preferred)
    • Option 2: Informational Interview
    • Option 3: Getting informed about a suspicion!

​

​

26 of 34

May 30: Stakeholder Outreach and Refined Intervention

Aka: What you’re doing between May 7 and May 30

Revise your initial proposal to include a summary of your outreach activities

  • What outreach did you do, why that?
  • Summarize your activity and the experiences
  • Distill insights from your outreach activities
    • Discuss how they inform your problem space, intervention focus, design concept, etc.

Capture your design discussions, include how and why your intervention design has changed.

27 of 34

Week 10~11 (June 8, 12): Final Presentation & Report

​

​

Your final presentation should showcase your proposed design and explain how you arrived at this design. Presentations should last about 10 minutes.

​

Your final report should outline your proposed design and explain how you arrived at this design. You should cite prior work and/or discuss your engagement with stakeholders to justify design decisions. You should also reflect on potential barriers to implementation or other limitations of your proposal. The final report should be about 3-5 pages (2000-3000 Words).

28 of 34

Break + Rearrange

Relocate post-break to be seated with your project groups

29 of 34

[OLD SLIDE]

Critical analysis identifies structural harms — how will you fix them?

Quarter-long project to design an intervention for your chosen focus context

  • Identifying Stakeholders & Stakeholder Outreach
    • You must talk to people outside of this class and this department
  • Iterative Design and Workshopping
    • Matching intent to outcome
    • Structural change must outlast your direct investment — how?
  • Making Your Case
    • Why should others invest time (or money, or..) in your approach?
    • How will you measure success?

30 of 34

@50m —

Interactive Exercise

31 of 34

Stakeholder Mapping Exercise

  • Step 1 Who might be directly affected by issues you are thinking about?
    • [5 Min] Instructors show
    • [10 Min] Try to find one news article that is talking about each of the direct stakeholders and their relations to this issue. Good sources:
      • Wired
      • The Atlantic
      • The New York Times
    • [10 Min] Shareback with class
  • Step 2 Who might be indirectly affected by the issues you are thinking about?
    • [5 Min] Instructors show
    • [10 Min] Thinking about second order effects, relationships of your direct stakeholders.
      • List your reasons about how they might be affected. Why might it be important to get their input
    • [10 Min] Shareback with class
  • Homework to include in project proposal: Finding a few references. Include at least 2 academic sources and 2 newspaper articles per person in your group.

32 of 34

How far does this go? [ https://anatomyof.ai/ ]

33 of 34

How far should you go?

Rule-of-thumb:

Direct affecteds and one-hop indirect

Consider:

Direct affecteds can have non-trivial separation

Any ‘major’ n-hop indirects

34 of 34

What to prepare for Thursday

[ Intervention Design Workshop Day ]

  1. Initial Intervention Proposal
    • 1-2 page (500-1000 Words) extended abstract
  2. Lighting round of proposal presentations and critiques
    • Give the “elevator speech”
      • One slide max, if any
      • Truly just 30s — hear back from group what they understood
    • Give the “VC Pitch”
      • 5m — Give the full story