Ethics in

Software Engineering Practice

David Gray Widder (he/him)�Jonathan Aldrich (he/him) �Christian Kästner

Legal vs Ethical

In September 2015, Shkreli received widespread criticism when Turing obtained the manufacturing license for the antiparasitic drug Daraprim and raised its price by a factor of 56 (from USD 13.5 to 750 per pill), leading him to be referred to by the media as "the most hated man in America" and "Pharma Bro". -- Wikipedia

"I could have raised it higher and made more profits for our shareholders. Which is my primary duty." -- Martin Shkreli

​

With a few lines of code...

With a few lines of code...

“Update Jun 17: Wow—in just 48 hours in the U.S., you recorded 5.1 years worth of music—40 million songs—using our doodle guitar. And those songs were played back 870,000 times!“

Concerns

Safety

Weapons, Surveillance

Addiction

Polarization

Deskilling

Discrimination

Stress, mental health

Monopolies

...

Harms beyond “Killing People”

Technological Harms (contd)

• widen inequality: checkout kiosks, displace workers, excess productivity going to the those who own or operate the software/ tech who are usually already wealthy
• invade privacy: NSA’s warrantless wiretaps
• make wars less costly, possibly more likely, CMU’s NREC’s warfighting robots
• obscure societal problems under veil of “objectivity”, ie CMU’s predictive policing algorithm

Technological Solutionism and Humility

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." – Abraham Maslow

• Tech schools (eg, CMU) and tech companies foster an environment where tech is seen as “good” in and of itself, such that it is considered the “right” or “best” way to solve any problem.
• Eg: hackathons: “disrupting” to solve deep seated societal issues with an app.
• Tech solutions often shift responsibility to the individual for things which require systemic reforms.
• This makes it hard to think about non-technological solutions, ask to slow down, not build the tech, or express concerns.
• Tech’s high salaries worsen this, situating tech workers as high experts compared with those with lived experience, cultural knowledge or expertise from other domains.

“Technological possibilities are irresistible to [hu]man[s]” – John von Neumann

Does Software Have Politics? Yes.

• Langdon Winner’s article argued that things - bridges, tomato harvesters, atomic bombs - encourage specific kinds of political outcomes, social orders, and power relations
• Politics: “the debate or conflict among individuals or parties having or hoping to achieve power”
• How a system is designed (the “affordances it offers”) of software make certain uses more likely than others.
• Building software is a political act - it makes things possible (or easier) that weren’t otherwise.
• “Yes or No”: Should we build this tech? Why might it be hard to ask this question in tech companies or tech schools?

Gorr et. al's 2014 "Early Warning System for Temporary Crime Hot Spots" according to CMU’s Metro21, “did not use racial, demographic or socioeconomic data” Hmm.

Responsibility: “It’s just the system/algorithm”

• “Tool Neutrality”, but also,
• Software can “launder” accountability for the design choices that software engineers encode into it, because the software engineers are “alienated” from where the software acts in the hands of the customer.
• Should software engineers be held accountable for software errors?

Software and Data Encode Values

• Often those of its creators and operators, or by using datasets which reflect dominant culture
• Algorithm design often encodes values, usually implicitly
• Recidivism Prediction, GenderMag, Boeing 737

1860 census of Enslaved People, recorded solely with owner’s name and physical characteristics

2020 census demographic collection form reflects socially constructed racial and gender categories

GenderMag

• What is the goal of GenderMag?
• Recognize that people think differently, and that people designing often have a specific thinking style which they reflect in what they build.
• What does Gendermag leave out?
• Risk of re-enforcing gender essentialism
• Nonbinary gender excluded
• Intersectionality - new “InclusiveMag”

Recidivism Prediction

• Tools which predict if an accused or convicted person will re-offend
• How do we decide what is “Good Enough” for recidivism prediction?
• 99% accurate? What about the 1% misclassified?
• Many definitions of fairness, each is value laden. Who gets to choose what “Fair” means?
• Predictive parity: “that the rate at which a tool generates true predictions should be the same for different groups.”
• Equalized odds: “should not be more likely to generate false predictions for one group than another.”

Exercise: Personal Ethical Red Lines

Imagine that in five years from now, you’ve earned your PhD and are working at a software company in a role relevant to your research area.

Take 5-10 mins to write down 3 “ethical red lines”: things which, if you were asked to do them, you would refuse on ethical grounds.

Examples:

• “I will not be involved with building Lethal Autonomous Weapons.”
• “I will not be involved with building a system which collects people’s data without their meaningful consent.”

Now we discuss. What themes do we notice? What might make these hard or easy to stick to going forward?

Boeing 737 MAX 8 Disaster

• What values were represented here?
• Whose “fault” is it?
• What things could be done to stop this happening again?
• Thinking beyond this case, how should we regulate tech? How should this work? How to balance interests?

“As airplanes became more complex and the gulf between what the FAA could pay and what an aircraft manufacturer could pay grew larger, more and more of those engineers migrated from the public [FAA] to the private sector. Soon the FAA had no in-house ability to determine if a particular airplane’s design and manufacture were safe. So the FAA said to the airplane manufacturers, ‘Why don’t you just have your people tell us if your designs are safe?’”

Let note to self: let student into zoom room

Ethics in

Software Engineering Research

David Widder (he/him), Jonathan Aldrich (he/him), Christian Kästner

Prelude: Ethics in Research vs Practice?

• SE Ph.D. program is ~15 years old, with ~22 graduated students.
• 4 are in academia
• 6 are doing industrial research
• 12 are in industry (software engineering)

​

• Meta Discussion Question: How might ethical implications be different in research vs software engineering practice?

UMN Case: A Series of Ethical Lapses by Good Faith Actors

• This study:
• Passed researcher’s own sense of ethics
• [presumably] passed internal SSSG-like presentations, lab discussions
• Passed University IRB review
• Passed peer review, including ethics, by a prestigious IEEE venue
• What were the risks and harms?
• What could have gone differently?

“I believe that they acted in good faith.”

​

“there may be some mismatch between what the academic community accepted as ethical conduct, and what the subjects perceived as ethical conduct”

​

“Our community does not appreciate being experimented on, and being “tested” by submitting known patches that are either do nothing on purpose, or introduce bugs on purpose.” - Linux chief

What does an IRB do? What doesn’t it do?

IRBs do…

• Examine human subject studies for compliance with specific federal rules, derived from ethical principles.

IRBs do not…

• Review for wider possibly negative societal impact
• Consider harms outside of federal rules
• Apply to research at non federally-funded institutions
• Determine whether research is ethical.

Therefore, you should:

• Take CITI Human Subjects training, submit IRB protocols for human subjects research, take this seriously, but also:
• Question your own research: how could this be misused? What unintended effects might it or my results have on society? How can I control these?
• Respectfully question your friends, your lab mates research: you might have thought of concerns they haven’t.

Raising ethical concerns about research you know about is your duty as a member of a scholarly community. You must do this, and do it respectfully.

But how? 'Have you thought about...'

• Humility, be open to understanding their worldview and adjusting yours.
• Opportunity to save face: discuss the research, not the person or their character
• People think they’re doing good.
• Private before public
• What to do when you reach an impasse?
• What might make this harder for you to do?

Raising ethical concerns about research you know about is your duty as a member of a scholarly community. You must do this, and do it respectfully.

“Ethics are discursive—ethical understandings emerge from conversation.”

​

“Peer reviewers should require that papers and proposals rigorously consider all reasonable broader impacts, both positive and negative” [ACM Future of Computing]

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

• What societal impact concerns does this paper raise?
• Climate
• Bias
• What epistemological concerns for NLP does this paper raise?
• What is meant by Parrot?
• Going Meta: Why was Gebru fired?
• What might this say about industrial research?

“Is it fair or just to ask, for example, that the residents of the Maldives (likely to be underwater by 2100) [to] pay the environmental price of training and deploying ever larger English LMs, when similar large-scale models aren’t being produced for Dhivehi?”

​

“However, no actual language understanding is taking place in LM-driven approaches to [question and answer] tasks, as can be shown by careful manipulation of the test data to remove spurious cues the systems are leveraging”

​

“Reddit, Twitter, and Wikipedia present themselves as open and accessible to anyone, there are structural factors including moderation practices which make them less welcoming to marginalized populations.”

​

Critical Computing Education

• Volunteer to summarize and react to the talk.
• What does Amy mean by Critical?
• How do you think we’ve learned these? Do we disagree?
• “Many of us think of computing as endlessly powerful.”
• “Computing is not neutral.”
• “Recognize your power and the responsibility that comes with it.”
• What do we think of the self critiques Professor Ko raises on the left?

“My research amplified algorithmic bias” helping developers write biased algorithms faster and more correctly than ever before

“My research centralized and privatized power:” My inventions largely served powerful platforms owned by Amazon, Google, etc

“My research replaced people with machines: Our research created two dozen jobs replaced tens of thousands of customer service agents with information retrieval algorithms

6 Tanks CMU Researchers Have Prototyped

• What do we think of this reading?
• Whose military?

UMN Case

• “IRB determinations are also not always sufficient to establish that a paper is ethical.”

So, how do we

• “IRB determinations are also not always sufficient to establish that a paper is ethical.”

Critical Computing Education - Amy J. Ko, CMU PhD

A How to Talk To Your Friends About Research Ethics

• ​

When and where (2021)

Confirmed:

• Sept 30, 3pm TCS 460 - SE Practice
• Nov 11, 3pm TCS 460 - SE Research

Notes from CK and David Widder after class 27/Sept/22

• Remove google doodle, substitute with random airplane seating example
• Streamline, one core idea per slide. Slides built, but David felt some was repetitive
• Show ties between Gendermag and Tech solutionism
• Read RE module, examine confluences (esp with Jackson paper). Suggest making this module after RE module.
• Integrate Widder preprint paper better
• Cut 787 Max paper
• Winner: assign full paper? Full paper with summary?