Ethics and Responsibility:�Societal Impact of Language Technology

Rishi Bommasani

CS 329X Spring 2023

1

Outline

• Backdrop
• Taxonomy of harms
• Beyond prototypical harms in NLP
• Mechanisms

2

Five distinctions in terminology

1. individual vs. group
2. minority vs. marginalized
3. risk vs. harm
4. model vs. system
5. harm vs. (societal) impact

3

Harms

4

Case Study: Algorithmic Hiring

5

What happened?

Setting.

1. Candidates apply to Amazon.
2. Amazon needs to decide whether to interview and hire them. Amazon’s decisions are mediated by a resume screening algorithm.

​

Harm.

The hiring algorithm systematically discriminated against women.

6

How did it happen?

The algorithm downweighted terms like “women’s”

​

The algorithm upweighted terms like “executed”

7

Why did it happen?

Amazon’s workforce of software engineers is heavily male-biased.

​

Using historical data as training data, the algorithm perpetuated and reified this bias.

8

What does “harm” mean?

Be concrete in reasoning about how people are affected

​

• Psychological and representational harms
• Allocational and material harms

Bommasani 9

Taxonomizing harm

1. Malicious vs. unintentional
2. Instance-level vs. distributional
3. Individual vs. group

​

Reason about the frequency and severity

10

Sources:

1. Ethical and social risks of harm from language models
2. On the opportunities and risks of foundation models

​

Enumerating common risks in NLP

• Bias, fairness, discrimination, equity, stereotypes, toxicity
• Misinformation, disinformation, manipulation, fraud, persuasion
• Privacy and security
• Overreliance, mismanaged expectations

11

Bommasani 12

Bommasani 13

Characterizing harms (transparency)

Measurement

- Severity, frequency

- Validity, reliability

​

User experience

- User studies, surveys

- Feedback, monitoring

Bommasani 14

Standard approaches to harm in NLP are inadequate

Claim: The harm and impact of technology is poorly characterized by “deliberate” vs “unintentional” harm

​

Technology often re-structures society

• Internet
• Social networks
• Search engines

​

Structural risk (Zwetsloot and Dafoe, 2019)

Bommasani 15

Structural risks

• Copyright and license violations
• Environmental harm
• Job displacement and economic impact; labor practices
• Reduction of diversity, homogenization/standardization
• Centralization of power

​

Bommasani 16

New structure: same algorithm�New harm: homogeneous outcomes

Bommasani 17

Decision-maker perspective

Applicant perspective

Note: companies mentioned are merely for illustrative purposes�Source: Picking on the same person: Does algorithmic monoculture lead to outcome homogenization?

Cultural homogenization?

Language models can influence culture

Will they lead to a reduction of cultural diversity?

Bommasani 18

Mechanisms to address harm

• Transparency (documentation, evaluation, visualization)
• Technical methods (e.g. RLHF, various bias mitigation techniques)
• Ethics
• Policy (law, governance – Irene will cover this)

​

Bommasani 19

Documentation

• Data sheets, model cards, system cards, ecosystem graphs
• Examples:

https://huggingface.co/bert-base-uncased

https://crfm.stanford.edu/ecosystem-graphs/index.html?asset=GitHub%20CoPilot

Bommasani 20

Are these mechanisms enough?

No – language technology is evolving very quickly

​

Transparency and technical responses are just the starting point

​

More flexible and broad-coverage/general approaches

Bommasani 21

Ethics

Personal ethics: individual preferences and values

​

While people have different ethics, this is not a non-starter

​

Professional and organizational ethics: collective norms and decisions

Bommasani 22

Bommasani 23

Pace

Bommasani 24

Investment

Bommasani 25

Existing policy and regulation

Europe

• GDPR (flagship privacy regulation)
• DSA and DMA (flagship consumer protections of platform)

​

US

• NIST Risk Management Framework
• AI Accountability Act

Bommasani 26

Bommasani 27

Policy: UK

Bommasani 28

Policy: US

Bommasani 29

Policy: China

Bommasani 30

Policy: EU

Bommasani 31