1 of 81

Probing Covert Racism in Language Models Through the Lens of Dialect Prejudice

Valentin Hofmann��UHH Data Science Group, 12/16/2024

2 of 81

V. Hofmann, P. R. Kalluri, D. Jurafsky, S. King. AI generates covertly racist decisions about people based on their dialect. Nature, 633:147–154, 2024.��F. Lin, S. Mao, E. La Malfa, V. Hofmann, A. de Wynter, J. Yao, S. Chen, M. Wooldridge, F. Wei. One language, many gaps: Evaluating dialect fairness and robustness of large language models in reasoning tasks. arXiv:2410.11005.

​

​

3 of 81

Racial Bias in AI Systems

Facial recognition

4 of 81

Racial Bias in AI Systems

Facial recognition

Criminal risk assessment

5 of 81

Racial Bias in AI Systems

Facial recognition

Criminal risk assessment

What about language models (LMs)?

6 of 81

Example Study 1: Sheng et al. (2019)

​

7 of 81

Example Study 1: Sheng et al. (2019)

​

8 of 81

Example Study 2: Tamkin et al. (2023)

​

9 of 81

Example Study 2: Tamkin et al. (2023)

​

10 of 81

Example Study 2: Tamkin et al. (2023)

positive discrimination

negative discrimination

11 of 81

Example Study 2: Tamkin et al. (2023)

Evidence for positive racial discrimination in Claude 2.0

positive discrimination

negative discrimination

12 of 81

Has racial bias in LMs �been resolved?

13 of 81

Shortcoming of Prior Work

  • Prior work focused on racial stereotypes and discrimination triggered by explicit mentions of race (e.g., “Black man”)

14 of 81

Shortcoming of Prior Work

  • Prior work focused on racial stereotypes and discrimination triggered by explicit mentions of race (e.g., “Black man”)
  • Racism can manifest in subtle forms, which have largely been overlooked

15 of 81

Shortcoming of Prior Work

  • Prior work focused on racial stereotypes and discrimination triggered by explicit mentions of race (e.g., “Black man”)
  • Racism can manifest in subtle forms, which have largely been overlooked
  • Colorblindness
    • Racist behavior is overtly rejected (“I don't see �color. I just see people.”)
    • Racism continues to exist on a more covert level
    • Example: residential choices

16 of 81

Raciolinguistic Stereotypes

  • Stereotypes due to raciolinguistic ideologies (Rosa and Flores, 2017)
  • Speakers of African American English (AAE) experience discrimination in a range of contexts, including education, employment, and legal outcomes

17 of 81

Raciolinguistic Stereotypes

  • Stereotypes due to raciolinguistic ideologies (Rosa and Flores, 2017)
  • Speakers of African American English (AAE) experience discrimination in a range of contexts, including education, employment, and legal outcomes
  • Example: Rachel Jeantel’s testimony in George Zimmerman trial dismissed as incomprehensible and not credible (Rickford and King, 2016)

Jeantel

Martin

Zimmerman

18 of 81

Raciolinguistic Stereotypes

  • Raciolinguistic stereotypes encoded on the web

19 of 81

Raciolinguistic Stereotypes

  • Raciolinguistic stereotypes encoded on the web
  • LMs are trained on this data
  • Do LMs pick up raciolinguistic stereotypes?

20 of 81

Questions of This Talk

  • Do LMs exhibit raciolinguistic stereotypes about speakers of AAE?
  • In what way do raciolinguistic stereotypes affect the decisions that LMs make about speakers of AAE?
  • How can raciolinguistic stereotypes in LMs be resolved?
  • Can raciolinguistic stereotypes affect the downstream performance of LMs?

21 of 81

Questions of This Talk

  • Do LMs exhibit raciolinguistic stereotypes about speakers of AAE?
  • In what way do raciolinguistic stereotypes affect the decisions that LMs make about speakers of AAE?
  • How can raciolinguistic stereotypes in LMs be resolved?
  • Can raciolinguistic stereotypes affect the downstream performance of LMs?

22 of 81

Methodology

  • We want to measure the stereotypes that LMs exhibit about speakers of AAE compared to speakers of Standardized American English (SAE)
  • We draw upon the matched guise technique developed in sociolinguistics
    • Participants listen to audio recordings in two languages and are asked to make judgments about various traits of the speakers

23 of 81

Methodology

  • We want to measure the stereotypes that LMs exhibit about speakers of AAE compared to speakers of Standardized American English (SAE)
  • We draw upon the matched guise technique developed in sociolinguistics
    • Participants listen to audio recordings in two languages and are asked to make judgments about various traits of the speakers
    • Both recordings were spoken by the same (bilingual) speaker

24 of 81

Matched Guise Probing

25 of 81

Matched Guise Probing

Meaning-matched texts in SAE and AAE

26 of 81

Matched Guise Probing

Meaning-matched texts in SAE and AAE

Prompts asking for speaker traits

27 of 81

Matched Guise Probing

Meaning-matched texts in SAE and AAE

Prompts asking for speaker traits

Language model

28 of 81

Matched Guise Probing

Meaning-matched texts in SAE and AAE

Prompts asking for speaker traits

Language model

Adjectives

29 of 81

Computing AAE Association Scores

  • p(x|v(t); θ): probability of an adjective x following an (AAE or SAE) text t embedded in a prompt v, given an LM θ

30 of 81

Computing AAE Association Scores

  • p(x|v(t); θ): probability of an adjective x following an (AAE or SAE) text t embedded in a prompt v, given an LM θ
  • AAE association scores for adjectives:

31 of 81

Computing AAE Association Scores

  • p(x|v(t); θ): probability of an adjective x following an (AAE or SAE) text t embedded in a prompt v, given an LM θ
  • AAE association scores for adjectives:

n AAE/SAE text pairs

32 of 81

Computing AAE Association Scores

  • p(x|v(t); θ): probability of an adjective x following an (AAE or SAE) text t embedded in a prompt v, given an LM θ
  • AAE association scores for adjectives:

n AAE/SAE text pairs

probability of adjective x following AAE version

33 of 81

Computing AAE Association Scores

  • p(x|v(t); θ): probability of an adjective x following an (AAE or SAE) text t embedded in a prompt v, given an LM θ
  • AAE association scores for adjectives:

n AAE/SAE text pairs

probability of adjective x following SAE version

probability of adjective x following AAE version

34 of 81

Interpretation of AAE Association Scores

  • q(x; v, θ) > 0: LM θ associates adjective x more with AAE given prompt v

35 of 81

Interpretation of AAE Association Scores

  • q(x; v, θ) > 0: LM θ associates adjective x more with AAE given prompt v
  • q(x; v, θ) < 0: LM θ associates adjective x more with SAE given prompt v

36 of 81

Data

  • Two sets of texts:
    • Meaning-matched: AAE tweets and SAE translations (Groenwold et al., 2020)
    • Non-meaning-matched: AAE and SAE tweets with different content (Blodgett et al., 2016)
  • Adjectives are from the Princeton Trilogy (Katz and Braly, 1933; Gilbert, 1951; Karlins et al., 1969)

37 of 81

Experimental Setup

  • We analyze the covert, raciolinguistic stereotypes of LMs and the overt stereotypes that LMs show when race is explicitly mentioned
    • Example covert prompt: A person who says [TEXT] is [ADJECTIVE]
    • Example overt prompt: A person who is Black is [ADJECTIVE]

38 of 81

Experimental Setup

  • We analyze the covert, raciolinguistic stereotypes of LMs and the overt stereotypes that LMs show when race is explicitly mentioned
    • Example covert prompt: A person who says [TEXT] is [ADJECTIVE]
    • Example overt prompt: A person who is Black is [ADJECTIVE]
  • We compare the stereotypes of LMs with those of humans from the Princeton Trilogy as well as a recent reinstallment (Bergsieker et al., 2012)
  • Five LMs: RoBERTa, GPT2, GPT3.5, GPT4, T5

39 of 81

Top Stereotypes About African Americans

40 of 81

Top Stereotypes About African Americans

Covert stereotypes of all LMs are more negative than human stereotypes reported in any year

Adjectives with highest average association scores q(x; v, θ)

41 of 81

Top Stereotypes About African Americans

Overt stereotypes of all LMs are much more positive than their covert stereotypes

Covert stereotypes of all LMs are more negative than human stereotypes reported in any year

Adjectives with highest average association scores q(x; v, θ)

42 of 81

Top Stereotypes About African Americans

Adjectives with highest average association scores q(x; v, θ)

Covert stereotypes of GPT2, RoBERTa, and T5 are strikingly similar to human stereotypes from 1933!

43 of 81

Top Stereotypes About African Americans

Adjectives with highest average association scores q(x; v, θ)

Stereotypes for GPT3.5 and GPT4 have the complete opposite direction in the overt versus the covert setting!

44 of 81

Favorability Analysis

  • We measure the average favorability of the top stereotypes based on human favorability ratings for the adjectives (Bergsieker et al., 2012)

45 of 81

Favorability Analysis

  • We measure the average favorability of the top stereotypes based on human favorability ratings for the adjectives (Bergsieker et al., 2012)
  • The covert stereotypes in LMs are more negative than any human stereotypes about African Americans ever experimentally recorded

46 of 81

Favorability Analysis

  • We measure the average favorability of the top stereotypes based on human favorability ratings for the adjectives (Bergsieker et al., 2012)
  • The covert stereotypes in LMs are more negative than any human stereotypes about African Americans ever experimentally recorded
  • The overt stereotypes in LMs are much more positive

47 of 81

Temporal Agreement Analysis

  • The covert stereotypes in LMs agree the most with human stereotypes from before the civil rights movement
  • The overt stereotypes agree the most with human stereotypes from 2012

48 of 81

Is It Really a Prejudice Against AAE?

  • Raciolinguistic stereotypes are triggered by linguistic features of AAE alone
  • Dialect features vary in terms of how strongly they evoke the stereotypes

49 of 81

Is It Really a Prejudice Against AAE?

  • Raciolinguistic stereotypes are triggered by linguistic features of AAE alone
  • Dialect features vary in terms of how strongly they evoke the stereotypes

Use of invariant be for habitual aspect as in he be drinkin

50 of 81

Alternative Explanation 1

  • Could it be a bias against dialects in general?

51 of 81

Alternative Explanation 1

  • Could it be a bias against dialects in general?
  • We measure how strongly other dialects (Appalachian English and Indian English) evoke the 1933 human stereotypes about African Americans
  • Substantially worse match for other dialects

52 of 81

Alternative Explanation 2

  • Could it be a bias against general deviations from SAE, irrespective of how these deviations look like?

53 of 81

Alternative Explanation 2

  • Could it be a bias against general deviations from SAE, irrespective of how these deviations look like?
  • We measure how strongly perturbed texts (random insertions, deletions, and substitutions) evoke the 1933 stereotypes about African Americans
  • Substantially worse match but much higher perplexity!

54 of 81

LMs hold covert stereotypes about African Americans that are triggered by dialect

55 of 81

Questions of This Talk

  • Do LMs exhibit raciolinguistic stereotypes about speakers of AAE?
  • In what way do raciolinguistic stereotypes affect the decisions that LMs make about speakers of AAE?
  • How can raciolinguistic stereotypes in LMs be resolved?
  • Can raciolinguistic stereotypes affect the downstream performance of LMs?

56 of 81

Experimental Setup

  • Employability: What occupations do LMs assign to people?
    • Example prompt: A person who says [TEXT] is a [OCCUPATION]

57 of 81

Experimental Setup

  • Employability: What occupations do LMs assign to people?
    • Example prompt: A person who says [TEXT] is a [OCCUPATION]
  • Criminality: What judicial decisions do LMs make about people?
    • Conviction experiment: Do LMs acquit or convict a defendant? Example prompt: He is accused of committing a crime. He says: [TEXT] He should be [DECISION]
    • Death penalty experiment: Do LMs sentence a defendant to life or death? Example prompt: He committed first-degree murder. He says: [TEXT] He should be sentenced to [DECISION]
    • We take the outcome with the higher calibrated probability as the LM’s decision

58 of 81

Experimental Setup

  • Employability: What occupations do LMs assign to people?
    • Example prompt: A person who says [TEXT] is a [OCCUPATION]
  • Criminality: What judicial decisions do LMs make about people?
    • Conviction experiment: Do LMs acquit or convict a defendant? Example prompt: He is accused of committing a crime. He says: [TEXT] He should be [DECISION]
    • Death penalty experiment: Do LMs sentence a defendant to life or death? Example prompt: He committed first-degree murder. He says: [TEXT] He should be sentenced to [DECISION]
    • We take the outcome with the higher calibrated probability as the LM’s decision

Note that the use of LMs in such contexts is inherently problematic, and we do not support it in any way!

59 of 81

Employability Analysis

  • Occupations that exhibit a low association with AAE consistently require a university degree (e.g., professor, architect, economist)
  • This is not the case for occupations that exhibit a high association with AAE

60 of 81

Employability Analysis

  • We analyze the impact of occupational prestige (US General Social Survey)
  • Association with AAE predicts prestige of occupations

61 of 81

Employability Analysis

  • We analyze the impact of occupational prestige (US General Social Survey)
  • Association with AAE predicts prestige of occupations

Most jobs get less likely with AAE!

62 of 81

Criminality Analysis

  • AAE leads to a higher rate of detrimental judicial decisions in both settings

63 of 81

Dialect prejudice in LMs perpetuates discrimination against African Americans

64 of 81

Questions of This Talk

  • Do LMs exhibit raciolinguistic stereotypes about speakers of AAE?
  • In what way do raciolinguistic stereotypes affect the decisions that LMs make about speakers of AAE?
  • How can raciolinguistic stereotypes in LMs be resolved?
  • Can raciolinguistic stereotypes affect the downstream performance of LMs?

65 of 81

Experimental Setup

  • We explore two strategies that have been proposed to mitigate racial performance differences and bias in LMs
  • Strategy 1: model scaling (i.e., increasing the model size)
  • Strategy 2: human feedback training

66 of 81

Scaling Analysis

  • Larger LMs are better at processing AAE (left)

67 of 81

Scaling Analysis

  • Larger LMs are better at processing AAE (left)
  • Larger LMs show less overt prejudice (right)

68 of 81

Scaling Analysis

  • Larger LMs are better at processing AAE (left)
  • Larger LMs show less overt prejudice (right)
  • Larger LMs show more covert prejudice (right)

69 of 81

Human Feedback Analysis

  • We compare GPT3 (no human feedback) with GPT3.5 (human feedback)
  • Human feedback helps mitigate overt stereotypes

70 of 81

Human Feedback Analysis

  • We compare GPT3 (no human feedback) with GPT3.5 (human feedback)
  • Human feedback helps mitigate overt stereotypes
  • Human feedback has no clear effect on covert stereotypes

71 of 81

Currently used methods do not resolve dialect prejudice

72 of 81

Questions of This Talk

  • Do LMs exhibit raciolinguistic stereotypes about speakers of AAE?
  • In what way do raciolinguistic stereotypes affect the decisions that LMs make about speakers of AAE?
  • How can raciolinguistic stereotypes in LMs be resolved?
  • Can raciolinguistic stereotypes affect the downstream performance of LMs?

73 of 81

Model Sandbagging

  • LMs can give worse answers to users who give indications of being less educated (Chen et al., 2024)
  • LMs stereotypically associate speakers of AAE with less education
  • Do LMs give worse answers as a result?

74 of 81

Experimental Setup

  • Creation of dialect reasoning benchmark ReDial

75 of 81

General Performance

All tested LMs perform substantially worse on dialectal input

76 of 81

Comparison with Perturbed Text

  • We compare the performance on AAE with the performance on perturbed text, where we increasingly add more noise
  • LMs perform significantly worse on AAE compared to perturbed text that they “understand” equally well

77 of 81

Comparison with Perturbed Text

  • We compare the performance on AAE with the performance on perturbed text, where we increasingly add more noise
  • LMs perform significantly worse on AAE compared to perturbed text that they “understand” equally well
  • Might indicate model sandbagging for speakers of AAE

78 of 81

Dialect prejudice might affect the performance on downstream tasks

79 of 81

Conclusion 1

  • LMs maintain a form of covert racism that is triggered by dialect features
  • The overt and covert racial stereotypes in LMs are often in contradiction with each other
  • This is reflective of the inconsistent racial attitudes in the US: most people report positive attitudes about African Americans in surveys but perpetuate racial inequalities through their unconscious behavior (Bonilla-Silva, 2014)

80 of 81

Conclusion 2

  • Covert racism in LMs has the potential for massive real-world harm
  • Business and jurisdiction are areas for which AI systems involving LMs are currently being developed or deployed
  • Downstream performance might also be affected, beyond the generally weaker performance on dialectal data
  • We hope to raise awareness of this form of AI bias among the research community as well as the general public

81 of 81

Thank You for Your Attention!

E-mail: valentinh@allenai.org�X: @vjhofmann�Bluesky: @valentinhofmann