1 of 39

CS 162: Natural Language Processing

Saadia Gabriel

Lecture 12:

Sequence Labeling Models & NLP Ethics

2 of 39

Announcements

  • Reminder: mid-project report due Wednesday 11:59pm
  • Opinion pieces are not required to and typically don’t have a technical component
  • Please respond to LA final survey for extra credit on homework 3 (Bruin Learn, week 8)

3 of 39

Quiz #4

https://forms.gle/oEfMK74ZZcE3FFYL9

4 of 39

Recap: Sequence Tagging

5 of 39

Recap: HMM Prediction

6 of 39

Goals of This Lecture

  • Introduce other models that can be used for sequence tagging (MEMMs, CRFs)
  • Discuss hybrid neural models
  • Explain the Viterbi algorithm

7 of 39

Viterbi Algorithm

  • Now we can introduce dynamic programming

8 of 39

Viterbi Algorithm

Observations

9 of 39

Viterbi Algorithm

10 of 39

Viterbi Algorithm

11 of 39

Viterbi Algorithm

12 of 39

Viterbi Algorithm

k

13 of 39

Viterbi Algorithm

14 of 39

Viterbi Algorithm

15 of 39

Viterbi Algorithm

16 of 39

Viterbi Algorithm

17 of 39

Applications of HMMs

18 of 39

Learning HMM Parameters

19 of 39

Supervised Learning

20 of 39

But what if…

21 of 39

Recap:

Generative vs. Discriminative Models

22 of 39

Maximum Entropy Markov Models

(MEMMs)

23 of 39

Decomposition Based on Markov Assumption

24 of 39

Modeling Conditional Probability

25 of 39

Combining MEMMs with Transformers

This Generalizes to CRFs

26 of 39

Inference for MEMMs

Guess?

27 of 39

Inference for MEMMs

28 of 39

HMM vs. MEMM

The model is biased towards states with fewer transitions due to normalization for each state, regardless of the observed sequence

29 of 39

Conditional Random Fields (CRFs)

  • CRFs address the label bias issue

Let’s normalize globally instead of locally

Global partition function ensures we have valid probabilities (a function of the observation sequence and not the current state).

Computed using forward-backward algorithm with more dynamic programming

(J&M Appendix A)

30 of 39

Ethics of NLP

31 of 39

Types of Sampling Bias in Naturalistic Data

32 of 39

Effect on Downstream Tasks

​

​

African-American English tweets are considerably more likely to be misclassified as offensive compared to White-aligned tweets

​

- Sap, Card and Gabriel et al. (2019)

33 of 39

Socially Biased Representations

and Generations

Recall HW1

Sheng et al. (2019)

34 of 39

Factually Inaccurate or Toxic Generated Language

35 of 39

Jailbreaking

Courtesy of VentureBeat

36 of 39

Biased Objectives

Ongoing NLP debate: sometimes the problems we work on shouldn’t be solved with AI

IQ Prediction

Automatic Criminal Sentencing

Hofmann et al. (2024)

37 of 39

“Ethics is a study of what are good and bad ends to

pursue in life and what it is right and wrong to do in the

conduct of life.

​

It is therefore, above all, a practical discipline.

Its primary aim is to determine how one ought to live and

what actions one ought to do in the conduct of one’s life.”

​

Introduction to Ethics, John Deigh

38 of 39

Summary

  • Perform careful analysis of data to understand what biases may be reflected
  • Think carefully about whether the problems we’re trying to solve with AI are well-specified enough to avoid harm
  • Evaluate extensively, considering the characteristics of deployment settings and populations who may be affected by algorithms

Paper peer review typically includes an ethics review

39 of 39

Student Presentations