Data Science

for Public Leaders

Class 06: Harnessing Insights from Text Data: Introduction to Text Mining and Natural Language Processing

Where we were:

Introduction to neural networks and machine vision/audio, and how they facilitate the collection and processing of data previously unsuitable for policy analysis

​

​

Where we are:

Introduction to natural language processing, and how it also facilitates the processing of text data previously unsuitable for policy analysis

Where we’re going:

Mid-semester check-in, followed by our unit on how predictive analytics can support valuable policy targeting

Class Overview

• Setting the Stage
• Text, text, everywhere, but not a measure to regress...
• Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)
• The (Likely) Future of NLP: Transformers
• Cautions and Concerns with Natural Language Processing
• Possible Uses of NLP
• Unit Overview

3

Text, Text, Everywhere...

At each station: For the provided policy area (e.g., health), work with your partners to brainstorm any “naturally accumulating” text documents in that policy area. Write any ideas you come up with on the sheet!

​

It may help to think about the sort of forms, statements, paperwork, etc. that are involved in the normal functioning of policy processes in that area.

...But Not a Measure to Regress

(c/o ChatterMill)

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

6

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

7

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

8

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

9

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

10

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

11

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

12

Topic Modeling Algorithm in a Nutshell:

​

1. Assume all “documents” are randomly assigned to be composed of some set of “topics”
2. Assume all “topics” are randomly assigned to be composed of some set of “words”
3. How well does this model explain what we see in the text data we have? If not so well, find document-topic and topic-word combinations that explain the text data better
4. Repeat steps 1-3 until you can’t really find a better set of combinations

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

13

Topic 1

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

14

Topic 1

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

15

Topic 1

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

16

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

17

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

18

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

19

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

20

Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)

21

We should always be keeping in mind:

​

1. What are we really trying to learn about?

​

2. What is the best evidence can we bring to bear for that phenomenon?

​

3. In what ways is our best evidence still imperfect?

In small groups, discuss:

​

1. What are we really trying to learn about?

​

1. Is this the best evidence can we bring to bear for that phenomenon?

​

1. In what ways is this evidence still imperfect?

A Small Aside: Word Embeddings

24

Word Embedding Algorithm in a Nutshell:

​

1. Assume all “words” are randomly assigned to be composed of some set of “concepts”
2. If two words are used nearly interchangeably across many examples of sentences, their “concepts” should be similar
• “I ran to the bank” “I ran down the street” “She ran a mile”
• “I walked to the bank” “I walked down the street” “She walked a mile”
3. How well does this model accomplish #2 given what we see in the text data we have? If not so well, find word-concept combinations that do #2 a little better
4. Repeat steps 1-3 until you can’t really find a better set of combinations

A Small Aside: Word Embeddings

25

A Small Aside: Word Embeddings

26

A Small Aside: Word Embeddings

27

A Small Aside: Word Embeddings

28

A Small Aside: Word Embeddings

29

A Small Aside: Word Embeddings

30

A Small Aside: Word Embeddings

31

A Small Aside: Word Embeddings

32

A Small Aside: Word Embeddings

33

Break

When we come back: The (Likely) Future of NLP: Transformers

The (Likely) Future of NLP: Transformers

The (Likely) Future of NLP: Transformers

The Transformer Architecture in a Nutshell:

1. Learn complex multi-word embeddings using *unbelievably large* text datasets
2. See many examples of a specific task -- complete input and output
3. Read the example input text to place its overall meaning in “vector space”
4. Read the example output text to place its overall meaning in “vector space”
5. Figure out how those two vector spaces relate
6. When given a new input text, “transform it” into the output in the same way as examples

The (Likely) Future of NLP: Transformers

The (Likely) Future of NLP: Transformers

The (Likely) Future of NLP: Transformers

https://transformer.huggingface.co/doc/gpt2-large

The (Likely) Future of NLP: Transformers

https://transformer.huggingface.co/doc/gpt2-large

Cautions and Concerns with NLP

• “Bias Encoded in Data”
• “Minimizing Average Error Fits Majority Populations”
• “The Need to Explore”

Cautions and Concerns with NLP

Ultimately, we need to always be asking:

​

1. Who is represented in the data? Who decides that?
2. How are they represented? How is that representation imperfect?
3. When the data are used, who will it benefit and harm? In what ways?
4. Who gets to choose how those benefits and harms are weighed?

Cautions and Concerns with NLP

Ultimately, we need to always be asking:

​

1. Who is represented in the data? Who decides that?
2. How are they represented? How is that representation imperfect?
3. When the data are used, who will it benefit and harm? In what ways?
4. Who gets to choose how those benefits and harms are weighed?

​

In small groups: Pick either the Quinn et al. or Fesler et al. paper. How would you answer the above questions?

Possible Uses of NLP

44

​

1. Data-Up:
1. What novel data can be collected? Then...
2. What relevant questions could we answer with these data?
2. Question-Down
• What question do we want to answer? Then...
• What relevant, scrapable data could we utilize to answer that question?

Possible Uses of NLP

45

​

1. Data-Up:
1. What novel data can be collected? Then...
2. What relevant questions could we answer with these data?

​

​

In small groups: Pick one of the text data sources we identified earlier in class. What’s a relevant question those data could be used and processed to answer?

Class Overview

• Setting the Stage
• Text, text, everywhere, but not a measure to regress...
• Introduction to Dictionary and Lexical Methods: Topic Modeling w/ Quinn et al. (2010)
• The (Likely) Future of NLP: Transformers
• Cautions and Concerns with Natural Language Processing
• Possible Uses of NLP
• Unit Overview

46

Unit Overview

• Data Acquisition in the Business-as-Usual
• Acquiring New Data:
• Web Scraping
• Processing “Old” Data:
• Machine Vision/Audio
• Natural Language Processing
• Fancy Methods Can’t Sidestep Fundamental Data Issues
• Keys Under the Lamppost
• Garbage-in, Garbage-out
• Fancy Methods Don’t Justify Bias and/or Invasion of Privacy
• Who do these methods work for?
• Who interprets and makes and deploys these models?

47