1 of 32

Towards�Human-Level�Text Coding�with LLMs

1

�The Case of Fatherhood Roles�in Public Policy Documents�

Lorenzo Lupo, Oscar Magnusson,�Dirk Hovy, Elin Naurin, Lena Wängnerud

2 of 32

LLMs for text annotation…

Methodology:

  • Step1: Data Selection
  • Step2: Codebook
  • Step3: (Little) Manual Annotation
  • Step4: Choose Model and Prompt

​

Discussion about tradeoffs all along.

​

2

3 of 32

Why?

    • Large text corpora are very expensive to evaluate manually
    • Supervised-learning techniques still require thousands of annotated samples
    • LLMs can annotate text without requiring training (data)

3

4 of 32

Step1: Data Selection

​

Data selection:

    • Keyword search (Pappa, pappor, fäder, fader, far..)
    • Part of speech (POS) tagger
      • to filter out “far” (SWE “to go”) as verb 🡪 1901 fathers

​

​

4

RQ: How have fatherhood roles evolved over time (in Sweden)?

Material: All public documents from 1993 to 2021 from riksdagen.se.

16390 documents: a mix of motions, protokoll, betänkanden, propositioner…

5 of 32

Step 2: Codebook

  • Labels for “Role of the father in the family”
    1. NA: not applicable
    2. PASSIVE: traditional breadwinner
    3. ACTIVE_NEGATIVE: father as a threat.
    4. ACTIVE_POS_CARING: as the stereotypical female parental role
      1. nurturing, caring, loving
    5. ACTIVE_POS_ CHALLENGING: stereotypical male parental role
      • strengthening activity, risk-taking, sport
    6. ACTIVE_POS_ OTHER: positive, but substantive content is lacking

​

  • Codebook
    • Labels
    • Explanations
    • Examples

​

5

6 of 32

Step 3: (Little) Manual Annotation

  • Not necessary for using LLMs…

​

  • …but useful for evaluation

​

  • Validation set = 350 randomly extracted sentences (out of 1910)
    • Check if performance is satisfying
    • Try out different prompt configurations
    • Try out different models

​

  • 3 coders individually annotate a sample of 350 sentences.
    • About 3,5 hour per coder (~ 100 sentences per hour)

6

7 of 32

Step 4: Choose Model and Prompt

7

8 of 32

Step 4: Choose Model

​

  1. GPT through OpenAI’s API

8

9 of 32

Step 4: Choose Model - GPT

  • Minimal coding
  • Can run on your laptop
  • Requirements
    • Python
    • Storage/RAM for data
    • Minimal CPU for
      • Pre-processing
      • Post-processing

9

import openai

�# Set your OpenAI API key

openai.api_key = 'your_api_key_here'

�# Load a list of texts to annotate

texts = your_loading_function()

�# Loop through each text in the list

annotations = []

for text in texts:

# Call the API to generate annotations for the text

response = openai.Completion.create(

prompt=text,

engine="text-davinci-003",

max_tokens=50,

temperature=0.0,

)

�# Extract the generated annotation from the API response

annotation = response.choices[0].text

�# Append the annotation to the list of annotations

annotations.append(annotation)

10 of 32

Step 4: Choose Model

  1. GPT through OpenAI’s API
  2. Open-source LLMs through Hugging Face:
    • https://huggingface.co/models

​

10

11 of 32

Step 4: Choose Model – OS LLMs

  • Minimal coding
  • Can NOT run on your laptop
  • Requirements
    • Python
    • Storage/RAM for data
    • Minimal CPU for
      • Pre-processing
      • Post-processing
    • GPU for processing

​

    • Bocconi HPC hpc@unibocconi.it

​

11

from transformers import AutoModelForCausalLM, AutoTokenizer

�# Load the model and tokenizer

model_name = "meta-llama/Llama-2-13b-hf"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(model_name)

​

# Load a list of texts to annotate

texts = your_loading_function()

�# Loop through each text in the list

annotations = []

for text in texts:

# Tokenize the text

inputs = tokenizer.encode(text, return_tensors="pt")

�# Generate annotations using the model

with torch.no_grad():

outputs = model.generate(

inputs, max_length=50, temperature=0.7)

�# Decode the generated annotations

annotation = tokenizer.decode(

outputs[0],skip_special_tokens=True)

�# Append the annotation to the list of annotations

annotations.append(annotation)

12 of 32

Step 4: Choose Model

  1. GPT through OpenAI’s API
  2. Open-source LLMs through Hugging Face:
    • https://huggingface.co/models
  3. Open-source LLMs through APIs
    • E.g., https://groq.com/
  4. Other proprietary LLMs through APIs
    • E.g., https://gemini.google.com/app

​

​

12

13 of 32

Step 4: Choose Model and Prompt

How to evaluate options:

  • Each coder’s average agreement with all (other) humans
    • Cohen’s Kappa
    • Accuracy
    • macro-F1

​

​

13

14 of 32

Step 4: Choose Prompt

14

15 of 32

Step 4: Choose Prompt

15

16 of 32

Step 4: Choose Prompt

16

17 of 32

Step 4: Choose Prompt

17

18 of 32

Step 4: Choose Prompt

18

19 of 32

Step 4: Choose Prompt

19

20 of 32

Step 4: Choose Prompt

20

21 of 32

Step 4: Choose Model

21

22 of 32

Step 5: Annotate the Full Corpus

  • GPT4 with the best prompt
    • Faster than humans (270x)
    • Cheaper: (90$*) vs U.S. human estimate for 19 hours work (234$)
    • More reliable over long working hours?

​

22

23 of 32

23

24 of 32

Accessibility

  • Our code is available on GitHub (https://github.com/lorelupo/pappa)
  • “BETA-testing” with Oscar:
    • Half a week to get the same results
  • Pre-print article:

​

24

25 of 32

Next step: Substantial article

  • Scaling up to “parents”:
    • 1910 sentences is not very impressive.
    • We want to scale up to “parents” (<150 000 sentences)
      • Price will increase --> Need to decrease sample. Any ideas?
        • Only motions? Propositions?
        • Random sample from election periods?
      • Scaling down evaluation process.
        • Human(s) say Yes/No to GPT’s annotation.
        • Instead of 3 humans independentely annotating.
        • Convincing?

​

​

25

26 of 32

26

27 of 32

Conclusion

Summary:

  • GPT outperform humans in terms of:
    • Time
    • Cost
    • Quality (on par or better)
  • This paper shows it works for 1) novel data and 2) novel theoretical task.
    • (That it most probably has not trained on before)

​

Challenges:

    • Reproducibility (The model da-vinci-003 that we produced some results with is gone.)
    • Validation for each task (but actually not different from human coders)
    • Hallucinations?
      • Easily detected and handled in this task (e.g. Poem instead of Label)

27

28 of 32

This presentation…

  • Example of how LLMs can be adopted to (semi-) automate downstream research tasks with minimal development skills.
  • Our case: Evolution of fatherhood roles over time in Sweden.
  • Coding w/ LLMs: step by step
  • LLMs vs Human coding?

28

29 of 32

Ideas

29

30 of 32

Escpaing the loop

30

GPT

190 Gold-labels

1721 labels

BERT

Human

Instructions + 10 example sentences

Kappa = .65

Eval = .68

Full dataset

Train

Train

Annotate

Annotate

Annotate

Annotate

31 of 32

31

32 of 32

32