Real or Fake Text?�Investigating Human Ability to Detect Boundaries Between Human-Written and Machine-Generated Text

Presenter: Amin Bonyad

Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, Chris Callison-Burch

2022-12-24

April 27, 2023

Topic

NLG

RoFT

Main Findings

Model Comparison Findings

Takeaways

Future Work

​

​

PRESENTATION TITLE

2

4/27/2023

How does Natural Language Generation (NLG) work?

Real or Fake Text?

3

4/27/2023

NLG system

Neural

language

model

(GPT-2, GPT-3, BLOOM, GROVER, etc.)

"Yesterday, I went to"

home

gym

cinema

job

⋮

“university "

university

How does natural language generation (NLG) work?

Real or Fake Text?

4

4/27/2023

"Yesterday, I went to"

“university "

"Yesterday, I went to university"

“by "

"Yesterday, I went to university by"

“bicycle "

Real or Fake Text?

5

4/27/2023

Real or Fake Text?

6

4/27/2023

ChatGPT

Real or Fake Text?

7

4/27/2023

Credit: InsideHigherEd, Washington Post, Nature, EuroNews

What is machine-generated text?

Credit: Jay Alammar

Chat GPT and Cheating

Real or Fake Text?

9

4/27/2023

Can people detect generated text?

​

Credit: New York Post

Previous Work: The Binary Detection Task

Real or Fake Text?

10

4/27/2023

Problems with this approach:

• They do not learn why the users selected Yes or No
• Setting is unrealistic
• Annotators get tired or bored
• No interactive feedback

Is this text written by a human?

​

Once upon a time there lived a boy.

He was a boy no longer, but a soldier. He was a soldier no longer, but a warrior. He was a warrior no longer, but a legend.

​

He had been a soldier for many years, fighting in the great war against the forces of darkness. He served under the great generals of the time, the likes of which would...

Research Goal

Real or Fake Text?

11

4/27/2023

Gamify the detection task in order to conduct a large scale study of human detection ability

Generations form many Genres

Real or Fake Text?

12

4/27/2023

Real or Fake Text?

13

4/27/2023

http://roft.io

Real or Fake Text?

14

4/27/2023

http://roft.io

Real or Fake Text?

15

4/27/2023

http://roft.io

Players earn 5 point for guessing on the boundry:

​

4 for guessing

One sentence after

​

Advantages of the RoFT Platform

Real or Fake Text?

16

4/27/2023

• More realistic annotation setting.
• Sentence-level interactivity forces annotators to read carefully.
• Leaderboard and personal stats pages encourage annotators to keep improving.
• They collect data on why decisions are made.
• Time tracking and other statistics.
• Can be used as an educational tool.

Where do the generations in RoFT come from?

Real or Fake Text?

17

4/27/2023

Systematically Vary

​

• Model Size (small, XL, GPT-2)

• Decoding Strategy ( )

• Fine-tuning (with, without)

GPT-2 XL

Using nucleus sampling with

Main System:

Study Population

Real or Fake Text?

18

4/27/2023

• 241 Students participated
• All senior undergrad and grad students from a UPenn AI Course.
• Most are computer science students

**Not reflective of global population!**

Data collected from RoFT platform

Real or Fake Text?

19

4/27/2023

Dates: September 20th 2021 – October 15th 2021

Data ref: https://github.com/liamdugan/human-detection

Main Findings

Can humans detect generated text?

Real or Fake Text?

21

4/27/2023

Detection ability varies substantially

Real or Fake Text?

22

4/27/2023

players were much better than the mean

These are the same players

Best Players Detection

Real or Fake Text?

23

4/27/2023

77.3%

90.09%

4.32

Best

Random

Chance

Players

10.0%

23.4%

“Perfect Guess” accuracy

50.0%

72.3%

“Correct side of boundary”

accuracy

1.31

2.08

Mean score

Good players agree with each other more

Real or Fake Text?

24

4/27/2023

Good Players make similar errors!

Can bad players train to perform like the good players?

Guide Document

Real or Fake Text?

25

4/27/2023

The goal of "Real or Fake Text" (RoFT) is to conduct research into how hard it is to detect when a text switches from human writing to being written by an Al. This guide is designed to give you a sense of the kind of things you should look out for when trying to identify whether a sentence was machine-generated.

​

​

Real or Fake Text Guide

https://github.com/liamdugan/human-detection/blob/main/data/guide.pdf

​

Guide Document

Real or Fake Text?

26

4/27/2023

Here are some of the signs you can look for to tell that a sentence might be machine-generated. Note that none of these are hard rules; rather they are general trends we have observed in machine-generated text.

[1] Text that is not grammatical.

Sometimes generated text will have weird, grammatical disfluencies. But remember, humans can write ungrammatically too (especially when they're posting on the internet).

[2] Text that substantially repeats previous text or itself.

Sometimes the Al system gets stuck in loops and writes the same thing multiple times in a row, or it repeats the same phrase multiple times but in slightly different ways.

[3] Text that is irrelevant or unrelated to the previous sentences.

Often, machine-generated text will meander into a new topic and never return to the original topic.

​

Signs of machine-generated text

Guide Document

Real or Fake Text?

27

4/27/2023

Incorrect sentence boundary segmentation

Our game tries to show you one sentence at a time. Sentence boundaries were automatically parsed, and could be wrong. Errors in sentence segmenting do not imply the text was machine-generated.

​

Bad writing

In domains like New York Times articles, bad writing may indicate that the text was machine-generated, but in domains like the Reddit Short Stories, it may not. Remember that amateur human writers often write text that is incoherent, typo-ridden, and otherwise not very good.

Features that may be misleading

Guide Document

Real or Fake Text?

28

4/27/2023

Some concrete examples

Real or Fake Text?

29

4/27/2023

Can people be trained to detect text?

• Students in Intro to Al course
• Received fixed extra credit
• No guide or help doc

• Students in Intro to Al course
• Received extra credit proportional to points scored
• Given guide document and tips for how to detect generated text

Group A

(Control Group)

Group B

(Study Group)

Real or Fake Text?

30

4/27/2023

Humans can learn to detect generated text!

w/o incentives (A)

with incentives and instructions (B)

Time

Time

Players get better over time!

Given the right incentives and some guidance on what to look for

Real or Fake Text?

31

4/27/2023

Error Taxonomy

What errors do models make?

Real or Fake Text?

32

4/27/2023

Grammar is not a reliable error type

People who notice common sense errors have almost

1.5x the score

Real or Fake Text?

33

4/27/2023

Result

• News and Stories are the hardest to detect
• Recipes are the easiest to detect
• Recipes are more structured and require more common sense reasoning

Real or Fake Text?

34

4/27/2023

Error types change across genres

Model Comparison Findings

Where do the generations in RoFT come from?

Real or Fake Text?

36

4/27/2023

Systematically Vary

​

• Model Size (small, XL, GPT-3)

• Decoding Strategy ( )

• Fine-tuning (with, without)

GPT-2 XL

Using nucleus sampling with

Main System:

PRESENTATION TITLE

37

4/27/2023

Are certain genres of text easier to detect?

• Model size has a very significant effect on detection accuracy
• This confirms previous work’s findings here
• Fine-tuning does not seem to affect detection accuracy
• Perplexity of fine-tuned model on recipes was about half that of the pre-trained model (!)
• May have been better in more specialized domain (e.g. legal or medical)
• Control code usage does not seem to affect detection accuracy
• Likely that the errors picked up on by humans are not domain related

PRESENTATION TITLE

38

4/27/2023

More diverse text easier to detect

• p = %age of probability mass sampled from when generating a new token
• Lower p = less diverse but also less noisy

​

• p=1.0 is significantly easier to detect
• Validating results from previous work
• p=0.0 and p=0.4 are both difficult to detect

​

• Models struggle to generate high quality text with the diversity of human text

Does domain familiarity affect detectability?

• No, self-reported domain familiarity has little effect on detectability
• Familiarity with NLG & reading our help guide do affect performance
• Interestingly, being a native speaker has higher mean but lower variance
• Most of our best annotators were non-native speakers!

Takeaways

PRESENTATION TITLE

41

4/27/2023

You can train yourself to spot generated text

Before

Detection Training

After

Detection Training

Real or Fake Text?

42

4/27/2023

Models make different errors on different genres

• Detecting skill may not transfer across Genre or Model
• Focus training on areas where generated text is most harmful

Takeaways

Real or Fake Text?

43

4/27/2023

• Introduced boundary detection as a novel formulation of the detection task
• useful for understanding sentence level features
• Confirmed findings from previous work at larger scale
• Humans over-condition on grammatical errors
• p=1.0 is easier to detect than p=0.0
• Bigger models are more difficult to detect
• There is significant variance in annotator ability
• Added new results of Their own
• Annotators may improve over time given the proper incentives
• Different domains & p-values have different distributions over error types
• Humans tend to think longer sentences with more named entities are generated
• Fine-tuning and control code usage do not seem to affect detection accuracy

Real or Fake Text?

44

4/27/2023

Future Work

Real or Fake Text?

45

4/27/2023

Question from Discord

Thank you

Amin Bonyad

Amin.bonyad.khalaj@umontreal.ca

www.AminBonyad.com