1 of 36

Amy Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B. Adams, Emmanuel Vincent, Jennifer 8. Lee, Martin Robbins, Ed Bice, Sandro Hawke, David Karger, and An Xiao Mina.

https://credibilitycoalition.org/results/

A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles.

2 of 36

2018 • April 8

credibilitycoalition.org • @credcoalition

an initiative led by

3 of 36

4 of 36

5 of 36

Who we are

6 of 36

Partners

7 of 36

Funders

8 of 36

Can we agree on scientific and systematic ways to assess the credibility of information, and whether they can be applied at scale? �

The big question

9 of 36

Amy Zhang, Aditya Ranganathan, Sarah Emlen Metz, Scott Appling, Connie Moon Sehat, Norman Gilmore, Nick B. Adams, Emmanuel Vincent, Jennifer 8. Lee, Martin Robbins, Ed Bice, Sandro Hawke, David Karger, and An Xiao Mina.

https://credibilitycoalition.org/results/

A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles.

10 of 36

Pilot

Credibility Coalition				Researchers, Platforms and AI systems
Story of unassessed credibility	Markup by participating organizations	Automated analysis, non-expert and expert review by humans	Aggregate testing data	Interpret data	Credibility scoring, algorithmic transparency	Story of assessed credibility

11 of 36

Pilot

50 articles total (40 in the paper)

9 annotators (6 in the paper)

16 indicators

12 of 36

Paper indicators

Content �

Title Representativeness
“Clickbait” Title
Quotes from Outside Experts
Citation of Organizations and Studies
Calibration of Confidence
Logical Fallacies ( e.g. false dilemma)
Tone
Inference

Context�

Originality
Fact-checked
Representative Citations
Reputation of Citations
Number of Ads
Number of Social Calls
“Spammy” Ads
Placement of Ads and Social Calls

Full definitions available at https://credweb.org/cciv/r/1\

13 of 36

Article selection

14 of 36

Article selection

15 of 36

Annotator process

6 journalism school students or recent graduates from Northwestern (3), Columbia (2), and Santa Clara (1)
3 UC Berkeley annotators came from Sense and Sensibility and Science course, selected for skills in scientific critical thinking
Each annotator received an annotation guide and compensation — either financial or class credit — with the assumption of about 10-20 hours of annotation.
Worked with a set of domain experts in public health and climate science to provide a “gold standard” assessment of credibility.

16 of 36

Annotator demographics

Average age: 22.1
Race/Ethnicity

4 Asian
4 White
1 Middle Eastern

Gender

5 Women
1 Man

Political Affiliation

3 Democrat
1 Republican
1 Independent
1 None

Economic Issues

3 Very or Moderately Liberal
1 Moderate
2 Very or Moderately Conservative

Social Issues

4 Very Liberal
2 Moderate

Publications Read

4 New York TImes
2 CNN
22 Other

17 of 36

Annotation • TextThresher

18 of 36

Annotation • Check

19 of 36

Preliminary findings

20 of 36

Some indicators do correlate with credibility

Representativeness of citations to outside sources
Results of a fact check from an IFCN signatory
Clickbait assessment of a title
Exaggerated or emotionally charged tone

Findings • Correlations with credibility

21 of 36

Clickbait Title

Listicle (“6 Tips on …”)
Defying convention (“Think Orange Juice is Good for you? Think Again!”, “Here are 5 Foods You Never Thought Would Kill You”)
Inducing fear (“Is Your Boyfriend Cheating on You?”)

Findings • Correlations with credibility

22 of 36

Exaggerated or emotionally charged tone

Findings • Correlations with credibility

23 of 36

The quantity of ads, content recommendation boxes and sponsored content do not have a significant correlation with credibility.

Findings • Advertising

24 of 36

The quantity of ads, content recommendation boxes and sponsored content do not have a significant correlation with credibility.

Findings • Advertising

25 of 36

The quantity of ads, content recommendation boxes and sponsored content do not have a significant correlation with credibility.

Findings • Advertising

26 of 36

Annotators’ assessment of ad placement as “aggressive” does suggest a difference between credible and non-credible content.

Findings • Advertising

27 of 36

“The page of the article calls for readers to share using multiple buttons, as well as the actual text, telling readers to share via social media.” — Annotator

Findings • Advertising

28 of 36

29 of 36

After answering annotation questions, annotators’ assessments of credibility aligned more closely with those of domain experts.

Findings • Media literacy

30 of 36

“The project has certainly got me thinking more about how to evaluate credibility…. I'm hoping that I was internally consistent as an evaluator.”

“Often times I found myself indecisive about how to treat certain ambiguities and giving rationale for my decisions.”

“From going through this process, I have a firm [belief] in the success of Credibility Coalition and where it will go. I have hopes that anyone can be an annotator.... This will be augmented with the help of AI, but only through a human partnership.”

“After having taken part in hours of article annotation, I can say that it is a tedious process that takes a toll on [one’s] mental stamina. Because it is quite challenging, those that don’t care about this work will be unwilling while those who do care will be hesitant; a result that would be detrimental to the means of fact checking.”

Findings • Annotator Feedback

31 of 36

Paper indicators

Content �

Title Representativeness
“Clickbait” Title
Quotes from Outside Experts
Citation of Organizations and Studies
Calibration of Confidence
Logical Fallacies ( e.g. false dilemma)
Tone
Inference

Context�

Originality
Fact-checked
Representative Citations
Reputation of Citations
Number of Ads
Number of Social Calls
“Spammy” Ads
Placement of Ads and Social Calls

Full definitions available at https://credweb.org/cciv/r/1

32 of 36

Data Set, Annotation Guides and Indicators

Data Set and Annotation Guides

https://data.world/credibilitycoalition/webconf-2018�

Indicators�https://credweb.org/cciv/r/1

Paper

https://credibilitycoalition.org/results/

33 of 36

Score? No
Structure, process, data? Yes!

Our goals, redux

34 of 36

More articles and annotators — goal is to develop data for 5,000 - 10,000 articles — with key goals around diversity and inclusion

Develop a series of ongoing research collaborations and a grant structure

Release an open data set following ethical data standards to support further research, and make available on Hypothesis on data.world

Work with W3C Credible Web Community Group to define indicator vocabulary

Explore potential use cases: media literacy, search, feeds, browser, citations

What’s next

35 of 36

Thank you!

Lead Researcher

Amy X. Zhang

axz@mit.edu

Credibility Coalition

hello@credibilitycoalition.org

Credible Web Community Group

Sandro Hawke and An Xiao Mina

team-credweb-chairs@w3.org

36 of 36

2018 • April

credibilitycoalition.org • @credcoalition

an initiative led by