Export from CCIV
 Share
The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss

 
$
%
123
 
 
 
 
 
 
 
 
 
 
 
 
 
 
ABCDEFGHIJKLMNOPQRSTUVWXYZAAABACAD
1
Signal LabelDefinitionTagsCommentFrame as a QuestionDefinitionTagsCategory
Potential Markup Effort
Potential Impact
Proposed Priority for WebConf Paper
Priority Rationale
Interoperability
Interoperability Example
Data TypeLocation
Rationale for Inclusion
Good Toy Example
Problematic Toy Example
Article Example URL
Dynamic; Change over Time?
Machine Generation from Article
Ideal Annotator Credential
Supporting Evidence
Known Weakness
Release Target
Source TeamNotes
2
Attribution of Non-Original ContentIf the content of the article is not original, was attribution given and if so, was the attribution accurate? *** (A) Attribution was not given
(B) Attribution was given but was inaccurate
(C) Attribution was given and was accurate
(D) Unclear which is the original
ideaIf the content of the article is not original, was attribution given and if so, was the attribution accurate?(A) Attribution was not given
(B) Attribution was given but was inaccurate
(C) Attribution was given and was accurate
(D) Unclear which is the original
Article Structure
MediumMediumMediumMultiple ChoiceArticle Context
Originality or non-originality is not by itself a sufficient condition for measuring credibility. Understanding the attribution of non-original content is going to be key.
1
Credibility Coalition Study: Web Conference 2018
3
Is OriginalHas the text of this article appeared in exactly the same words or very similar words in another publication? *** (A) Most likely original
(B) Appears to be a copy of one or more articles, with some portions different or remixed
(C) Extensive quoting from another source, with some original content
(D) A wholesale duplicate of another article
ideaHas the text of this article appeared in exactly the same words or very similar words in another publication?(A) Most likely original
(B) Appears to be a copy of one or more articles, with some portions different or remixed
(C) Extensive quoting from another source, with some original content
(D) A wholesale duplicate of another article
Article Structure
MediumVery High
Like doing a reverse image search, knowing that the article is original does not by itself guarantee credibility. However, knowing if it's *not* original can be a flag that the site is either duplicating content or running content from a newswire service. We might want to set this as scalar, as there might be slight modifications to original content.
Multiple ChoiceArticle Context
Sometimes article texts get repurposed in new publications. This can be due to licensing agreements from a wire service or the article can simply be stolen without crediting the original article. Sometimes the article is copied wholesale while other times some words are changed or a new article is created that copies from multiple articles. Finally, some articles will quote extensively from one or more other articles with only a small amount of original reporting or writing.
1
Credibility Coalition Study: Web Conference 2018
4
Publication (site)*** Parent of the Article.ideaParent of the Article.@
Article Structure
Very LowLow
5
Dateline (Date)When does the article claim it was published? *** From An: "i think technically this states both the date and location of the article’s publication. in which case it needs two forms of operationalizationideaWhen does the article claim it was published?From An: "i think technically this states both the date and location of the article’s publication. in which case it needs two forms of operationalization
Article Structure
LowLowDate and Timechecked
6
Headline[The article] has a title or headline.idea-GLWhat title or headline does this article offer for itself?
Article Structure
Very LowLowcheckedchecked
7
AuthorThe article has a byline identifying an author or authors.idea-GLArticles 1:M authorship. Note that not all articles have bylines, even in traditional news sources. Bylines also don't always start with 'by' (https://medium.com/@rchang/advice-for-new-and-junior-data-scientists-2ab02396cf5b)Who is the author of the article?Articles 1:M authorship. Note that not all articles have bylines, even in traditional news sources. Bylines also don't always start with 'by' (https://medium.com/@rchang/advice-for-new-and-junior-data-scientists-2ab02396cf5b)
Article Structure
LowLow
Schema.org,NewsML-G2 (IPTC)
http://schema.org/author, http://www.iptc.org/std/NewsML-G2/2.25/specification/XML-Schema-Doc-Power/NewsItem.html#Link60
Text
"By Abigail Beall" (https://www.newscientist.com/article/2151823-bitcoin-mining-uses-more-energy-than-ecuador-but-theres-a-fix/)
No known author [author is 'the publication']
https://www.newscientist.com/article/mg23631503-200-is-modern-life-making-todays-teenagers-more-depressed/
checked
8
Length[The article] contains [#] words.idea-GLNeed to count the words in the article.How many words does the article contain?
Article Structure
Very LowLowIntegercheckedchecked
9
Language[The article] language is [language].idea-GLNeeds a tool that identifies the language.What language is the article written in?
Article Structure
MediumLowMultiple Choicechecked
10
Correction/RedactionThe article contains a stated correction or redaction.idea-GLDoes the article contain a stated correction or redaction?
Article Structure
HighMedium
Boolean or Multiple Choice
checkedchecked
11
Article AwardsThe article identifies awards it or the website has received.idea-GL *** Awards are also assigned to specific Articles (but rarely)
Awards are also assigned to specific Articles (but rarely)
Article Structure
HighMedium
12
Article Locator *** URL, DOI, ideaURL, DOI,
Article Structure
HighLow
13
GenreWhat is the state genre, if available? *** Opinion, Feature, Biography -- but it will not always be labeled and i am not sure this will be defined consistently ideaWhat is the state genre, if available?Opinion, Feature, Biography -- but it will not always be labeled and i am not sure this will be defined consistently
Article Structure
MediumMedium
NewsCodes (IPTC)
http://show.newscodes.org/index.html?newscodes=genre&lang=en-GB&startTo=Show
Multiple Choice (should include Unknown/Unspecified)
Opinion
Quote (see IPTC genre) or, unknown
14
Dateline (Location)Where does the article claim it was published? ***ideaWhere does the article claim it was published?
Article Structure
LowLowGeolocationchecked
15
Is TranslationIs the article a translation? ***ideaIs the article a translation?
Article Structure
LowLowBoolean
16
Source Language (if translation)If the article is a translation, what is the source language? *** ideaIf the article is a translation, what is the source language?
Article Structure
LowLowMultiple Choice
17
Factual assertionsDoes the article contain factual, verifiable assertions or is entirely opinion based? *** Complements the Genre (is the content actually opinion piece? or verifiable information?)
ideaDoes the article contain factual, verifiable assertions or is entirely opinion based?Complements the Genre (is the content actually opinion piece? or verifiable information?)
Article Structure
HighMedium
Boolean or multiple choice

18
Subject AreaWhat is the article's genre? *** Indicates a subject of an item: Sports, EntertainmentideaWhat is the article's genre?Indicates a subject of an item: Sports, Entertainment
Article Structure
Medium
NewsCodes (IPTC)
http://show.newscodes.org/index.html?newscodes=subj&lang=en-GB&startTo=Show
Multiple Choice (though I think one is default)
Technology
Sports and Entertainment (just highlighting varying classifications)
19
Shows versions and changesDoes this article show revisions/diffs? (Most places do not) *** Shows versions, changes, diffs of an article.ideaDoes this article show revisions/diffs? (Most places do not)Shows versions, changes, diffs of an article.
Article Structure
Low
Multiple Choice (How rigorous the changes are. Is there a Newsdiff-type thing)
20
Subhead/Dek***idea
Article/Site Metadata
LowMedium
Just a note that this can be critical; people don't always read the articles they forward, they do depend a lot on this Dek/summary hook
checked
21
Publication Domain Registration Date***idea
Article/Site Metadata
MediumLowDatechecked
22
Publication Domain Registration Location***idea
Article/Site Metadata
MediumLowDatechecked
23
Article RightsWhat are the rights for this article? *** Explicit Copyright, Creative Commons, Unstated, etc.ideaWhat are the rights for this article?Explicit Copyright, Creative Commons, Unstated, etc.
Article/Site Metadata
HighLowMultiple Choicechecked
24
Photo/Video GeotagsWhat are the geolocations for the photos and videos on the article? ***ideaWhat are the geolocations for the photos and videos on the article?
Article/Site Metadata
Very HighLowGeolocationchecked
25
***idea
Article/Site Metadata
26
Ads.txt ExistsDoes an ads.txt file exist on the domain? *** ads.txt (Authorized Digital Sellers) is an Interactive Advertising Bureau initiative. It specifies a text file that companies can host on their web servers, listing the other companies authorized to sell their products or services. This is designed to allow online buyers to check the validity of the sellers from whom they buy, for the purposes of internet fraud prevention. (from https://en.wikipedia.org/wiki/Ads.txt)
ideaDoes an ads.txt file exist on the domain?ads.txt (Authorized Digital Sellers) is an Interactive Advertising Bureau initiative. It specifies a text file that companies can host on their web servers, listing the other companies authorized to sell their products or services. This is designed to allow online buyers to check the validity of the sellers from whom they buy, for the purposes of internet fraud prevention. (from https://en.wikipedia.org/wiki/Ads.txt)
Revenue Model
Article/Site Metadata
Very LowBoolean
Evolving standard. some initial brainstorming on other potential features: ```- the number of sellers in given file (could consider SSPs and seller accounts separately) - the balance of DIRECT vs. RESELLER flags - which SSPs are included (some are more quality/stringent than others) - weighting SSPs included by the number of accounts per SSP.``` I’m not sure that any of those would be strong misinfo signals especially since it seems like publishers are still figuring out how to handle ads.txt, but could be things to explore.
https://www.huffingtonpost.com/ads.txt
https://www.reddit.com/r/adops/comments/703k4s/is_there_a_legitimate_reason_to_accept_an_adstxt/
checkedchecked
https://wiki.appnexus.com/display/industry/AppNexus+Support+for+Ads.txt
Standards still under development
June 2018
Credibility Coalition Study: June 2018
Status: DRAFTING
27
OccupationWhat is the occupation of the author? *** What is the occupation of the author. Are they a full-time journalist or something elseideaWhat is the occupation of the author?What is the occupation of the author. Are they a full-time journalist or something else
Author Reputation
Very HighMediumCategorychecked
28
Education CredentialsWhat is the educational background of the author? *** What is the educational background of the author. Do they have a degree in journalism? Do they have any post-grad education?ideaWhat is the educational background of the author?What is the educational background of the author. Do they have a degree in journalism? Do they have any post-grad education?
Author Reputation
Very HighMedium
category or series of booleans?
checked
I don't know we should narrowly frame it as "education" as a top level indicator, rather as "expertise" or which education can be a key subpart?
29
Track recordHas the author already published articles containing misleading or credible information? ***ideaHas the author already published articles containing misleading or credible information?
Author Reputation
MediumHigh
A track record of credible or misleading reporting can be a very strong indicator of the author's current article. It may also be very difficult to reproduce at scale.
scale
Daily Mail journalist David Rose has published a number of misleading articles on the topic of climate change as shown by IPSO, the UK MetOffice, Climate Feedback...
http://www.lse.ac.uk/GranthamInstitute/news/uk-newspaper-regulator-acts-against-fake-news-story-about-climate-change-which-fooled-us-congressman/, https://climatefeedback.org/authors/david-rose/
checked
30
Number of publicationsHow many publications does this author have and in which venues? *** How many publications does this author have and in which venues.ideaHow many publications does this author have and in which venues?How many publications does this author have and in which venues.
Author Reputation
MediumMedium
integers tied with categories (publication names)
checked
I feel like "which publications" rather than number is a stronger indicator (you can have both, but if I had one, I'd go with which). Also like where would you get "number?" That's hard to track.
J8L
31
Public AccessibilityHow accessible/responsive is this author to the public? *** How accessible is this author to the public. Do they have a website? Does this author have a publicly available email address? Are they on Twitter or Facebook? Do they often respond to readers?ideaHow accessible/responsive is this author to the public?How accessible is this author to the public. Do they have a website? Does this author have a publicly available email address? Are they on Twitter or Facebook? Do they often respond to readers?
Author Reputation
HighHigh
@Amy Zhang Can you fill in some details here?
scale, or set of booleans
checked
32
Followers/ListenersWhat degree of attention does this author command from other individuals? *** How many people follow this author (on social media). How many other journalists follow this author? What speaking engagements does this author command?ideaWhat degree of attention does this author command from other individuals?How many people follow this author (on social media). How many other journalists follow this author? What speaking engagements does this author command?
Author Reputation
MediumHigh
@Amy Zhang can you fill in rationale here?
scale
Kind of vague and could be broken down. I don't know if raw Twitter follower count is quite right but some idea of who listens to or associates with this author.
checkedchecked
clearly can be gamed, by raw twitter follower count by bots or by other strategies like state-sponsored ecosystems
33
Has Author Bio***idea
Author Reputation
Has author bio, or author bio?
34
Claim has been fact checkedHas the central claim in this article been fact-checked by an IFCN Verified Signatory? *** A) Most likely not fact-checked by an IFCN Verified Signatory
B) Most likely not fact-checked by an approved source
C) Fact-checked and determined false
D) Fact-checked and determined true
E) Fact-checked with unclear results
F) Fact-checked with mixed results
ideaHas the central claim in this article been fact-checked by an IFCN Verified Signatory?A) Most likely not fact-checked by an IFCN Verified Signatory
B) Most likely not fact-checked by an approved source
C) Fact-checked and determined false
D) Fact-checked and determined true
E) Fact-checked with unclear results
F) Fact-checked with mixed results
ClaimVery HighVery HighVery HighSingle ChoiceArticle Context
As many IFCN signatories use the ClaimReview schema, understanding correlations between fact checks and credibility can help us understand how best to utilize the schema or improve it.
1
Credibility Coalition Study: Web Conference 2018
Deprecated for Credibility Coalition Study: Summer 2018
35
Claim contains logical fallacy***ideaClaim
36
Claim contains false assertion***ideaClaim
37
Claim contains false and misleading assertion***ideaClaim
38
Claim contains misleading assertion***ideaClaim
39
Claim contains bad data***ideaClaim
40
Fact-check results of claimWhat are the results of the fact check of the central claim in this article? *** A) Most likely not fact-checked
B) Fact-checked and determined false
C) Fact-checked and determined true
D) Fact-checked with unclear results
E) Fact-checked with mixed results
ideaWhat are the results of the fact check of the central claim in this article?A) Most likely not fact-checked
B) Fact-checked and determined false
C) Fact-checked and determined true
D) Fact-checked with unclear results
E) Fact-checked with mixed results
ClaimSingle Choice
Modification of https://credweb.org/cciv/#claim-has-been-fact-checked used in Web Conference 2018 study above.
Credibility Coalition Study: Summer 2018
One of two indicators replacing https://credweb.org/cciv/#claim-has-been-fact-checked
41
***ideaClaim
42
Links from other news sitesWhat are the 5 most popular (Alexa rating) news sites linking to this article? *** What other news sites are linking to this URL? Does it include credible sites or not?ideaWhat are the 5 most popular (Alexa rating) news sites linking to this article?What other news sites are linking to this URL? Does it include credible sites or not?
Inbound References
LowHigh
@Cameron Hickey 
Assumes that low-quality content links to other low-quality content.
checked
Depends on a catalogue of "trusted websites".
43
Facebook sharesHow many facebook shares? *** Number of times shared on facebook. Important to estimate the reach of the news article.
ideaHow many facebook shares?Number of times shared on facebook. Important to estimate the reach of the news article.
Inbound References
LowHigh
@Cameron Hickey 
Not directly related to credibility, but to reach.
checked
Not directly related to credibility
44
Facebook engagementHow many people engaged on facebook? *** Number of facebook engagement. Important to estimate the reach of articles.ideaHow many people engaged on facebook?Number of facebook engagement. Important to estimate the reach of articles.
Inbound References
MediumHigh
Engagements is a composite of comments, likes and shares. As a total number, it potentially reflects popularity of a given article on social media. This value, or any of the elements that comprise it could be used in conjunction with other indicators to try and extract the credibility (or lack-thereof) of the source or article. For example: if an article has a high value for engagement but the domain
Not directly related to credibility, but to reach.
checked
Not directly related to credibility
45
Ratio of comments to likes <10%Does it have many more likes than comments? *** Ratio of comments to likes on Facebook. Indicative of bot-promoted content.ideaDoes it have many more likes than comments?Ratio of comments to likes on Facebook. Indicative of bot-promoted content.
Inbound References
LowHigh
@Cameron Hickey 
Indicative of bot networks? (NY meeting)
checked
Can be easily gamed by posting dummy comments.
46
Links from social mediaWhat are the 5 most liked (Twitter, Facebook), upvoted (Reddit) social accounts that shared this link *** What social media accounts have shared the URL? Do they include credible accounts?
ideaWhat are the 5 most liked (Twitter, Facebook), upvoted (Reddit) social accounts that shared this linkWhat social media accounts have shared the URL? Do they include credible accounts?
Inbound References
LowHigh
@Cameron Hickey 
Assumes that low-quality social accounts link to low-quality content.
checked
Depends on a catalogue of "trusted accounts"
47
Verdict from fact-checking websitesA trusted fact-checking organisation has verified the textidea, stIf the page (or one with the same claim) has been fact-checked by one of the signatory organisations, show this inlink with the fact-checking verdict. Has a trusted fact-checking organisation verified it?If the page (or one with the same claim) has been fact-checked by one of the signatory organisations, show this inlink with the fact-checking verdict.
Inbound References
MediumVery High
Most reliable 3rd party validation, should be included for sure.
We should show any existing fact-checks from trusted organisations. This is a form of 3rd party validation.
checkedN/A
48
Facebook comment qualityAll or nearly all the Facebook comments are mono-sentence.idea, j8l*** Number and content of facebook comments. Important to estimate the reach and sentiment of articles.How many facebook comments?Number and content of facebook comments. Important to estimate the reach and sentiment of articles.
Inbound References
MediumHigh
@Cameron Hickey 
Not directly related to credibility, but to reach.
checked
Can be gamed with communities with a strong agenda or bots.
49
A significant number of the comments are multi-sentence.idea, new addition, j8lWe Split Facebook comment quantity to quantity and quality.
50
Facebook comment quantityThere are 10 or fewer Facebook commentsidea, new addition, j8l
51
There are between 11 and 49 Facebook comments
52
There are between 50 and 100 Facebook commentsidea, new addition, j8l
53
There are 100 or more acebook commentsidea, new addition, j8lcan increase or divide them differently depending on common findings
54
Facebook comment sentimentWhat is the sentiment of the facebook comments? *** Sentiment analysis on Facebook comments. Important to gauge the attitudes towards the news, and types of reactions they invoke. ideaWhat is the sentiment of the facebook comments?Sentiment analysis on Facebook comments. Important to gauge the attitudes towards the news, and types of reactions they invoke.
Inbound References
HighHigh
@Cameron Hickey 
Not directly related to credibility, but to reach.
checked
Can be gamed with communities with a strong agenda or bots.
55
Number of links in WikipediaMultiple Wikipedia articles point to the articleidea, stNumber of inbound links from Wikipedia main namespace.How many links on Wikipedia articles point to the article?Number of inbound links from Wikipedia main namespace.
Inbound References
LowMedium
Links to the article in Wikipedia indicate a level of social relevance that is more enduring than inlinks from social in a given moment. This can easily be gamed, but this also relies on the immune systems that Wikipedia has set up.
Integer
Wikipedia is the highest quality reference material on the internet, and links to articles (outside of writing about fake news, or lists of fake news websites) would be a form of third party validation.
checkedchecked
Context-dependent. E.g. the link might be coming from a list of fake news websites, would require some contextual filtering. Also, in principle anyone can add a link albeit in all likelihood for only a brief period of time before it is reverted.
56
***idea
Inbound References
57
Represents scientific literatureThe content fairly represents scientific literature on the issue at handidea, stDoes the content fairly represent scientific literature on the issue at hand?
Journalistic Rigor
Very HighVery High
Accurate representation of scientific literature reflects a certain level of knowledge of the literature that can be difficult to replicate. Presumably, only scientists and full-time science writers with some subject matter knowledge can do this effectively?
Scale
58
Represents scientific processThe content rigorously represents the scientific processidea, stDoes the content rigorously represent the scientific process?
Journalistic Rigor
HighVery High
Not simply regurgitating scientific findings but setting them in the context of the scientific method might be the mark of a paid science writer, who offers a critical perspective for the reader. This may be difficult to reproduce by misinformation agents?
Boolean
59
Presents multiple perspectivesThe content fairly represent multiple perspectives on an issueidea, stDoes the content fairly represent multiple perspectives on an issue?
Journalistic Rigor
MediumMediumBoolean?
False balance. Enforcing a "multiple perspectives" approach to science topics can lead to pathological results over-representing fringes, eg vaccines and climate change.
60
Cites wire servicesThe article clearly states a known wire service as a sourceidea, stDoes the article clearly state if the content comes from a known wire service?
Journalistic Rigor
LowMediumBoolean?
61
Dependency on anonymous sourcesTHe premise of this article is based on anonymous sources. idea, ngThis can be made more nuanced depending on how many sources, and how critical it is to the main premise of the article.Does this article cite anonymous sources for the premise of the article?Does this premise of this article rest on anonymous sources. This can be made more nuanced depending on how many sources, and how critical it is to the main premise of the article.
Journalistic Rigor
LowHigh
Multiple Choice (premise of article is based on anonymous sources to more mild) or Boolean
Critical factor in "showing your work" or "replicability." This will likely interplay with the institution that publishes it. Should likely have different statuses of how critical the anonymous source is. "Rape victim" vs. "government official." Also can be multiple choice. "Premise based entirely on single anonymous source." Premise based on two anonymous source." "Premise based on two more more anonymous sources."
62
Gives motivation of anonymous sources for revealing informationThis article explains the motivation for all of the anonymous sources revealing information.idea, ogDoes this article explain the motivation of any of the anonymous sources reveal information?
Journalistic Rigor
Medium
Multiple Choice or Boolean
Knowing the motivation of the anonymous source is important to assessing the credibility of the article. Was this a purposeful leak? Was it only confirmed after reporters approached them. Examples: "Sexual assault victim," "fears for safety," "government official revealing secret information."
63
Shares any documents cited in pieceThis article publishes (or links to?) any important documents cited within it. idea, ngDoes this refer to inlinks within the article? or if the creators of the article only cites articles they themselves publishDoes this article publish any important documents cited in the article?
Journalistic Rigor
Low
Multiple Choice or Boolean
"Show your work." Also allows readers to judge for themselves.
64
Primary subjects of article have opportunity to respondThe main subjects of the article have a chance to respond (to the main points of the artice? in a direct or indirect quote after being interviewed?) idea, ngNot sure if the responses metric refers to responses within the article, or responses via commentingDo the main subjects of the article have a chance to respond?
Journalistic Rigor
LowHigh
Multiple Choice or Boolean
Giving subjects a chance to respond or say "no comment" is basic journalism rigor. Many articles spiral because premise is taken out of context.
https://thehill.com/homenews/administration/332132-fcc-probing-colberts-trump-putin-joke
Can be made up.
65
Premise of article is disputed by primary subjects***idea
Journalistic Rigor
Multiple Choice or Boolean
Disputed does not mean not credible. In fact may be more credible. Again interplay. But worth noting. Dispute can also be internal or external to the article.
https://www.bloomberg.com/news/features/2018-10-04/the-big-hack-how-china-used-a-tiny-chip-to-infiltrate-america-s-top-companies
Dispute could possible be in the article, or afterwards in "fallout", which would not be captured in the original article.
66
Corroboration?***idea
Journalistic Rigor
67
Straw Man ArgumentThe [text] presents a counterargument as a weaker, more foolish version of the real counterargument (uses a Straw Man Argument).idea,shDoes the author present the counterargument as a weaker, more foolish version of the real counterargument (use a Straw Man Argument)? If so, highlight the relevant section(s).Presentation of a counterargument as a weaker, more foolish version of the real counteargument
Logic/Reasoning
Very HighHighLikertArticle Content
"Proponents of nuclear energy have argued that it is safer than it used to be. But we don’t want energy that is barely safer than Chernobyl."
1Public Editor
68
False DilemmaThe [text] presents a complicated choice as if it were binary (constructs a false dilemma).idea, ogDoes the author present a complicated choice as if it were binary (construct a false dilemma)? If so, highlight the relevant section(s).Presentation of a complicated choice as if it is binary
Logic/Reasoning
Very HighVery HighLikertArticle Content
"America; love it or leave it! If you're not happy with U.S. policies, you should just move somewhere else."
1Public Editor
69
Slippery Slope ArgumentThe text says that one small change will lead to a major changeidea, stSlippery slope argument.Does the author say that one small change will lead to a major change (use a slippery slope argument)? Highlight the relevant section(s).Argument that one small change will lead to a major change
Logic/Reasoning
Very HighHighLikertArticle Content
"If we allow human cloning, we will soon be overrun by armies of human clones."
1Public Editor
70
Naturalistic FallacyThe [text] suggests that something is good because it is natural, or bad because it is not natural (the naturalistic fallacy).idea, ogDoes the author suggest that something is good because it is natural, or bad because it is not natural (the naturalistic fallacy)?Suggestion that something is good because it is natural or bad because it is not natural
Logic/Reasoning
MediumHighLikertArticle Content
"Vaccines aren't natural; they're full of all kinds of lab-manufactured chemicals. That stuff can't possibly be good for you."
1Public Editor
71
Calibrating Confidence - JustificationThe author's confidence in their claims is well justifiedidea, ogTo what extent does their confidence in their claims seem justified?Expressed confidence in a claim, with justification
Logic/Reasoning
Very HighVery HighLikertArticle Content
"According to Mass, the idea that human-caused climate change had any effect on Harvey is more than far-fetched — it’s downright not true."
http://www.theblaze.com/news/2017/09/02/scientist-shuts-down-climate-change-alarmists-with-new-report-about-hurricane-harvey/
1Public Editor
72
Appeal to Fear FallacyThe [text] exaggerates the dangers of a situation and uses scare tactics to persuade (uses the appeal to fear fallacy).idea, DKDoes the author exaggerate the dangers of a situation and use scare tactics to persuade (the appeal to fear fallacy)?Exaggeration of the dangers of a situation and use of scare tactics in an attempt to persuade
Logic/Reasoning
Very HighLikertArticle Content1Public Editor
73
Calibrating Confidence - Level of ConfidenceThe [text] acknowledges uncertainty or the possibility that things might be otherwise (expresses level of confidence in a claim).idea, DKDo they acknowledge uncertainty or the possibility that things might be otherwise? If so, highlight the relevant section(s).Expressed confidence in a claim
Logic/Reasoning
Very HighVery HighLikertArticle Content
"According to Mass, the idea that human-caused climate change had any effect on Harvey is more than far-fetched — it’s downright not true."
http://www.theblaze.com/news/2017/09/02/scientist-shuts-down-climate-change-alarmists-with-new-report-about-hurricane-harvey/
1Public Editor
74
Causal Claim TypesIs a general or singular causal claim made? Highlight the section(s) that supports your answer. *** General Causal Claim
Singular Causal Claim
No Causal Claim
ideaIs a general or singular causal claim made? Highlight the section(s) that supports your answer.General Causal Claim
Singular Causal Claim
No Causal Claim
Logic/Reasoning
Very HighHighMultiple ChoiceArticle Content1Public Editor
75
Draws sound conclusions from available evidence***idea
Logic/Reasoning
Very HighHigh
@Emlen Metz  @ADITYA RANGANATHAN 
76
Confirmation Bias***idea
Logic/Reasoning
Very HighHigh
@Emlen Metz  @ADITYA RANGANATHAN 
Scale
“Lots of bad things have happened in the last month. Since bad things have happened under every president, maybe we shouldn’t count that as additional evidence that the current president is awful.”
“I think the world is getting worse. So many bad things have happened in the last few years; just think about it! Famine, mass murder, hurricanes, wildfires. Everything is going wrong.”
Difficult to assess
Public Editor
77
Begging the Question***idea
Logic/Reasoning
Very HighMediumScale
"Capital punishment is wrong because it is immoral to inflict death as a punishment for a crime."
Public Editor
78
Mistaking Noise for Signal***idea
Logic/Reasoning
Very HighVery High
@Emlen Metz  @ADITYA RANGANATHAN 
Scale
"The seemingly miraculous Twitter prediction isn’t new, but normally we see it from some never-used account that is kept private and meticulously stocked with every possible permutation — only to be made public and get attention at a later date. That’s not the case for @RaysFanGio. He uses his account all the time — this was a one-off prediction and we should all be scared."
https://www.sbnation.com/lookit/2016/11/2/13506174/2014-tweet-predicts-world-series-ninth-inning-game-7-cubs-indians
Public Editor
79
Orders of Understanding***idea
Logic/Reasoning
Very HighVery High
@Emlen Metz  @ADITYA RANGANATHAN 
Scale
"We need to do a better job exploring the complex set of causes that produced 9/11, among them the rise of Islamic fundamentalism and postcolonial nationalism, as well as the spread of globalism and regional conflicts."
"The Hurricane Harvey disaster was caused entirely by society creating and perpetuating vulnerability to these natural events."
Public Editor
80
Number of Enthymemes (Arguments with Missing Premises)***idea
Logic/Reasoning
Very HighMediumHigh
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature: http://aclweb.org/anthology/W/W16/W16-2804.pdf
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
81
Numbers of Argument ComponentsThe argumentation and logic in the article is complexidea, stHow complex is the argumentation and logic in the article?
Logic/Reasoning
Very HighLowHigh
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
82
Number of ClaimsHow far could this article go wrong? ***ideaHow far could this article go wrong?
Logic/Reasoning
HighLowHigh
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
83
Number of Arguments Againstthe text is biased against one set of premisesidea, stHow biased is the article towards one set of premises
Logic/Reasoning
HighMediumHigh
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
84
Number of Supporting PremisesThe article backs up arguments with clear hypothesesidea, stHow far does the article back up arguments with clear hypotheses
Logic/Reasoning
Very HighMediumHigh
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
85
Average Number of Premises Per Claim***idea
Logic/Reasoning
Very HighLowMedium
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
86
Number of Arguments Forthe text is biased towards one set of premisesidea, stHow biased is the article towards one set of premises
Logic/Reasoning
HighMediumMedium
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
87
Number of Attacking PremisesThe text strongly contradicts itselfidea, stHow far does the article contradict itself
Logic/Reasoning
Very HighMediumMedium
Not sure about the priority here but I would say very difficult for WebConf
Integer
From argument mining literature
N/A - has to include example URLs
N/A - has to include example URLs
checked
Argument mining is relatively recent as an ML technique
Factmata
88
Use of conspiratorial thinkingThe article suggests a conspiracyidea, stDoes the article suggest a conspiracy?
Logic/Reasoning
Very HighHigh
@ADITYA RANGANATHAN @Emlen Metz 
boolean
"It’s all hogwash. The “official” narrative of how things went down..."
https://www.naturalnews.com/2017-10-02-lone-gunman-theory-of-las-vegas-shooter-is-complete-nonsense-stephen-paddock.html
Corruption does exist, not all conspiracy theories are BS
89
Identifiable Victim Effect***idea
Logic/Reasoning
90
Just World Fallacy***idea
Logic/Reasoning
91
Supporting Claim TypesWhat evidence is given for the primary claim? Select all that apply. *** Types of evidence for a claim:

Correlation
Cause precedes effect
The correlation appears across multiple independent contexts
A plausible mechanism is proposed
An experimental study was conducted (natural experiments OK)
Experts are cited
Other kind of evidence
No evidence given
ideaWhat evidence is given for the primary claim? Select all that apply.Types of evidence for a claim:

Correlation
Cause precedes effect
The correlation appears across multiple independent contexts
A plausible mechanism is proposed
An experimental study was conducted (natural experiments OK)
Experts are cited
Other kind of evidence
No evidence given
Logic/Reasoning
Multiple Choice
92
Contains Link to Scientific JournalsThe text includes a link to original contentidea, stA simple link to the a scientific journal article that backs up the assertion made. This may also be paired with a URL to the specific article.Is a link provided in the article to where the original content came from?A simple link to the a scientific journal article that backs up the assertion made. This may also be paired with a URL to the specific article.
Outbound References
LowMediumMediumBooleanArticle Context1
Credibility Coalition Study: Web Conference 2018
93
Accuracy of representation of source articleThe text properly characterizes the methods and conclusions of its sources idea, stThis article properly characterizes the methods and conclusions of the cited or quoted source. In addition to a Likert measure, two other options are possible: (A) Unable to find source, (B) Source is behind a paywallDoes this article properly characterize the methods and conclusions of the original source?This article properly characterizes the methods and conclusions of the cited or quoted source. In addition to a Likert measure, two other options are possible: (A) Unable to find source, (B) Source is behind a paywall
Outbound References
Very HighHighVery High
Ability to accurately reflect the source articles in outbound references requires a good deal of knowledge of the field and of the scientific method in general and may be difficult to replicate.
Likert, with Multiple Choice
Article Context
This ideally would be done per source article
"It’s sad that the blogger did not understand what this study is about..."
https://climatefeedback.org/evaluation/breitbart-misrepresents-research-58-scientific-papers-falsely-claim-disprove-human-caused-global-warming-james-delingpole/
1
Credibility Coalition Study: Web Conference 2018
94
Source Types Which of the following types of sources are cited in the article? Check all that apply. If Other, please highlight. *** None
Experts
Studies
Organizations
Other
idea Which of the following types of sources are cited in the article? Check all that apply. If Other, please highlight.None
Experts
Studies
Organizations
Other
Outbound References
MediumMediumMultiple ChoiceArticle Context1
Credibility Coalition Study: Web Conference 2018
95
Academic Journal Impact FactorWhat is the impact factor of the journal or conference cited? *** From Wikipedia: The impact factor (IF) or journal impact factor (JIF) of an academic journal is a measure reflecting the yearly average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field; journals with higher impact factors are often deemed to be more important than those with lower ones. https://en.wikipedia.org/wiki/Impact_factorideaWhat is the impact factor of the journal or conference cited?From Wikipedia: The impact factor (IF) or journal impact factor (JIF) of an academic journal is a measure reflecting the yearly average number of citations to recent articles published in that journal. It is frequently used as a proxy for the relative importance of a journal within its field; journals with higher impact factors are often deemed to be more important than those with lower ones. https://en.wikipedia.org/wiki/Impact_factor
Outbound References
MediumMediumMediumIntegerArticle Context1
96
Quotes reputable scientists***idea
Outbound References
MediumMedium
97
Databases*** Databases as a 'Publication' owned by a PublisherideaDatabases as a 'Publication' owned by a Publisher
Outbound References
MediumMedium
98
Agencies for Authority *** I think that Agencies could also be considered Publishers.
But Publisher captures I think the relationship to the Article
ideaI think that Agencies could also be considered Publishers.
But Publisher captures I think the relationship to the Article
Outbound References
MediumMedium
99
Quotes outside expertsDoes the article quote experts who are not part of the study but are part of the field? ***ideaDoes the article quote experts who are not part of the study but are part of the field?
Outbound References
MediumVery High
A good science writer should make sure to quote experts in the field who are not the original authors. This is a miniature reflection of the peer review system. Quotes could be fabricated, of course.
Boolean
Quotes or experts could be fabricated
100
Contains original quotesThe article contains original quotes that appear to be sourced directly by the reporter. idea, DKDoes the article contain original quotes that appear to be sourced directly by the reporter?
Outbound References
MediumVery High
Original quotes could be fabricated, but if they are not, they show an attention to getting opinions from subject matter experts. However, they could also just indicate a competent mid-level blogger who doesn't have the resources to seek outside perspectives.
Boolean
Loading...
Main menu