| A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z | AA | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Title | Authors | Year | Journal | Treatment (IV) | Outcome (DV) | Source of Variation | Study Sample | Relevant Intervention (primary) | Relevant Intervention (secondary) | Key Finding | Posited Mechanism | Abstract | Notes | DOI/Links | Representative Sample? | Treatment Type | Outcome Type | Social Media Platform | Reduces False Beliefs | Reduces Sharing | Accuracy Nudge Effect | Partisan Bias | Source Credibility Effect | ||||
2 | Editing episodic memory following the identification of error | A. L. Wilkes and M. Leatherbarrow | 1988 | The Quarterly Journal of Experimental Psychology | Participants read a series of statements about an event regarding a fictional fire or accident and some participants received correction of misinformation | Subjects wrote an account of what they recalled about the event | Corrected vs. uncorrected misinformation | University of Dundee students | Disinformation disclosure | "When the old information played a salient role in the message sequence, not even an explicit and direct denial was sufficient to purge the memory record of all of its implications" | "Episodic records are not always changed easily, but it does occur within limits… If a record was extensively altered after every contradiction that arose, it could soon become unusable as its internal structure began to break down." | "The editing of an episodic memory record in order to remove incorrect information embedded within naturalistic communications is an important though underinvestigated phenomenon. Experiment 1 deals with the recall and comprehension of a sequence of messages following the delayed identification of one of the messages as being incorrect. Two styles of correction were employed, and it was found that in neither case was the memory record edited effectively. Inferences based upon the old information continued to be drawn although subjects had clearly recalled that it had been subsequently corrected. Experiment 2 showed that editing could be effective if the old information did not play a central role in the message sequence. It is concluded that the observed difficulties in editing arise when old information has to be excised from the episodic record; the uncontested insertion of new information retrospectively did not present the same difficulty. Reading span was used to monitor subjects' editing strategies, and from its association with performance measures it is concluded that contradictions in the memory record are not dealt with immediately but are resolved locally when comprehension is questioned. At this time inferences are drawn based upon the most recent version of the contradictory messages. This recency strategy breaks down when the old information provides a better fit to the question posed. Some implications of these findings for models of memory storage are discussed." | https://doi.org/10.1080/02724988843000168 | No | Experimental | Knowledge | No | |||||||||||
3 | Investigating and Improving the Accuracy of US Citizens’ Beliefs About the COVID-19 Pandemic: Longitudinal Survey Study | Aart Van Stekelenburg, Gabi Schaap, Harm Veling, and Moniek Buijzen | 2021 | Journal of Medical Internet Research | Participants rated belief of a number of true and false statements about COVID-19 with or without exposure to an intervention | Change in participant beliefs to true and false information about COVID-19 over time | "Half of the participants were exposed to an intervention aimed at increasing belief accuracy. The intervention consisted of a short infographic that set out three steps to verify information by searching for and verifying a scientific consensus." | U.S. adults | Disinformation literacy | "Looking at each set of statements, we found a small but significant increase in belief accuracy over time. This indicates that the general public is quite able to figure out what is true and what is not in times of crisis. Moreover, a small but robust correlation suggests that accurate beliefs about the pandemic might be important for coronavirus-related behavior. Associations with belief accuracy suggest that the processes of belief formation and correction might be affected by individuals’ trust in scientists and political orientation, as well as their news habits. Finally, the boosting intervention yielded no significant increase in belief accuracy over the control condition, demonstrating that the boosting infographic was not successful in helping people figure out what is true and what is false. Exploratory analyses suggested that the intervention did, however, inhibit a decline in trust in scientists as a source of information about the coronavirus among more conservative, Republican participants." | "However, when looking for potential explanations of this study’s findings, we should consider the possibility that COVID-19 misinformation is not as prevalent as expected. Perhaps misinformation makes up only a small portion of the average US citizen’s media diet. The second possibility is that we are indeed facing a COVID-19 infodemic, but that the public is not very susceptible to it... A third possibility that should be considered is that the public is more careful in forming beliefs in times of crisis, especially in the relatively early days of a crisis, making a well-informed public not unique to the COVID-19 pandemic." | "Background: The COVID-19 infodemic, a surge of information and misinformation, has sparked worry about the public’s perception of the coronavirus pandemic. Excessive information and misinformation can lead to belief in false information as well as reduce the accurate interpretation of true information. Such incorrect beliefs about the COVID-19 pandemic might lead to behavior that puts people at risk of both contracting and spreading the virus. Objective: The objective of this study was two-fold. First, we attempted to gain insight into public beliefs about the novel coronavirus and COVID-19 in one of the worst hit countries: the United States. Second, we aimed to test whether a short intervention could improve people’s belief accuracy by empowering them to consider scientific consensus when evaluating claims related to the pandemic. Methods: We conducted a 4-week longitudinal study among US citizens, starting on April 27, 2020, just after daily COVID-19 deaths in the United States had peaked. Each week, we measured participants’ belief accuracy related to the coronavirus and COVID-19 by asking them to indicate to what extent they believed a number of true and false statements (split 50/50). Furthermore, each new survey wave included both the original statements and four new statements: two false and two true statements. Half of the participants were exposed to an intervention aimed at increasing belief accuracy. The intervention consisted of a short infographic that set out three steps to verify information by searching for and verifying a scientific consensus. Results: A total of 1202 US citizens, balanced regarding age, gender, and ethnicity to approximate the US general public, completed the baseline (T0) wave survey. Retention rate for the follow-up waves— first follow-up wave (T1), second follow-up wave (T2), and final wave (T3)—was high (≥85%). Mean scores of belief accuracy were high for all waves, with scores reflecting low belief in false statements and high belief in true statements; the belief accuracy scale ranged from –1, indicating completely inaccurate beliefs, to 1, indicating completely accurate beliefs (T0 mean 0.75, T1 mean 0.78, T2 mean 0.77, and T3 mean 0.75). Accurate beliefs were correlated with self-reported behavior aimed at preventing the coronavirus from spreading (eg, social distancing) (r at all waves was between 0.26 and 0.29 and all P values were less than .001) and were associated with trust in scientists (ie, higher trust was associated with more accurate beliefs), political orientation (ie, liberal, Democratic participants held more accurate beliefs than conservative, Republican participants), and the primary news source (ie, participants reporting CNN or Fox News as the main news source held less accurate beliefs than others). The intervention did not significantly improve belief accuracy. Conclusions: The supposed infodemic was not reflected in US citizens’ beliefs about the COVID-19 pandemic. Most people were quite able to figure out the facts in these relatively early days of the crisis, calling into question the prevalence of misinformation and the public’s susceptibility to misinformation." | https://doi.org/10.2196/24069 | Yes | Real-world events | Beliefs | ||||||||||||
4 | Processing political misinformation: comprehending the Trump phenomenon | Adam Briony Swire, Adam J. Berinsky, Stephan Lewandowsky, and Ullrich K. H. Ecker | 2017 | Royal Society Open Science | Presentation of statements made by Donald Trump on the campaign trail in 2015 or statements without attribution and corrections or affirmation of facts | "Participants rated their belief in the statements both before and after the corrective/affirmative explanation" | Statements explicitly attributed to Trump or presented without attribution; second belief rating was immediate or followed a one-week delay | US residents from Amazon's Mechanical Turk | Disinformation disclosure | "We found that participants' opinion of Donald Trump influenced their assessment of information, that is, how valid they perceived it to be… Republicans and Democrats seemed to take into account their Trump-related biases and overcorrected for them one week after the explanations: Republican supporters by assuming that Trump's facts were false and Democrats by assuming that Trump's misinformation was true." | "Given that attitude homophily is a crucial component of source credibility [6], coupled with the notion that higher source credibility results in an increased perception of information credibility [46], it is reasonable that political figures such as Donald Trump act as a heuristic when evaluating the veracity of information." | "This study investigated the cognitive processing of true and false political information. Specifically, it examined the impact of source credibility on the assessment of veracity when information comes from a polarizing source (Experiment 1), and effectiveness of explanations when they come from one's own political party or an opposition party (Experiment 2). These experiments were conducted prior to the 2016 Presidential election. Participants rated their belief in factual and incorrect statements that President Trump made on the campaign trail; facts were subsequently affirmed and misinformation retracted. Participants then re-rated their belief immediately or after a delay. Experiment 1 found that (i) if information was attributed to Trump, Republican supporters of Trump believed it more than if it was presented without attribution, whereas the opposite was true for Democrats and (ii) although Trump supporters reduced their belief in misinformation items following a correction, they did not change their voting preferences. Experiment 2 revealed that the explanation's source had relatively little impact, and belief updating was more influenced by perceived credibility of the individual initially purporting the information. These findings suggest that people use political figures as a heuristic to guide evaluation of what is true or false, yet do not necessarily insist on veracity as a prerequisite for supporting political candidates." | https://doi.org/10.1098/rsos.160802 | Unclear | Experimental | Beliefs | Yes | Yes | ||||||||||
5 | Rumors and Health Care Reform: Experiments in Political Misinformation | Adam J. Berinsky | 2015 | British Journal of Politial Science | Presentation of news stories with real-world rumors about health care reform (related to the 2010 Affordable Care Act), varying the presentation of corrections and news source | Participants rated veracity of rumors and the extent to which they rejected rumors | Varied according to "(1) the pairing of rumor and correction, (2) the partisanship of the source of the correction of the rumor and (3) the degree to which the respondent was induced to rehearse the rumor" | Online sample of American adults | Disinformation disclosure | "Though partisanship colors how citizens process information about public policy, my studies show that under the right circumstances – with the right arguments made by the right people – corrections can increase rumor rejection rates among the mass public, regardless of their partisan predilections. In particular, corrections acquire credibility when politicians make statements that run counter to their personal and political interests." | "Drawing upon research from psychology on ‘fluency’ – a state of mind that characterizes the ease of information processing – I argue that rumors acquire their power through familiarity. Attempting to quash rumors through direct refutation may instead facilitate their diffusion by increasing their fluency" | "This article explores belief in political rumors surrounding the health care reforms enacted by Congress in 2010. Refuting rumors with statements from unlikely sources can, under certain circumstances, increase the willingness of citizens to reject rumors regardless of their own political predilections. Such source credibility effects, while well known in the political persuasion literature, have not been applied to the study of rumor. Though source credibility appears to be an effective tool for debunking political rumors, risks remain. Drawing upon research from psychology on ‘fluency’ – the ease of information recall – this article argues that rumors acquire power through familiarity. Attempting to quash rumors through direct refutation may facilitate their diffusion by increasing fluency. The empirical results find that merely repeating a rumor increases its power." | https://doi.org/10.1017/S0007123415000186 | Yes | Experimental | Beliefs | Yes | |||||||||||
6 | "Fact-checking" videos reduce belief in, but not the sharing of fake news on Twitter | Alexander Bor, Mathias Osmundsen, Stig Hebblestrup Rye Rasmussen, Anja Bechmann, and Michael Bang Petersen | 2020 | Field experiment exposing participants to a popular fact-checking video; exposure to fake and real news items | Participant belief in and sharing of fake news among Twitter users; performance on a fake news quiz | Participants exposed to "one of six professionally produced fact-checking videos from government-funded or non-governmental organizations" | Field experiment on 1600 Twitter users | Disinformation literacy | Disinformation disclosure | "We found that the educational materials on how to spot fake news improved participants' ability to assess the credibility of news story headlines… Yet, while these widely available educational interventions may have succeeded in lowering beliefs in fake news," they did not prevent participants from sharing false and untrustworthy news sources on Twitter." | "Partisan motivations create benefits to posting misinformation, which may outweigh the potential costs of spreading false rumors." | "Fake news" are widely acknowledged as an important challenge for Western democracies.Yet, surprisingly little effort has been devoted to measuring the effects of various counter-strategies. We address this void by running a pre-registered field experiment analyzing the causal effects of popular fact-checking videos on both believing and sharing fake news among Twitter users (N = 1,600). We find that the videos improve truth discernment ability as measured by performance in a fake news quiz immediately after exposure. However, the videos have not reduced sharing links from verified "fake news" websites on Twitter in the weeks following the exposure. Indeed, we find no relationship between truth discernmentability and fake news sharing. These results imply that the development of effective interventions should be based on a nuanced view of the distinct psychological motivations of sharing and believing "fake news"." | This is a preprint working paper not yet published in a journal | https://doi.org/10.31234/osf.io/a7huq | No | Real-world social media | Observed online behaviors | |||||||||||
7 | Correcting misinformation by health organizations during measles outbreaks: A controlled experiment | Anat Gesser-Edelsburg, Alon Diamant, Rana Hijazi, Gustavo S. Mesch | 2018 | PLoS One | Participants presented with a situation involving vaccination-related concerns about sending a child to kindergarten and a mother's Facebook post detailing these concerns, then varying forms of corrective information | Trust in health response; satisfaction with response; self-efficacy; desire to search for more information; behavioral intention | Participants received "common vs. recommended theory-based information correction" | Graduate students from Haifa University | Disinformation disclosure | "Some of the subjects received a common response (i.e., the tone and the wording that can be found on Health Ministries' websites), versus a response based on the theory of health and risk communication. The findings of the study indicated that the experiment participants who received a Health Ministry response in keeping with the theory, expressed a higher level of satisfaction, trust and self-efficacy than those who received a common Health Ministry response." | "These findings support the assumptions of the approach of risk communication, namely that when health organizations provide full and transparent information and address the emotional element, they are more effective than when they deliver one-dimensional, partial responses that do not address the public's fears and concerns." | "Background During epidemic crises, some of the information the public receives on social media is misinformation. Health organizations are required to respond and correct the information to gain the public’s trust and influence it to follow the recommended instructions. Objectives (1) To examine ways for health organizations to correct misinformation concerning the measles vaccination on social networks for two groups: pro-vaccination and hesitant; (2) To examine the types of reactions of two subgroups (pro-vaccination, hesitant) to misinformation correction; and (3) To examine the effect of misinformation correction on these two subgroups regarding reliability, satisfaction, self-efficacy and intentions. Methods A controlled experiment with participants divided randomly into two conditions. In both experiment conditions a dilemma was presented as to sending a child to kindergarten, followed by an identical Facebook post voicing the children mothers’ concerns. In the third stage the correction by the health organization is presented differently in two conditions: Condition 1 –common information correction, and Condition 2 –recommended (theory-based) information correction, mainly communicating information transparently and addressing the public’s concerns. The study included (n = 243) graduate students from the Faculty of Social Welfare and Health Sciences at Haifa University. Results A statistically significant difference was found in the reliability level attributed to information correction by the Health Ministry between the Control condition and Experimental condition (sig<0.001), with the average reliability level of the subjects in Condition 2 (M = 5.68) being considerably higher than the average reliability level of subjects in Condition 1 (4.64). A significant difference was found between Condition 1 and Condition 2 (sig<0.001), with the average satisfaction from the Health Ministry’s response of Condition 2 subjects (M = 5.75) being significantly higher than the average satisfaction level of Condition 1 subjects (4.66). Similarly, when we tested the pro and hesitant groups separately, we found that both preferred the response presented in Condition 2. Conclusion It is very important for the organizations to correct misinformation transparently, and to address the emotional aspects for both the pro-vaccination and the hesitant groups. The pro-vaccination group is not a captive audience, and it too requires a full response that addresses the public's fears and concerns." | https://doi-org.ezproxy.princeton.edu/10.1371/journal.pone.0209505 | No | Experimental | Intended action/behavior | Yes | |||||||||||
8 | Polarity and attitude effects in the continued-influence paradigm | Andrew Gordon, Ullrich K. H. Ecker, Stephan Lewandowsky | 2019 | Journal of Memory and Language | Participants presented with different versions of a fictitious news report about a robbery or stabbing, varying demographics of characters in the story and misinformation correction | "Participants’ understanding of the report and their level of reliance on the critical race-related information was tested using an open-ended questionnaire booklet" | Experiment 1: "2 (racial prejudice: high vs. low) × 4 (condition: only-negated, negated-then-reinstated, only-affirmed, affirmed-then-retracted)" Experiment 2: "5 (condition: only-negated, negated-then-reinstated, only-affirmed, affirmed-then-retracted, control) × 2 (islamophobia: high vs. low)" | Experiment 1: Undergraduate students at the University of Western Australia Experiment 2: U.S. based participants from Amazon's Mechanical Turk | Disinformation disclosure | "Previous research into the CIE has exclusively focused on examining the impact of corrective information in a single polarity – the retraction of initially-affirmed information. In two experiments we examined how the CIE would be affected by a reversal of this polarity – providing a reinstatement of initially-negated information... Instead the data suggested that the continued influence of initially presented information is not modulated by its polarity. Corrective information appears to engender only partial belief updating, regardless of whether it corrects initially-affirmed or initially-negated material." | "Such a ‘reverse-CIE’ effect – the finding that a reinstatement cannot entirely offset an initial negation – is concordant with the previous literature demonstrating continued influence in the typical affirmation-negation (i.e., misinformation-retraction) paradigm. However, although the results are quite clear, the particular cognitive mechanisms underlying this reverse-CIE, much like those that underlie traditional CIE effects, remain uncertain. Given the symmetry of these effects, our results lend support to the notion outlined in the Introduction that an initial negation is recoded into a single meaningful informational unit." | "Misinformation – information that is false or inaccurate – can continue to influence people’s memory and reasoning even after it has been corrected. Researchers have termed this the continued influence effect (CIE). However, to date, research has focused exclusively on examining the CIE in a single polarity, namely the ongoing effect of initially affirmed material that is later negated. No research has yet examined how reliance on outdated information may be affected if this polarity is reversed, that is, if initially-negated information is reinstated. It also remains unclear how participants’ pre-existing beliefs may impact the acceptance of a correction, with prior evidence showing conflicting results. To investigate these questions, across two experiments we presented participants scoring high versus low on measures of relevant attitudes with fictional news reports that contained a piece of critical attitude-relevant information. This information was either true throughout, false throughout, initially-affirmed then retracted, or initially-negated then reinstated. Participants’ reliance on the critical information was subsequently measured with the use of inferential-reasoning items. Reinstatement of initially-negated information was insufficient to bring reliance on that information to a baseline level – that is, reliance on information presented as true throughout was greater than reliance on negated and then reinstated information. This result was symmetrical with the conventional CIE observed with a reversed polarity. The effect of participants’ pre-existing attitudes on continued reliance was equivocal. The results therefore suggest that the CIE is not contingent on polarity, raising questions about the cognitive mechanisms underlying the effect." | https://doi-org.ezproxy.princeton.edu/10.1016/j.jml.2019.104028 | Unclear | Experimental | Knowledge | Unclear | |||||||||||
9 | The Potential for Narrative Correctives to Combat Misinformation | Angeline Sangalang, Yotam Ophir, and Joseph N. Cappella | 2019 | Journal of Communication | Participants presented with a scenario in which a character provided misinformation about tobacco and then viewed varying forms of a correction | Measured participants' emotional responses and misinformed beliefs | Experimental conditions varied the emotional narrative of the correction | Current smokers recruited from Survey Sampling International | Disinformation disclosure | "Study 2 found that both narrative corrective conditions (simple and enhanced emotional) reduced all three misinformation outcomes relative to the no-correction control condition, though it is particularly noteworthy that the enhanced emotional corrective narrative ending produced less favorable attitudes toward natural tobacco products than the simple corrective narrative and that the difference was statistically significant." | "Previous work demonstrated that affective aspects of misinformation, which in turn impact intentions and behaviors, can remain uncorrected even when corrective information is accepted (e.g., Sherman & Kim, 2002; Thorson, 2016). The findings from Study 2 indicate emotional language in corrective messages might have some degree of success at combating this affective perserverence, or belief echoes. This is also consistent with other work that suggests rationally-based attitudes must be addressed with rational messages, while emotionally-based attitudes should be countered with emotional messages." | "Misinformation can influence personal and societal decisions in detrimental ways. Not only is misinformation challenging to correct, but even when individuals accept corrective information, misinformation can continue to influence attitudes: a phenomenon known as belief echoes, affective perseverance, or the continued influence effect. Two controlled experiments tested the efficacy of narrative-based correctives to reduce this affective residual in the context of misinformation about organic tobacco. Study 1 (N = 385) tested within-narrative corrective endings, embedded in four discrete emotions (happiness, anger, sadness, and fear). Study 2 (N = 586) tested the utility of a narrative with a negative, emotional corrective ending (fear and anger). Results provide some evidence that narrative correctives, with or without emotional endings, can be effective at reducing misinformed beliefs and intentions, but narratives consisting of emotional corrective endings are better at correcting attitudes than a simple corrective. Implications for misinformation scholarship and corrective message design are discussed." | https://doi.org/10.1093/joc/jqz014 | No | Experimental | Beliefs | Yes | |||||||||||
10 | How Stories in Memory Perpetuate the Continued Influence of False Information | Anne Hamby, Ullrich Ecker, and David Brinberg | 2020 | Journal of Consumer Psychology | Participants viewed a story about a character experiencing drug complications and information to explain complications which was then retracted | Participants acceptance of poststory misinformation | Provided an explanation for complications or no explanation | U.S. undergraduate students | Disinformation disclosure | "Across three studies, the current work demonstrated that information presented after a story that is subsequently revealed to be false can continue to influence readers’ reasoning and beliefs. This effect is pronounced when the information enhances comprehension of a story in memory (Study 1) and can fill a causal “gap” in an individual's mental model of a story (i.e., the effect is less likely to occur with noncausally related information; Study 2). The influence of poststory misinformation can extend to behavior (Study 2; though results were marginal). However, the continued influence of misinformation is attenuated when the story ends in a negative manner (Study 3)" | "Past work on the continued influence effect (CIE) of misinformation has repeatedly demonstrated that it occurs for information embedded in a narrative, such as a news story or report (Johnson & Seifert, 1994; van Oostendorp & Bonebakker, 1996). This effect has been shown to occur for causal information that is encountered in the context of processing a single story. As Seifert (2014, p. 45) noted about the seminal Johnson and Seifert (1994) study, the target misinformation “had to be part of the story line” in order to influence participants’ answers about an event cause. The present study refutes the notion that the target information must be processed as part of the story, and demonstrates that the CIE can result from misinformation received after a story is encoded, with the magnitude of the effect influenced by the story representation in memory; specifically, whether the associated mental event model and its cause‐effect structure is complete." | "People often encounter information that they subsequently learn is false. Past research has shown that people sometimes continue to use this misinformation in their reasoning, even if they remember that the information is false, which researchers refer to as the continued influence effect. The current work shows that the continued influence effect depends on the stories people have in memory: corrected misinformation was found to have a stronger effect on people's beliefs than information that was topically related to the story if it helped to provide a causal explanation of a story they had read previously. We argue this effect occurs because information that can fill a causal “gap” in a story enhances comprehension of the story event, which allows people to build a complete (if inaccurate) event model that they prefer over an accurate but incomplete event model. This effect is less likely to occur for stories in memory that end in a negative way, presumably because people are more motivated to accurately understand negative outcome events." | https://doi.org/10.1002/jcpy.1135 | No | Experimental | Beliefs | Unclear | |||||||||||
11 | Combating Fake News on Social Media with Source Ratings: The Effects of User and Expert Reputation Ratings | Antino Kim, Patricia Moravec, Alan Dennis | 2019 | Journal of Management Information Systems | Exposure to 8 news headlines accompanied by a form of source rating | Participants "reported the believability of each article and their likelihood of reading, liking, commenting on, and sharing it" | "Between-subjects treatments: expert rating, user article rating, or user source rating" | US adults from a Qualtrics panel | Content labeling | "Overall, our results show that presenting source reputation ratings directly influences the extent to which users believe articles on social media. In general, low ratings decrease believability, while high ratings do not… We also found an unexpected second-order effect. Once users saw ratings on some sources, they became more skeptical of articles from sources without ratings, despite instructions that the lack of ratings had no meaning" | "Confirmation bias plays an important role in users’ beliefs of the social media articles and also strongly influences their engagement with the articles (i.e., reading, liking, commenting, and sharing). Ratings affected believability, but only when sources were rated low: articles from low-rated sources were less believable—when the ratings were from experts or users evaluating articles—but articles from highrated sources were not more believable. Believability and confirmation bias both affected user actions" | "As a remedy against fake news on social media, we examine the effectiveness of three different mechanisms for source ratings that can be applied to articles when they are initially published: expert rating (where expert reviewers fact-check articles, which are aggregated to provide a source rating), user article rating (where users rate articles, which are aggregated to provide a source rating), and user source rating (where users rate the sources themselves). We conducted two experiments and found that source ratings influenced social media users’ beliefs in the articles and that the rating mechanisms behind the ratings mattered. Low ratings, which would mark the usual culprits in spreading fake news, had stronger effects than did high ratings. When the ratings were low, users paid more attention to the rating mechanism, and, overall, expert ratings and user article ratings had stronger effects than did user source ratings. We also noticed a second-order effect, where ratings on some sources led users to be more skeptical of sources without ratings, even with instructions to the contrary. A user’s belief in an article, in turn, influenced the extent to which users would engage with the article (e.g., read, like, comment and share). Lastly, we found confirmation bias to be prominent; users were more likely to believe — and spread — articles that aligned with their beliefs. Overall, our results show that source rating is a viable measure against fake news and propose how the rating mechanism should be designed." | https://doi-org.ezproxy.princeton.edu/10.1080/07421222.2019.1628921 | Yes | Experimental | Intended action/behavior | Yes | Yes | Yes | |||||||||
12 | Adapting Security Warnings to Counter Online Disinformation | Ben Kaiser, Jerry Wei, Elena Lucherini, Kevin Lee, J. Nathan Matias, Jonathan Mayer | 2020 | arXiv | In a lab setting, participants "assumed the persona of an academic researcher trying to find answers to four questions using Google search" and were exposed to misinformation warnings | Measured participant clickthrough rate (CTR) and alternative visit rate (AVR), and interviewed participants about understanding of disinformation | Participants exposed to contextual or interstitial disinformation warnings | Students from Princeton University | Content labeling | Disinformation disclosure | "The interstitial warning was distinctly more noticeable and comprehensible than the contextual warning, and also far more effective at inducing behavioral changes." | "The interstitial warning’s strong effect was due, in part, to the friction it introduced into the task workflow. Some participants preferred to choose another source rather than read the warning, decide whether to believe and comply with it, and click through it. As with the “fear of harm” mechanism, friction must be carefully calibrated to avoid inducing warning fatigue or habituating users to ignore warnings. Friction also has serious drawbacks as a causal mechanism: it degrades the user experience, makes platforms more difficult to use, and does not rely on an informed decision about disinformation." | "Disinformation is proliferating on the internet, and platforms are responding by attaching warnings to content. There is little evidence, however, that these warnings help users identify or avoid disinformation. In this work, we adapt methods and results from the information security warning literature in order to design and evaluate effective disinformation warnings. In an initial laboratory study, we used a simulated search task to examine contextual and interstitial disinformation warning designs. We found that users routinely ignore contextual warnings, but users notice interstitial warnings--and respond by seeking information from alternative sources. We then conducted a follow-on crowdworker study with eight interstitial warning designs. We confirmed a significant impact on user information-seeking behavior, and we found that a warning's design could effectively inform users or convey a risk of harm. We also found, however, that neither user comprehension nor fear of harm moderated behavioral effects. Our work provides evidence that disinformation warnings can -- when designed well -- help users identify and avoid disinformation. We show a path forward for designing effective warnings, and we contribute repeatable methods for evaluating behavioral effects. We also surface a possible dilemma: disinformation warnings might be able to inform users and guide behavior, but the behavioral effects might result from user experience friction, not informed decision making." | Study "adapted contextual and interstitial disinformation warnings from modern security warnings used by Google" | https://arxiv.org/abs/2008.10772 | No | Simulated social media | Observed online behaviors | Unclear | |||||||||
13 | Fake news, fast and slow: Deliberation reduces belief in false (but not true) news headlines. | Bence Bago, David G. Rand, and Gordon Pennycook | 2020 | Journal of Experimental Psychology | Exposure to 16 actual headlines, varying along political ideology and truthfulness, with participants judging headlines with or without deliberation | Rating of headline accuracy | One-response condition versus two-response condition (deliberation between initial judgement of headline and later interpretation) | Participants from Amazon's Mechanical Turk | Other | "Broadly, we found that people made fewer mistakes in judging the veracity of headlines – and in particular were less likely to believe false claims – when they deliberated, regardless of whether or not the headlines aligned with their ideology. Conversely, we found no evidence that deliberation influenced the level of partisan bias/polarization." | "The classical account of reasoning… argues that when people engage in deliberation, it typically helps uncover the truth. In the context of misinformation, the classical account therefore posits that it is the lack of deliberation that promotes belief in fake news, while deliberation results in greater truth discernment" | "What role does deliberation play in susceptibility to political misinformation and “fake news”? The Motivated System 2 Reasoning (MS2R) account posits that deliberation causes people to fall for fake news, because reasoning facilitates identity-protective cognition and is therefore used to rationalize content that is consistent with one’s political ideology. The classical account of reasoning instead posits that people ineffectively discern between true and false news headlines when they fail to deliberate (and instead rely on intuition). To distinguish between these competing accounts, we investigated the causal effect of reasoning on media truth discernment using a 2-response paradigm. Participants (N = 1,635 Mechanical Turkers) were presented with a series of headlines. For each, they were first asked to give an initial, intuitive response under time pressure and concurrent working memory load. They were then given an opportunity to rethink their response with no constraints, thereby permitting more deliberation. We also compared these responses to a (deliberative) 1-response baseline condition where participants made a single choice with no constraints. Consistent with the classical account, we found that deliberation corrected intuitive mistakes: Participants believed false headlines (but not true headlines) more in initial responses than in either final responses or the unconstrained 1-response baseline. In contrast—and inconsistent with the Motivated System 2 Reasoning account—we found that political polarization was equivalent across responses. Our data suggest that, in the context of fake news, deliberation facilitates accurate belief formation and not partisan bias." | https://psycnet.apa.org/doi/10.1037/xge0000729 | Unclear | Real-world media | Beliefs | ||||||||||||
14 | When Readers Believe Journalists: Effects of Adjudication in Varied Dispute Contexts | Benjamin A. Lyons | 2018 | International Journal of Public Opinion Research | "Participants read a news story on a state legislature’s dispute over the economic cost–benefit of a proposed nuclear power project." | Measured factual belief, news quality, satisfaction of informational needs, and likely future news use | "2 (one-sided adjudication/none) × 2 (intraparty/polarized dispute) design" | U.S. participants from Amazon’s Mechanical Turk platform | Disinformation disclosure | "This study instead tested the effects of one-sided journalistic adjudication in two theoretically promising contexts—factual disputes within parties, and interparty disagreement on an issue over which the parties have not become bitterly and symbolically opposed. The results show that adjudication was effective overall in influencing factual beliefs in both contexts. Moreover, these findings show no differential effectiveness because of directional motivated reasoning... Across contexts, adjudication increased satisfaction of informational needs, and had no effect on perceived bias or future news use. This effect also was not conditional on whether the adjudication supported or contradicted the reader’s party." | "the findings suggest that accuracy motivation played a role in the tendency of respondents to reach conclusions in line with the evidence presented in the article rather than their partisanship or ideology (see Druckman, 2012 for a discussion). Because this study explicitly focused on conditions most likely to produce broadly successful adjudication, the logical next step would be a more stringent test, a contentiously polarized dispute." | "Journalists are often criticized for passive reporting of factual disputes in politics, but researchers have only recently begun exploring conditions in which they may successfully influence readers’ beliefs—scenarios less likely to produce partisan bias. Intraparty disputes and those which are polarized, but not contentious, may be two alignments of elite cues that vitiate motivated reasoning and allow for influential adjudication. This experiment (N = 523) used a 2 (one-sided adjudication/none) × 2 (intraparty/polarized dispute) design to test this hypothesis. In both cases, adjudication’s effects on factual beliefs were not conditional on ideological or partisan cues. Adjudication did not increase perceived bias, and increased satisfaction of readers’ informational needs." | https://doi.org/10.1093/ijpor/edx013 | Unclear | Experimental | Intended action/behavior | Yes | |||||||||||
15 | When Corrections Fail: The Persistence of Political Misperceptions | Brendan Nyhan and Jason Reifler | 2010 | Political Behavior | Reading "mock newspaper articles containing a statement from a political figure that reinforces a widespread misperception" with or without corrective information | Responses to factual and opinion questions | Presence of corrective information after a false or misleading statement vs. control | Undergraduates at a Catholic university in the Midwest | Disinformation disclosure | "We find that responses to corrections in mock news articles differ significantly according to subjects' ideological views. As a result, the corrections fail to reduce misperceptions for the most committed participants. Even worse, they actually strengthen misperceptions among ideological subgroups in several cases." | "The backfire effects that we found seem to provide further support for the growing literature showing that citizens engage in "motivated reasoning." While our experiments focused on assessing the effectiveness of correction the results show that direct factual contradictions can actually strengthen ideologically grounded factual beliefs - an empirical finding with important theoretical implications." | "An extensive literature addresses citizen ignorance, but very little research focuses on misperceptions. Can these false or unsubstantiated beliefs about politics be corrected? Previous studies have not tested the efficacy of corrections in a realistic format. We conducted four experiments in which subjects read mock news articles that included either a misleading claim from a politician, or a misleading claim and a correction. Results indicate that corrections frequently fail to reduce misperceptions among the targeted ideological group. We also document several instances of a "backfire effect" in which corrections actually increase misperceptions among the group in question." | https://www.jstor.org/stable/40587320 | No | Experimental | Knowledge | Unclear | |||||||||||
16 | The Effect of Fact-Checking on Elites: A Field Experiment on U.S. State Legislators | Brendan Nyhan and Jason Reifler | 2015 | American Journal of Political Science | Sent a series of letters "about the risks to [state legislator's] reputation and electoral security if they were caught making questionable statements" | Measured likeliness of receiving a "negative fact-checking rating" or having "their accuracy questioned publicly" | Randomly assigned state legislators to receive letters or not | "Field experiment on a diverse group of state legislators from nin U.S. states" | Other | Find that "the randomized provision of a series of letters highlighting the electoral and reputational risks of having questionable statements exposed by fact-checkers significantly reduced the likelihood that legislators in nine U.S. states would receive a negative fact-checking rating or have the accuracy of their claims questioned publicly" | "We found no evidence that these results were driven by legislators speaking less frequently or receiving less coverage, suggesting instead that they were less likely to make inaccurate statements rather than being silenced more generally" | "Does external monitoring improve democratic performance? Fact-checking has come to play an increasingly important role in political coverage in the United States, but some research suggests it may be ineffective at reducing public misperceptions about controversial issues. However, fact-checking might instead help improve political discourse by increasing the reputational costs or risks of spreading misinformation for political elites. To evaluate this deterrent hypothesis, we conducted a field experiment on a diverse group of state legislators from nine U.S. states in the months before the November 2012 election. In the experiment, a randomly assigned subset of state legislators was sent a series of letters about the risks to their reputation and electoral security if they were caught making questionable statements. The legislators who were sent these letters were substantially less likely to receive a negative fact-checking rating or to have their accuracy questioned publicly, suggesting that fact-checking can reduce inaccuracy when it poses a salient threat." | https://www.jstor.org/stable/24583087 | No | Real-world events | Observed real-world behaviors | ||||||||||||
17 | Taking Fact-checks Literally But Not Seriously? The Effects of Journalistic Fact-checking on Factual Beliefs and Candidate Favorability | Brendan Nyhan, Ethan Porter, Jason Reifler, Thomas Wood | 2019 | Political Behavior | "Exposure to realistic journalistic fact-checks of claims made by Donald Trump during his convention speech and a general election debate" | "Accuracy of respondents’ factual beliefs" and attitudes toward Trump | Assignment to one of five conditions, with treatments "based on actual news events during the 2016 Republican National Convention" | Clinton and Trump supporters, with participants operating through Morning Consult and Mechanical Turk | Disinformation disclosure | "Fact-checking messages related to President Trump improved the accuracy of respondents’ factual beliefs, even among his supporters, but had no measurable effect on attitudes toward Trump" | Suggest "exposure to counterattitudinal information decreases perceptions of the accuracy of our stimulus article and the source of counter-attitudinal information," but doesn't necessarily lead to belief updating | "Are citizens willing to accept journalistic fact-checks of misleading claims from candidates they support and to update their attitudes about those candidates? Previous studies have reached conflicting conclusions about the effects of exposure to counterattitudinal information. As fact-checking has become more prominent, it is therefore worth examining how respondents respond to fact-checks of politicians — a question with important implications for understanding the effects of this journalistic format on elections. We present results to two experiments conducted during the 2016 campaign that test the effects of exposure to realistic journalistic fact-checks of claims made by Donald Trump during his convention speech and a general election debate. These messages improved the accuracy of respondents’ factual beliefs, even among his supporters, but had no measurable effect on attitudes toward Trump. These results suggest that journalistic fact checks can reduce misperceptions but often have minimal effects on candidate evaluations or vote choice." | https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2995128 | No | Experimental | Beliefs | Yes | |||||||||||
18 | The Hazards of Correcting Myths About Health Care Reform | Brendan Nyhan, Jason Reifler, and Peter A. Ubel | 2013 | Medical Care | Participants read an article containing false information about the Affordable Care Act and some received corrective information | Assessed "belief in the death panels myth" and approval for the Affordable Care Act | Participants did or did not receive corrective information | U.S. residents recruited via SurveySpot | Disinformation disclosure | "These results show that corrective information can reduce health policy misinformation for some groups. Most notably, our correction decreased misperceptions about death panels and increased approval of ACA among low-knowledge respondents who viewed Palin favorably. However, the correction backfired among high-knowledge respondents who viewed Palin unfavorably, increasing misperceptions about death panels and strong disapproval of ACA." | "These results demonstrate that corrective informationn may not be sufficient to overcome motivated reasoning among more sophisticated members of the public." | "Context: Misperceptions are a major problem in debates about health care reform and other controversial health issues. Methods: We conducted an experiment to determine if more aggressive media fact-checking could correct the false belief that the Affordable Care Act would create “death panels.” Participants from an opt-in Internet panel were randomly assigned to either a control group in which they read an article on Sarah Palin’s claims about “death panels” or an intervention group in which the article also contained corrective information refuting Palin. Findings: The correction reduced belief in death panels and strong opposition to the reform bill among those who view Palin unfavorably and those who view her favorably but have low political knowledge. However, it backfired among politically knowledgeable Palin supporters, who were more likely to believe in death panels and to strongly oppose reform if they received the correction. Conclusions: These results underscore the difficulty of reducing misperceptions about health care reform among individuals with the motivation and sophistication to reject corrective information." | https://www.jstor.org/stable/41714666 | Yes | Real-world media | Beliefs | Unclear | Yes | ||||||||||
19 | Emotions, Partisanship, and Misperceptions: How Anger and Anxiety Moderate the Effect of Partisan Bias on Susceptibility to Political Misinformation | Brian E. Weeks | 2015 | Journal of Communication | Participants "read a news article attributed to the Associated Press that discussed how public misperceptions were circulating around the political debate on either immigration or the death penalty," and some participants viewed corrected false information | Participants "reported their beliefs in the claims presented in the article" | Treatment: manipulated the issue (immigration reform vs. death penalty), emotion (anger vs. anxiety), presence of a correction on false claims, political affiliation of the source of original news Control: made to feel relaxed | 768 U.S. asults recruited by Qualtrics | Disinformation disclosure | Content labeling | "This study provides causal evidence that the independent experience of anger and anxiety at times has different consequences for political misperceptions by heightening or dampening the influence of partisanship. When initially faced with inaccurate claims about politics, angry people are more likely to process the information in a partisan manner, which results in beliefs that reinforce their party affiliation. In contrast, anxiety reduces the reliance on partisanship at this initial stage and leads to beliefs that are consistent with information contained in the message" | "The results here suggest that the partisan motivated reasoning process may be facilitated by anger rather than anxiety or general negative affect, as anger resulted in initial beliefs that were consistent with party identification. This raises the possibility that the “automatic affective processes” that are thought to lead to biased processing of information might be limited to the discrete emotion anger rather than other negative emotions or general negative affect." | "Citizens are frequently misinformed about political issues and candidates but the circumstances under which inaccurate beliefs emerge are not fully understood. This experimental study demonstrates that the independent experience of two emotions, anger and anxiety, in part determines whether citizens consider misinformation in a partisan or open-minded fashion. Anger encourages partisan, motivated evaluation of uncorrected misinformation that results in beliefs consistent with the supported political party, while anxiety at times promotes initial beliefs based less on partisanship and more on the information environment. However, exposure to corrections improves belief accuracy, regardless of emotion or partisanship. The results indicate that the unique experience of anger and anxiety can affect the accuracy of political beliefs by strengthening or attenuating the influence of partisanship." | https://doi-org.ezproxy.princeton.edu/10.1111/jcom.12164 | No | Experimental | Beliefs | Yes | ||||||||||
20 | They Might Be a Liar But They’re My Liar: Source Evaluation and the Prevalence of Misinformation | Briony Swire-Thompson, Ullrich K. H. Ecker, Stephan Lewandowsky, Adam J. Berinsky | 2020 | Political Psychology | Participants read "true/false statements made by either Trump or Sanders" and were later presented with fact-checks | Participants rated statements on a belief scale and feelings towards each candidate before and after fact-checks | Participants reviewed true/false statements from either Donald Trump or Bernie Sanders | U.S. residents from Amazon's Mechanical Turk | Disinformation disclosure | "We aimed to investigate whether feelings towards favored political candidates were reduced when many inaccurate statements were corrected and only a single factual statement affirmed, as opposed to a “balanced” presentation of equal numbers of false and factual statements. In a clear replication of Swire et al. (2017), supporters of the politicians reduced their belief in misinformation once corrected, yet they did not reduce their feelings towards the political figure if misinformation was presented alongside an equal number of facts. There was, however, a slight reduction in feelings if more misinformation was corrected than factual statements affirmed... Our final aim was to examine whether correcting misinformation changed participants’ view of a politician’s general veracity. We found that participants did not shift their perceived general veracity in accordance with the number of corrections they received during the experiment" | "Hahl and colleagues argue that a politician’s willingness to be regarded as a pariah by the establishment and flout the norms of honesty only increases perceived authenticity among supporters. Alternatively, people may perceive their preferred candidate’s false statements as unintentional errors rather than a deliberate intent to deceive. It is possible that being perceived as inaccurate is less costly than being perceived to be dishonest." | "Even if people acknowledge that misinformation is incorrect after a correction has been presented, their feelings towards the source of the misinformation can remain unchanged. The current study investigated whether participants reduce their support of Republican and Democratic politicians when the prevalence of misinformation disseminated by the politicians appears to be high in comparison to the prevalence of their factual statements. We presented U.S. participants either with (1) equal numbers of false and factual statements from political candidates or (2) disproportionately more false than factual statements. Participants received fact‐checks as to whether items were true or false, then rerated both their belief in the statements as well as their feelings towards the candidate. Results indicated that when corrected misinformation was presented alongside equal presentations of affirmed factual statements, participants reduced their belief in the misinformation but did not reduce their feelings towards the politician. However, if there was considerably more misinformation retracted than factual statements affirmed, feelings towards both Republican and Democratic figures were reduced—although the observed effect size was extremely small." | Replicated Aird et al. (2018, next listed study), and found that Australian voters "did find a sizable reduction in feelings towards Australian politicans when participants were presented with disproportionately more misinformation than factual statements" while effect was small for American voters. Indicates that "veracity sometimes matters to voters and that the cultural context may determine people’s expectations of truthfulness among politicians and how they respond to lies" | https://doi-org.ezproxy.princeton.edu/10.1111/pops.12586 | Unclear | Experimental | Beliefs | Yes | ||||||||||
21 | The role of familiarity in correcting inaccurate information | Briony Swire, Ullrich K. H. Ecker, and Stephan Lewandowsky | 2017 | Journal of Experimental Psychology | Presented "incorrct and correct claims (i.e., myths and facts), then corrected false claims in a way that boosted their familiarity" | Measured participants' belief in statements before and after the manipulation | "The experiment used a 2 x 2 x 3 within-between design, with within- subjects factors type of item (myth vs. fact) and type of explanation (the veracity of each statement was explained either briefly or in some detail), and the between-subjects factor retention interval (immediate, 30-min, or 1-week)" | Experiment 1: Undergraduate population from the University of Western Australia Experiment 2: older adults (over age 50) recruited by the University of Western Australia | Disinformation disclosure | Content labeling | "Across both experiments, we found a striking asymmetry in that belief change was more sustained after fact affirmation compared with myth retraction—retractions thus seemingly have an 'expiration date.'… Overall, the pattern of belief change over time — and in particular the asymmetry between facts and myths — was similar in young and older participants... However, 'old' participants aged 65 and over were found to be comparatively worse than those aged 50-64 ('middle-aged' participants) at sustaining their postcorrection belief that myths are inaccurate." | "The present research provides evidence for familiarity causing an increase in postcorrection myth belief after a delay; this meshes well with previous studies that similarly reported that myths are often “misremembered” as facts over time (Peter & Koch, 2016; Skurnik et al., 2005; Skurnik et al., 2007). However, we found no evidence for the existence of a true familiarity-based backfire effect. As in these previous studies, the corrections did help participants update their beliefs in the right direction—that is, myth beliefs were reduced by the corrections. Corrections repeating the myth were simply less effective (compared with fact affirmations) rather than backfiring." | "People frequently continue to use inaccurate information in their reasoning even after a credible retraction has been presented. This phenomenon is often referred to as the continued influence effect of misinformation. The repetition of the original misconception within a retraction could contribute to this phenomenon, as it could inadvertently make the "myth" more familiar-and familiar information is more likely to be accepted as true. From a dual-process perspective, familiarity-based acceptance of myths is most likely to occur in the absence of strategic memory processes. Thus, we examined factors known to affect whether strategic memory processes can be utilized: age, detail, and time. Participants rated their belief in various statements of unclear veracity, and facts were subsequently affirmed and myths were retracted. Participants then rerated their belief either immediately or after a delay. We compared groups of young and older participants, and we manipulated the amount of detail presented in the affirmative or corrective explanations, as well as the retention interval between encoding and a retrieval attempt. We found that (a) older adults over the age of 65 were worse at sustaining their postcorrection belief that myths were inaccurate, (b) a greater level of explanatory detail promoted more sustained belief change, and (c) fact affirmations promoted more sustained belief change in comparison with myth retractions over the course of 1 week (but not over 3 weeks), This supports the notion that familiarity is indeed a driver of continued influence effects." | Sample from western Australia | https://doi.org/10.1037/xlm0000422 | No | Experimental | Beliefs | Yes | |||||||||
22 | Reminders of Everyday Misinformation Statements Can Enhance Memory for and Beliefs in Corrections of Those Statements in the Short Term | Christopher N. Wahlheim, Timothy R. Alexander, and Carson D. Peske | 2020 | Psychological Science | Participants exposed to statements of unclear veracity and thenn exposed to affirmations of facts and corrections of misinformation | "Measured memory and beliefs for corrections and misinformation" | "Within-subjects variable statement type (repeated vs. control vs. correction vs. correction + reminder)" | U.S. university undergraduates | Disinformation disclosure | "We examined the effects of providing misinformation reminders before fake-news corrections on memory and belief accuracy. Our study included everyday fake-news misinformation that was corrected by fact-check-verified statements. Building on research using fictional, yet naturalistic, event narratives to show that reminders can counteract misinformation reliance in memory reports (e.g., Ecker et al., 2017), we showed improvements in both memory and belief accuracy when reminders increased misinformation recollection." | "These findings suggest that reminders of misinformation before corrections enhanced conflict salience, recollection of corrections, and source memory, thereby reducing familiarity-backfire effects." | "Fake-news exposure can cause misinformation to be mistakenly remembered and believed. In two experiments (Ns = 96), we examined whether reminders of misinformation could improve memory for and beliefs in corrections. Subjects read factual statements and misinformation statements taken from news websites and then read statements that corrected the misinformation. Misinformation reminders appeared before some corrections but not others. Subjects then attempted to recall facts, indicated their belief in those recalls, and indicated whether they remembered corrections and misinformation. In Experiment 1, we did not constrain subjects’ report criteria. But in Experiment 2, we encouraged conservative reporting by instructing subjects to report only information they believed to be true. Reminders increased recall and belief accuracy. These benefits were greater both when misinformation was recollected and when subjects remembered that corrections had occurred. These findings demonstrate one situation in which misinformation reminders can diminish the negative effects of fake-news exposure in the short term." | https://doi.org/10.1177%2F0956797620952797 | No | Experimental | Beliefs | Yes | |||||||||||
23 | Flagging Fake News on Social Media: An Experimental Study of Media Consumers’ Identification of Fake News | Dongfang Gaozhao | 2020 | SSRN Scholarly Paper | Participants read 30 news items of varying political ideology and truthfulness and forms of fact-checks | Participants rated the authenticity of each news story | "In the control group, subjects received news stories with no flag assessment while participants in two treatment groups received news stories with fact-checkers’ and peers’ flag assessments, respectively." | Participants from Amazon's Mechanical Turk | Content labeling | Disinformation disclosure | "In terms of flag assessments’ effects, both accurate and inaccurate flags were equally powerful to change subjects’ perceived genuineness since the effect sizes of accurate and inaccurate flags were almost equal in their absolute values… In terms of the comparisons between professional fact-checkers versus crowdsourcing, the insignificant differences between two treatment groups imply that the expert source and the laymen source might not make a huge difference in the eyes of survey respondents in this experiment." | "It is clear that fact-checking flags may change participants’ perceived genuineness of news items. These perception changes were not a consequence of skepticism but a result of fact-checking flags because subjects in two treatment groups were more likely to identify real and fake news items as real when those news items had flags claiming the news to be true. Therefore, this finding disconfirmed the speculation of Lewandowsky et al. (2012, 2013) that people may be “a temporary state of skepticism.”" | "Policymakers are taking actions to protect their citizens and democratic systems from online misinformation. However, media consumers usually have a hard time differentiating misinformation from authentic information. There are two explanations for this difficulty, namely lazy reasoning and motivated reasoning. While lazy reasoning suggests that people may feel reluctant to conduct critical reasoning when consuming online information, the motivated reasoning theory points out that individuals are also thinking in alignment with their identities and established viewpoints. A proposed approach to address this issue is adding fact-checking flags in the hope that flags could alert people to information falsehoods and stimulate critical thinking. This study examines the impact of fact-checking flags on media consumers’ identification of fake news. Conducting an experiment (n = 717) on Amazon Mechanical Turk, the study finds that experimental participants with different political backgrounds depend heavily on flag-checking results provided by flags. Flags are powerful to influence people’s judgments in a way that participants have blind beliefs in flags even if the flag assessments are inaccurate. Furthermore, the study’s results indicate that flag assessments made by professional fact-checkers or crowd-sourcing are equally influential in shaping participants’ identification. These observations provide public and private leaders with suggestions that fact-checking flags can significantly affect media consumers’ identification of fake news. However, flags appear to have little ability to promote critical thinking in this experiment." | https://dx.doi.org/10.2139/ssrn.3669375 | Unclear | Experimental | Beliefs | Yes | ||||||||||
24 | Do Corrective Effects Last? Results from a Longitudinal Experiment on Beliefs Toward Immigration in the U.S. | Dustin Carnahan, Daniel E. Bergan and Sangwon Lee | 2020 | Political Behavior | Participants presented with stories about immigration and fact-checks | "The key outcome in our analyses is the accuracy of participants’ beliefs regarding a series of claims commonly invoked in the immigration debate in the U.S. Participants were asked to rate the factual veracity of six claims (four false, two true) on a four-point scale ranging from “definitely true” to “definitely false.”" | Control group; varies correction treatments with myth-fact presentation or presenting fact checks only | U.S. participants recruited by Research Now/SSI | Disinformation disclosure | "This study offers evidence that corrective effects show considerable stability over time; improvements in belief accuracy in response to corrective information were detectable several weeks later. Furthermore, we found no evidence that durability was influenced by whether the correction was belief-consistent and only limited evidence that participants’ media habits—isolated only to social media use—during the time periods intervening each wave of study might influence the long-term efficacy of corrective efforts. Repeated exposure to corrective messages, in addition to fostering greater immediate gains in belief accuracy, promoted greater correction durability, suggesting continued engagement with accurate information as a means of attaining longer-lasting impacts on beliefs." | "On this latter point, these findings—when considered in the context of the various schools of thought regarding belief formation and maintenance—appear to yield evidence supporting the idea that belief change might follow a somewhat Bayesian logic. Whereas affective and mental model explanations of how beliefs are formed and organized might suggest any effects of corrections on belief change to be fleeting, evidence of a long-term impact of these messages suggest that people are generally successful in updating their beliefs regardless of attitudinal or cognitive biases." | "Although interest in the efficacy of efforts to correct false beliefs has peaked in recent years, the extent to which corrective effects endure over time remains understudied. Drawing on insights from related literatures in the psychology of belief, persuasion and media effects to inform theoretical expectations, this study uses a longitudinal experiment to observe both contemporaneous and long-term changes in participants’ belief accuracy in response to corrective information within an ongoing, contentious political debate. We measured factors thought to either promote durability (e.g. repeated exposure to corrective information) or cause decay (e.g. political predispositions, media behaviors) in assessing moderators of the magnitude and longevity of corrections. Corrective effects were found to be quite durable, detectable up to 4 weeks after exposure to the initial message, while repeated exposure to corrective information further promoted the longevity of these effects." | https://doi.org/10.1007/s11109-020-09591-9 | Yes | Experimental | Beliefs | Yes | |||||||||||
25 | Not wallowing in misery – retractions of negative misinformation are effective in depressive rumination | Ee Pin Chang, Ullrich K. H. Ecker, and Andrew C. Page | 2019 | Cognition and Emotion | "Participants were presented with a report including a piece of critical information that was or was not subsequently retracted." | "Participants’ understanding of the report and their inferential reasoning concerning the information presented in the report were assessed via questionnaire." | Experiment 1: "2 (retraction: no vs. yes) × 2 (scenario: specific vs. general) × 2 (group: depressive rumination low vs. high) between-subjects design" Experiment 2: "2 (retraction: no vs. yes; within-subjects) × 2 (valence: positive vs. negative; within-subjects) × 2 (group: depressive rumination low vs. high; between-subjects) mixed design" | undergraduates from the University of Western Australia | Disinformation disclosure | "The aim of the present study was to investigate potential memory updating deficits in depressive rumination (DR) in a real-world context. We used the continued-influence effect (CIE; Johnson & Seifert, 1994) paradigm, varying the specificity (Experiment 1) and valence (Experiment 2) of the fictional news reports presented, to assess participants’ reliance on retracted misinformation... Contrary to our hypothesis, we found evidence against this hypothesis in both experiments. Retractions of worldview-congruent, negative misinformation were as effective as (Experiment 1) or even more effective than (Experiment 2) retractions of worldview-incongruent misinformation. That is, across both experiments, retractions of negative misinformation were at least as effective in depressive ruminators as they were in control participants, and they were more effective than retractions of positive misinformation in depressive ruminators." | "It is important to reiterate that the CIE occurs reliably even with no involvement of specific worldviews or cognitive biases, as a consequence of generic integration or retrieval failures (see Lewandowsky et al., 2012). Our results suggest that with emotionally valent information, the effect is modulated by an attentional bias towards information that is worldview-congruent, which may boost references to this information if it is non-retracted but facilitate its updating and revision following retraction. We speculate that this modulation is salience-driven; that is, in our sample of depressive ruminators, we argue that the updating facilitation arose for negative information because this information was high in salience based on its valence-related worldview congruence." | "People often continue to rely on misinformation in their reasoning after they have acknowledged a retraction; this phenomenon is known as the continued-influence effect. Retractions can be particularly ineffective when the retracted misinformation is consistent with a pre-existing worldview. We investigated this effect in the context of depressive rumination. Given the prevalence of depressotypic worldviews in depressive rumination, we hypothesised that depressive rumination may affect the processing of retractions of valenced misinformation; specifically, we predicted that the retraction of negative misinformation might be less effective in depressive ruminators. In two experiments, we found evidence against this hypothesis: in depressive ruminators, retractions of negative misinformation were at least as effective as they were in control participants, and more effective than retractions of positive misinformation. Findings are interpreted in terms of an attentional bias that may enhance the salience of negative misinformation and may thus facilitate its updating in depressive rumination." | https://doi-org.ezproxy.princeton.edu/10.1080/02699931.2018.1533808 | No | Experimental | Knowledge | Yes | |||||||||||
26 | Better Liked than Right: Trustworthiness and Expertise as Factors in Credibility | Elliott McGinnies and Charles D. Ward | 1980 | Personality and Social Psychology Bulletin | Participants read a booklet with an argument about extending territorial boundaries from varying sources | Measured participants' opinion on the issue | Manipulated expert vs. non-expert source; trustworthiness of the source | 289 U.S. students from the University of Maryland; 275 Japanese subjects from Tokyo colleges; 221 New Zealander students from the University of Auckland; 270 Australian students from the University of Sydney | Disinformation disclosure | Content labeling | "Averaged across countries, the results provide convincing evidence that the trustworthiness of the source was more important in our study than expertise. The most favorable condition for persuasion in both the American and New Zealand samples was one in which the source was described as both expert and trustworthy. In Australia and Japan, the trustworthy source was more effective regardless of whether it was paired with high or low expertise" | Not specifically stated. Authors say that "[e]xpertise undoubtable lends credibility to a communicator, but it may be less important under some circumstances than trustwothiness" | "A persuasive message on the subject of international maritime boundaries was presented in pamphlet form to 1055 students in four countries. Trustworthiness and expertise of the source were manipulated in a 2 x 2 x 4factorial design of the after-only type to assess the relative impact of each component on the communicator's persuasiveness. Main effects were found for both country and trustworthiness. Overall, the expert and trustworthy source generated the most opinion change. However, the trustworthy communicator was more persuasive, whether expert or not." | Samples from U.S., Japan, New Zealand, and Australia | https://doi.org/10.1177%2F014616728063023 | No | Experimental | Beliefs | Unclear | |||||||||
27 | Nudge Effect of Fact-Check Alerts: Source Influence and Media Skepticism on Sharing of News Misinformation in Social Media | Elmie Nekmat | 2020 | Social Media + Society | Fact-check alert vs. no fact-check alert; legacy mainstream news source vs. unfamiliar non-mainstream news source | Likelihood of sharing news story on social media | Random assignment to one of four conditions varying the presence of a fact-check and type of nes source | Representative sample of Singapore residents owning a personal social media account | Disinformation disclosure | "News sharing likelihood was overall lower for non-mainstream news than mainstream news, but showed a greater decrease for mainstream news when nudged with a fact-check" | Authors "found no conditional moderation from [users'] media skepticism... [though] skepticism of mainstream media amplified the nudge effect (unwillingness to share news labeled as fake) for news from legacy mainstream media" | "This study extends the nudge principle with media effects and credibility evaluation perspectives to examine whether the effectiveness of fact-check alerts to deter news sharing on social media is moderated by news source and whether this moderation is conditional upon users’ skepticism of mainstream media. Results from a 2 (nudge: fact-check alert vs. no alert)×2 (news source: legacy mainstream vs. unfamiliar non-mainstream) (N=929) experiment controlling for individual issue involvement, online news involvement, and news sharing experience revealed significant main and interaction effects from both factors. News sharing likelihood was overall lower for non-mainstream news than mainstream news, but showed a greater decrease for mainstream news when nudged. No conditional moderation from media skepticism was found; instead, users’ skepticism of mainstream media amplified the nudge effect only for news from legacy mainstream media and not unfamiliar non-mainstream source. Theoretical and practical implications on the use of fact-checking and mainstream news sources in social media are discussed." | https://journals.sagepub.com/doi/full/10.1177/2056305119897322 | Yes | Simulated social media | Intended action/behavior | General social media use | Yes | Yes | |||||||||
28 | I do not believe you: how providing a source corrects health misperceptions across social media platforms | Emily K. Vraga and Leticia Bode | 2018 | Information, Communication & Society | Participants read through a simulated feed from either Facebook or Twitter containing misperceptions about the Zika virus | Participants answered questions measuring belief in misinformation about the Zika outbreak | Platform: Twitter vs. Facebook; Zika information: control, corrective response without source, corrective response with source | Participants from a Mid-Atlantic university | Disinformation disclosure | Content labeling | "It seems that when everyday users are correcting one another, the provision of a source confirming the correction being offered is required in order to successfully address and mitigate misperceptions. Social correction absent this corroborating evidence did not significantly reduce misperceptions compared to the control... On both Facebook and Twitter, social correction with a source was successful in reducing misperceptions as compared to a control condition. However, the mechanisms that were successful for social correction with a source differed in comparison to social corrections without a source. On Facebook, providing an external source to substantiate the correction significantly improved perceptions of the corrective comments, while adding this source did not affect evaluations of the corrective replies on Twitter. However, the moderated mediation analysis suggested that these higher evaluations of the corrective replies did not directly translate into lower misperceptions on the Zika issue on Facebook. In other words, even though individuals saw a social correction that provided a source as more credible on Facebook, this heightened credibility did not lead them to reduce their misperceptions compared to a correction without a source. In contrast, on Twitter, individuals with higher evaluations of the social correction were more likely reduce their misperceptions regarding Zika compared to those with lower evaluations of the correction – but adding a source to the correction was not responsible for these heightened credibility perceptions." | "This study highlights that the mechanism underlying the effectiveness of including a source as part of social correction differs on Facebook versus Twitter. We expect these differences are rooted in different technical and social affordances are likely to promote some types of information sharing over others." | "Social media are often criticized as serving as a source of misinformation, but in this study we examine how they may also function to correct misperceptions on an emerging health issue. We use an experimental design to consider social correction that occurs via peers, testing both the type of correction (i.e., whether a source is provided or not) and the platform on which the correction ocratcurs (i.e., Facebook versus Twitter). Our results suggest that a source is necessary to correct misperceptions about the causes of the Zika virus on both Facebook and Twitter, but the mechanism by which such correction occurs differs across platforms. Implications for successful social media campaigns to address health misinformation are addressed." | Finds differences in the mechanisms for social corrections of misinformation on Facebook vs. Twitter | https://doi-org.ezproxy.princeton.edu/10.1080/1369118X.2017.1313883 | No | Simulated social media | Knowledge | Twitter, Facebook | Yes | ||||||||
29 | Using Expert Sources to Correct Health Misinformation in Social Media | Emily K. Vraga and Leticia Bode | 2017 | Science Communication | Participants viewed a simulated Twitter news feed with misinformation about the Zika virus and corrections presented 1 or 2 times from varying sources | Participants answered questions about misperceptions about Zika, credibility of the CDC, and trustworthiness of other users | Manipulated a correction for a false news story about the Zika outbreak in the U.S.: "from a single user, from the CDC, from a user followed by the CDC, and from the CDC followed by a user" | Participants from a Mid-Atlantic university | Disinformation disclosure | Content labeling | "We find that a single correction from a reputable source like the CDC was sufficient to reduce misperceptions about the causes of the spread of the Zika virus that can be created when users are posting misinformation on the topic. Adding a correction from the CDC after another user has refuted the information—even with a link to the same content that the user posted—again enhances corrective effects. Moreover, the credibility of the CDC was not harmed—but neither was it improved—when it engaged in this type of corrective action. This may suggest that the public views such behaviors as appropriate for the CDC, creating opportunities for scientific and health organizations to engage with the public." | "Social media also offers a space for what we term observational correction, which occurs when social media users update their own attitudes after witnessing another user being corrected." | "This study tests whether the number (1 vs. 2) and the source (another user vs. the Centers for Disease Control and Prevention [CDC]) of corrective responses affect successful reduction of misperceptions. Using an experimental design, our results suggest that while a single correction from another user did not reduce misperceptions, the CDC on its own could correct misinformation. Corrections were more effective among those higher in initial misperceptions. Notably, organizational credibility was not reduced when correcting misinformation, making this a low-cost behavior for public health organizations. We recommend that expert organizations like the CDC immediately and personally rebut misinformation about health issues on social media." | https://doi.org/10.1177/1075547017731776 | No | Simulated social media | Knowledge | Yes | ||||||||||
30 | Creating News Literacy Messages to Enhance Expert Corrections of Misinformation on Twitter | Emily K. Vraga, Leticia Bode, and Melissa Tully | 2020 | Communication Research | "Participants read a simulated Twitter feed with 6 tweets; four previously validated control tweets on social and news topics and two manipulated tweets," then viewed varying forms of corrections | Study 1: Participant perception of scientific consensus Study 2: Participant misperceptions about the flu | Study 1: "2 (Correction: Misinformation-only vs. Pew correction) × 3 (Promoted tweet: Texting [control], Your job, Citizen job)" Study 2: "2 (Correction: Misinformation-only, AMA correction) × 2 (Promoted tweet: Texting [control], NL tweet)" | Participants from Amazon's Mechanical Turk | Disinformation disclosure | "Despite calls to correct misinformation on social media and to improve media literacy education to facilitate its identification and correction, this study suggests such efforts are complicated. While this study found additional evidence for observational correction from two different expert sources, it highlights the difficulty of crafting NL tweets that break through the clutter on social media, and the unlikelihood that they enhance the effectiveness of expert correction. Three different NL tweets across two controversial health issues showed little ability to enhance the effectiveness of correction by expert organizations." | "There are several possible reasons that the NL tweets did not produce the expected results. First, these messages were lost in the noise of a busy social media feed—we found that roughly 60% of participants recalled seeing the NL tweet in Study 1 and Study 2—a much lower percentage than who reported seeing the texting and driving tweet (roughly 85% in both studies), the GMO tweet in Study 1 (94%), or the flu tweet in Study 2 (95%)... Second, our results echo previous findings that general warning messages and forewarnings of misinformation—which are similar to the NL message used here—may be less effective than specific corrective responses and sometimes generate cynicism toward all information rather than helping people distinguish misinformation from other types of content" | "A number of solutions have been proposed to address concerns about misinformation online, including encouraging experts to engage in corrections of misinformation being shared and improving media literacy among the American public. This study combines these approaches to examine whether news literacy (NL) messages on social media enhance the effectiveness of expert correction of misinformation on Twitter. Two experiments suggest that expert organizations can successfully correct misinformation on social media across two controversial issues with a single tweet. However, three different NL messages did not improve the effectiveness of expert corrections. We discuss the difficulties of crafting NL messages that break through the clutter on social media and suggest guidelines for organizations attempting to address misinformation online." | https://doi.org/10.1177%2F0093650219898094 | Unclear | Simulated social media | Knowledge | Yes | |||||||||||
31 | Testing the Effectiveness of Correction Placement and Type on Instagram | Emily K. Vraga, Sojung Claire Kim, John Cook, and Leticia Bode | 2020 | The International Journal of Press/Politics | "All participants were told they would see a series of Instagram posts on the topic of climate change and asked to review them as if they had searched for “climate change” on Instagram. All participants viewed six control posts on the topic of climate change, as well as a post containing misinformation claiming that CO2 is plant food and good for plants." Some participants were treated with prebunking corrections, and some viewed corrections after misinformation. | "After viewing the Instagram feed, participants reported their misperceptions regarding the plant myth being promoted by the misinformation, and then rated the credibility of the misinformation post and the correction post that they saw." | "The four correction conditions used a 2 (logic-focused or fact-focused correction) × 2 (prebunking or debunking) experimental design to expose participants to a post that corrects the misinformation that CO2 is good for plants." | Nationally representative U.S. sample recruited by Lucid | Disinformation disclosure | Disinformation literacy | "Our results suggest that observational correction is effective on visual platforms like Instagram, even when the correction is not directly threaded to the misinformation post, but both the correction type and its placement matter for these effects. Overall, we find that logic-focused corrections, which highlight the rhetorical flaws of the misinformation claim, outperform fact-focused corrections, which debunk the claim itself by highlighting scientific facts on the issue, because the logic-focused corrections are impervious to placement." | "Our mediation analyses provide insight into the process by which logic-focused corrections function differently than fact-focused corrections. Logic-focused corrections produce an indirect effect on plant misperceptions through reduced credibility ratings for the misinformation post—an indirect pathway that is not significant for those viewing a fact-focused condition (although it is directionally consistent). However, this indirect pathway for logic-focused corrections via the credibility of the misinformative post is only part of the story—a significant direct pathway remains, reducing misperceptions. More research is needed into what mechanisms explain this direct pathway in reducing misperceptions for both the logic-focused and fact-focused approaches, but this study suggests that perceptions of message credibility explain some of the effectiveness of logic-focused corrections." | "Despite concerns about misinformation across social media platforms, little attention has been paid to how to correct misinformation on visual platforms like Instagram. This study uses an experimental design on a national sample to test two features of user-based correction strategies on Instagram for a divisive issue on which misinformation abounds: the issue of climate change. First, we unite the inoculation and correction literature to test the efficacy of prebunking corrections that come before exposure to the misinformation versus debunking strategies that occur after exposure. Second, we compare fact-focused corrections that provide accurate information to rebut the misinformation against logic-focused corrections that highlight the rhetorical flaw underpinning the misinformation. Our findings suggest that these strategies intersect to reduce misperceptions. Logic-focused corrections effectively reduce misperceptions regardless of their placement before or after the misinformation, and appear to function in part by reducing perceptions of the credibility of the misinformation post. Fact-focused corrections only reduce misperceptions when they occur after the misinformation, but are seen as more credible than logic-focused corrections. We discuss the implications of our findings for understanding the theoretical mechanism by which correction can occur and the practical guidelines to best correct misinformation in visual social media spaces." | Study misinformation on Instagram, which authors say is a "popular but understudied visual platform" | https://doi.org/10.1177%2F1940161220919082 | Yes | Simulated social media | Knowledge | Yes | |||||||||
32 | Belief Echoes: The Persistent Effects of Corrected Misinformation | Emily Thorson | 2015 | Political Communication | Participants read fake news articles with manufactured misinformation. Some participants were exposed to misinformation corrections. Experiment 1: Control, exposure to uncorrected vs. corrected misinformation Experiment 2: (corrected misinformation, uncorrected misinfo, no misinfo) x (same-party candidate, opposing-party candidate) Experiment 3: similar to experiment 2, but participants completed a recall task about candidates immediately before or after evaluation | Experiment 1: candidate evaluation and factuality recall Experiment 2: candidate evaluation and assessment of belief in misinformation Experiment 3: Candidate evaluation and recall task | Experiment 1 introduced a delay between exposure to misinformation and participant response Experiment 2 made misinformation corrections immediately available and varied party alignment Experiment 3 encouraged reflecting on misinformation/correction with a recall task | U.S. participants from Amazon’s Mechanical Turk platform | Disinformation disclosure | "I show that belief echoes can be created through either an affective or a deliberative process, consistent with the dual-process understanding of retrieval outlined in Ecker and Colleagues (2010). When retrieval is automatic, affective belief echoes are created because the misinformation has a larger impact than its correction (as in Experiments 1 and 2). In this process, party identification plays a smaller role. However, when individuals actively recall the correction (as in Experiment 3), party identification affects the inferences that they draw, which in turn shapes their attitudes" | "I show that exposure to misinformation creates belief echoes: Lingering effects on attitudes that persist even after the misinformation is effectively corrected" | "Across three separate experiments, I find that exposure to negative political information continues to shape attitudes even after the information has been effectively discredited. I call these effects “belief echoes.” Results suggest that belief echoes can be created through an automatic or deliberative process. Belief echoes occur even when the misinformation is corrected immediately, the “gold standard” of journalistic fact-checking. The existence of belief echoes raises ethical concerns about journalists’ and fact-checking organizations’ efforts to publicly correct false claims." | https://doi-org.ezproxy.princeton.edu/10.1080/10584609.2015.1102187 | Unclear | Experimental | Beliefs | Unclear | |||||||||||
33 | The Implied Truth Effect: Attaching Warnings to a Subset of Fake News Headlines Increases Perceived Accuracy of Headlines Without Warnings | Gordon Pennycook, Adam Bear, Evan Collins, David Rand | 2020 | Management Science | Experiment 1 and 2: Presentation of true and false headlines with or without associated warnings. | Experiment 1: Participant's perceived accuracy of a headline and willingness to share the headline on social media Experiment 2: Participant's willingness to share a headline on Facebook | Experiment 1: Attachment of a warning to some fake headlines vs. control Experiment 2: Presence of a more explicit "False" label on fake news; presence of "False" and "True" labels; unlabeled control | American residents recruited from Amazon's Mechanical Turk | Content labeling | Experiment 1: "Taken together, the results of Study 1 confirm the predictions of our model: the “disputed” warning decreases belief in items that are tagged (the warning effect) but increases belief in items that are untagged (the implied truth effect)" Experiment 2: "Taken together, these results provide further support for the existence of an implied truth effect, a demonstration that this effect generalizes beyond the outcome and specific “disputed” warning used in Study 1" | "An implied truth effect, whereby false headlines that fail to get tagged are considered validated and thus are seen as more accurate. With a formal model, we demonstrate that Bayesian belief updating can lead to such an implied truth effect" | "What can be done to combat political misinformation? One prominent intervention involves attaching warnings to headlines of news stories that have been disputed by third-party fact-checkers. Here we demonstrate a hitherto unappreciated potential consequence of such a warning: an implied truth effect, whereby false headlines that fail to get tagged are considered validated and thus are seen as more accurate. With a formal model, we demonstrate that Bayesian belief updating can lead to such an implied truth effect. In Study 1 (n = 5,271 MTurkers), we find that although warnings do lead to a modest reduction in perceived accuracy of false headlines relative to a control condition (particularly for politically concordant headlines), we also observed the hypothesized implied truth effect: the presence of warnings caused untagged headlines to be seen as more accurate than in the control. In Study 2 (n = 1,568 MTurkers), we find the same effects in the context of decisions about which headlines to consider sharing on social media. We also find that attaching verifications to some true headlines—which removes the ambiguity about whether untagged headlines have not been checked or have been verified— eliminates, and in fact slightly reverses, the implied truth effect. Together these results contest theories of motivated reasoning while identifying a potential challenge for the policy of using warning tags to fight misinformation—a challenge that is particularly concerning given that it is much easier to produce misinformation than it is to debunk it." | https://doi.org/10.1287/mnsc.2019.3478 | No | Experimental | Intended action/behavior | Yes | Yes | ||||||||||
34 | Fighting COVID-19 Misinformation on Social Media: Experimental Evidence for a Scalable Accuracy-Nudge Intervention | Gordon Pennycook, Jonathon McPhetres, Yunhao Zhang, Jackson G. Lu, David G. Rand | 2020 | Psychological Science | Some participants were nudged to think about accuracy by rating a headline, then all participants were presented false and true headlines relating to COVID-19 in the form of Facebook posts | Asked about accuracy of the headline or willingness to share | In study 2 in the treatment condition, participants "rated the accuracy of a single headline (unrelated to COVID-19) before beginning the news-sharing task" | Participants from Lucid | Other | "Participants were far less discerning if they were asked about whether they would share a headline on social media than if they were asked about its accuracy… In Study 2, we demonstrated the promise of a behavioral intervention informed by this inattention-based account. Prior to deciding which headlines they would share on social media, participants were subtly primed to think about accuracy by being asked to rate the accuracy of a single non-COVID-related news headline. This minimal, content-neutral intervention nearly tripled participants’ level of discernment between sharing true and sharing false headlines." | "Theoretically, our findings shed new light on the perspective that inattention plays an important role in the sharing of misinformation online. By demonstrating the role of inattention in the context of COVID-19 misinformation (rather than politics), our results suggest that partisanship is not, apparently, the key factor distracting people from considering accuracy on social media. Instead, the tendency to be distracted from accuracy on social media seems more general. Thus, it seems likely that people are being distracted from accuracy by more fundamental aspects of the social media context." | "Across two studies with more than 1,700 U.S. adults recruited online, we present evidence that people share false claims about COVID-19 partly because they simply fail to think sufficiently about whether or not the content is accurate when deciding what to share. In Study 1, participants were far worse at discerning between true and false content when deciding what they would share on social media relative to when they were asked directly about accuracy. Furthermore, greater cognitive reflection and science knowledge were associated with stronger discernment. In Study 2, we found that a simple accuracy reminder at the beginning of the study (i.e., judging the accuracy of a non-COVID-19-related headline) nearly tripled the level of truth discernment in participants’ subsequent sharing intentions. Our results, which mirror those found previously for political fake news, suggest that nudging people to think about accuracy is a simple way to improve choices about what to share on social media." | https://doi-org.ezproxy.princeton.edu/10.1177%2F0956797620939054 | Yes | Simulated social media | Intended action/behavior | ||||||||||||
35 | Prior Exposure Increases Perceived Accuracy of Fake News | Gordon Pennycook, Tyrone Cannon, and David Rand | 2018 | Journal of Experimental Psychology | Participants viewed fake news headlines as they appeared on Facebook, varying the familiarity of the information through exposure and delays in recall checks. Experiment 1 and 2: Presentation of false and true statements Experiment 3: repeated Experiment 2 with the same participants after a weeklong delay | Experiment 1: Rating the plausability of statements Experiment 2 and 3: rating statements for familiarity and accuracy | Experiment 1: Previous exposure to statements (familiarization) Experiment 2 and 3: presence of fact-check tag on fake news or not | Mechnical Turk participants | Content distribution/sharing | Experiment 1: "We observed no significant effect of repetition on accuracy judgments for statements that were patently false" Experiment 2: "The results of Study 2 indicate that a single prior exposure is sufficient to increase perceived accuracy for both fake and real news. This occurs even (a) when fake news is labeled as Disputed by 3rd Party Fact-Checkers during the familiarization stage (i.e., during encoding at first exposure), (b) among fake (and real) news headlines that are inconsistent with one’s political ideology, and (c) when isolating the analysis to news headlines that participants were not consciously aware of having seen in the familiarization stage" Experiment 3: "The effect of repetition on perceived accuracy persisted after a week and increased with an additional repetition. This suggests that fake-news credulity compounds with increasing exposures and maintains over time" | "Our repetition effect was likely driven, at least in part, by automatic (as opposed to strategic) memory retrieval… More broadly, these effects correspond with prior work demonstrating the power of fluency to influence a variety of judgments" | "The 2016 U.S. presidential election brought considerable attention to the phenomenon of “fake news”: entirely fabricated and often partisan content that is presented as factual. Here we demonstrate one mechanism that contributes to the believability of fake news: fluency via prior exposure. Using actual fake-news headlines presented as they were seen on Facebook, we show that even a single exposure increases subsequent perceptions of accuracy, both within the same session and after a week. Moreover, this “illusory truth effect” for fake-news headlines occurs despite a low level of overall believability and even when the stories are labeled as contested by fact checkers or are inconsistent with the reader’s political ideology. These results suggest that social media platforms help to incubate belief in blatantly false news stories and that tagging such stories as disputed is not an effective solution to this problem. It is interesting, however, that we also found that prior exposure does not impact entirely implausible statements (e.g., “The earth is a perfect square”). These observations indicate that although extreme implausibility is a boundary condition of the illusory truth effect, only a small degree of potential plausibility is sufficient for repetition to increase perceived accuracy. As a consequence, the scope and impact of repetition on beliefs is greater than has been previously assumed." | Suggests that labeling fake news is ineffective at reducing perceived accuracy — a single prior exposure increases perceived accuracy of fake news regardless of the label | http://dx.doi.org/10.1037/xge0000465 | Unclear | Simulated social media | Beliefs | |||||||||||
36 | Shifting attention to accuracy can reduce misinformation online | Gordon Pennycook, Ziv Epstein, Mohsn Mosleh, Antonio Arechar, Dean Eckles, David Rand | 2020 | Experiment 1: Presentation of a headline, lede sentence, and image for 36 actual news stories from social media (half true and half false) Experiment 2: Presentation of 24 news headlines Experiment 3: Twitter message asking users to rate the accuracy of one non-political headline | Experiment 1: rated veracity of a headline or willingness to share Experiment 2: Participant willingness to share a headline on Facebook Experiment 3: Links shared by Twitter users | Experiment 1: Judged headline's veracity or indicated whether they would share each headline online Experiment 2: In treatment condition, participants rated the accuracy of one non-partisan headline at the beginning of the study Experiment 3: Randomized date on which users received treatment message | Experiment 1 and 2: American participants from Amazon's Mechanical Turk Experiment 3: "5,379 Twitter users who had previously shared links to websites that publish misleading or false news content" | Content labeling | Experiment 1: "In the Accuracy condition, true headlines received much higher accuracy ratings than false headlines. Conversely, politically concordant headlines were only rated as slightly more accurate than politically discordant headlines… Turning to the Sharing condition, the pattern was the opposite of what we saw for accuracy: participants were only slightly more likely to consider sharing true headlines than false headlines, but were much more likely to consider sharing politically concordant headlines than politically discordant headlines" Experiment 2: "Participants in the Treatment were significantly less likely to consider sharing false headlines compared to the Control, but equally likely to consider sharing true headlines" Experiment 3: "We find clear evidence that the single accuracy message made users more discerning in their subsequent sharing decisions. Relative to baseline, the accuracy message increased the average quality of the news sources shared, p=.009, and the total quality of shared sources summed over all posts, p=.011. This translates into increases of 4.8% and 9.0% respectively when estimating the treatment effect for user-days on which tweets would occur in treatment" | "According to the inattention-based account, the Treatment influences sharing by drawing participants’ attention to the concept of accuracy – and because most participants have a preference to avoid sharing inaccurate content, this attentional shift increases the likelihood that participants will discern between true and false content when deciding what to share" | "Why do people share false and misleading news content on social media, and what can be done about it? In a first survey experiment (N=1,015), we demonstrate a disconnect between accuracy judgments and sharing intentions: Even though true headlines are rated as much more accurate than false headlines, headline veracity has little impact on sharing. Although this may seem to indicate that people share inaccurate content because, for example, they care more about furthering their political agenda than they care about truth, we propose an alternative attentional account: Most people do not want to spread misinformation, but the social media context focuses their attention on factors other than truth and accuracy. Indeed, when directly asked, most participants say it isimportant to only share news that is accurate. Accordingly, across four survey experiments (total N=3,485) and a digital field experiment on Twitter in which we messaged users who had previously shared news from websites known for publishing misleading content (N=5,379), we find that subtly inducing people to think about accuracy increases the quality of the news they subsequently share. These results, together with additional computational analyses, challenge the narrative that people no longer care about accuracy. Instead, the findings support our inattention-based account wherein people fail to implement their preference for accuracy due to attentional constraints – particularly on social media. Furthermore, our research provides evidence for scalable anti-misinformation interventions that are easily implementable by social media platforms." | This is a peer-reviewed working paper that has not yet been published in a journal | https://doi.org/10.31234/osf.io/3n9u8 | Unclear | Experimental | Intended action/behavior | Unclear | Yes | Yes | |||||||||
37 | Trust It or Not: Effects of Machine-Learning Warnings in Helping Individuals Mitigate Misinformation | Haeseung Seo, Aiping Xiong, and Dongwon Lee | 2019 | Proceedings of the 10th ACM Conference on Web Science | Participants received warnings about fake news and read news headlines formatted as Facebook posts | Participants made decisions on recognizing, detecting, and sharing fake and real news | Experimental groups: control, two machine-learning warnings (MLG), and one fact-checking warning (FC) | U.S. adults from Amazon's Mechanical Turk | Disinformation disclosure | Content labeling | "Across two experiments, we proposed three machine-learning warnings and evaluated their effects and a fact-checking warning in helping individuals mitigate fake news. Both decision rates and SDT measures showed the effect of MLG warning in helping participants differentiate fake news from real ones. When no warnings were displayed in Phase 2, although the MLG warning did not impact individuals’ detection decisions, participants increased their uncertainty in sharing fake news but reduced their uncertainty in sharing real news, suggesting a short-term effect of the warning. We obtained that the effect of FC warning increased participants’ correct detection of both fake and real news when the source was included in news headlines but not when sources were excluded. Although the FC warning did not impact individuals’ detection decisions when the source was excluded, they increased participants’ uncertainty in sharing fake news and reduced their uncertainty in sharing real news when the warning was not displayed in Phase 2, suggesting a short-term effect." | "With the FC warning, participants not only increased the correct detection of fake news to which the warning was attached but also the correct detection of real news, suggesting that participants probably relied on the presence and absence of the warning to make the detection decision." | "Despite increased interests in the study of fake news, how to aid users' decision in handling suspicious or false information has not been well understood. To obtain a better understanding on the impact of warnings on individuals' fake news decisions, we conducted two online experiments, evaluating the effect of three warnings (i.e., one Fact-Checking and two Machine-Learning based) against a control condition, respectively. Each experiment consisted of three phases examining participants' recognition, detection, and sharing of fake news, respectively. In Experiment 1, relative to the control condition, participants' detection of both fake and real news was better when the Fact-Checking warning but not the two Machine-Learning warnings were presented with fake news. Post-session questionnaire results revealed that participants showed more trust for the Fact-Checking warning. In Experiment 2, we proposed a Machine-Learning-Graph warning that contains the detailed results of machine-learning based detection and removed the source within each news headline to test its impact on individuals' fake news detection with warnings. We did not replicate the effect of the Fact-Checking warning obtained in Experiment 1, but the Machine-Learning-Graph warning increased participants' sensitivity in differentiating fake news from real ones. Although the best performance was obtained with the Machine-Learning- Graph warning, participants trusted it less than the Fact-Checking warning. Therefore, our study results indicate that a transparent machine learning warning is critical to improving individuals' fake news detection but not necessarily increase their trust on the model." | Machine learning fake news warning may be effective, but "participants showed limited trust on it" | https://doi.org/10.1145/3292522.3326012 | Unclear | Simulated social media | Intended action/behavior | Yes | |||||||||
38 | Sources of the Continued Influence Effect: When Misinformation in Memory Affects Later Inferences | Holly N. Johnson and Colleen Seifert | 1994 | Journal of Experimental Psychology: Learning, Memory, and Cognition | Presentation of misinformation reports and a correction received at varying times after the misinformation Experiment 3: Proivided a "plausible causal alternative" | Participants completed a questionnaire about understanding of reports | Experiment 1: Correction received immediately or later Experiment 2: Manipulated "the availability of the misinformation within the comprehension context" | University of Michigan undergraduates | Disinformation disclosure | Content labeling | "Misinformation can still influence inferences one generates after a correction has occurred; however, providing an alternative that replaces the causal structure it affords can reduce the effects of misinformation" | "One may involve misinformation in inferences when it is the only material that affords causal structure to the account, and without it, one could not construct as good a representation of the event. This is consistent with other proposals that subjects use discredited information when they do not have other 'relevant' information available... In summary, our results support the hypothesis that continued influence from misinformation depends on its propensity for becoming involved in causal inferences and does not depend on simple heightened availability in memory" | "Several lines of research have found that information previously encoded into memory can influence inferences and judgments, even when more recent information discredits it. Previous theories have attributed this to difficulties in editing memory: failing to successfully trace out and alter inferences or explanations generated before a correction. However, in Exps 1A and 1B, Ss who had received an immediate correction made as many inferences based on misinformation as Ss who had received the correction later in the account (despite presumably having made more inferences requiring editing). In a 2nd experiment, the availability of the misinformation within the comprehension context was tested. Results showed that Ss continued to make inferences involving discredited information when it afforded causal structure, but not when only incidentally mentioned or primed during an intervening task. Exps 3A and 3B found that providing a plausible causal alternative, rather than simply negating misinformation, mitigated 1 effect. The findings suggest that misinformation can still influence inferences one generates after a correction has occurred; however, providing an alternative that replaces the causal structure it affords can reduce the effects of misinformation." | https://doi.org/10.1037/0278-7393.20.6.1420 | No | Experimental | Beliefs | Yes | ||||||||||
39 | How Warnings about False Claims Become Recommendations | Ian Skurnik, Carolyn Yoon, Denise C. Park, Norbert Schwarz | 2005 | Journal of Consumer Research | Exposure to claims about consumer products explicitly labeled as "false" or "true" and repeated varying numbers of times | Participants recalled the truth value of the claims | Exposed to claims once or three times; recalled truth of claims after 30 minutes or 3 days Experiment 2 also placed a truth disclosure with every presentation of a claim or only with the last presentation | "Younger adults at a large university" and "community-dwelling older adults" | Content labeling | "Our experiments document a paradoxical effect of warnings: the more often older adults were told that a given claim was false, the more likely they were to accept it as true after several days have passed (experiment 1). Similarly, warning older adults that a previously seen claim of unknown validity is actually false increases acceptance of this claim as true" | "Together, findings from the two studies suggest that people have multiple bases for making constructive judgments about the truth of remembered claims. One basis can be remembered contextual details, such as explicit “true” and “false” designations. A second basis can be partial information about prior exposure, such as experienced familiarity. When incomplete information such as familiarity is the only available cue for judging truth, people generally tend to infer that the information is true" | "Telling people that a consumer claim is false can make them misremember it as true. In two experiments, older adults were especially susceptible to this “illusion of truth” effect. Repeatedly identifying a claim as false helped older adults remember it as false in the short term but paradoxically made them more likely to remember it as true after a 3 day delay. This unintended effect of repetition comes from increased familiarity with the claim itself but decreased recollection of the claim's original context. Findings provide insight into susceptibility over time to memory distortions and exploitation via repetition of claims in media and advertising." | https://doi-org.ezproxy.princeton.edu/10.1086/426605 | No | Experimental | Knowledge | No | |||||||||||
40 | State media warning labels can counteract the effects of foreign misinformation | Jack Nassetta and Kimberly Gross | 2020 | Harvard Kennedy School (HKS) Misinformation Review | Exposure to a YouTube video with or without disclaimers about source | Participant responses to a series of questions about RT, trust in outlets representing mainstream media, etc. | Exposure to a non-political control video, an RT video without a disclaimer, an RT video with the real disclaimer, or the RT video with a custom implementation of the disclaimer superimposed onto the video frame | Amazon Mechnical Turk participants | Content labeling | "YouTube’s state-controlled media labels, which, in the case of RT, state “RT is funded in part or in whole by the Russian government,” have the purpose of informing the user of a media channel’s government connection. Our findings indicate that these labels can be successful, but success depends on their implementation. In our April 2020 experiment, the YouTube label was in a light grey box below the video. In that case, there was no increase in knowledge that RT was state funded between those who saw the real disclaimer and those who saw the video without it. By July 2020, when the second experiment was conducted, YouTube had tweaked the disclaimer to be in a blue, instead of a grey box. This change alone appears to have made the label more effective." | Suggests that the prominence of state media labels mitigates their effectiveness | "In order to test the efficacy of YouTube’s disclaimers, we ran two experiments presenting participants with one of four videos: A non-political control, an RT video without a disclaimer, an RT video with the real disclaimer, or the RT video with a custom implementation of the disclaimer superimposed onto the video frame. The first study, conducted in April 2020 (n = 580) used an RT video containing misinformation about Russian interference in the 2016 election. The second conducted in July 2020 (n = 1,275) used an RT video containing misinformation about Russian interference in the 2020 election. Our results show that misinformation in RT videos has some ability to influence the opinions and perceptions of viewers. Further, we find YouTube’s funding labels have the ability to mitigate the effects of misinformation, but only when they are noticed, and the information absorbed by the participants. The findings suggest that platforms should focus on providing increased transparency to users where misinformation is being spread. If users are informed, they can overcome the potential effects of misinformation. At the same time, our findings suggest platforms need to be intentional in how warning labels are implemented to avoid subtlety that may cause users to miss them." | https://doi.org/10.37016/mr-2020-45 | Unclear | Real-world social media | Beliefs | YouTube | Yes | ||||||||||
41 | What Debunking of Misinformation Does and Doesn't | Jeong-woo Jang, Eun-Ju Lee, and Soo Yun Shin | 2019 | Cyberpsychology, Behavior, and Social Networking | Viewed a mock-up Facebook page with a news article about the "pet tax" and varying forms of corrections | Measured pre- and post-test attitude toward the source, agreement with news position and "perceived credibility of social media as a news platform" | Debunking (present vs. absent), News Position (pro vs. con) | 960 Facebook users from South Korea | Disinformation disclosure | "Overall, the results showed that when the falsehood of information was exposed, participants became less favorable toward the immediate source who shared the misinformation (H1a), but their initial source attitude also moderated their reactions by inducing different attribution processes... Consistent with the continued influence of misinformation account, participants' agreement with the news position was not attenuated by the explicit post hoc correction." | "Our findings suggest that news readers' predisposition toward the source may contribute to the difficulty of correcting misinformation. Fake news often spreads over social media through interpersonal contacts, rather than formal news agencies. As such, familiarity and liking of an interpersonal news source can make shared information seem more credible14 and discourage people from independently validating received information. Moreover, even when the information was verified to be false, instead of revisiting their positive evaluation of the source, participants blamed the media platform instead, while strengthening their initial attitudes toward the source." | "A web-based experiment (n = 960) examined how debunking of publicly shared news on social media affects viewers' attitudes toward the source who shared the fake news, their agreement with the news position, and perceived credibility of social media as a news platform. Exposure to debunking information did not lower participants' agreement with the news position, but led them to derogate (1) the source who shared the misinformation and (2) social media as a news platform. However, participants who initially favored the source were less likely to attribute the sharing of fake news to the source's dispositions, rather than situational factors, thereby maintaining their positive attitudes toward the source." | https://doi-org.ezproxy.princeton.edu/10.1089/cyber.2018.0608 | No | Simulated social media | Beliefs | No | |||||||||||
42 | Evaluation of a template for countering misinformation—Real-world Autism treatment myth debunking | Jessica Paynter, Sarah Luskin-Saxby, Deb Keen, Kathryn Fordyce, Grace Frost, Christine Imms, Scott Miller, David Trembath, Madonna Tucker, and Ullrich Ecker | 2019 | PLoS One | Exposure to various kinds of training and support for non-empirically-supported autism treatments | "The selected ratings were (1) a rating of the evidence base for a given treatment (rated on a scale from 0 [ineffective] to 4 [established]); (2) a rating of intended future use of a given treatment (from 0 [never] to 4 [frequently]); and (3) a rating of the likelihood of recommending a given treatment to parents (from -3 [will not recommend with high confidence] to +3 [will recommend with high confidence])" | "Between-subjects factor condition (optimized debunking vs. control) and the within-subjects factor time (time 1, pre-intervention; time 2, post-intervention; time 3, delayed follow-up)" | "Participants were recruited from four different autism early intervention centers from the Autism Specific Early Learning and Care Centres (ASELCC) across four organisations, and four states of Australia" | Disinformation disclosure | "We demonstrated that an optimized-debunking intervention was more effective than a treatment-as-usual intervention at reducing support for non-empirically-supported treatments." | "Our research confirmed the positive effects of weight-of-evidence information and graphical representations, while avoiding backfire effects potentially arising from emotive or confrontational debunkings" | "Misinformation poses significant challenges to evidence-based practice. In the public health domain specifically, treatment misinformation can lead to opportunity costs or direct harm. Alas, attempts to debunk misinformation have proven sub-optimal, and have even been shown to “backfire”, including increasing misperceptions. Thus, optimized debunking strategies have been developed to more effectively combat misinformation. The aim of this study was to test these strategies in a real-world setting, targeting misinformation about autism interventions. In the context of professional development training, we randomly assigned participants to an “optimized-debunking” or a “treatment-as-usual” training condition and compared support for non-empirically-supported treatments before, after, and six weeks following completion of online training. Results demonstrated greater benefits of optimized debunking immediately after training; thus, the implemented strategies can serve as a general and flexible debunking template. However, the effect was not sustained at follow-up, highlighting the need for further research into strategies for sustained change." | https://doi-org.ezproxy.princeton.edu/10.1371/journal.pone.0210746 | No | Experimental | Intended action/behavior | Yes | |||||||||||
43 | Neutralizing misinformation through inoculation: Exposing misleading argumentation techniques reduces their influence | John Cook, Stephan Lewandowsky, and Ullrich K. H. Ecker | 2017 | PLoS One | Participants were shown information about climate change or media coverage and a news article that "presented mainstream scientific views alongside contrarian scientists' views" | Measured participant views on climate change, contrarian scientists, etc. | Varied type of information read prior to misinformation (the inoculation) | U.S. representative sample recruited through Qualtrics | Disinformation literacy | "Although Experiments 1 and 2 employed different styles of misinformation, both found that inoculation neutralized the negative influence of misinformation on perceived consensus. Experiment 2 also showed that inoculation neutralizes the polarizing influence of misinformation across acceptance of AGW, perceived consensus, and policy support." | "Given the difficulties associated with correcting misinformation once it has been processed [10], an alternative approach is to neutralize potential misinformation before it is encoded, a technique colloquially known as “prebunking”… These studies indicate that pre-existing attitudes influence how people respond to new information (or misinformation). Similarly, inoculation theory proposes that people can be “inoculated” against misinformation by being exposed to a refuted version of the message beforehand" | "Misinformation can undermine a well-functioning democracy. For example, public misconceptions about climate change can lead to lowered acceptance of the reality of climate change and lowered support for mitigation policies. This study experimentally explored the impact of misinformation about climate change and tested several pre-emptive interventions designed to reduce the influence of misinformation. We found that false-balance media coverage (giving contrarian views equal voice with climate scientists) lowered perceived consensus overall, although the effect was greater among free-market supporters. Likewise, misinformation that confuses people about the level of scientific agreement regarding anthropogenic global warming (AGW) had a polarizing effect, with free-market supporters reducing their acceptance of AGW and those with low free-market support increasing their acceptance of AGW. However, we found that inoculating messages that (1) explain the flawed argumentation technique used in the misinformation or that (2) highlight the scientific consensus on climate change were effective in neutralizing those adverse effects of misinformation. We recommend that climate communication messages should take into account ways in which scientific content can be distorted, and include pre-emptive inoculation messages." | https://doi.org/10.1371/journal.pone.0175799 | Yes | Experimental | Beliefs | ||||||||||||
44 | The fake news game: actively inoculating against the risk of misinformation | Jon Roozenbeek and Sander Van der Linden | 2019 | Journal of Risk Research | Participants play a fake news game to inoculate against fake news | Participants evaluated fake news articles, assessing familiarity, persuasion, agreement, and judgement | Groups assigned to one of four characters representing "common ways in which information is presented in a misleading manner," namely the denier, the alarmist, the clickbait monger, and the conspiracy theorist | Public high school students in the Netherlands | Disinformation literacy | "Our experimental pilot study found that the process of active inoculation induced by the fake news game reduced the perceived reliability and persuasiveness of fake news articles about the refugee crisis." | "First and foremost, because traditional inoculation messages are often passively delivered in the refutational-same format, i.e. participants are inoculated against the same information to which they will later be exposed, whereas this study specifically tested a refutational-different format, where the article students are ‘trained’ on is related but not the same as the ‘validation’ article. The goal of the game was to train participants to recognize fake news tactics on a more general level by actively rehearsing different roles/sides of the argument, so that the inoculation is more likely to offer broad resistance." | "The rapid spread of online misinformation poses an increasing risk to societies worldwide. To help counter this, we developed a ‘fake news game’ in which participants are actively tasked with creating a news article about a strongly politicized issue (the European refugee crisis) using misleading tactics, from the perspective of different types of fake news producers. To pilot test the efficacy of the game, we conducted a randomized field study (N = 95) in a public high school setting. Results provide some preliminary evidence that playing the fake news game reduced the perceived reliability and persuasiveness of fake news articles. Overall, these findings suggest that educational games may be a promising vehicle to inoculate the public against fake news." | https://doi.org/10.1080/13669877.2018.1443491 | No | Experimental | Beliefs | ||||||||||||
45 | Real Solutions for Fake News? Measuring the Efectiveness of General Warnings and Fact‑Check Tags in Reducing Belief in False Stories on Social Media | Katherine Clayton, Spencer Blair, Jonathan A. Busam, Samuel Forstner, John Glance, Guy Green, Anna Kawata, Akhila Kovvuri, Jonathan Martin, Evan Morgan, Morgan Sandhu, Rachel Sang, Rachel Scholz-Bright, Austin T. Welch, Andrew G. Wolff, Amanda Zhou & Brendan Nyhan | 2020 | Political Behavior | Participants viewed false headlines accompanied by varying forms of fact-checking tags | Evaluated accuracy of a headline and self-reported likeliness that participant would "like" and share the story on Facebook | Control group or one of six experimental conditions: no fact-cheking tag, tag of "disputed," or tag of "rated false; general warning (yes or no) | 2994 participants recruited from Amazon Mechanical Turk | Content labeling | "Both “Disputed” and “Rated false” tags modestly reduce belief in false news… Encouragingly, we fnd no consistent evidence that the effects of these tags varies by the political congeniality of the headlines or that exposure to the tags increases the perceived accuracy of unlabeled false headlines... By contrast, though general warnings about false news also appear to decrease belief in false headlines, the effect of a general warning is small compared to either type of tag. Moreover, general warnings also reduce belief in real news and do not enhance the efects of the “Rated false” and “Disputed” tags, suggesting that they are a less efective approach" | "One potential explanation is that warnings about false news prime people to think about misleading information online, making them less likely to trust any articles they see on social media. Another possible interpretation is that we are observing a “tainted truth” efect in the context of political misinformation. In social cognition research, such an efect occurs when eyewitnesses who are warned about the infuence of misinformation overcorrect for this threat and identify fewer true items than eyewitnesses who are not warned" | "Social media has increasingly enabled “fake news” to circulate widely, most notably during the 2016 U.S. presidential campaign. These intentionally false or misleading stories threaten the democratic goal of a well-informed electorate. This study evaluates the efectiveness of strategies that could be used by Facebook and other social media to counter false stories. Results from a pre-registered experiment indicate that false headlines are perceived as less accurate when people receive a general warning about misleading information on social media or when specifc headlines are accompanied by a “Disputed” or “Rated false” tag. Though the magnitudes of these efects are relatively modest, they generally do not vary by whether headlines were congenial to respondents’ political views. In addition, we fnd that adding a “Rated false” tag to an article headline lowers its perceived accuracy more than adding a “Disputed” tag (Facebook’s original approach) relative to a control condition. Finally, though exposure to the “Disputed” or “Rated false” tags did not afect the perceived accuracy of unlabeled false or true headlines, exposure to a general warning decreased belief in the accuracy of true headlines, suggesting the need for further research into how to most efectively counter false news without distorting belief in true information." | https://link.springer.com/article/10.1007/s11109-019-09533-0 | Unclear | Experimental | Intended action/behavior | Yes | |||||||||||
46 | In Related News, That was Wrong: The Correction of Misinformation Through Related Stories Functionality in Social Media | Leticia Bode and Emily Vraga | 2015 | Journal of Communication | Participants view a Facebook post containing misinformation and related stories which confirm, correct, or confirm and correct the misinformation. | Evaluation of viewed news story | Related story position: confirm misperception, debunk misperception, mixed to misperceptions, unrelated to misperception Controversial issue: GMOs cause illness or vaccines cause autism | Study 1: mid-Atlantic university students Study 2: Mechanical Turk participants | Disinformation disclosure | "Attitude change related to GMOs can be achieved with regard to misperceptions by virtue of exposure to corrective information within social media" | "In this study, although participants successfully denigrated the value of the related stories when they disagreed with their beliefs, those holding misperceptions on the GMO issue still adjusted their attitudes after reading debunking news stories. It may be that although the stories were seen as less credible, they maintained sufficient credibility to impact attitudes. A second possibility may be that participants reached their affective “tipping point” of incongruence, where motivated reasoning still operates in evaluating stories but does not prevent updating of attitudes" | "Research on social media and research on correcting misinformation are both growing areas in communication, but for the most part they have not found common ground. This study seeks to bridge these two areas, considering the role that social media may play in correcting misinformation. To do so, we test a new function of Facebook, which provides related links when people click on a link within Facebook. We show users a post containing misinformation, and then manipulate the related stories to either confirm, correct, or both confirm and correct the misinformation. Findings suggest that when related stories correct a post that includes misinformation, misperceptions are significantly reduced." | https://doi.org/10.1111/jcom.12166 | Unclear | Simulated social media | Beliefs | Yes | |||||||||||
47 | See Something, Say Something: Correction of Global Health Misinformation on Social Media | Leticia Bode and Emily Vraga | 2018 | Health Communication | Exposure to a simulated Facebook feed with misinformation about the Zika virus with varying forms of corrections | Participants responded to questions measuring belief in misinformation about the Zika outbreak | Participants in the control saw three pages of posts about social interactions and news; both treatment groups saw an additional post with a false news story about the Zika outbreak; one group saw an algorithmic correction (stories from Snopes.com and the CDC generated by the Facebook algorithm); one group saw a social correction where "two individual commenters discredit the information and provide links to the same two debunking news stories" | Participants from a Mid-Atlantic university | Disinformation disclosure | Content labeling | "This study examines how everyday users—in addition to platform-generated algorithms—can reduce health misperceptions on social media. The results suggest that social comments are as effective as related stories produced by the platform at health misinformation correction, at least for breaking health issues for which public beliefs likely remain malleable." | "Future research should examine the mechanisms by which social versus algorithmic correction occur. This study suggests that the Facebook “related stories” algorithm, which necessarily provides concrete sources to support their claims, was seen as equally credible and trustworthy to social commenters who use the same sources to explicitly debunk the health misinformation provided. Previous research argued that the seemingly unbiased nature of an algorithm may contribute to its effectiveness (Bode & Vraga, 2015), limiting the motivated reasoning that might otherwise forestall successful persuasion efforts. In this case, the social corrections that individuals were exposed to were not known members of someone’s social networks, but anonymous others." | "Social media are often criticized for being a conduit for misinformation on global health issues, but may also serve as a corrective to false information. To investigate this possibility, an experiment was conducted exposing users to a simulated Facebook News Feed featuring misinformation and different correction mechanisms (one in which news stories featuring correct information were produced by an algorithm and another where the corrective news stories were posted by other Facebook users) about the Zika virus, a current global health threat. Results show that algorithmic and social corrections are equally effective in limiting misperceptions, and correction occurs for both high and low conspiracy belief individuals. Recommendations for social media campaigns to correct global health misinformation, including encouraging users to refute false or misleading health information, and providing them appropriate sources to accompany their refutation, are discussed." | Compares the Facebook algorithm for correcting misinformation to social comments and suggests "encouraging users to refute false or misleading health information" with evidence, particularly given the difficulty of encouraging social media companies to impost algorithmic corrections | https://doi.org/10.1080/10410236.2017.1331312 | No | Simulated social media | Knowledge | Yes | |||||||||
48 | Do the right thing: tone may not affect correction of misinformation on social media | Leticia Bode, Emily K. Vraga, and Melissa Tully | 2020 | Harvard Kennedy School (HKS) Misinformation Review | Participants shown a simulated Twitter feed with misinformation and varying forms of correction | Participants asked about beliefs about safety and nutrition of raw milk | Some participants received a correction; varied correction's tone (uncivil, affirmational, or neutral) | Participants from Amazon's Mechanical Turk | Disinformation disclosure | "Participants who saw any of the corrections expressed lower levels of misperceptions that raw milk is more nutritious than pasteurized milk as compared to people who saw the misinformation without any correction. Moreover, there are no differences in misperceptions about raw milk’s nutritional value among people who saw a neutral (factual-only) correction, an uncivil correction, or an affirmative correction." | Does not explicitly state mechanism, but says "research increasingly shows that correcting misinformation is effective at getting people to update their beliefs – when you give them new facts, they tend to reduce their beliefs in misinformation" | "An experiment conducted with 610 participants suggests that corrections to misinformation – pointing out information that is wrong or misleading and offering credible information in its place – on social media reduce misperceptions regardless of the correction’s tone (uncivil, affirmational, or neutral). There is also an opportunity to correct secondary but related misperceptions (dealing with the same topic but with a different specific fact) when responding to misinformation on social media. Our findings emphasize that correction on social media could operate as part of a broader strategy to reduce beliefs in misinformation, and users should be encouraged to bring additional relevant information into the conversation, using whatever tone feels most comfortable for them." | https://doi.org/10.37016/mr-2020-026 | Unclear | Simulated social media | Beliefs | Yes | |||||||||||
49 | A preregistered replication of “Inoculating the public against misinformation about climate change” | Matt Williams and Christina Bond | 2020 | Journal of Environmental Psychology | Exposed to information about the scientific consensus surrounding climate change | Post-treatment, respondents rate perception of scientific consensus surrounding climate change | Replicated Van der Linden et al. (2017), row 61 | Participants from Amazon's Mechanical Turk | Disinformation disclosure | "We found that providing information about the scientific consensus on climate change increased perceptions of scientific consensus, as did an inoculation provided prior to provision of misinformation. However, we were unable to replicate their finding that an inoculation counteracted the effect of misinformation to a greater extent than simply providing information about scientific consensus—although this may have been due to a ceiling effect in our study" | "There are several potential explanations for the relatively small effect of the counter-message in our study, and by extension the high level of perceived scientific consensus observed in condition 4 (consensus-treatment + counter-message). One possibility is that not all participants read the counter-message properly, thus reducing its effect on perceived scientific consensus, and reducing our capacity to determine whether the inoculation buffered the effect of the counter-message to a greater extent than the consensus-treatment did. This explanation is plausible because the counter-message (an image of the Oregon petition) was presented in the form of an image file (as was the case in the original study), and the text on this image could have been difficult to read for some participants—especially those viewing the survey on mobile phones.' | "A strong consensus exists amongst climate scientists that the Earth is warming, and that this warming is due to anthropogenic greenhouse gas emissions. Members of the public are nevertheless frequently exposed to misinformation about climate change (and the degree of scientific consensus on climate change). One strategy to combat such misinformation is attitudinal “inoculation”—exposing individuals to a weakened item of misinformation along with a pre-emptive refutation. Van der Linden, Leiserowitz, Rosenthal, and Maibach (2017) reported that exposing participants to misinformation reduced the extent to which the participants perceived there to be a scientific consensus about climate change, but that the effect of this misinformation was successfully countered by an attitudinal inoculation. In this study, we report a preregistered replication of van der Linden et al. (2017). Our replication study used a mixed between-within design, with data collected via Mechanical Turk (N = 792). We were able to replicate some (but not all) of van der Linden et al.’s findings. Specifically, we found that providing information about the scientific consensus on climate change increased perceptions of scientific consensus, as did an inoculation provided prior to provision of misinformation. However, we were unable to replicate their finding that an inoculation counteracted the effect of misinformation to a greater extent than simply providing information about scientific consensus—although this may have been due to a ceiling effect in our study. Our preregistration and data are available at" | Replicates Van der Linden et al. (2017) | https://doi-org.ezproxy.princeton.edu/10.1016/j.jenvp.2020.101456 | Unclear | Experimental | Beliefs | Yes | ||||||||||
50 | The Role of Source, Headline and Expressive Responding in Political News Evaluation | Maurice Jakesch, Moran Koren, Anna Evtushenko, Mor Naaman | 2018 | Presentation of news claims randomly associated with news sources, with some participants receiving financial compensation for correct answers | Rated truthfulness of each headline | News headlines randomly assigned to news sources; treatment group received bonus payments depending on how many headlines were accurately determined to be true or false | Participants from Amazon's Mechanical Turk | Other | "Our results show that, when evaluating trust in partisan claims, the “brand” of the publisher plays a minor role: In an experimentally controlled setting where the New York Times and Fox News publish identical claims, left- and right-leaning participants evaluate the sources similarly. In contrast, the political valence of the claims significantly influences who believes it and who does not: both Democrats and Republicans found the headlines that supported their own view significantly more credible... When incentivized to answer correctly, right-leaning readers reported higher credence to pro-Democrat claims and lower credence to pro-Republican claims." | "Right-leaning respondents held beliefs closer to the ones reported by left-leaning respondents in the first place but without incentives, “partisan cheerleading” was more attractive than providing accurate responses" | "Studies have observed that readers are more likely to trust news sources that align with their own political leanings. We ask: is the higher reported trust in politically aligned news sources due to perceived institutional trustworthiness or does it merely reflect a preference for the political claims aligned sources publish? Furthermore, do respondents report their actual beliefs about news or do they choose to express their political commitments instead? We conducted a US-based experiment (N=400) using random association of news claims to news sources as well as financial incentives to robustly identify the main drivers of trust in news and to evaluate response bias. We observe a comparatively weak effect of source on news evaluation and find that response differences are largely due to the alignment of the respondents' politics and the news claim. We also find significant evidence for expressive responding, in particular among right-leaning participants." | This manuscript is an SSRN scholarly paper not yet published in a journal | https://dx.doi.org/10.2139/ssrn.3306403 | Unclear | Experimental | Beliefs | ||||||||||||
51 | Good News about Bad News: Gamified Inoculation Boosts Confidence and Cognitive Immunity Against Fake News | Melisa Basol, Jon Roozenbeek, and Sander van der Linden | 2020 | Journal of Cognition | Playing the Bad News game covering six misinformation techniques | "An assessment of the reliability of 18 misinformation headlines in the form of Twitter posts" | Played Bad News game vs. control (played Tetris); completed assessment before or after playing game | 197 U.K. participasnts recruited through Prolific Academic | Disinformation literacy | "This study successfully demonstrated the efficacy of a “broad-spectrum” inoculation against misinformation in the form of an online fake news game" | "Lastly, the current study also significantly advances our understanding of the theoretical mechanisms on which the intervention acts. For example, while inoculated individuals improved in their reliability assessments of the fake news items, the average confidence they expressed in their judgements also increased significantly and substantially. Importantly, the intervention only significantly increased confidence amongst those who updated their judgments in the right direction (i.e. correctly judging manipulative items to be less reliable). These findings are supported by previous literature demonstrating the certainty-bolstering effects of inoculation treatments (Tormala & Petty, 2004) and may suggest that confidence plays a key role in both prophylactic and therapeutic inoculation approaches" | "Recent research has explored the possibility of building attitudinal resistance against online misinformation through psychological inoculation. The inoculation metaphor relies on a medical analogy: by pre-emptively exposing people to weakened doses of misinformation cognitive immunity can be conferred. A recent example is the Bad News game, an online fake news game in which players learn about six common misinformation techniques. We present a replication and extension into the effectiveness of Bad News as an anti-misinformation intervention. We address three shortcomings identified in the original study: the lack of a control group, the relatively low number of test items, and the absence of attitudinal certainty measurements. Using a 2 (treatment vs. control) × 2 (pre vs. post) mixed design (N = 196) we measure participants’ ability to spot misinformation techniques in 18 fake headlines before and after playing Bad News. We find that playing Bad News significantly improves people’s ability to spot misinformation techniques compared to a gamified control group, and crucially, also increases people’s level of confidence in their own judgments. Importantly, this confidence boost only occurred for those who updated their reliability assessments in the correct direction. This study offers further evidence for the effectiveness of psychological inoculation against not only specific instances of fake news, but the very strategies used in its production. Implications are discussed for inoculation theory and cognitive science research on fake news." | https://dx.doi.org/10.5334%2Fjoc.91 | No | Experimental | Beliefs | ||||||||||||
52 | Designing and Testing News Literacy Messages for Social Media | Melissa Tully, Emily K. Vraga, and Leticia Bode | 2020 | Mass Communication and Society | On a fictitious Twitter feed, some participants exposed to News Literacy tweets and all participants exposed to misinformation about one of two health issues | Measured participants' perception of tweet credibility, self-perceived media literacy, value for media literacy, and GMO misperceptions | "3 (promoted tweet: control, your job, citizen’s job) × 2 (misinformation: control, GMO misinformation)" | Participants from Amazon's Mechanical Turk | Disinformation literacy | "However, our study suggests that it is difficult to alter misinformation perceptions and bolster NL beliefs with a single message. Although the NL tweets in Study 1 boosted self-perceived media literacy as expected, they did not increase skepticism of misinformation or boost perceptions of the societal value of media literacy. In contrast, the NL tweet in Study 2 increased skepticism of misinformation but did not boost SPML or VML. In short, the NL tweets were somewhat successful in affecting credibility perceptions and NL beliefs." | "Ideally, NL messages on sites like Twitter would encourage people to be more critical consumers of news and information—encouraging skepticism toward low-quality or false information and bolstering perceptions of high-quality information. Important to note, these kinds of interventions are proactive, rather than reactive, and have been shown to be effective at addressing misinformation and promoting critical engagement with news and information online." | "As concerns grow about the spread of misinformation through social media, scholars have called for improving the public’s media literacy as a potential solution. This study examines the effectiveness of deploying news literacy (NL) messages on social media by testing whether NL tweets are able to affect perceptions of information credibility and NL beliefs. Using two experiments, this study tests NL tweets designed to (a) mitigate the impact of exposure to misinformation about two health issues (genetically modified foods and the flu vaccine) and (b) boost people’s perceptions of their own media literacy and media literacy’s value to society broadly. Findings suggest that NL messages are able to alter misinformation perceptions and NL beliefs, but not with a single message, suggesting the need to develop tailored and targeted NL campaigns that feature multiple messages and calls to action." | https://doi.org/10.1080/15205436.2019.1604970 | Unclear | Simulated social media | Beliefs | ||||||||||||
53 | Mobilizing Users: Does Exposure to Misinformation and Its Correction Affect Users’ Responses to a Health Misinformation Post? | Melissa Tully, Leticia Bode, and Emily K. Vraga | 2020 | Social Media + Society | Presentation of a page of Twitter posts with five neutral posts and one manipulated post about unpasteurized milk; varying forms of corrections | Participants rated likelihood of responding to the raw milk post and explained what they would say if they did reply | On fake post about raw milk, some participants saw no response, some saw responses debunking misinformation with a link to the CDC, and some replies reinforced the CDC correction | Amazon Mechnical Turk participants | Disinformation disclosure | "This study examines whether social media users say they would respond to misinformation about raw milk on Twitter and the content of these hypothetical replies. Results suggest that not only do people say they are relatively unlikely to respond, they often do not provide substantive comments even when prompted. Only 25% of our sample reported that they were even ambivalent in their willingness to respond to the misinformation tweet. In addition, exposure to other corrections did not increase the likelihood of replying." | "Notably, this (un)likelihood of responding was not impacted by whether or not participants were exposed to other users’ corrective replies. It appears neither a bystander nor bandwagon effect occurred. If a bandwagon effect occurred, exposure to corrections could have led people to respond upon seeing other corrections. If a bystander effect occurred, whereby people fail to respond because they assume that others’ responses are sufficient and thus absolve them of the need to respond, then exposure to corrections would have produced a lower likelihood of replying. Future research should continue to engage with these questions and seek to explore how and when users will respond to misinformation and what will motivate them to do so. Perhaps, interventions that evoke descriptive and injunctive norms could be designed to motivate user correction and could potentially be more impactful when paired with existing corrections" | "Misinformation spreads on social media when users engage with it, but users can also respond to correct it. Using an experimental design, we examine how exposure to misinformation and correction on Twitter about unpasteurized milk affects participants’ likelihood of responding to the misinformation, and we code open-ended responses to see what participants would say if they did respond. Results suggest that participants are overall unlikely to reply to the misinformation tweet. However, content analysis of hypothetical replies suggests they largely do provide correct information, especially after seeing other corrections. These results suggest that user corrections offer untapped potential in responding to misinformation on social media but effort must be made to consider how users can be mobilized to provide corrections given their general unwillingness to reply." | https://doi-org.ezproxy.princeton.edu/10.1177%2F2056305120978377 | Unclear | Simulated social media | Intended action/behavior | Yes | |||||||||||
54 | Separating truth from lies: comparing the effects of news media literacy interventions and fact-checkers in response to political misinformation in the US and Netherlands | Michael Hameleers | 2020 | Information, Communication & Society | Exposure to "evidence-based or fact-free anti-immigration misinformation, fact-checkers and/or a media literacy intervention" | "The perceived accuracy of the (fictional) news items… issue agreement after exposure to different forms of misinformation and corrective efforts" | Varied the type of argumentation used in misinformation: "evidence-based or factual coverage versus anti-experts and people-centric coverage resonating with opinions and experience;" media literacy intervention; varying the fact check (true, false, or absent) | A representative sample of U.S. and Dutch participants | Disinformation disclosure | Disinformation literacy | "Evidence-based misinformation is seen as more accurate than fact-free misinformation, and the combination of news media literacy interventions and fact-checkers is most effective in lowering issue agreement and perceived accuracy of misinformation across countries" | "Media literacy messages or warnings have weaker effects than corrections... but fact-checkers may have a hard time correcting existing schemata and stored cognitions if the correction is delayed (which is very likely in the digital communication setting characterized by information overload and fragmentation)" | "Although previous research has offered important insights into the consequences of mis- and disinformation and the effectiveness of corrective information, we know markedly less about how different types of corrective information – news media literacy interventions and fact-checkers – can be combined to counter different forms of misinformation. Against this backdrop, this paper reports on experiments in the US and the Netherlands (N = 1,091) that exposed people to evidence-based or fact-free anti-immigration misinformation, fact-checkers and/or a media literacy intervention. The main findings indicate that evidence-based misinformation is seen as more accurate than fact-free misinformation, and the combination of news media literacy interventions and fact-checkers is most effective in lowering issue agreement and perceived accuracy of misinformation across countries. These findings have important implications for journalism practice and policy makers that aim to combat mis- and disinformation." | No clearly defined posited mechanism | https://www.tandfonline.com/doi/full/10.1080/1369118X.2020.1764603 | Yes | Experimental | Beliefs | Yes | |||||||||
55 | Susceptibility to mis- and disinformation and the effectiveness of fact-checkers: Can misinformation be effectively combated? | Michael Hameleers | 2019 | SCM Studies in Communication and Media | Exposure to varying forms of misinformation and fact-checks | Measure of participant issue agreement and misperceptions | "2 (framing of mis-disinformation: evidencebased versus anti-experts and empirical evidence) x 2 (fact-checker: absent versus present)" | Participants recruited from the US and the Netherlands by an international polling firm | Disinformation disclosure | Content labeling | "The main findings indicate that two different formats or frames of misinformation – people-centric/anti-experts and evidence-based information – are equally credible, and yield similar levels of issue agreement… Another key findiing is that fact-checkers can help to correct misinformation, and that they do not result in a backfire effect among participants in the US and Netherlands" | "This implies that mis- or information cannot be made more effective when the communicator aims to feed off the legitimacy of objective journalism by using rhetorical tools of truth and objectivity. On a more pessimistic note, this finding indicates that the objective status of (political) information is up for debate, as facts are not more credible than experiences – which ties in with the notion of post-factual relativism" | "The online dissemination of mis- and disinformation may pose vexing problems on democracy. The factual basis of (political) information may be challenged by opposed partisans or issue publics, and misinformation may impact decision-making as confirmation biases may outweigh accuracy motivations. In this setting, fact-checkers that refute the false claims of misinformation may be regarded as an important tool to combat misinformation. Yet, the effectiveness of corrective information may be contingent upon partisan lenses, or the framing used in misinformation. In this study, the effectiveness of fact-checkers that refute different forms of misinformation on the polarizing issue of crime rates related to anti-immigration framing was assessed in the US and Netherlands. The main findings indicate that exposure to fact-checkers can correct misperceptions on immigration, and lowers the credibility of misinformation. Fact-checkers are more effective in the Netherlands than the US. These findings have important ramifications for understanding citizens’ susceptibility to (partisan) misinformation and rebuttals." | Found fact-checking to be more effective in the Netherlands compared to the US | https://doi.org/10.5771/2192-4007-2019-4-523 | Unclear | Experimental | Knowledge | Yes | |||||||||
56 | Misinformation and Polarization in a High-Choice Media Environment: How Effective Are Political Fact-Checkers? | Michael Hameleers, Toni G. L. A. van der Meer | 2019 | Communication Research | Participants read an article framed as pro- or counter-immigration in experiment 1, pro-climate change or counter-climate change policies in experiment 2; some were exposed to a fact-check | "We asked participants to what extent they agreed with the forwarded problem definition, causal interpretation, and treatment recommendation for the versions of the immigration story they were exposed to." | "2 (attitudinal stance: pro- versus counter-immigration framing of news article) × 3 (exposure to a fact-checker: no fact-checker versus forced exposure to a fact-checker versus free to select or avoid fact-checker) between-subject factorial design" | U.S. participants from Amazon’s Mechanical Turk platform | Disinformation disclosure | Content labeling | "The results of two experimental studies (N = 1,117) consistently indicate that fact-checkers do have the potential to correct attitude-congruent misinformation. This means that partisans used the information provided in the fact-checkers to reconsider their prior attitudes and to moderate their issue positions. Importantly, individuals who already disagreed with the misinformation were not guided by the fact-checkers. In other words, when people already disagreed with the misinformation based on their prior attitude, neither the exposure to misinformation or a fact-checker altered their view as this was already correct (i.e., in line with the fact-checker) in the first place." | "This can be interpreted as evidence for the effectiveness of fact-checkers in discrediting misinformation in a political polarized context. This is in line with recent research that has indicated that corrective attempts presented in fact-checkers can help counter misperceptions, even among those with partisan identities." | "One of the most fundamental changes in today’s political information environment is an increasing lack of communicative truthfulness. To explore this worrisome phenomenon, this study aims to investigate the effects of political misinformation by integrating three theoretical approaches: (1) misinformation, (2) polarization, and (3) selective exposure. In this article, we examine the role of fact-checkers in discrediting polarized misinformation in a fragmented media environment. We rely on two experiments (N = 1,117) in which we vary exposure to attitudinal-congruent or incongruent political news and a follow-up fact-check article debunking the information. Participants were either forced to see or free to select a fact-checker. Results show that fact-checkers can be successful as they (1) lower agreement with attitudinally congruent political misinformation and (2) can overcome political polarization. Moreover, dependent on the issue, fact-checkers are most likely to be selected when they confirm prior attitudes and avoided when they are incongruent, indicating a confirmation bias for selecting corrective information. The freedom to select or avoid fact-checkers does not have an impact on political beliefs." | https://doi.org/10.1177/0093650218819671 | Unclear | Experimental | Beliefs | Yes | ||||||||||
57 | Does truth matter to voters? The effects of correcting political misinformation in an Australian sample | Michael J. Aird, Ullrich K. H. Ecker, Briony Swire, Adam J. Berinsky, and Stephan Lewandowsky | 2018 | Royal Society Open Science | Participants read true/false statements from politicians of various ideologies followed by fact-checks | Participants rated belief in statements, voting intentions, and feelings regarding the politicans | Manipulated ratio of true to false statements; received fact-check before or after reading the statement; statements said to come from different politicians; participants' political orientation | 100 undergraduate students from the University of Western Australia and 355 online participants who wre residents of Australia | Disinformation disclosure | Content labeling | "The main result regarding myth belief and fact belief change was that participants updated their beliefs in accordance with the information received: affirmations increased fact beliefs, and refutations reduced myth beliefs. There was no evidence of motivated reasoning, and in fact, refutations reduced beliefs in myths more strongly if the myths came from a supported politician—that is, when there was congruence between the party affiliation of the participant and the politician (as discussed earlier, this counter-motivational effect was driven mainly by left-wing participants). The main result regarding politician support was that support decreased when mostly false statements were fact-checked (the 4 : 1 condition) but not when participants received an identical number of facts and myths (the 4 : 4 condition)." | "One explanation for these attitude effects is that attitude-incongruent corrections induce motivated reasoning—the processing of new information such that existing attitudes are able to be maintained. Proposed mechanisms of motivated reasoning include generating counterarguments to attitude-incongruent messages, bringing to mind reasons for holding one's initial attitude [28], and derogating sources of attitude-incongruent messages." | "In the ‘post-truth era’, political fact-checking has become an issue of considerable significance. A recent study in the context of the 2016 US election found that fact-checks of statements by Donald Trump changed participants' beliefs about those statements—regardless of whether participants supported Trump—but not their feelings towards Trump or voting intentions. However, the study balanced corrections of inaccurate statements with an equal number of affirmations of accurate statements. Therefore, the null effect of fact-checks on participants’ voting intentions and feelings may have arisen because of this artificially created balance. Moreover, Trump's statements were not contrasted with statements from an opposing politician, and Trump's perceived veracity was not measured. The present study (N = 370) examined the issue further, manipulating the ratio of corrections to affirmations, and using Australian politicians (and Australian participants) from both sides of the political spectrum. We hypothesized that fact-checks would correct beliefs and that fact-checks would affect voters’ support (i.e. voting intentions, feelings and perceptions of veracity), but only when corrections outnumbered affirmations. Both hypotheses were supported, suggesting that a politician's veracity does sometimes matter to voters. The effects of fact-checking were similar on both sides of the political spectrum, suggesting little motivated reasoning in the processing of fact-checks." | Australian study sample | https://doi.org/10.1098/rsos.180593 | No | Experimental | Intended action/behavior | Yes | |||||||||
58 | Refuting fake news on social media: nonprofits, crisis response strategies and issue involvement | Michail Vafeiadis, Denise S. Bortree, Christen Buckley, Pratiti Diddi, and Anli Xiao | 2019 | Journal of Product & Brand Management | Show participants "rumors about consumer privacy when non-profit organizations are targeted on social media" and varying forms of response type | Efficacy of the rumor response strategy (message credibility) | "2 (response type: denial vs attack) × 2 (privacy concerns: low vs high)" | Not specified; participants recruited via Qualtrics | Content labeling | Disinformation disclosure | "Data showed that attacking the source of fake news (as a crisis response) reduces the message’s credibility more than denying fake news. Furthermore, highly involved individuals are more likely to centrally process information and develop positive supportive intentions toward the affected non-profit brand. High issue involvement also predicted organizational and response credibility. Conversely, an attack rebuttal message increased the credibility of the circulated malicious rumors for low involved individuals." | Not specified | "The dissemination of fake news has accelerated with social media and this has important implications for both organizations and their stakeholders alike. Hence, the purpose of this study is to shed light on the effectiveness of the crisis response strategies of denial and attack in addressing rumors about consumer privacy when non-profit organizations are targeted on social media." | https://doi.org/10.1108/JPBM-12-2018-2146 | Unclear | Experimental | Beliefs | Unclear | ||||||||||
59 | Correcting Political and Consumer Misperceptions: The Effectiveness and Effects of Rating Scale Versus Contextual Correction Formats | Michelle A. Amazeen, Emily Thorson, Ashley Muddiman, Lucas Graves | 2016 | Journalism & Mass Communication Quarterly | Participants shown fictitious claims from a non-political figure or Congressman and varying forms of corrections | Participants rated feelings about people mentioned in fact check and about the fact check outlet | "Correction Type: rating scale, context only, no correction x Statement Type: same party, opposing party, non-political" | Nationally representative U.S. sample recruited by YouGov | Disinformation disclosure | Content labeling | "The results of this study suggest that truth scales can substantially increase people’s understanding of the world around them and that their use comes with few disadvantages. The effectiveness of rating scales is especially pronounced when they are used to correct non-political misinformation" | "Consistent with the ELM theory of persuasion, the cues provided by the ratings icon may have facilitated the ability of participants to process the information given the low-involvement nature of the [non-political] stimulus… Overall, these results reinforce the difficulty of overcoming partisan-driven motivated reasoning. Partisanship consistently moderated the political correction’s effectiveness in educating the public" | "While fact-checking has grown dramatically in the last decade, little is known about the relative effectiveness of different formats in correcting false beliefs or overcoming partisan resistance to new information. This article addresses that gap by using theories from communication and psychology to compare two prevailing approaches: An online experiment examined how the use of visual “truth scales” interacts with partisanship to shape the effectiveness of corrections. We find that truth scales make fact-checks more effective in some conditions. Contrary to theoretical predictions and the fears of some journalists, their use does not increase partisan backlash against the correction or the organization that produced it." | https://doi-org.ezproxy.princeton.edu/10.1177%2F1077699016678186 | Yes | Experimental | Beliefs | Yes | ||||||||||
60 | Exposure to Social Engagement Metrics Increases Vulnerability to Misinformation | Mihai Avram, Nicholas Micallef, Sameer Patil, Filippo Menczer | 2020 | Harvard Kennedy School Misinformation Review | Playing the game Fakey which "simulates fact checking on a social media feed" | Participants choose to interact with articles in a variety of ways | Included articles from "mainstream as well as low-credibility sources" | Users who downloaded the game in the U.S., Australia, UK, Canada, Germany, and Bulgaria | Content labeling | "High levels of social engagement results in lower fact checking and higher liking/sharing, especially for low-credibility content… People are more vulnerable to low-credibility content with high social engagement." | "Our findings show that displayed engagement metrics can strongly influence interaction with low-credibility information. The higher the shown engagement, the more prone people were to share questionable content and less to fact check it." | "News feeds in virtually all social media platforms include engagement metrics, such as the number of times each post is liked and shared. We find that exposure to these social engagement signals increases the vulnerability of users to low-credibility information. This finding has important implications for the design of social media interactions in the misinformation age. To reduce the spread of misinformation, we call for technology platforms to rethink the display of social engagement metrics. Further research is needed to investigate whether and how engagement metrics can be presented without amplifying the spread of low-credibility information." | Intrepret not showing engagement metrics as the implied intervention. Not doing so would presumaably reduce sharing intentions. | https://arxiv.org/abs/2005.04682 | No | Simulated social media | Observed online behaviors | General social media use | Unclear | Yes | ||||||||
61 | When I Learn the News is False: How Fact-Checking Information Stems the Spread of Fake News Via Third-Person Perception | Myojung Chunng and Nuri Kim | 2021 | Human Communication Research | Participants read a fake news story presented as a Facebook post from a fictitious enironmental organization, and some saw fact-checks | "Participants filled out a posttest questionnaire that measured the perceived quality of the news story, perceived influence of the story on the self and others, and intentions to share the news story on social media." | Participants in the fact-checking conditions saw fact-check banners beneath the fake news story | Participants from Amazon's Mechanical Turk | Disinformation disclosure | Content labeling | "The current study investigated whether fact-checking information could lessen the spread of fake news on social media and, if so, why. Two experiments found that participants who viewed a fake news story on Facebook without fact-checking information perceived the news to be more credible, accurate, and informative, and presumed that they were influenced by the news to a similar extent as others. However, when the fake news was debunked by fact-checkers, participants evaluated the news to be of lower quality and presumed that others were more influenced by the news than themselves" | Third-person perception (TPP): "We found that a perceived discrepancy in media influence on the self and others (i.e., TPP) accounts for individuals’ intentions to share fake news on social media; fact-checking information alerted people that the news they read was fake and thus increased TPP. Increased TPP, in turn, led individuals to be less likely to share the news on social media." | "While fact-checking has received much attention as a potential tool to combat fake news, whether and how fact-checking information lessens intentions to share fake news on social media remains underexplored. Two experiments uncovered a theoretical mechanism underlying the effect of fact-checking on sharing intentions, and identified an important contextual cue (i.e., social media metrics) that interacts with fact-checking effects. Exposure to fake news with fact-checking information (vs. fake news without fact-checking information) yielded more negative evaluations of the news and a greater belief that others are more influenced by the news than oneself (third-person perception [TPP]). Increased TPP, in turn, led to weaker intentions to share fake news on social media. Fact-checking information also nullified the effect of social media metrics on sharing intentions; without fact-checking information, higher (vs. lower) social media metrics induced greater intentions to share the news. However, when fact-checking debunked the news, such an effect disappeared." | https://doi-org.ezproxy.princeton.edu/10.1093/hcr/hqaa010 | Unclear | Simulated social media | Intended action/behavior | Yes | Yes | |||||||||
62 | Unchecked vs. Uncheckable: How Opinion-Based Claims Can Impede Corrections of Misinformation | Nathan Walter, Nikita A. Salovich | 2021 | Mass Communication and Society | Participants received an article in favor of gun rights or gun control containing false information. Participants were then exposed to a fact-checking message. | Participants answered questions measuring manipulation check, speaker's credibility, and message accuracy | Study 1: "2 (misinformation-congruence: pro-attitudinal or counter-attitudinal) X 2 (message-checkability: combination of fact- and opinion-based statements or only fact-based statements)" Study 2: "2 (misinformation-congruence: pro-attitudinal or counter-attitudinal) x 2 (fact-checking: present or absent) x 2 (message-checkability detection: with or without 3 ) between-subjects factorial design" | Participants from Qualtrics panels | Disinformation disclosure | "In each experiment reported, pro-attitudinal fact-checking outperformed corrective information that ran counter to participants’ ideological views." | "Ostensibly, the nonpartisan and independent nature of fact-checkers makes them ideal contenders to address political misinformation. Yet, as others have also observed (e.g., Jarman, 2016; Thorson, 2016; Walter et al., 2020), the robust effects of motivated reasoning, whereby individuals do not welcome information that contradicts their pre-established beliefs but show too great a readiness to believe things that align with their existing worldview, equally applies to fact-checking and to unverified information from partisan sources." | "Although the prominence of fact-checking in political journalism has grown dramatically in recent years, empirical investigations regarding the effectiveness of fact-checking in correcting misperceptions have yielded mixed results. One understudied factor that likely influences the success of fact-checking initiatives is the presence of opinion statements in fact-checked messages. Recent work suggests that people may have difficulty differentiating opinion- from fact-based claims, especially when they are congruent with preexisting beliefs. In three experiments, we investigated the consequences of opinion-based claims to the efficacy of fact-checking in correcting misinformation regarding gun policy. Study 1 (N = 152) demonstrated that fact-checking is less effective when it attempts to correct statements that include both fact- and opinion-based claims. Study 2 (N = 561) replicated and expanded these findings showing that correction is contingent on people’s ability to accurately distinguish facts from opinions. Study 3 (N = 389) illustrated that the observed effects are governed by motivated reasoning rather than actual inability to ascertain fact-based claims. Together these results suggest that distinguishing facts from opinions is a major hurdle to effective fact-checking." | https://doi.org/10.1080/15205436.2020.1864406 | Unclear | Experimental | Beliefs | Yes | |||||||||||
63 | Effective strategies for rebutting science denialism in public discussions | Philipp Schmid and Cornelia Betsch | 2019 | Nature Human Behaviour | Participants read or listened to an audio discussion with a science denier, varying the topic of discussion and method of fact correction | Participants reported attitude and intention regarding topic of discussion | Varied topic across experiments; presence or absence of a science advocate in discussion; advocate "corrected the facts about the topic, unncovered the technique of the denier or used a combination of both methods" | Adult participants recruited by Questback | Disinformation disclosure | "The results show that public discussions with a science denier have a damaging effect on the audience, as revealed by negative changes in attitudes (Supplementary Fig. 2) and intentions (Fig. 2): pre- and post-measures showed that the attitude towards a behaviour favoured by science and the intention to perform this behaviour were reduced by reading or listening to a discussion with a science denier (attitude: g = −0.32, 95% confidence interval (CI): −0.46, −0.17; intention: g = −0.21, 95% CI: −0.35, −0.08). When no advocate for science was present, the denier had the strongest effects compared with conditions where an advocate was present." | "Altogether, the results do not support the backfire hypothesis in attempts to rebut science denial in public discussions. Instead, the results suggest that both topic and technique rebuttal as single strategies or as a combined strategy can reduce the impact of a science denier. Moreover, it is especially beneficial to use rebuttal strategies among audiences whose prior beliefs or ideology render them particularly vulnerable to science deniers." | "Science deniers question scientific milestones and spread misinformation, contradicting decades of scientific endeavour. Advocates for science need effective rebuttal strategies and are concerned about backfire effects in public debates. We conducted six experiments to assess how to mitigate the influence of a denier on the audience. An internal meta-analysis across all the experiments revealed that not responding to science deniers has a negative effect on attitudes towards behaviours favoured by science (for example, vaccination) and intentions to perform these behaviours. Providing the facts about the topic or uncovering the rhetorical techniques typical for denialism had positive effects. We found no evidence that complex combinations of topic and technique rebuttals are more effective than single strategies, nor that rebutting science denialism in public discussions backfires, not even in vulnerable groups (for example, US conservatives). As science deniers use the same rhetoric across domains, uncovering their rhetorical techniques is an effective and economic addition to the advocates’ toolbox." | https://doi-org.ezproxy.princeton.edu/10.1038/s41562-019-0632-4 | No | Experimental | Intended action/behavior | Yes | |||||||||||
64 | Weight-of-Evidence Strategies to Mitigate the Influence of Messages of Science Denialism in Public Discussions | Philipp Schmid, Marius Schwarzer, and Conelia Betsch | 2020 | Journal of Cognition | Participants watched a mock TV discussion related to vaccines with varying forms of corrections | "Participants indicated their attitudes towards vaccination and intention to get vaccinated before and after the TV discussion" | Experiment 1: "2 (rebuttal vs. advocate silent; between subjects) × 2 (outnumbering 5:1 vs. equal proportion of guests 3:3; between subjects) × 2 (measurement before vs. after the debate; within subjects)" Experiment 2: added forewarning vs. no forewarning Experiment 3: varied science advocates outnumbering deniers | Students from the University of Erfurt | Disinformation disclosure | "The results also showed that it is necessary to use such measures consciously, as in all three experiments, the science denier damaged study participants’ vaccination-related attitudes and intention, and reduced their confidence in vaccines’ safety and efficacy. While the damage can be mitigated through clever rebuttals from the advocate (replicating Schmid & Betsch, 2019), it cannot always be guaranteed that rebuttals will be delivered successfully – or at all" | "In the light of the present findings, we expect that forewarnings about the false-balance effect should help reduce damage when screened prior to a falsely balanced discussion. The results from the present experiments reveal that such forewarnings are an effective weight-of-evidence strategy that can mitigate science denialism’s influence, independent of whether a rebuttal is delivered and independent of audience characteristics. The results are consistent with an increasing body of evidence showing that using prior information as a prebunking is an effective strategy against damage from misinformation" | "In mass media, the positions of science deniers and scientific-consensus advocates are repeatedly presented in a balanced manner. This false balance increases the spread of misinformation under the guise of objectivity. Weight-of-evidence strategies are an alternative, in which journalists lend weight to each position that is equivalent to the amount of evidence that supports the position. In public discussions, journalists can invite more advocates of scientific consensuses than science deniers (outnumbering) or they can employ warnings about the false-balance effect prior to the discussions (forewarning). In three pre-registered laboratory experiments, we tested the efficacy of outnumbering and forewarning as weight-of-evidence strategies to mitigate science deniers’ influence on individuals’ attitudes towards vaccination and their intention to vaccinate. We explored whether advocates’ responses to science deniers (rebuttal) and audiences’ issue involvement moderate the efficacy of these strategies. A total of N = 887 individuals indicated their attitudes towards vaccination and their intention to vaccinate before and after watching a television (TV) discussion. The presence and absence of forewarning, outnumbering and rebuttal were manipulated between subjects; participants also indicated their individual issue involvement. We obtained no evidence that outnumbering mitigates damage from denialism, even when advocates served as multiple sources. However, forewarning about the false-balance effect mitigated deniers’ negative effects. Moreover, the protective effect was independent of rebuttal and issue involvement. Thus, forewarnings can serve as an effective, economic and theory-driven strategy to counter science denialism in public discussions, at least for highly educated individuals such as university students." | https://dx.doi.org/10.5334%2Fjoc.125 | No | Experimental | Intended action/behavior | Yes | |||||||||||
65 | The promise and peril of real-time corrections to political misperceptions | R. Kelly Garrett and Brian E. Weeks | 2013 | Proceedings of the 2013 conference on Computer supported cooperative work | Participants read a news article about technology containing false information and viewed corrections | Measured memory of the article, participants' feelings on a number of issues, and assessment of the fact-checking message | Participants given a delayed correction, an immediate correction, or no correction | U.S. sample recruited by Survey Sampling International | Disinformation disclosure | "Despite the obvious appeal of providing immediate corrections to false information online, this approach has some negative consequences that need to be addressed. Annotating a misleading message by highlighting inaccuracies and embedding fact-checking information accentuates individuals’ tendency to view these corrections through an attitudinally biased lens. We should be careful not to exaggerate the significance of this behavior: real-time corrections are often as effective as traditional postexposure correction strategies. The problem is that these techniques actually increase resistance to the correction among those whose attitudes are most strongly supported by the misperception." | "We believe that the key theoretical insight for guiding future design work is our argument that people respond to fact-checking messages in ways that are comparable to their response to propaganda and persuasion." | "Computer scientists have responded to the high prevalence of inaccurate political information online by creating systems that identify and flag false claims. Warning users of inaccurate information as it is displayed has obvious appeal, but it also poses risk. Compared to post-exposure corrections, real-time corrections may cause users to be more resistant to factual information. This paper presents an experiment comparing the effects of real-time corrections to corrections that are presented after a short distractor task. Although real-time corrections are modestly more effective than delayed corrections overall, closer inspection reveals that this is only true among individuals predisposed to reject the false claim. In contrast, individuals whose attitudes are supported by the inaccurate information distrust the source more when corrections are presented in real time, yielding beliefs comparable to those never exposed to a correction. We find no evidence of real-time corrections encouraging counterargument. Strategies for reducing these biases are discussed." | https://doi.org/10.1145/2441776.2441895 | Yes | Experimental | Beliefs | Unclear | |||||||||||
66 | Combatting climate change misinformation: Evidence for longevity of inoculation and consensus messaging effects | Rakoen Maertens, Frederik Anseel, and Sander Van Der Linden | 2020 | Journal of Environmental Psychology | Exposure to a message about scientific consensus surrounding climate change, potential exposure to misinformation | One week later, participants related belief in climate change and other related topics | Control group; participants received message about scientific consensus on climate change; consensus plus misinformation one week later; consensus message followed by inoculation and misinformation one wek later | US adults recruited by Prolific Academic | Disinformation disclosure | "We replicated the initial positive effect of the scientific consensus message by itself and across ideology and party affiliation (Lewandowsky et al., 2013; van der Linden et al., 2015, van der Linden et al., 2019a, van der Linden et al., 2019b), but also found that this consensus effect shows partial decay over the course of one week. We also replicated the finding that the misinformation message counteracts the consensus message and brings perceived scientific consensus back to baseline. Finally, we found that an inoculation message is able to protect the positive effects of the consensus message against doubt-sowing misinformation presented with a one-week delay, without any decay in the inoculation effect over time." | "These findings may indicate that participants in our study gave equal weight to the misinformation as to the facts, which resulted in a net change of zero, consistent with van der Linden et al. (2017). Although Williams and Bond (2020) found a significant inoculation effect compared to a no-treatment control, they found no additional benefit compared to the false-balance condition. These results suggests that the consensus message may have been strong enough to remain significant on its own, and did not need an inoculation as extra protection. We also found a protective benefit of the consensus message on its own, but this effect was less strong, or, alternatively, our misinformation message was more potent" | "Despite the fact that there is a 97% consensus among climate scientists that humans are causing global warming, the spread of misinformation continues to undermine public support for climate action. Previous studies have found that resistance to misinformation can be induced by cognitively inoculating individuals against doubt-sowing about climate change. However, the long-term effectiveness of this approach is currently unknown. In a preregistered replication and extension experiment we combined a scientific consensus message with an inoculation treatment, and exposed participants to an influential misinformation message one week later. We explored 1) whether we can replicate the finding that inoculation is able to protect against a misinformation attack, and 2) whether or not the consensus and inoculation effects remain stable over the course of one week. Successfully replicating the effects of the original study, we found a strong initial consensus effect that is sensitive to doubt-sowing misinformation. Importantly, we also found that the consensus effect can be inoculated against misinformation. Extending the replication, we found that the consensus effect shows partial decay over time, while the inoculation effect remains stable for at least one week. We discuss the impact of our findings for inoculation theory, climate change psychology, and public policy." | Replicates previous study, Linden et al. (2017) | https://doi.org/10.1016/j.jenvp.2020.101455 | Unclear | Experimental | Beliefs | Yes | ||||||||||
67 | Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments | Rakoen Maertens, Jon Roozenbeek, Melisa Basol, and Sander Van der Linden | 2020 | Journal of Experimental Psychology: Applied | Participants played Bad News (inoculation game) or Tetris (control) | "Rated the reliability of news headlines that either used a misinformation technique or not" | Treatment group played Bad News game, control group played Tetris | Participants recruited through Prolific | Disinformation literacy | "Overall, across the three experiments, we successfully replicate the inoculation treatment effect reported by Roozenbeek and van der Linden (2019b), but with more rigorous experimental designs. We show that after playing Bad News, participants find fake news headlines significantly less reliable than before playing the game." | "However, we argue that an alternative theoretical model could be based on memory strength and forgetting. After an inoculation intervention participants have bolstered their psychological “immune system,” but the techniques used in the attacks have to be remembered, and are subject to interference and potentially catastrophic forgetting (Hardt, Nader, & Nadel, 2013). Indeed, we can link various key concepts of inoculation theory to a potential memory model to explain decreases in the inoculation effect. As associative networks have been linked to the long-term memory system (Collins & Loftus, 1975; Smith, 1998), inoculation effect decreases over time could be researched through the lens of neural network simulations of memory networks (Hardt et al., 2013). Over time, the memory network could suffer from forgetting (Frankland & Bontempi, 2005), with interference as the mechanism." | "This study investigates the long-term effectiveness of active psychological inoculation as a means to build resistance against misinformation. Using 3 longitudinal experiments (2 preregistered), we tested the effectiveness of Bad News, a real-world intervention in which participants develop resistance against misinformation through exposure to weakened doses of misinformation techniques. In 3 experiments (NExp1 = 151, NExp2 = 194, NExp3 = 170), participants played either Bad News (inoculation group) or Tetris (gamified control group) and rated the reliability of news headlines that either used a misinformation technique or not. We found that participants rate fake news as significantly less reliable after the intervention. In Experiment 1, we assessed participants at regular intervals to explore the longevity of this effect and found that the inoculation effect remains stable for at least 3 months. In Experiment 2, we sought to replicate these findings without regular testing and found significant decay over a 2-month time period so that the long-term inoculation effect was no longer significant. In Experiment 3, we replicated the inoculation effect and investigated whether long-term effects could be due to item-response memorization or the fake-to-real ratio of items presented, but found that this is not the case. We discuss implications for inoculation theory and psychological research on misinformation." | https://psycnet.apa.org/doi/10.1037/xap0000315 | Unclear | Experimental | Observed real-world behaviors | ||||||||||||
68 | Inoculating the Public against Misinformation about Climate Change | Sander Van Der Linden, Anthony Leiserowitz, Seth Rosenthal, and Edward Maibach | 2017 | Global Challenges | Participants exposed to information about the scientific consensus surrounding climate change, and some participants viewed a countermessage | "Respondent's (pre and post) estimate of the current level of scientific agreement on human‐caused climate change" | Groups: Control; Consennsus treatment; Countermessage; Consensus followed by countermessage; Consensus + general inoculation followed by countermessage; Consensus + detailed inoculation followed by countermessage | Participants from Amazon's Mechanical Turk | Disinformation disclosure | "This study finds that public attitudes about climate change can be effectively “inoculated” against influential misinformation. In particular, our results point to three important conclusions. First, consistent with prior work, we find strong support for the efficacy of communicating the scientific consensus on human‐caused climate change.7, 19, 38, 43, 44, 47 Second, this research further extends these findings by presenting information about the consensus in a politically “contested” information environment, that is, countered by a real petition claiming that there is no scientific consensus on human‐caused climate change... Third, the current study also found that much of the initial consensus‐effect was preserved (up to two‐thirds) by the inoculation messages, which, importantly, proved equally effective across the political spectrum" | "Some scholars have argued that because people sometimes engage in “identity‐protective motivated reasoning,” highlighting scientific consensus will only cause or exacerbate existing attitude polarization.37 Yet, “true” attitude polarization in response to mixed evidence is relatively infrequent (ref. 64-66 and recent research suggests that political polarization on climate change is more likely the result of selective exposure to partisan media rather than motivated reasoning alone.48, 50, 67, 68 Moreover, this study finds no support for the hypothesis that inoculating people about the scientific consensus backfires among those who are ideologically predisposed to be skeptical about climate change (e.g., Republicans), which is both promising and consistent with other research on inoculation theory." | "Effectively addressing climate change requires significant changes in individual and collective human behavior and decision‐making. Yet, in light of the increasing politicization of (climate) science, and the attempts of vested‐interest groups to undermine the scientific consensus on climate change through organized “disinformation campaigns,” identifying ways to effectively engage with the public about the issue across the political spectrum has proven difficult. A growing body of research suggests that one promising way to counteract the politicization of science is to convey the high level of normative agreement (“consensus”) among experts about the reality of human‐caused climate change. Yet, much prior research examining public opinion dynamics in the context of climate change has done so under conditions with limited external validity. Moreover, no research to date has examined how to protect the public from the spread of influential misinformation about climate change. The current research bridges this divide by exploring how people evaluate and process consensus cues in a polarized information environment. Furthermore, evidence is provided that it is possible to pre‐emptively protect (“inoculate”) public attitudes about climate change against real‐world misinformation." | https://doi.org/10.1002/gch2.201600008 | Unclear | Experimental | Knowledge | Yes | |||||||||||
69 | Misinformation lingers in memory: Failure of three pro-vaccination strategies | Sara Pluviano, Caroline Watt, Sergio Della Sala | 2017 | PLoS One | Participants completed a questionnaire on baseline beliefs about vaccines and were exposed to corrective information | Participants completed a post-manipulation questionnaire to assess how beliefs and attitudes changed | Varied corrective information (myth vs. fact, visual correction, fear correction) | University students in Europe | Disinformation disclosure | "Specifically, we found that the myths vs. facts format, at odds with its aims, induced stronger beliefs in the vaccine/autism link and in vaccines side effects over time, lending credit to the literature showing that countering false information in ways that repeat it may further contribute to its dissemination. Also the exposure to fear appeals through images of sick children led to more increased misperceptions about vaccines causing autism. Moreover, this corrective strategy induced the strongest beliefs in vaccines side effects, highlighting the negative consequences of using loss-framed messages and fear appeals to promote preventive health behaviours. Our findings also suggest that no corrective strategy was useful in enhancing vaccination intention. Compared to the other techniques, the usage of fact/icon boxes resulted in less damage but did not bring any effective result." | "Multiple explanations have been proposed for the continued influence of misinformation. A strong argument is that, once a belief is formed, people generate explanations that fit and further reinforce this belief and tend to vigorously reject counter-arguments that make them uncomfortable, regardless of their validity" | "People’s inability to update their memories in light of corrective information may have important public health consequences, as in the case of vaccination choice. In the present study, we compare three potentially effective strategies in vaccine promotion: one contrasting myths vs. facts, one employing fact and icon boxes, and one showing images of non-vaccinated sick children. Beliefs in the autism/vaccines link and in vaccines side effects, along with intention to vaccinate a future child, were evaluated both immediately after the correction intervention and after a 7-day delay to reveal possible backfire effects. Results show that existing strategies to correct vaccine misinformation are ineffective and often backfire, resulting in the unintended opposite effect, reinforcing ill-founded beliefs about vaccination and reducing intentions to vaccinate. The implications for research on vaccines misinformation and recommendations for progress are discussed." | https://doi.org/10.1371/journal.pone.0181640 | No | Experimental | Beliefs | No | |||||||||||
70 | Educative Interventions to Combat Misinformation: Evidence From a Field Experiment in India | Sumitra Badrinathan | 2020 | CASI Workinig Paper (University of Pennsylvania) | Treated participants with an in-person media literacy training focused on identifying misinformation | "Whether the intervention positively affected respondents’ ability to identify misinformation" based on responses about news stories | Random assignment to treatment | 1224 participants from the city of Gaya, India | Disinformation literacy | "The most striking finding to emerge from this study demonstrates that the intervention improved misinformation identification skills for one set of respondents (non-BJP respondents) but not another (BJP partisans). Paralleling results seen in developed contexts, the perceptual screen (Campbell et al. 1960) of BJP partisanship shaped how respondents interacted with this treatment, with BJP partisans demonstrating a tendency to cheerlead for their party and discredit pro-party stories despite them being false" | "These findings of motivated reasoning demonstrate that citizen attachments to political parties are heightened during elections (Michelitch and Utych 2018) and that strong partisans engage in strategic ignorance, pushing away information and facts that get in the way of feelings" | "Misinformation makes democratic governance harder, especially in developing countries. Despite its real-world import, little is known about how to combat misinformation outside of the U.S., particularly in places with low education, accelerating Internet access, and encrypted information sharing. This study uses a field experiment in India to test the efficacy of a pedagogical intervention on respondents’ ability to identify misinformation during the 2019 elections (N=1224). Treated respondents received in-person media literacy training in which enumerators demonstrated tools and tips to identify misinformation in a coherent learning module. Receiving this hour-long media literacy intervention did not significantly increase respondents’ ability to identify misinformation on average. However, treated respondents who support the ruling party became significantly less able to identify proattitudinal stories. These findings point to the resilience of misinformation in India and the presence of motivated reasoning in a traditionally non-ideological party system." | Finds a backfire effect although "there is little evidence of such backfire effects in the American context" | https://casi.sas.upenn.edu/sites/default/files/research/Educative%20Interventions%20to%20Combat%20Misinformation%20-%20Sumitra%20Badrinathan%20%28CASI%20Working%20Paper%2C%2012-2020%29.pdf | No | Experimental | Observed real-world behaviors | |||||||||||
71 | The Elusive Backfire Effect: Mass Attitudes’ Steadfast Factual Adherence | Thomas Wood and Ethan Porter | 2019 | Political Behavior | Presented participants with "real instances of misstatements by political leaders from both sides of the aisle," and some participants saw a correction | Subjects were "asked whether they agreed with the original misstatement" | "Some subjects were randomly vended a correction, consisting of neutral data from governmental sources." Two studies presented misstatements as excerpts from newspaper articles; three studies presented complete fictitious news articles | Studies 1-4: Mechanical Turk subjects Study 5: nationally representative survey panel | Disinformation disclosure | "We find that backfire is stubbornly difficult to induce, and is thus unlikely to be a characteristic of the public’s relationship to factual information. Overwhelmingly, when presented with factual information that corrects politicians—even when the politician is an ally—the average subject accedes to the correction and distances himself from the inaccurate claim." | "Our findings are consistent with one of the most well-documented aspects of mass public opinion: respondents shy away from cognitive effort, and will deploy shrewd strategies to avoid it (Lippmann 1922). The backfire hypothesis proposes that a subject, when furnished facts inconsistent with her ideological commitments, will resolve the challenge of these facts by concocting new considerations to offset the threatening information. Developing counter-arguments would be unusually effortful, as sophisticated respondents can simply filter out, rather than counterargue, unwelcome facts" | "Can citizens heed factual information, even when such information challenges their partisan and ideological attachments? The “backfire effect,” described by Nyhan and Reifler, says no: rather than simply ignoring factual information, presenting respondents with facts can compound their ignorance. In their study, conservatives presented with factual information about the absence of Weapons of Mass Destruction in Iraq became more convinced that such weapons had been found. The present paper presents results from five experiments in which we enrolled more than 10,100 subjects and tested 52 issues of potential backfire. Across all experiments, we found no corrections capable of triggering backfire, despite testing precisely the kinds of polarized issues where backfire should be expected. Evidence of factual backfire is far more tenuous than prior research suggests. By and large, citizens heed factual information, even when such information challenges their ideological commitments." | https://doi-org.ezproxy.princeton.edu/10.1007/s11109-018-9443-y | Yes | Experimental | Knowledge | Yes | |||||||||||
72 | Corrections of Political Misinformation: No Evidence for an Effect of Partisan Worldview in a U.S. Convenience Sample | Ullrich Ecker, Brandon Sze, and Matthew Andreotta | 2020 | "Participants were presented with a fictitious report revolving around a research study that purportedly found that politicians of either the Democratic or Republican Party were three times as likely to embezzle funds compared to the rival party," and some participants viewed corrections | "The test questionnaire consisted of nine inferential-reasoning items, three memory items, and two retraction-awareness items" | "2 × 2 × 2 between-subjects design, with factors group (Democrats, Republicans), worldview congruence (congruent, incongruent), and retraction condition (no-retraction, retraction)" | Adult U.S. participants | Disinformation disclosure | "In line with hypotheses, we found that (1) retractions reduced reliance on misinformation, and (2) participants were more likely to use worldview-congruent (mis)information in their reasoning. Contrary to predictions, however, we found that (3) the effect of worldview was not stronger in politically-conservative participants. In other words, retractions were effective in reducing references to misinformation in both Democrats and Republicans, irrespective of whether the misinformation challenged participants’ worldview" | "This study therefore failed to replicate the asymmetry observed by Ecker and Ang (2019), and instead suggests that the processing of misinformation corrections is not impacted by worldview, even though reasoning in general is biased towards worldviewcongruent information (in line with Ditto et al., 2019; Ecker et al., 2014; Guay & Johnston, 2020). These results add further weight to the emerging consensus that worldview biases information selection during reasoning but does not generally lead to dismissal of misinformation corrections in the CIE paradigm, let alone backfire effects" | "Misinformation often has a continuing effect on people’s reasoning despite clear correction. One factor assumed to affect post-correction reliance on misinformation is worldview-driven motivated reasoning. For example, a recent study with an Australian undergraduate sample found that when politically-situated misinformation was retracted, political partisanship influenced the effectiveness of the retraction. This worldview effect was asymmetrical, that is, particularly pronounced in politically-conservative participants. However, the evidence regarding such worldview effects (and their symmetry) has been inconsistent. Thus, the present study aimed to extend previous findings by examining a sample of 429 pre-screened U.S. participants supporting either the Democratic or Republican Party. Participants received misinformation suggesting that politicians of either party were more likely to commit embezzlement; this was or was not subsequently retracted, and participants’ inferential reasoning was measured. While political worldview (i.e., partisanship) influenced the extent to which participants relied on the misinformation overall, retractions were equally effective across all conditions. There was no impact of political worldview on retraction effectiveness, let alone evidence of a backfire effect, and thus we did not replicate the asymmetry observed in the Australian-based study. This pattern emerged despite some evidence that Republicans showed a stronger emotional response than Democrats to worldview-incongruent misinformation." | Preprint in press at Philosophical Transactions of the Royal Society | https://doi.org/10.31234/osf.io/bszm4 | Unclear | Experimental | Knowledge | Yes | |||||||||||
73 | Reminders and Repetition of Misinformation: Helping or Hindering Its Retraction? | Ullrich Ecker, Joshua Hogan, and Stephan Lewandowsky | 2017 | Journal of Applied Research in Memory and Cognition | Participants read fictional reports, some of which retracted previously presented information | Measurement of inferential reasoning via a questionnaire; belief in and reliance on retracted misinformation | Fictional reports varied in the extent to which they contained retractions serving as misinformation reminders | undergraduates from the University of Western Australia | Content labeling | "Retractions that explicitly repeated the misinformation were more effective in reducing misinformation effects than retractions that avoided repetition, presumably because of enhanced salience." | "It has been proposed that one reason why corrections are so ineffective is that a myth is often repeated when it is corrected… We found contrary to the popular recommendation that corrections were more effective when they explicitly repeated the myth." | "People frequently rely on information even after it has been retracted, a phenomenon known as the continued-influence effect of misinformation. One factor proposed to explain the ineffectiveness of retractions is that repeating misinformation during a correction may inadvertently strengthen the misinformation by making it more familiar. Practitioners are therefore often encouraged to design corrections that avoid misinformation repetition. The current study tested this recommendation, investigating whether retractions become more or less effective when they include reminders or repetitions of the initial misinformation. Participants read fictional reports, some of which contained retractions of previous information, and inferential reasoning was measured via questionnaire. Retractions varied in the extent to which they served as misinformation reminders. Retractions that explicitly repeated the misinformation were more effective in reducing misinformation effects than retractions that avoided repetition, presumably because of enhanced salience. Recommendations for effective myth debunking may thus need to be revised." | https://doi.org/10.1016/j.jarmac.2017.01.014 | No | Experimental | Knowledge | Yes | |||||||||||
74 | Explicit warnings reduce but do not eliminate the continued influence of misinformation | Ullrich Ecker, Stephan Lewandowsky, and David Wang | 2010 | Memory & Cognition | Participants read fictitious scenarios accompanied by various forms of retractions. Experiment 1: Exposure to a false story and retraction Experiment 2: Exposure to a form of fact-check warning | Experiment 1 and 2: Participant responses to inference, fact-recall, and manipulation-check questions | Experiment 1: Presentation of a retraction, retraction and alternative information, retraction and pre-warning, no retraction Experiment 2: presence of specific warning only or specific warning with the provision of an alternative | Undergraduate psychology students | Disinformation disclosure | Content labeling | Experiment 1: "The novel finding of Experiment 1 was that a specific warning—by providing a detailed account of the CIE— reduced people’s reliance on misinformation. In doing so, the specific warning was as successful as the provision of an alternative account" Experiment 2: "Showed that the combined effect of the specific warning and the provision of an alternative account reduced reliance on misinformation more than the constituent strategies alone" | "A warning might, therefore, lead participants to rely less on automatic processing at retrieval, thus guarding against the unfiltered use of strongly activated, but misleading, information, and instead lead participants to engage in more strategic retrieval processing, with a focus on source monitoring" | "Information that initially is presumed to be correct, but that is later retracted or corrected, often continues to influence memory and reasoning. This occurs even if the retraction itself is well remembered. The present study investigated whether the continued influence of misinformation can be reduced by explicitly warning people at the outset that they may be misled. A specific warning—giving detailed information about the continued influence effect (CIE)—succeeded in reducing the continued reliance on outdated information but did not eliminate it. A more general warning—reminding people that facts are not always properly checked before information is disseminated—was even less effective. In an additional experiment, a specific warning was combined with the provision of a plausible alternative explanation for the retracted information. This combined manipulation further reduced the CIE but still failed to eliminate it altogether." | https://doi.org/10.3758/MC.38.8.1087 | No | Experimental | Knowledge | Yes | ||||||||||
75 | Do False Allegations Persist? Retracted Misinformation Does Not Continue to Influence Explicit Person Impressions | Ullrich K. H. Ecker and Arnold E. Rodricks | 2020 | Journal of Applied Research in Memory and Cognition | Participiants "received examples of behaviors that a fictitious person had allegedly engaged in," with behaviors which were or were not retracted | Participants rated traits of fictitious person and behavior predictions | In experimental scenario, domestic-violence behavior was or was not retracted | Undergraduate students from the University of Western Australia | Disinformation disclosure | "Two experiments produced a clear result: A strongly negative piece of misinformation had the expected effect on trait ratings and behavior-prediction measures, but no continued impact after being retracted. Thus, we demonstrate that direct person-impression measures can be unaffected by person-specific misinformation after its retraction." | "As outlined in the Introduction, one way to interpret this result is that forming and updating an impression of a person may differ from building and updating a mental model of an event. Specifically, while model-updating theory assumes that retraction of a core piece of information can threaten event-model coherence (e.g., Ecker et al., 2010), impression formation is a more holistic process (e.g., Park, 1986), where retraction of one piece of information may not produce comparable incoherence. Thus, continued influence may arise more readily with event reports because events by their very nature demand an explanation (something must have started the fire!), and it may therefore be rational to hold onto initially-provided causal information even if it is credibly retracted (Connor et al., 2020), whereas credibly retracted information about a person is less likely to still be of relevance post-retraction. This supports the notion of efficient impression updating (also see Mende-Siedlecki, 2018) and is in line with theoretical models that specify belief updating as a sequential anchoring-and-adjustment (Hogarth & Einhorn, 1992) or averaging process" | "Corrected misinformation often continues to influence reasoning; this is known as the continued-influence effect (CIE). It is unclear whether this effect also occurs in impression formation, with some arguing that person impressions are readily updated. The present study tested if a retracted allegation influences person impressions. Participants received examples of behaviors that a fictitious person had allegedly engaged in. The set did or did not include a domestic-violence behavior, which subsequently was or was not retracted. Discredited misinformation was found to influence neither trait ratings of the person nor behavior predictions. This held even when the person’s name implied a cultural background stereotypically associated with domestic violence. This provides evidence that under some circumstances, people can fully discount discredited misinformation when building person impressions. However, there was some tentative evidence that corrected misinformation did influence a more indirect measure, namely ratings of the fictitious person’s face." | Study of Australian participants | https://doi-org.ezproxy.princeton.edu/10.1016/j.jarmac.2020.08.003 | No | Experimental | Beliefs | Unclear | ||||||||||
76 | Can you believe it? An investigation into the impact of retraction source credibility on the continued influence effect | Ullrich K. H. Ecker and Luke M. Antonio | 2021 | Memory & Cognition | Participants presented with a scenario and various kinds of retractions | Participants' reliance on critical information, measured using a questionnaire regarding inferential reasoning; participant belief in critical information and belief in retraction | "Source expertise (low vs. high) and source trustworthiness (low vs. high)" | Participants recruited from Australia, the U.S., and the U.K. Participants from the University of Western Australia | Disinformation disclosure | Content labeling | "The first conclusion to be drawn from this study, when seen in conjunction with previous research (in particular, Guillory & Geraci, 2013), is that the trustworthiness of retraction sources matters, at least under standard processing conditions such as those employed in Experiment 1. In Experiment 1, we found that only retractions from sources perceived to be trustworthy reduced reliance on retracted misinformation, while the perceived source expertise had no impact. As such, retractions from expert sources were ineffective if trustworthiness was low." | "An effect of perceived trustworthiness meshes well with the evidence provided by both experiments that retraction belief is an important determinant of post-retraction misinformation reliance and thus the CIE. Our results are consistent with the suggestion that people may continue to rely on retracted misinformation partly because they do not believe the retraction." | "The continued influence effect refers to the finding that people often continue to rely on misinformation in their reasoning even if the information has been retracted. The present study aimed to investigate the extent to which the effectiveness of a retraction is determined by its credibility. In particular, we aimed to scrutinize previous findings suggesting that perceived trustworthiness but not perceived expertise of the retraction source determines a retraction’s effectiveness, and that continued influence arises only if a retraction is not believed. In two experiments, we found that source trustworthiness but not source expertise indeed influences retraction effectiveness, with retractions from low-trustworthiness sources entirely ineffective. We also found that retraction belief is indeed a predictor of continued reliance on misinformation, but that substantial continued influence effects can still occur with retractions designed to be and rated as highly credible." | Study of participants in Australia, the U.S., and the U.K. | https://doi.org/10.3758/s13421-020-01129-y | No | Experimental | Beliefs | Yes | Yes | ||||||||
77 | Political Attitudes and the Processing of Misinformation Corrections | Ullrich K. H. Ecker, Li Chang Ang | 2019 | Political Psychology | Presentation of fictional scenarios involving politician misconduct and in some cases the presentation of a retraction. Scenarios were presented via a PowerPoint presentation. Experiment 1: "Political party preference (Labor, Liberal), scenario (singular, general), and retraction condition (no-retraction, retraction)" Experiment 2: "Party preference (Labor, Liberal), misconduct party (Labor, Liberal), and retraction condition (no-retraction, retraction)" | "Mean number of references to the critical information participants made in response to the questionnaire" | Targeted political affiliation in misinformation story; political party of participant; misinformation event specificity; presence of misinformation retraction | Undergraduate students at the University of Western Australia | Disinformation disclosure | "Across all conditions in both experiments, retractions tended to be more effective if they retracted attitude-dissonant misinformation" | "This is in line with theoretical accounts of motivated reasoning and the cultural cognition thesis… This might best be explained by the notion that worldview effects are mediated by the negative emotional impact of self-concept threatening corrections, combined with the assumption that conservatives are (1) particularly susceptible to such threats... and (2) generally less likely to engage with worldview-dissonant information based on their greater need for certainty, ambiguity intolerance, and closed-mindedness" | "Misinformation often continues to influence people’s memory and inferential reasoning after it has been retracted; this is known as the continued influence effect (CIE). Previous research investigating the role of attitude-based motivated reasoning in this context has found conflicting results: Some studies have found that worldview can have a strong impact on the magnitude of the CIE, such that retractions are less effective if the misinformation is congruent with a person’s relevant attitudes, in which case the retractions can even backfire. Other studies have failed to find evidence for an effect of attitudes on the processing of misinformation corrections. The present study used political misinformation—specifically fictional scenarios involving misconduct by politicians from left-wing and right-wing parties—and tested participants identifying with those political parties. Results showed that in this type of scenario, partisan attitudes have an impact on the processing of retractions, in particular (1) if the misinformation relates to a general assertion rather than just a specific singular event and (2) if the misinformation is congruent with a conservative partisanship." | Study of Australian participants | https://doi-org.ezproxy.princeton.edu/10.1111/pops.12494 | No | Experimental | Beliefs | Yes | ||||||||||
78 | Can corrections spread misinformation to new audiences? Testing for the elusive familiarity backfire effect | Ullrich K. H. Ecker, Stephan Lewandowsky, and Matthew Chadwick | 2020 | Cognitive Research: Principles and Implications | Presentation of "true and false claims and/or associated affirmative or corrective fact-checks" in the form of mock social media posts | Measuring belief in claims | Conditions: "no-exposure control (NE), claim-only (CO), fact-check-only (FCO), and claim-plus-fact-check (CFC; in this condition, participants first received all claims without any indication of validity, and then received the fact-checks separately)" Experiment 2: measured outcome immediately or after 1 week | US-based adult Amazon Mechanical Turk participants | Disinformation disclosure | "Both Experiments 2 and 3 yielded substantial evidence against the presence of a familiarity backfire effect, even under conditions that should maximize reliance on familiarity and thus facilitate occurrence of familiarity backfire" | "Broadly speaking, these results support the view that memory-based evaluation processes determine inferential reasoning and endorsement of claims much more than metacognitive judgments of fluency" | "Misinformation often continues to influence inferential reasoning after clear and credible corrections are provided; this effect is known as the continued influence effect. It has been theorized that this effect is partly driven by misinformation familiarity. Some researchers have even argued that a correction should avoid repeating the misinformation, as the correction itself could serve to inadvertently enhance misinformation familiarity and may thus backfire, ironically strengthening the very misconception that it aims to correct. While previous research has found little evidence of such familiarity backfire effects, there remains one situation where they may yet arise: when correcting entirely novel misinformation, where corrections could serve to spread misinformation to new audiences who had never heard of it before. This article presents three experiments (total N = 1718) investigating the possibility of familiarity backfire within the context of correcting novel misinformation claims and after a 1-week study-test delay. While there was variation across experiments, overall there was substantial evidence against familiarity backfire. Corrections that exposed participants to novel misinformation did not lead to stronger misconceptions compared to a control group never exposed to the false claims or corrections. This suggests that it is safe to repeat misinformation when correcting it, even when the audience might be unfamiliar with the misinformation." | https://doi.org/10.1186/s41235-020-00241-6 | Unclear | Experimental | Beliefs | Yes | |||||||||||
79 | Correcting false information in memory: manipulating the strength of misinformation encoding and its retraction | Ullrich K. H. Ecker, Stephan Lewandowsky, Briony Swire, Darren Chang | 2011 | Psychonomic Bulletin & Review | Participants read messages (misinformation) aloud as well as retractions of misinformation | Participants completed a causal inference questionnaire about the events described in messages | Varied the strength of misinformation and the strength of retraction (number of repetitions) Experiment 2 added a cognitive load (dividing attention between two tasks) for some participants | Undergraduate students at the University of Western Australia | Disinformation disclosure | Content labeling | "We found that stronger encoding of misinformation resulted in increased levels of continued influence… The results of both experiments also suggested that greater misinformation effects required stronger retractions. to substantially reduce continued influence. More interestingly, however, the results of both experiments suggested that the strength of retraction is immaterial if misinformation is only encoded relatively weakly" | "We focused our attention on the junction of memory and reasoning, in particular the way in which the memory system might support inferences. The simplest mechanism would involve random sampling: If misinformation were randomly sampled from memory, and hence were more likely to be sampled if more misinformation was represented, and if the impact of misinformation were largely but not entirely offset by retractions, could this explain the observed pattern?" | "Information that is presumed to be true at encoding but later on turns out to be false (i.e., misinformation) often continues to influence memory and reasoning. In the present study, we investigated how the strength of encoding and the strength of a later retraction of the misinformation affect this continued influence effect. Participants read an event report containing misinformation and a subsequent correction. Encoding strength of the misinformation and correction were orthogonally manipulated either via repetition (Experiment 1) or by imposing a cognitive load during reading (Experiment 2). Results suggest that stronger retractions are effective in reducing the continued influence effects associated with strong misinformation encoding, but that even strong retractions fail to eliminate continued influence effects associated with relatively weak encoding. We present a simple computational model based on random sampling that captures this effect pattern, and conclude that the continued influence effect seems to defy most attempts to eliminate it." | Australian study sample | https://doi.org/10.3758/s13423-011-0065-1 | No | Experimental | Beliefs | No | |||||||||
80 | Terrorists brought down the plane!--No, actually it was a technical fault: processing corrections of emotive information | Ullrich K. H. Ecker, Stephan Lewandowsky, Joe Apai | 2011 | Quarterly Journal of Experimental Psychology | Presented information about a fictitious plane crash; some participants received a retraction | Responses to inference questions about the plane crash | Manipulated the emotionality of the cause of the plain crash (e.g. intentional vs. unintentional violence); provided a retraction to some participants; provided a more neutral alternative cause to some participants | Psychology students | Other | Content labeling | "We again found no suggestion that emotionality determines the continued influence of misinformation, despite increased statistical power, a reduction in references to initial misinformation following retraction alone (thus removing a possible ceiling effect), and demonstrable emotional impact of the scenarios. The latter finding is particularly relevant because it confirms that memory updating took place at a time when participants were demonstrably emotional." | "Our finding that emotionality did not influence the recall of peripheral detail is not very surprising. As discussed in the introduction, the positive effects of emotion on memory are largely confined to central aspects of an event" | "It is well known that people often continue to rely on initial misinformation even if this information is later corrected and even if the correction itself is remembered. This article investigated the impact of emotionality of the material on people's ability to discount corrected misinformation. The focus was on moderate levels of emotionality comparable to those elicited by real-world news reports. Emotionality has frequently been shown to have an impact upon reasoning and memory, but the generality of this influence remains unclear. In three experiments, participants read a report of a fictitious plane crash that was initially associated with either an emotionally laden cause (terrorist attack) or an emotionally more neutral cause (bad weather). This initial attribution was followed by a retraction and presentation of an alternative cause (faulty fuel tank). The scenarios demonstrably affected participants' self-reported feelings. However, all three experiments showed that emotionality does not affect the continued influence of misinformation." | https://doi.org/10.1080/17470218.2010.497927 | No | Experimental | Knowledge | |||||||||||
81 | Refutations of Equivocal Claims: No Evidence for an Ironic Effect of Counterargument Number | Ullrich K. H. Ecker, Stephan Lewandowsky, Kalpana Jayawardana, and Alexander Mladenovic | 2019 | Journal of Applied Research in Memory and Cognition | Participants exposed to argumentative claims and varying numbers of refutations | Participants rated belief in claims | Claims followed by 0, 2, or 5 counterarguments | undergraduates from the University of Western Australia | Disinformation disclosure | Content labeling | "In Experiment 1, five counterarguments reduced beliefs in equivocal claims to a similar extent as two counterarguments; however, only five counterarguments led to reduced claim support on the inferential reasoning measure. In Experiment 2, only four, not two, reasonably strong counterarguments reduced claim belief relative to control, without however affecting the indirect measure. Experiment 3 showed that six counterarguments led to more belief change than two, a result that was mirrored in the indirect measure. Although there were some discrepancies across measures, the findings from all three experiments point to the same conclusion: there was no overkill backfire effect, and a larger number of counterarguments was conducive to greater belief revision." | "The fact that irrelevant counterarguments were found to have no impact suggests that the observed effects of argument number were not purely a result of a numerosity heuristic. Rather, the results support the basic assumption of rational belief updating, namely that more evidence against a claim leads to a stronger belief reduction. This is in line with a benchmark finding from the literature on the continued influence effect of misinformation: that the provision of alternative factual information is crucial when retracting misinformation" | "This study investigated the refutation of equivocal claims using counterarguments. Common sense suggests that more counterarguments should be more effective at inducing belief change. However, some researchers have argued that in persuasive reasoning, using too many arguments might lead to counterproductive skepticism and reactance. Thus, there have been calls to actively curtail the number of counterarguments used in refutations to avoid risking an “overkill backfire effect”—an ironic strengthening of beliefs from too many counterarguments. In three experiments, we tested whether calls to limit the number of counterarguments are justified. We found that a larger number of counterarguments (between four and six) led to as much or more belief reduction compared to a smaller number of (two) counterarguments. This was not merely an effect arising from a simple numerosity heuristic, as counterarguments had to be relevant to affect beliefs: irrelevant counterarguments failed to reduce beliefs even though perceived as moderately persuasive." | Australian study sample | https://doi.org/10.1016/j.jarmac.2018.07.005 | No | Experimental | Beliefs | Yes | |||||||||
82 | Do people keep believing because they want to? Preexisting attitudes and the continued influence of misinformation | Ullrich K. H. Ecker, Stephan Lewandowsky, Olivia Fenton, and Kelsey Martin | 2014 | Memory & Cognition | Presentation of a various forms of a "fictitious news report about a liquor store robbery," varying the identity of the characters | Assessed participants' understanding of the story using a questionnaire | Participants had high vs. low racial prejudice; participants did or did not receive a retraction Experiment 1 presented misinformation in which a robbery was committed by an Aboriginal person, and Experiment 2 presented misinformation in which an Aboriginal person prevented a robbery | Undergraduate students at the University of Western Australia | Disinformation disclosure | Content labeling | "In two experiments, we found that people use race-related information in their inferential reasoning mainly when this information is congruent with their attitudes… In contrast, people’s racial attitudes did not determine the effectiveness of retractions. Retractions reduced reliance on the critical information, but they did so equally for people in the high- and low-prejudice groups." | "If accepting a retraction does not require a shift in attitudes, it will seemingly be followed even when it is attitude-incongruent. Attitude-incongruent retractions may also be effective when people can use strategies to avoid attitude change. For example, people can accommodate exceptions to stereotypes, and thus maintain them by way of a process known as stereotype subtyping. This means, for example, that people with high racial prejudice might be able to accept an Aboriginal hero while maintaining their negative stereotype regarding Aboriginal people if they can identify a seemingly atypical attribute and use it to subtype the "deviant exemplar."" | "Misinformation—defined as information that is initially assumed to be valid but is later corrected or retracted—often has an ongoing effect on people’s memory and reasoning. We tested the hypotheses that (a) reliance on misinformation is affected by people’s preexisting attitudes and (b) attitudes determine the effectiveness of retractions. In two experiments, participants scoring higher and lower on a racial prejudice scale read a news report regarding a robbery. In one scenario, the suspects were initially presented as being Australian Aboriginals, whereas in a second scenario, a hero preventing the robbery was introduced as an Aboriginal person. Later, these critical, race-related pieces of information were or were not retracted. We measured participants’ reliance on misinformation in response to inferential reasoning questions. The results showed that preexisting attitudes influence people’s use of attitude-related information but not the way in which a retraction of that information is processed." | Australian study sample | https://doi-org.ezproxy.princeton.edu/10.3758/s13421-013-0358-x | No | Experimental | Knowledge | Unclear | |||||||||
83 | The effectiveness of short-format refutational fact-checks | Ullrich K. H. Ecker, Ziggy O'Reilly, Jesse S. Reid, and Ee Pin Chang | 2020 | British Journal of Psychology | Participants presented with true and false claims, rated belief, and viewed retractions or refutations | Participants "rerated their claim beliefs and responded to a series of inferential‐reasoning questions relating to the claims" | Manipulated claim veracity; provided retractions and detailed refutations, refutations only, refutations only without seeing the original claim, or no exposure to claims or corrections | U.S. residents recruited via Amazon's Mechanical Turk | Disinformation disclosure | Content labeling | "The results were clear‐cut and showed that simple retractions – messages that repeated a false claim while tagging it as false – did not backfire relative to either baseline. In fact, retractions substantially reduced belief in false claims relative to the pre‐correction level in the same sample, as well as relative to the level of belief expressed by a different sample after initial, unchallenged exposure to the false claims" | "Thus, the present study provides no support for the existence of familiarity backfire effects (in line with Swire, Ecker et al., 2017)2; there does not seem to be any harm associated with simple false‐tag fact‐checks. Moreover, additional claim repetition was generally associated with enhanced accuracy at test: Across both experiments, refutations were more powerful if participants were previously exposed to the myth, which runs counter to the assumption that greater familiarity with claims drives greater endorsement" | "Fact‐checking has become an important feature of the modern media landscape. However, it is unclear what the most effective format of fact‐checks is. Some have argued that simple retractions that repeat a false claim and tag it as false may backfire because they boost the claim's familiarity. More detailed refutations may provide a more promising approach, but may not be feasible under the severe space constraints associated with social‐media communication. In two experiments, we tested whether (1) simple ‘false‐tag’ retractions can indeed be ineffective or harmful; and (2) short‐format (140‐character) refutations are more effective than simple retractions. Regarding (1), simple retractions reduced belief in false claims, and we found no evidence for a familiarity‐driven backfire effect. Regarding (2), short‐format refutations were found to be more effective than simple retractions after a 1‐week delay but not a one‐day delay. At both delays, however, they were associated with reduced misinformation‐congruent reasoning." | https://doi.org/10.1111/bjop.12383 | Unclear | Experimental | Beliefs | Yes | ||||||||||
84 | Effects of Credibility Indicators on Social Media News Sharing Intent | Waheeb Yaqub, Otari Kakhidze, Morgan L. Brockman, Nasir Memon, Sameer Patil | 2020 | Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems | Presentation of 12 headlines and varying forms of accompanying credibility indicators | Participants indicated "whether they would share the corresponding article with their friends on social media" | Control group and exposure to one of four types of credibility indicators: Fact Checkers; News Media; Public ("A Majority of Americans disputes the credibility of this news); Artificial Intelligence | American users from Amazon's Mechanical Turk | Content labeling | Disinformation disclosure | "For the most part, the mere presence of a credibility indicator of any kind served to decrease the propensity to share Non-true headlines. Moreover, our results show that the influence of our credibility indicators on reducing sharing intent applies across the various categories of non-true news. Based on the individual characteristics of the participants, some indicators were more effective than others. The Fact Checkers indicator was a notable exception as it was found to be the most effective regardless of the influence of other factors." | No explicit posited mechanism, but authors generally suggest that the presence of a credibility indicator helps users identify fake stories and thus reduces the propensity to share | "In recent years, social media services have been leveraged to spread fake news stories. Helping people spot fake stories by marking them with credibility indicators could dissuade them from sharing such stories, thus reducing their amplification. We carried out an online study (N = 1,512) to explore the impact of four types of credibility indicators on people's intent to share news headlines with their friends on social media. We confirmed that credibility indicators can indeed decrease the propensity to share fake news. However, the impact of the indicators varied, with fact checking services being the most effective. We further found notable differences in responses to the indicators based on demographic and personal characteristics and social media usage frequency. Our findings have important implications for curbing the spread of misinformation via social media platforms." | https://doi-org.ezproxy.princeton.edu/10.1145/3313831.3376213 | Unclear | Experimental | Intended action/behavior | Yes | Yes | Yes | ||||||||
85 | Scaling Up Fact-Checking Using the Wisdom of Crowds | Jennifer Allen, Antonio Arechar, Gordon Pennycook, David Rand | 2020 | Participants viewed article headlines and ledes and some viewed article source | Rating accuracy of an article based on the headline and lede | All participants were given the headline and lede of 20 articles; half were shown the domain of the article's published (i.e., the source) | American laypeople from Mechanical Turk | Content labeling | "We find that, after judging merely the headline and lede of an article, a politically-balanced crowd of approximately 10 laypeople can match the performance of fact-checkers researching the full article. Interestingly, selecting raters whose characteristics make them more likely to agree with the fact-checkers (e.g. more deliberative, higher political knowledge, more liberal) or providing more information (the new article’s publisher) leads to only minimal improvements in the crowd’s agreement with the fact-checkers." | "our findings of greater cognitive reflection and political knowledge being associated with higher fact-checker agreement support the "classical reasoning" account whereby reasoning leads to more accurate judgments" | "Misinformation on social media has become a major focus of research and concern in recent years. Perhaps the most prominent approach to combating misinformation is the use of professional fact-checkers. This approach, however, is not scalable: Professional fact-checkers cannot possibly keep up with the volume of misinformation produced every day. Furthermore, many people see fact-checkers as having a liberal bias and thus distrust them. Here, we explore a potential solution to both of these problems: leveraging the “wisdom of crowds'' to identify misinformation at scale using politically-balanced groups of laypeople. Using a set of 207 news articles flagged for fact-checking by an internal Facebook algorithm, we compare the accuracy ratings given by (i) three professional fact-checkers after researching each article and (ii) 1,128 Americans from Amazon Mechanical Turk after simply reading the headline and lede sentence. We find that the average rating of a politically-balanced crowd of 10 laypeople is as correlated with the average fact-checker rating as the fact-checkers’ ratings are correlated with each other. Furthermore, the layperson ratings can predict whether the majority of fact-checkers rated a headline as “true” with high accuracy, particularly for headlines where all three fact-checkers agree. We also find that layperson cognitive reflection, political knowledge, and Democratic Party preference are positively related to agreement with fact-checker ratings; and that informing laypeople of each headline’s publisher leads to a small increase in agreement with fact-checkers. Our results indicate that crowdsourcing is a promising approach for helping to identify misinformation at scale." | This is a preprint working paper not yet published in a journal Should this be included since treatment was manipulated? | https://doi.org/10.31234/osf.io/9qdza | Unclear | Experimental | Observed real-world behaviors | Unclear | |||||||||||
86 | Emphasizing publishers does not effectively reduce susceptibility to misinformation on social media | Nicholas Dias, Gordon Pennycook, David Rand | 2020 | Harvard Kennedy School (HKS) Misinformation Review | Exposure to actual headlines from social media with or without publisher information | Participants rated perception of headlines as accurate; trustworthiness of a range of publishers | Presentation of visible publisher information vs. removing publisher information | American participants from Mechanical Turk and Lucid | Content labeling | "The research we present here confirms previous findings that U.S. adults, on average, tend to be good at identifying which sources are not credible (Pennycook & Rand, 2019a). Nonetheless, we find that, for most headlines, emphasizing publisher information has no meaningful impact on evaluations of headline accuracy. Thus, our findings suggest that emphasizing sources as a way to counter misinformation on social media may be misguided. Moreover, our analysis of plausible headlines from fake and hyperpartisan news sites suggests that source-based interventions of this kind may even be counterproductive, as plausible headlines from distrusted sources are often truthful" | No explicit posited mechanism, but study suggests that although social media users can identify sources which are not credible, this information has minimal impact on the assessment of credibility of a particular headline | "Survey experiments with nearly 7,000 Americans suggest that increasing the visibility of publishers is an ineffective, and perhaps even counterproductive, way to address misinformation on social media. Our findings underscore the importance of social media platforms and civil society organizations evaluating interventions experimentally, rather than implementing them based on intuitive appeal." | Paper reports that: "Neither Facebook nor YouTube has released data about the effectiveness of their source-based interventions, and the existing academic literature is inconclusive." | https://doi.org/10.37016/mr-2020-001 | Yes | Simulated social media | Beliefs | No | No | |||||||||
87 | Fighting the Past: Perceptions of Control, Historical Misperceptions, and Corrective Information in the Israeli‐Palestinian Conflict | Brendan Nyhan, Thomas Zeitzoff | 2017 | International Society of Political Psychology | Exposure to a text that "denied the Jewish role in the Palestinian exodus and instead attributed it to an anti‐Israeli conspiracy;" some participants then received corrective information | "We measured misperceptions and conspiracy beliefs about the Palestinian exodus; beliefs and attitudes on other key issues related to the conflict (tolerance for Nakba protests, final status of settlements, Jerusalem, etc.); and respondents' more general views about Israel and feelings toward Jewish Israelis, Arab Israelis, and Palestinians in the West Bank and Gaza." | "Manipulated level of control (low vs. high) and exposure to corrective information (historical denial vs. denial plus corrective information)" | 2,170 adult Jewish Israelis | Disinformation disclosure | "Exposure to a message‐denying evidence of the forced expulsion of many Palestinians during the creation of the state of Israel in 1948 increased misperceptions only when people were reminded of feelings of a lack of control. However, corrective information was quite effective in countering this effect—misperceptions about the Palestinian exodus declined significantly even for respondents in the low‐control condition and those whom we might expect to be most motivated to resist this information (e.g., right‐wing Jewish Israelis)." | "First, our findings suggest that intergroup conflicts can make people more vulnerable to misperceptions by inducing a feeling of a lack of control resulting from violence or oppression. Under these circumstances, people may be more likely to endorse misperceptions they hear or read or to blame outgroups for past ingroup behavior." | "What makes people deny wrongdoing that their group has inflicted on others? Prior research argues that refusing to acknowledge past misbehavior contributes to intergroup conflict, making historical misinformation important to understand and address. In particular, feeling a lack of control may make people more vulnerable to these misperceptions—a claim we test in a preregistered survey experiment examining beliefs about the Palestinian exodus during the creation of the state of Israel. Consistent with expectations, Jewish Israelis who were asked to recall an event in which they lacked control were more vulnerable to arguments (incorrectly) denying any Jewish responsibility for the exodus. By contrast, corrective information successfully reduced misperceptions regardless of feelings of control. However, corrections had no effect on attitudes toward the outgroup or support for the peace process, which suggests that historical misperceptions may be more of a symptom of intergroup conflict than a cause of its persistence." | Should this be included since treatment was manipulated? | https://doi-org.ezproxy.princeton.edu/10.1111/pops.12449 | No | Experimental | Beliefs | Yes | ||||||||||
88 | Will the Crowd Game the Algorithm?: Using Layperson Judgments to Combat Misinformation on Social Media by Downranking Distrusted Sources | Ziv Epstein, Gordon Pennycook, David Rand | 2020 | Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems | "To study the tendency of people to game the system, half of the participants were told their responses would inform social media ranking algorithms" | Americans indicated trust in numerous news sites | "Half of the participants were told their responses would inform social media ranking algorithms" | 1,130 Americans from Lucid | Content labeling | Content reporting | "The results we have presented here suggest that using crowdsourcing to identify outlets that produce misinformation, and then using those ratings as an input to social media ranking algorithms has promise for reducing the amount of misinformation on social media platforms. Specifically, we find that layperson trust ratings are quite effective in discerning between high and low quality news outlets. Rather than being blinded by partisanship, our participants tended to trust mainstream sources much more than hyper-partisan or fake news sources. Critically, in this work we find that layperson discernment is unaffected by informing participants that their responses will influence ranking algorithms: While this knowledge does indeed increase polarization of responses, these increases cancel out when calculating overall trust ratings" | Does not explicitly define a mechanism, but says that familiarity with a source is necessary but not sufficient for trusting that source | "How can social media platforms fight the spread of misinformation? One possibility is to use newsfeed algorithms to downrank content from sources that users rate as untrustworthy. But will laypeople be handicapped by motivated reasoning or lack of expertise, and thus unable to identify misinformation sites? And will they "game" this crowdsourcing mechanism in order to promote content that aligns with their partisan agendas? We conducted a survey experiment in which =984 Americans indicated their trust in numerous news sites. To study the tendency of people to game the system, half of the participants were told their responses would inform social media ranking algorithms. Participants trusted mainstream sources much more than hyper-partisan or fake news sources, and their ratings were highly correlated with professional fact-checker judgments. Critically, informing participants that their responses would influence ranking algorithms did not diminish these results, despite the manipulation increasing the political polarization of trust ratings." | https://doi.org/10.1145/3313831.3376232 | Yes | Experimental | Beliefs | Unclear | ||||||||||
89 | You’re Definitely Wrong, Maybe: Correction Style Has Minimal Effect on Corrections of Misinformation Online | Cameron Martel, Mohsen Mosleh, and David G. Rand | 2021 | Forthcoming in Media and Communication | Participants shown "up to 28 actual headlines that appeared on social media, half of which were factually accurate (real news) and half of which were entirely untrue (fake news)" | "After each headline, participants were asked whether or not they would share that article on social media publicly" After receiving a negative comment and being given the option to reply, participants reported perceived accuracy of the shared article | If choosing to share a fake news article, participants were presented with a comment received in response to the post which "varied by strength (direct, hedged) and depth (simple explanation, detailed explanation)" | 2,228 participants recruited via Lucid | Content labeling | "Our results suggest several conclusions about the effects of different styles of corrective messages on engagement with and replies to corrections of misinformation on social media. We find that hedged corrections are perceived as politer and less aggressive than direct corrections, and that hedged corrections result in a more positive perception of the corrector. Despite this, however, we do not find that hedged corrections are any more effective at eliciting replies to corrective messages, or promoting acceptance of corrective information. We consistently found no main effect of correction strength (direct, hedged) or explanatory depth (simple explanation, detailed explanation) on reply likelihood or reply sentiment." | "Our research also extends previous research on cognitive style and misinformation, which has found that people who are more reflective are less likely to believe false news headlines, and that deliberation causally reduces belief in false claims -- regardless of their partisan alignment... We found that analytic thinking and actively open-minded thinking (as assessed by CRT and AOT scales) predicted increased acceptance of corrective misinformation" | "How can online communication most effectively respond to misinformation posted on social media? Recent studies examining the content of corrective messages provide mixed results—several studies suggest that politer, hedged messages may increase engagement with corrections, while others favor direct messaging which does not shed doubt on the credibility of the corrective message. Furthermore, common debunking strategies often include keeping the message simple and clear, while others recommend including a detailed explanation of why the initial misinformation is incorrect. To shed more light on how correction style affects correction efficacy, we manipulated both correction strength (direct, hedged) and explanatory depth (simple explanation, detailed explanation) in response to participants from Lucid (N = 2,228) who indicated they would share a false story in a survey experiment. We found minimal evidence suggesting that correction strength or depth affects correction engagement, both in terms of likelihood of replying, and accepting or resisting corrective information. However, we do find that analytic thinking and actively open-minded thinking are associated with greater acceptance of information in response to corrective messages, regardless of correction style. Our results help elucidate the efficacy of user-generated corrections of misinformation on social media. | https://psyarxiv.com/w3tfb/download | Yes | Simulated social media | Intended action/behavior | General social media use | Yes | ||||||||||
90 | Prebunking interventions based on “inoculation” theory can reduce susceptibility to misinformation across cultures | Jon Roozenbeek, Sander Van Der Linden, Thomas Nygren | 2020 | Harvard Kennedy School (HKS) Misinformation Review | Exposure to fake news Twitter posts and "credible" news items before and after playing the Bad News game | Participants rated reliability of Twitter posts after playing the game | 5,061 participants recruited "by driving traffic through media reports linking to the game, as well as through promotional activities conducted by the Foreign and Commonwealth Office, Uppsala University, and local collaborating media literacy organizations" | Disinformation literacy | "We find significant and meaningful reductions in the perceived reliability of manipulative content across all languages, indicating that participants’ ability to spot misinformation significantly improved. Relevant demographic variables such as age, gender, education level, and political ideology did not substantially influence the inoculation effect." | "Previous work has shown that inoculation theory can be used to combat various types of online misinformation, for example about conspiracy theories (Banas & Miller, 2013), climate change (van der Linden et al., 2017), and immigration (Roozenbeek & van der Linden, 2018). Crucially, recent work indicates that online games that rely on inoculation theory can be effective at conferring psychological resistance against misinformation strategies rather than just individual examples of misleading information (Roozenbeek & van der Linden, 2019). We have dubbed this approach prebunking." | "This study finds that the online “fake news” game, Bad News, can confer psychological resistance against common online misinformation strategies across different cultures. The intervention draws on the theory of psychological inoculation: analogous to the process of medical immunization, we find that “prebunking,” or preemptively warning and exposing people to weakened doses of misinformation, can help cultivate “mental antibodies” against fake news. We conclude that social impact games rooted in basic insights from social psychology can boost immunity against misinformation across a variety of cultural, linguistic, and political settings." | https://misinforeview.hks.harvard.edu/article/global-vaccination-badnews/ | No | Simulated social media | Observed real-world behaviors | |||||||||||||
91 | Correcting Citizens’ Misperceptions about non-Western Immigrants: Corrective Information, Interpretations, and Policy Opinions | Frederik Juhl Jørgensen and Mathias Osmundsen | 2020 | Journal of Experimental Political Science | Participants reported "best estimates" of certain immigrant facts, then some received corrective information | Participants re-reported best estimates of immigration facts, immigration policy opinions, and how they interpreted immigration facts | Participants did or did not receive corrective information | A nationally representative sample of Danes | Disinformation disclosure | "In this study, we examined whether corrective information about the consequences of immigration could alter natives’ immigration policy opinions. We sought to replicate and extend prior work by moving the analysis to a new country (i.e., Denmark) and by examining reactions to different types of immigrant information (i.e., welfare dependency rates, crime rates) about a specific and hotly contested immigrant group (i.e., non-Western immigrants). Importantly, we also examined the information processing strategies people can use when exposed to correct information. Our findings support the conclusions from earlier work that people update their factual beliefs in light of correct information but fail to change their policy views." | "We demonstrate that the link between facts and policy beliefs break down because people interpret information in a belief-consistent manner" | "Can corrective information change citizens’ misperceptions about immigrants and subsequently lead to favorable immigration opinions? While prior studies from the USA document how corrections about the size of minority populations fail to change citizens’ immigration-related opinions, they do not examine how other facts that speak to immigrants’ cultural or economic dependency rates can influence immigration policy opinions. To extend earlier work, we conducted a large-scale survey experiment fielded to a nationally representative sample of Danes. We randomly expose participants to information about non-Western immigrants’ (1) welfare dependency rate, (2) crime rate, and (3) proportion of the total population. We find that participants update their factual beliefs in light of correct information, but reinterpret the information in a highly selective fashion, ultimately failing to change their policy preferences." | https://doi.org/10.1017/XPS.2020.35 | Yes | Experimental | Knowledge | Yes | |||||||||||
92 | Countering misinformation via WhatsApp: Preliminary evidence from the COVID-19 pandemic in Zimbabwe | Jeremy Bowles, Horacio Larreguy, Shelley Liu | 2020 | PLoS One | Exposure to WhatsApp messages with information about COVID-19 or debunking COVID-19 misinformation | Surveyed effects on "1) knowledge of the information disseminated in the messages, and 2) behavior relating to social distancing" | Randomized timing of messages on WhatsApp | WhatsApp users in Zimbabwe | Disinformation disclosure | "In sum, our results indicate encouraging positive changes in knowledge and behavior among WhatsApp subscribers of a trusted source. While WhatsApp has been identified as a platform through which misinformation easily spreads, we show that trusted CSOs can also leverage WhatsApp’s reach to successfully get individuals to reassess their misconceptions and correct related behavior." | "In conjunction with the experimental results we present above, this evidence suggests that a trusted source of information can use the same social media channels to disseminate information that both combats misinformation and changes related behavior. Thus, while we caution generalizing our results to general public in Zimbabwe, our results speak specifically to the important role that trusted sources play, particularly in confusing informational situations such as health crises, and in an authoritarian context where trust in information might be low" | "We examine how information from trusted social media sources can shape knowledge and behavior when misinformation and mistrust are widespread. In the context of the COVID-19 pandemic in Zimbabwe, we partnered with a trusted civil society organization to randomize the timing of the dissemination of messages aimed at targeting misinformation about the virus to 27,000 newsletter WhatsApp subscribers. We examine how exposure to these messages affects individuals’ beliefs about how to deal with the virus and preventative behavior. In a survey of 864 survey respondents, we find a 0.26σ increase in knowledge about COVID-19 as measured by responses to factual questions. Through a list experiment embedded in the survey, we further find that potentially harmful behavior—not abiding by lockdown guidelines—decreased by 30 percentage points. The results show that social media messaging from trusted sources may have substantively large effects not only on individuals’ knowledge but also ultimately on related behavior." | https://doi.org/10.1371/journal.pone.0240005 | No | Real-world social media | Observed real-world behaviors | Yes | |||||||||||
93 | Sex Trafficking, Russian Infiltration, Birth Certificates, and Pedophilia: A Survey Experiment Correcting Fake News | Ethan Porter, Thomas Wood, and David Kirby | 2018 | Journal of Experimental Political Science | Exposure to fake news stories and corrections | Participant agreement with the position advanced by the fake news story | Presence of a correction on a fake news story | 2,742 Mechanical Turk participants | Disinformation disclosure | Content labeling | "On every issue, corrected subjects on average became significantly less convinced by the fake news story. Corrections improved accuracy overall, even among those ideological cohorts who had a clear political interest in a fake news story… To be sure, there was some evidence of differential response to corrections by ideology. Furthermore, uncorrected subjects were credulous of the claims made by the fake stories" | No specific posited mechanism. Says that "upon seeing a correction, Americans are willing to disregard fanciful accounts and hew to the truth" | "Following the 2016 U.S. election, researchers and policymakers have become intensely concerned about the dissemination of “fake news,” or false news stories in circulation (Lazer et al., 2017). Research indicates that fake news is shared widely and has a pro-Republican tilt (Allcott and Gentzkow, 2017). Facebook now flags dubious stories as disputed and tries to block fake news publishers (Mosseri, 2016). While the typical misstatements of politicians can be corrected (Nyhan et al., 2017), the sheer depth of fake news’s conspiracizing may preclude correction. Can fake news be corrected?" | https://doi.org/10.1017/XPS.2017.32 | Unclear | Experimental | Beliefs | Yes | ||||||||||
94 | Winning the war on state-sponsored propaganda: Gains in the ability to detect disinformation a year and a half after completing a Ukrainian news media literacy program | Erin Murrock, Joy Amulya, Mehri Druckman, and Tetiana Liubyva | 2018 | International Research and Exchanges Board (IREX) | Some participants received a disinformation literacy training, then participants analyzed an objective news story and a false news story | Scores on analysis of objective and fake news stories | Treated participants received a Learn to Discern (L2D) curriculum | 412 Ukrainian participants | Disinformation literacy | "Compared to the control group, L2D participants had better disinformation news media analysis skills and more knowledge of the news media environment compared to the general population a year and a half after the end of the training. L2D participants also had a slightly greater sense of control over how they are influenced by the media as measured by media locus of control." | "Thus, the lack of a difference between the L2D and control groups on the objective news media assessment could be interpreted as a natural consequence of greater skepticism by those who are more news media literate. This may be the result of the fact that the L2D training emphasized deconstructing deliberately manipulative news stories rather than discerning the markers of more objective news reporting." | "In this context, IREX (International Research and Exchanges Board) designed and implemented Learn to Discern (L2D),4 a “demand-side” response to the problem of manipulative information, an essential companion to “supply-side” solutions such as supporting independent, ethical, and truthful journalism. Citizens must be able to separate fact from fiction, recognize manipulation and hate speech, and demand and seek out independent, fact-based journalism. From October 2015 through March 2016, IREX implemented L2D with funding from the Canadian government and in partnerships with local organizations Academy of Ukrainian Press and StopFake. Through intensive skill-building seminars, L2D reached more than 15,000 people of all ages and professional backgrounds. Remarkably, L2D also reached more than 90,000 people indirectly: direct participants shared what they learned with family, co-workers, and peers. Accompanying public service announcements and billboard messages alerting Ukrainian citizens to the danger of fake news reached an estimated 2.5 million people." | Published by a nonprofit, not in a journal Treatment involved a curriculum designed by IREX on understanding information and propaganda, particularly in the Ukrainian context | https://www.irex.org/sites/default/files/node/resource/impact-study-media-literacy-ukraine.pdf | Unclear | Experimental | Observed real-world behaviors | |||||||||||
95 | Fake news game confers psychological resistance against online misinformation | Jon Roozenbeek and Sander Van Der Linden | 2019 | Palgrave Communications | Participants played a 'Bad News' game | "Respondents’ ability to recognise misinformation strategies in the form of misleading tweets and news headlines" | Used a within-subjects design without variation; included 6 questions answered before and after game, 2 of which were controls | Participants recruited through a university press release and news outlets who opted in to the study | Disinformation literacy | "We find preliminary evidence that the process of active inoculation through playing the Bad News game significantly reduced the perceived reliability of tweets that embedded several common online misinformation strategies." | "The observation that participants rated both control questions as roughly equally reliable before and after playing the game underlines this point: it shows that active inoculation does not merely make participants more skeptical, but instead trains people to be more attuned to specific deception strategies. Or, to continue the metaphor, the psychological “vaccine” only activates specific “antibodies”. Achieving this is much more challenging because participants are tasked with using active reasoning to recognise a range of common misinformation strategies in different contexts (rather than just retrieving facts from memory)." | "The spread of online misinformation poses serious challenges to societies worldwide. In a novel attempt to address this issue, we designed a psychological intervention in the form of an online browser game. In the game, players take on the role of a fake news producer and learn to master six documented techniques commonly used in the production of misinformation: polarisation, invoking emotions, spreading conspiracy theories, trolling people online, deflecting blame, and impersonating fake accounts. The game draws on an inoculation metaphor, where preemptively exposing, warning, and familiarising people with the strategies used in the production of fake news helps confer cognitive immunity when exposed to real misinformation. We conducted a large-scale evaluation of the game with N = 15,000 participants in a pre-post gameplay design. We provide initial evidence that people’s ability to spot and resist misinformation improves after gameplay, irrespective of education, age, political ideology, and cognitive style." | https://doi-org.ezproxy.princeton.edu/10.1057/s41599-019-0279-9 | No | Experimental | Observed real-world behaviors | ||||||||||||
96 | A Longitudinal Study of Corrective Advertising | Robert Dyer and Phillip Kuehl | 1978 | Journal of Marketing Research | Exposure to corrective advertising | Belief-based measures related to deceptive claims from Listerine ads | Seemingly no manipulated variation | Four national samples of adult mouthwash users | Disinformation disclosure | "A reduction of approximately 20 percent in the level of deception due to the corrective advertising campaign" | No clearly posited mechanism. State that "it is clear that correcting false beliefs is difficult, particularly when the false beliefs are long-held or when personal experience may be perceived as supporting the false belief" | "The purpose of the current study was to explore the impact of the recent Listerine corrective advertising campaign. More specifically, the study sought to determine to what extent, if any, the corrective effort reduced salient deception concerning the ability of Listerine to ameliorate, prevent, and cure colds. The study also explored the role of individual differences on the impact of the corrective advertising and the effectiveness of various measures used in the study. If individual differences have a significant impact on the effectiveness of corrective advertising, public policymakers might have to consider a market segmentation approach to remedying de- ceptive advertising. Different segments might require different corrective approaches or emphases." | https://www.jstor.org/stable/3150399 | No | Real-world media | Beliefs | Yes | |||||||||||
97 | Tracing the Boundaries of Motivated Reasoning: How Deliberative Minipublics Can Improve Voter Knowledge | Kristinn Már and John Gastil | 2019 | Political Psychology | Receipt of a Citizens' Statement (CIR) from a group of voters about a ballot measure | Participants assessed claims on factuality | Treatment received CIR, control did not | Survey participants from Jackson County, Oregon | Disinformation literacy | "Overall, respondents who read the CIR Statement were significantly less likely to be incorrect and more confidently picked the correct answers. Importantly, across all eight factual items, there was no instance in which reading the CIR Statement led to significantly more bias." | "We found no evidence of partisan motivated reasoning of any sort. Exposure to the CIR statement improved factual knowledge among both partisan groups on a number of relevant items, although not universally. More importantly, we found instances in which those who if directionally motivated should have resisted the new information the most were in fact those who improved the most. This suggests that the deliberative minipublic's statement induced an accuracy motivation." | "A large body of work shows that reasoning motivated by partisan cues and prior attitudes leads to unreflective decisions and disparities in empirical beliefs across groups. Surprisingly little research, however, has tested the limits of motivated reasoning. We argue that the publicly circulated findings of deliberative minipublics can spark a more reflective motivation in voters when these bodies provide policy‐relevant factual information. To test that proposition, we conducted a survey experiment using information generated by one such minipublic during an election. Results showed that exposure to the minipublic's findings improved the accuracy of voters' empirical beliefs regarding a ballot proposition on the regulation of genetically modified seeds. This treatment effect transcended voters' partisan identities and prior environmental attitudes. In some instances, the respondents showing the greatest knowledge gains were those who a directional motivated‐reasoning account would have expected to resist the treatment most effectively, owing to party identity or prior attitudes." | Find that information derived from a group could cut through partisan motivated reasoning to improve factual accuracy | https://doi.org/10.1111/pops.12591 | No | Real-world events | Beliefs | |||||||||||
98 | A Picture Paints a Thousand Lies? The Effects and Mechanisms of Multimodal Disinformation and Rebuttals Disseminated via Social Media | Michael Hameleers, Thomas E. Powell, Toni G.L.A. Van Der Meer, and Lieke Bos | 2020 | Political Communication | Subjects exposed to messages about school shootings and refugees in the form of tweets | Participants' perceived credibility of disinformation | "2 (Disinformation: text versus multimodal Twitter post) x 2 (Source: ordinary citizen versus news agency) x 3 (Rebuttal: text versus multimodal versus absent)" | U.S. participants recruited by Survey Sampling | Other | "First of all, there is partial evidence that multimodal disinformation was perceived as slightly more credible than textual disinformation… Across topics, the presence of textual and visual fact checkers resulted in lower levels of credibility, and these effects hold both for congruent and incongruent issue publics." | "This connects to visuals’ quality of indexicality (Messaris & Abraham, 2001). Words are abstract symbols that need to be reconstructed into a mental image of reality (Grabe & Bucy, 2009). Visuals, in contrast, offer a direct index of reality (Messaris & Abraham, 2001) and, compared to textual disinformation, the use of visuals lowers citizens’ suspicion of being presented with a manipulated reconstruction of reality. The most pressing threat to communicative truthfulness may thus come from the manipulation of visual information." | "Today’s fragmented and digital media environment may create a fertile breeding ground for the uncontrolled spread of disinformation. Although previous research has investigated the effects of misinformation and corrective efforts, we know too little about the role of visuals in disinformation and fact checking. Against this backdrop, we conducted an online experiment with a diverse sample of U.S. citizens (N = 1,404) to investigate the credibility of textual versus multimodal (text-plus-visual) disinformation, and the effects of textual and multimodal fact checkers in refuting disinformation on school shootings and refugees. Our findings indicate that, irrespective of the source, multimodal disinformation is considered slightly more credible than textual disinformation. Fact checkers can help to overcome the potential harmful consequences of disinformation. We also found that fact checkers can overcome partisan and attitudinal filters – which points to the relevance of fact checking as a journalistic discipline." | https://doi.org/10.1080/10584609.2019.1674979 | No | Simulated social media | Beliefs | ||||||||||||
99 | Counting the Pinocchios: The effect of summary fact-checking data on perceived accuracy and favorability of politicians | Alexander Agadjanian, Nikita Bakhru, Victoria Chi, Devyn Greenberg, Byrne Hollander, Alexander Hurt, Joseph Kind, Ray Lu, Annie Ma, Brendan Nyhan, Daniel Pham, Michael Qian, Mackinley Tan, Clara Wang, Alexander Wasdahl, Alexandra Woodruff | 2019 | Research & Politics | Exposure to various kinds of fact checks attributed to different politicians or a control condition | Measured "favorability toward McConnell or Reid, perceived accuracy of that senator, and favorability toward fact-checking" | Study 1: "3 × 2 between-subjects design that randomly varied fact-check type and politician partisanship" | U.S. participants from Amazon's Mechanical Turk | Disinformation disclosure | Content labeling | "The results of this study confirmed that summary fact-checking is more effective at influencing the perceived accuracy and favorability of selected politicians than is individual fact-checking. These results did not vary by party or other preregistered moderators." | "Our theoretical expectations were that people would be less likely to dismiss a falsehood as an isolated incident and would instead view a politician’s behavior as more problematic when presented with summary data, which offers stronger evidence of a pattern of inaccuracy.1 Exposure to summary fact-checking might promote greater updating of respondent views toward a candidate compared to a fact-check of an individual statement through various mechanisms. These include a memory-based “running tally” (e.g., Fiorina, 1981) in which candidate inaccuracy is more likely to be registered as a negative consideration, online processing of negative affect inspired by information about a sustained record of inaccuracy (e.g., Lodge and Taber, 2005), or a Bayesian process in which more information about past inaccuracy leads to greater updating of candidate attitudes (Zechman, 1979)." | "Can the media effectively hold politicians accountable for making false claims? Journalistic fact-checking assesses the accuracy of individual public statements by public officials, but less is known about whether this process effectively imposes reputational costs on misinformation-prone politicians who repeatedly make false claims. This study therefore explores the effects of exposure to summaries of fact-check ratings, a new format that presents a more comprehensive assessment of politician statement accuracy over time. Across three survey experiments, we compared the effects of negative individual statement ratings and summary fact-checking data on favorability and perceived statement accuracy of two prominent elected officials. As predicted, summary fact-checking had a greater effect on politician perceptions than individual fact-checking. Notably, we did not observe the expected pattern of motivated reasoning: co-partisans were not consistently more resistant than supporters of the opposition party. Our findings suggest that summary fact-checking is particularly effective at holding politicians accountable for misstatements." | https://doi.org/10.1177%2F2053168019870351 | Unclear | Experimental | Beliefs | Yes | ||||||||||
100 | Checking and Sharing Alt-Facts | Emeric Henry, Ekaterina Zhuravskaya, and Sergei Guriev | 2020 | SSRN Scholarly Paper | Participants shown "two misleading statements about the European Union (EU) made by the leaders of the far-right party Rassemblement National (RN), with links to the exact source," then varying forms of fact-checks | "Participants had the opportunity to share the alt-facts—the two statements by RN leadership—on their Facebook pages" | "One-third of the subjects, randomly drawn, were exposed to fact-checking information compiled from media sources (we refer to this treatment group as Imposed Fact-Check) while another third was given the choice of accessing or not accessing this fact-checking information (Voluntary Fact-Check treatment group). The remaining third (Alt-Facts treatment group) was not shown fact-checking information and was not given an option to access it." | 2,537 French voting-age Facebook users | Disinformation disclosure | Content labeling | "The central result of our paper comes from the comparison of sharing behavior among the three groups of participants: those who were exposed to alt-facts only, those who were exposed to alt-facts and fact-checking, and those who were given an option to view fact-checking after being exposed to alt-facts. We find that both voluntary and imposed fact-checking substantially and significantly reduce sharing of alt-facts" | "One of our central results is that exposure to fact-checking significantly reduces sharing of altfacts. In this section, without claiming to be exhaustive, we discuss several possible explanations for this effect. First, it’s possible that such exposure changes the internal costs of sharing, and in particular the moral cost of sharing false information. There is an extensive literature on the moral cost of lying (see Abeler et al., 2019, for a survey). If fact-checking increases the perceived likelihood that the news is false, the sharer could face a higher moral cost of sharing something potentially incorrect Sperber and Baumard (see, for instance, 2012). Second, it could be that such exposure increases the salience of accuracy and therefore affects the decision to share (Pennycook et al., 2020b). This interpretation could also explain why Nonviewers in the Voluntary Fact-Check treatment group share alt-facts at a lower rate compared to what is predicted by their characteristics: being offered the possibility to view the fact-checking content makes the issue of accuracy more salient." | "Using an online randomized experiment in the context of the 2019 European elections campaign in France, we study how fact-checking affects sharing of false news on social media. We exposed a random sample of French voting-age Facebook users to statements on the role of the European Union made by the far-right populist party Rassemblement National. A randomly selected subgroup of participants was also presented with factchecking of these statements; another subgroup was offered a choice whether to view the fact-checking information. Then, all participants could choose whether to share the false statements on their Facebook pages. We show that (i) both imposed and voluntary fact-checking reduce sharing of false statements by about 45%; (ii) the size of the effect is similar between imposed and voluntary fact-checking; and (iii) each additional click required to share false statements substantially reduces sharing. These results inform the debate about policy proposals aimed at limiting propagation of false news on social media." | Study conducted on French Facebook users | https://dx.doi.org/10.2139/ssrn.3597191 | No | Simulated social media | Intended action/behavior | Yes | Yes |