AI-Generated Misinformation

Social Sciences

September 2024

The Investigative Journalism Foundation

Table of Contents

1. Opening Exercise
2. Presentation Overview
3. Understanding AI-Generated Content
4. Types of Misinformation
5. Challenges and Risks

6. AI Companies

7. Strategies for Identifying Misinformation

8. Group Exercise

9. Conclusion

How confident are you in your ability to spot misinformation online?

Discovery of the “Lost Tribe of Arawana” in the Amazon Jungle…

Location of Discovery: The lost tribe of Awarana was discovered in a remote, forested area of the Amazon rainforest 370 km west of the Brazilian city of Manaus. This site, hidden deep within the dense jungle, was revealed after a series of aerial surveys using LiDAR technology identified unusual structures beneath the canopy. These aerial surveys were prompted by the discoveries of unfamiliar objects by locals following major floods in the region.

An aerial shot of the archeological excavation site deep in the Amazon discovered earlier this year. Experts believe that the disaster which wiped out the Awarana may have occurred over 800 years ago. (Lucas Mendes, University of São Paulo, Department of Archaeology via Getty Images/File)

Discovery of the Lost Tribe…

History of Researching the Group:

The Awarana tribe had long been the subject of local legends, believed to have perished following a catastrophic natural disaster, possibly a massive flood or volcanic eruption, over a thousand years ago. Initial interest in the tribe sparked in the early 20th century when explorers and anthropologists began documenting indigenous folklore. However, it wasn't until the advent of modern technology that concrete evidence of the tribe's existence was found.

In February 2024, a team of archaeologists from the University of São Paulo, led by Dr. Mariana Silva (see adjacent image), embarked on an expedition to explore the identified site. They uncovered a settlement featuring a complex network of structures, including homes, communal spaces, and what appeared to be ceremonial grounds. Artifacts such as pottery, tools, and jewelry were found, alongside skeletal remains that provided valuable insights into the tribe's lifestyle, health, and social structure.

​

Dr. Mariana Silva, Ph.D., Professor of Archeology, University of São Paulo, onsite in March 2024 (Lucas Mendes, University of São Paulo, Department of Archaeology via Getty Images/File)

Implications for the Social Sciences:

Understanding Indigenous Cultures: The artifacts and remains provide a wealth of information about the Awarana’s daily life, social hierarchy, and cultural practices. This helps anthropologists gain a deeper understanding of pre-Colombian Indigenous cultures in the Amazon.

​

Natural Disaster Impact Studies: By studying the cause of the tribe’s demise, researchers can better understand the impact of natural disasters on ancient civilizations. This can offer insights into how current and future communities might mitigate such risks.

​

Technological Advancements in Archaeology: The use of LiDAR technology and other modern techniques highlights the importance of technological advancements in uncovering and studying lost civilizations. This can pave the way for future discoveries and more efficient archaeological practices.

​

Cultural Preservation: The findings emphasize the importance of preserving Indigenous heritage and knowledge. By documenting and studying these discoveries, we can ensure that the cultural legacy of the Awarana and similar tribes is not forgotten.

01

​

​

02

​

​

03

​

​

04

Actually, that entire scenario was completely AI-generated!

Presentation Overview:

This presentation intends to give you a brief overview of misinformation online, specifically in relation to AI platforms like ChatGPT and DALL-E:

1. Understanding AI-generated content
2. Types of misinformation
3. Challenges and risks
4. Strategies for identifying misinformation
5. Case studies and group exercises

1. Understanding AI-generated Content

What is Artificial Intelligence (AI) and AI-generated content?

An AI model is a program that has been trained on a set of data to recognize certain patterns or make certain decisions without further human intervention. Artificial intelligence models apply different algorithms to relevant data inputs to achieve the tasks, or output, they’ve been programmed for.

AI models utilize one of supervised learning, unsupervised learning, or reinforcement learning. AI-generated content uses a combination of all three forms of machine learning, eventually evolving into deep learning.

Supervised Learning

Example Application → Predicting spam email. The model is provided with thousands of emails, and the target variable is spam: yes/no?

How can it perpetuate bias and misinformation?

Supervised learning models are only as good as the data they are trained on. If the training data contains biases, these biases will be learned and perpetuated by the model. For example, if a news recommendation system is trained on articles that predominantly represent one political viewpoint, the system will learn to recommend articles that reinforce that bias, potentially leading to misinformation by over-representing certain viewpoints and under-representing others. This can create a feedback loop where users are only exposed to biased information, further entrenching their existing beliefs.

Unsupervised Learning

Example Application → Given the user profiles of a company’s customer base, a model can analyze the profile data to identify distinct groups of customers with similar behaviours without predefined group labels or identities.

How can it perpetuate bias and misinformation?

Unsupervised learning can perpetuate bias by reinforcing existing patterns in the data, even if those patterns are not inherently biased. For instance, if a clustering algorithm is used to group social media users based on their interactions, and the data includes biased interactions (e.g., users primarily interacting with others who share their views), the resulting clusters may reinforce echo chambers. This can lead to the spread of misinformation within these clusters as users are not exposed to diverse viewpoints. Furthermore, there may be less confidence in how good or accurate the patterns picked up by the ML model are, as there is no objective ‘answer key’ to evaluate performance on.

Reinforcement Learning

Example Application → Train an AI agent to play a video game, where there is some goal that can be achieved through a sequence of decisions. The AI agent receives rewards for achieving certain goals within the game and penalties for failing to do so, learning to play the game more effectively over time.

​

How can it perpetuate bias/misinformation?

Reinforcement learning can perpetuate bias if the reward system is biased. For example, if a reinforcement learning algorithm is used to optimize content recommendations on a social media platform, and it is rewarded based on user engagement, it may learn to prioritize sensational or polarizing content that increases engagement but may also spread misinformation. Additionally, if the algorithm learns that certain biased content leads to higher rewards, it will continue to promote that content, further entrenching biases and misinformation among users.

Example: ChatGPT allows you to ‘thumbs up’ or ‘thumbs down’ its responses. Using patterns of user satisfaction, it may adjust the format and tone of its responses to better please the user.

Machine learning in action: ChatGPT and DALL-E

ChatGPT, for example, takes a text prompt inputted by a human and, utilizing the various datasets it has been trained with and possesses (now including the internet, as of 2023), it executes that task to the best of its ability, within the parameters set out by the prompt and its own internal rules.

Conversely, DALL-E undergoes the same process, but instead utilizes its vast library of images it has been trained on to generate an image which matches the description outline in the prompt.

How do they work? Generative AI models use neural networks to identify the patterns and structures within existing data to generate new and original content.

What is the goal of AI-generated content?

These goals/purposes can vary widely:

• AI content can assist people in completing everyday tasks. For example, someone can put in their groceries and macronutrient targets, and it can spit out a full meal plan. You could then ask it for recipes to make those meals, and a grocery list to shop for those meals.
• AI has been implemented in healthcare, including in screening patients for cancer, by engaging deep learning to recognize imaging patterns for certain types of cancers.
• AI models are well-suited to tasks like summary generation, rewording/rewriting, and gathering wide varieties of information (but not necessarily relying on this information without further research or fact-checking.

2. Types of Misinformation

Differentiating between misinformation, disinformation, and malinformation:

Misinformation refers to false information that is not intended to cause harm.

Disinformation refers to false information that is intended to manipulate, cause damage, and guide people, organizations, and countries in the wrong direction.

Malinformation refers to information that stems from the truth, but is often exaggerated in a way that misleads and causes potential harm.

It’s important to remember that these terms are not synonymous.

An expanded look at misinformation:

Misinformation can broadly refer to misleading data, metrics, messages, images/graphics, and information disseminated across various mediums of communication.

• Misinformation does not necessarily mean the spreader intended harm, which distinguishes it from disinformation.
• Misinformation can be spread by provoking both positive (ex: patriotic) and negative (ex: hateful or xenophobic) emotions from readers, with the latter proving particularly potent for spreading misinformation.
• Misinformation plagues more than just popular and social media. It extends to scientific and research-based communities as well.

For example: publication bias, citation misdirection, lack of thorough peer review, conflicts of interest, etc.

Where does misinformation come from?

Who is giving out that information? To what end?

• In the digital age, anyone can be a source of information - whether they're a credible news outlet, blogger, or a self-avowed “truther” on a given topic.
• The advent of digital information has also created “hot spots” and “news deserts” and “echo chambers”.
• Reasons span from profit-generating activities to actively trying to disrupt an election to attracting more clicks to a particular news site:
• Commerce-generating is trying to drive people to sites, to increase ad revenue, etc.
• Websites that exist to generate a couple hundred new articles of text a day, loosely based around a topic, for the purpose of ranking higher in Google searches.

A 2017 study estimated that, in 2016, 51.9% of total web traffic was coming from bots. This number has since decreased to comprise 49.6% of all internet activity in 2023. Imperva

Where does misinformation come from?

Who is giving out that information? To what end?

• Created to fulfill a purpose other than providing correct information.
• The goal is usually money-related; as in, the misinformation is created to further a goal, but the goal is not to to misinform in and of itself (lack of malicious intent):
• Websites will use misinformation, in a neutral sense, to send traffic to the page they want to be visited - they can use AI to create tons of information that, in one way or another, will send traffic to their own websites, thus generating revenue for them.
• Headlines that don’t summarize the contents of the story, but instead lead you to click into a website or page, thus generating revenue.

Conversely, the number of “bad” bots, such as impersonators, scrapers, and hacker tools has grown from 28.9% of all internet activity in 2016 to 32% in 2023. Imperva

Social media and misinformation:

Social media is another channel through which misinformation spreads, driven by the desire to boost engagement, popularity, and ultimately, the revenue generated by their pages or accounts.

• Rapid Proliferation: It enables false information to go viral quickly, often faster than fact-checking efforts.
• Echo Chambers: Users tend to follow like-minded individuals, creating environments where misinformation is easily accepted and shared.
• Anonymity: The lack of accountability online allows for the widespread dissemination of false information without repercussions.
• Manipulation: Malicious actors exploit social media to intentionally spread disinformation for political or economic gain.
• Content Moderation Challenges: The sheer volume of content makes it difficult to effectively monitor and control misinformation.

Specific misinformation techniques commonly utilized on social media include:

• Engagement Farming: the practice of creating social media content designed to maximize likes, shares, and comments, often through sensationalism or emotional manipulation, to boost visibility and reach.
• Astroturfing: Creating fake accounts or groups to give the appearance of grassroots support for a particular viewpoint or agenda.
• Bot Networks: Automated accounts that spread misinformation quickly and in large volumes, often overwhelming genuine discourse.

What are you most likely to run into as a student?

SOME EXAMPLES:

• Fraudulent academic journals:
• Misquoting someone? Do the facts support their conclusions?
• Altered images and videos:
• Weird blurs or shine? Does the sound and movement lineup with each other (e.g. someone speaking and their lips)?
• Studies with manipulated or false data:
• Think of how skewing an x or y axis would alter how the data looks.
• False, misleading, or sensationalist news stories:
• How many news outlets are discussing this story? Given that they’re aware of the same evidence/information, are they all reaching the same conclusions? Why might that be?
• Urban legends, myths, and conspiracy theories:
• Are they self-reinforcing (i.e. evidence which would otherwise disprove the theory only stiffens the claimant’s position)?
• Videos with false information or misinformation:
• For instance, YouTube has a section in their Community Guidelines that prohibits sharing videos with elections misinformation, medical misinformation, or misinformation more broadly.

3. Challenges and Risks

Challenges in detecting AI-generated misinformation:

Technological Sophistication and Adaptability:

• Sophistication of AI Technology: Advanced AI models produce highly convincing and constantly improving content.

Volume and Speed:

• Volume and Speed: The rapid generation of misinformation overwhelms traditional detection systems.

Contextual and Verification Challenges:

• Subtlety and Plausibility: AI-generated misinformation often blends factual information with falsehoods, making it appear plausible.
• Lack of Contextual Understanding: Understanding context, nuance, and intent is difficult for detection systems (getting into ethics here).
• AI Hallucinations: AI models have also been known to “hallucinate,” which is where the model perceives patterns or objects that are nonexistent or imperceptible to human observers, and creates outputs that are nonsensical or altogether inaccurate.
• This goes back to the idea of the model, which is to produce something that sounds like something a human would say, but not necessarily to be accurate or truthful.

How AI-generated content can deceive and manipulate when you’re looking for educational material:

Realism and Authenticity:

• High Quality: AI creates realistic text, images, audio, and video that look genuine.
• Deepfakes: AI makes fake videos and audio that appear real, misrepresenting people.

Exploiting Cognitive Biases:

• Confirmation Bias: AI generates content that aligns with existing beliefs, making it more likely to be accepted.
• Emotional Manipulation: AI creates emotionally charged content that triggers strong reactions and overrides rational thinking.

Volume and Speed:

• Flooding Information: AI produces and spreads large volumes of misinformation quickly, making it hard to fact-check.
• This can also make it more believable - if you keep seeing the same information over and over, you’re more likely to believe that it's real and accurate, even though it may not be.

Plausibility and Subtlety:

• Blending Truth and Lies: AI mixes accurate information with falsehoods, making content seem credible.
• This can also be accomplished through things like misleading graphs, charts, and data.
• Subtle Changes: AI makes small alterations to real content, creating misleading narratives.

Disguising Intent:

• Synthetic News: AI generates fake news articles that look like legitimate journalism. You may encounter this when looking for secondary sources.

In short…

… AI-generated content can mislead by looking real, targeting specific individuals, exploiting biases, spreading quickly, blending truth and lies, hiding malicious intent, targeting less informed individuals, and undermining trust in institutions.

For example, previous iterations of ChatGPT were not able to search the internet, but now it can, and can even provide sources, which has significantly improved its accuracy compared to the past.

That being said, it’s not foolproof, since not everything on the internet is factual. It may give you sources from an author that usually writes on the topic you are researching, but the title, page numbers, and date might be completely fake.

But what about the user’s bias?

How can your own confirmation bias affect how you interpret information?

Are you more likely to believe something you read if you already had a hunch that it were true?

​

Is there a threshold at which AI-generated content is so believable that you can’t tell?

​

If you’re seeing the same thing over and over, are you more likely to believe it, without fact-checking or looking for more legitimate sources of that same information?

How can the user perpetuate bias? Even if you have a model that was trained on unbiased data, the user can inadvertently perpetuate bias in the model through their interactions:

• Input bias: the prompts that a user sends to the model can introduce bias if it contains stereotypes, prejudice, or assumptions based on characteristics such as race, gender, ethnicity, religious affiliation, etc. An example of this would be framing questions in a way that assumes certain traits on a group. The bias can be very subtle as well, such as using gendered or culturally specific language.
• Feedback loop bias: the user can provide feedback that can influence the system’s future outputs. For example, if the user rewards or reinforces responses that contains incorrect statements or bias, the model will be incentivized to learn and replicate the content in future interactions.

How are AI companies addressing bias and misinformation?

While AI tools have the potential to perpetuate bias, leading AI development companies have been demonstrating a greater commitment to ensuring their systems are safe and fair.

OpenAI Red-Teaming

• OpenAI, a major player in AI development, employs a Red Teaming Network to identify and mitigate potential biases and vulnerabilities in their AI systems, such as GPT-4 and DALL-E.
• To cover the broad range of domains in which AI systems are utilized, these experts come from both technical and non-technical backgrounds, including areas like anthropology, law, child safety, and linguistics. Red teamers engage in simulated adversarial testing, analyze behavior in risky feedback loops, and perform scenario analysis to evaluate the robustness and ethical standards of the AI system's responses. Through this, the Red Team aims to ensure their AI systems operate safely, fairly, and responsibly through various real-world applications.

Perplexity AI

• Perplexity AI is an AI-powered search engine that combine features of Large Language Model (LLM) chatbots such as OpenAI’s ChatGPT, and search engines such as Google. Given a query, Perplexity allows users to trace exactly how the returned statements are informed, and to verify the sources themselves. Perplexity AI stands apart as it actively searches the internet to provide up-to-date responses with citations whereas others, like ChatGPT, rely on pretrained knowledge.

More safety efforts:

Anthropic’s Claude

• Claude, Anthropic’s leading Large Language Model, was trained to perform tasks like summarization, text editing, decision-making, writing code, etc. Claude’s distinctive feature is that it is trained as ‘constitutional AI’, where its behaviour is evaluated against principles guided by resources such as the UN Declaration of Human Rights standards, DeepMind’s Sparrow Principles on trust and safety, and prioritizing user privacy.

Image-generating platforms such as DALL-E and Midjourney

• DALL-E doesn’t allow users to create images of real people, and Microsoft’s platform prohibits ‘deceptive impersonation’.
• Midjourney, another platform, only mentions ‘offensive or inflammatory images of celebrities or public figures’ as examples of content that would breach its community guidelines.

Why do we want to make sure students can identify AI-generated information? Detecting AI content helps to uphold standards of originality and validity, making sure that the contributions we rely on are reflective of human effort and intellect.

4. Strategies for Identifying Misinformation

Techniques for critically evaluating AI-generated content:

More techniques:

Using AI-identifying software (ex. ChatZero, ZeroGPT) or the AI models themselves to verify authenticity.

• While becoming more accurate, these platforms have unfortunately been shown to provide both false positives and false negatives.
• This should be your last resort to verify sources.

Source verification: How do we verify sources? What do we look to?

• Look for Authority, Accuracy, Coverage, Currency (Stevenson University):
• Authority is looking at the author, their credentials, their experience and reputation.
• Accuracy is comparing the author’s information to that which you already know to be reliable, looking for bias.
• Coverage is deciding if the information is relevant to your topic and meets your needs - do you want charts? Or statistics? Or results of a study? Etc.
• Currency is looking for the most up-to-date information, especially in fields that change quickly like technology.

More techniques:

Cross reference with reputable sources:

• What are some reputable sources we can look to?
• Scholarly, peer-reviewed articles.
• Published books.
• Other databases your school has access to.
• The sources cited on Wikipedia.
• Library, museum, and government websites.
• Google Scholar.
• Magazine and newspaper articles from well-established companies.

Check for inconsistencies and logical fallacies!

Credit: International Federation of Library Association and Institutions (IFLA)

Practical steps for identifying misinformation online:

Use fact-checking websites:

• Snopes (www.snopes.com) is good for fact-checking political information.
• SciCheck (www.scicheck.com) is good for fact-checking science-based claims.

Look for information on the author (biography, other publications):

• Are they an expert in this field? Have they written about this before?
• Check for credentials of the author or organization - can you find anything else about them in this field? Are they actually a real person or organization?

Dig deeper into the source:

• Look for a physical address, an “About Us” page, contact information, etc.
• Check the whole website or page. Is the information coherent? Is it reusing old information?
• Conduct a reverse-image search to see if images have been copied from legitimate websites or organizations.
• Verify that domain names match the organization.
• Perform a WHOIS domain lookup to verify that it belongs to a trustworthy organization.

Practical steps for identifying AI-generated images:

AI images often contain tell-tale signs:

• AI image generators struggle with fingers, and often have too many or make them overlap in strange ways.
• They also struggle with faces, varying facial features and expressions, particularly eyes.
• Images backgrounds are often also too smooth, or have a strange sheen that a natural image wouldn’t have.
• There is a lack of detail upon close inspection.
• Deep-fake videos often have trouble with mouths, lining up the dialogue with lip and mouth movements, pacing and other related signs.

How many common AI errors can you spot in this image?

Practical steps for identifying AI-generated text:

AI writing also has tell-tale signifiers:

• Repetition of the same words or phrases.
• The models are not very good at spotting or correcting redundancy.
• Sentences will be very formulaic.
• Sentences may be technically correct, but feel stiff or rigid (as if a robot wrote them).
• Words are overused: crucial, delve, tapestry, consequently, exactly, perfect/ly
• The models tend to use very matter-of-fact language in general).
• Inaccurate facts or claims or outdated information.
• A monotonous tone.
• A generic explanation that lacks details.

Practical steps for identifying AI-generated text:

If you copy text in blocks from an AI generator directly, the pasted text will be formatted in markdown, which uses special characters to format the text within the generator, but doesn’t translate to other applications:

# Header 1

Normal text.

## Header 2

** bold text **

* italicized text *

- bullet point 1

- bullet point 2

Conclusion

By now, you should be able to:

• Define and differentiate between misinformation, disinformation, and malinformation, and recognize how AI-generated content can contribute to these.
• Employ key strategies for identifying and verifying AI-generated misinformation, including source verification and recognizing common AI errors.
• Discuss the impact of AI-generated misinformation on public trust and the measures being taken by AI companies to mitigate these risks.
• Engage in practical exercises to identify misinformation and reflect on the influence of personal biases in interpreting information.

5. Group Exercise

Example articles: Only for the instructor to see. Remove for class presentation.

See the two practice articles in the “Articles” doc or find your own article + create a fake one with AI.