Evaluating AI's Ability to Perform Journalism Tasks

https://go.umd.edu/ai-journalism

Kind of terrible Image by DALL-E

Running with Scissors

Derek Willis

Lecturer in Data and Computational Journalism

Philip Merrill College of Journalism

University of Maryland

dpwillis@umd.edu

What is an AI Journalism Class, Anyway?

• 17 students of varying skill level (one non-journalism major)
• Access to multiple AI services (some paid, some free)
• Mostly web-based services (no technical requirements)
• Apply AI to individual journalism tasks

​

The Catch: what is a “right” answer?

Image by Gemini

Applying AI: Summarizing Ideas from a Report

• Use an evaluation template
• Describe the process
• Documentation is key

​

Evaluating AI

• Expectations vs. outcomes
• Is this helpful?
• Is it better than humans alone

Spare a thought for Team Luddite

More Tasks

• Story ideas
• Source archetypes
• Editing for style
• Extracting information
• Converting audio to text
• Describing images
• Data analysis
• Generate social copy

​

Our tools: the usual suspects, but new things are happening

​

​

Paid is (mostly) better

For now.

• ChatGPT (OpenAI)
• Claude (Anthropic)
• Gemini (Google)
• NotebookLM (Google)
• Llama3 (Meta)

​

Major Assignments

One solo, one group.

Solo: Adopt a news organization & draft policies/processes for its use of AI.

Group: Write a more general guide to accomplishing a specific set of tasks using AI.

Some Lessons

• AI is not a fact machine
• AI use is conversation-based
• Human role is critical
• Expectations usually don’t survive contact
• AI is trained on human work

• Chat-based interfaces are good, until they aren’t
• Generating good questions is the skill required for using AI
• Different AI systems work differently

​

Questions? Comments?

Email: dpwillis@umd.edu

Slides

Syllabus