Generative AI�How it works & applications for media orgs�Tim Olson, KQED tolson@kqed.org
http://bit.ly/48ccxIo
AGENDA
How foundation models work
Challenges - bias and hallucinations
Changes - media business & tools
Examples - media orgs applications
Discussion - talk with each other, headlines
HOW FOUNDATION MODELS WORK
Set goal
Lots of�data
Neural network
Train
Fine-tune
Identify an object in a photo
Given a sequence of text, guess what comes next
Predict the three-dimensional shapes of proteins
Set goal
Wikipedia, Reddit, Twitter, books, manuals, comments...
Photos
Amino acid sequences
Lots of�data
> data = model improves
A complex web of interconnected nodes (or “neurons”) that process and store information
Neural network
> compute = model improves
Break words into tokens - basic units than can be encoded.
Produce a vector (aka word embedding) (list of number values) -�- hundreds of values, each representing a different aspect of a words meaning
Neural network
Transformer model of neural network can analyze multiple pieces of tokens at the same time�(Chat GPT = Generative Pretrained Transformer)
Neural network
Financial Times - Generative AI exists because of the transformer
As it analyzes the data, token by token, it identifies patterns and relationships
Train
learns from the data itself
Develops a sense of context, it can also pick up other, unexpected abilities, such as knowing how to code
Train
Reinforcement learning with human feedback
Fine-tune
Retrieval Augmented Generation
Grounding
survey
website
medical
financial
legal
foundational model +�specific data set
Challenges
Data source - e.g. Reddit, web, articles… develop bias due to untruths, hate speech ..
Generate wrong info (hallucination) - generate (untruth) text not in the training data. Unlike classical computing where given the same inputs, you get the same output, Generative AI systems spin out multiple possibilities from a single prompt (given trillions of variables)
Generative is� non deterministic� (not traditional programming, results are probabilistic)�
Good for Poor for
Explore possibilities Reliable answer finder
Ideation Determining truth
Code development Decision maker
Data analysis A replacement for creativity
via: Toshi Anders Hoo, Institute for the Future
Cautions
AI makes mistakes
Be transparent about use of AI
Do not upload sensitive information e.g. proprietary data
�Do not publish any AI generated text without human editorial
�Appropriate: Data analysis, repetitive tasks, process improvement, first drafts, generating ideas
Not appropriate: Ethical concerns (e.g. promotion decisions), legal risks, privacy and security violation, discrimination risk, replace human interaction (e.g. difficult conversation)�
Nebraska Public Media Framework for Generative AI Experimentation
Principles for Using Generative A․I․ in The Times’s Newsroom | The New York Times Company
AI - THE NEXT BIG SHIFT
PC & Servers
Web & Internet
Cloud & Mobile
AI
Satya Nadella, Microsoft CEO - My annual letter: Leading in a new era
AI - THE NEXT BIG SHIFT
natural language (text)
vision
audio
foundation model
completing, summarizing, detecting anomalies, identify patterns, surface insights…
data
traffic, ocean currents, amino-acids, crop disease, radiology, law, stocks, star light spectrum…
vision
bird classifications
vision
blight
weed
soil
vision
tissue type
pneumothorax
skeletal anomaly
language
audio
Wendy’s menu
language
audio
crow vocal repertoires
trained on historical data to learn complex systems and generate projections, whereas numerical models use physics equations — and weather observations — to produce simulations of future conditions
GraphCast
weather
RFM-1
physics
space
measurement
language
audio
Physical AI�perceive, reason and act in�3D space and time
discovered more than 2.2 million hypothetical materials, including 381,000 stable new materials
microchips
batteries
photovoltaics
GNoME
inorganic crystals
generate biomolecular structure predictions containing proteins, DNA, RNA, ligands, ions
AlphaFold
protein structure
drug development�from discovery to engineering
TECH INVESTS TO BE THE AI “PLATFORM”
natural language (text)
vision
audio
foundation model
completing, summarizing, detecting anomalies, identify patterns, surface insights..
data
traffic, amino-acids, crop disease, radiology, law, stocks, star light spectrum
agent
agent
agent
agent
agent
ERNIE 4.0
GPT-4.o
Gemini
LLaMa-3.2
Mistral Large
Claude 3
Apple Intelligence
Invests in Anthropic
DOWNSIDES
Misinformation �- deep fakes, people stop trusting true information, the “liar’s dividend”
Security �- cyber attack, spoofing, microtargeting phishing…
Energy consumption�- training models and queries take more energy; though smaller models are being developed too
–
Jobs lost, added, & changed
Change is accelerating�
How do we help society understand and adapt?
How do we make the positive changes more equitable?
What will the market not address that society needs?
MEDIA�BUSINESS
AI driven intelligent devices
language
audio
vision
Transcription, translation, dubbing
Translation tipping point - Apple 17.4. podcast transcription, Samsung Galaxy S24 live translation
Localize to multiple markets - translation, captoning, subtitling, dubbing, lip synch..
language
audio
Personalized advertising
6,000 taglines
Project brief
Brand archetypes
Example taglines
location
weather
language
audio
character
companion
language
audio
Search
“Zero-click”
Links→Answers
�Threaded
Conversational
language
Summarization� at scale and personalized
AI chips on mobile device summarize news + for user
�Your “assistant” for calendar, events, news, traffic, weather
Sign
Sue
or
Grounding
Opinion | Ex-Google director: The real wolf menacing the news business is AI - The Washington Post �Jim Albrecht, senior director of news ecosystem products at Google from 2017 to 2023
“generative AI products tend to rely on a process known as “grounding,” in which the statements made by the AI are checked against relevant source documents to ensure that the AI is not making things up. This process is especially critical if a user is asking about a recent event in which the relevant facts did not exist at the time of the LLM’s training. In such cases, the AI can only answer accurately if it retrieves those facts from recent grounding documents. These documents are the essence of the work newspapers do — sourcing and reporting new facts — and the fruits of that labor should reasonably belong to those who perform it”
MEDIA�PRODUCTION
AI MEDIA PRODUCTION TOOLS
TEXT | IMAGE |
| |
AUDIO | VIDEO |
|
AI - YOUR ASSISTANT (co-pilot, creative assistant)
AI - YOUR ASSISTANT (co-pilot, creative assistant)
high school intern
college grad
postgrad grad
PH.D
?
Lead with our values
People first
WHAT SHOULD WE DO
Learn & play - understand and help our community understand, try it
Processes - how can this save our staff’s time, reduce mundane tasks?
Explore - how we might leverage the tools for public service?
LEARN
PLAY
labs.google.com/search
search
personalize chatbot
homework
gramar
lensa - images
research documents
images
PROCESS
Generate draft text
Analyse a document
Audio workflow
Video workflow
EXPLORE �EXAMPLES OF PUBLIC FACING APPLICATIONS IN JOURNALISM
Context aware search of audio archives
How KQED is enriching its 'Forum' archive with generative AI - Current �KQED
Audio�transcription�
Investigative journalism
(perception)
Investigative journalism
The California Reporting Project ��Law enforcement documents� - 7.5 TB of data (written, text, body cam video..)
Newsgathering collaboration - request and analyze records from law enforcement agencies to report on use of force and misconduct cases around the state
OCR
LLM C�check LLM B
Toss all data, keep structure & redo
Missing data
LLM A
LLM B
Image generation�Photoshop Generative Expand
Filter�Photoshop
Style�After Effects
Branding & color�After Effects
ChatBots based on publisher data
Content into other formats
NY Times Audio- automated voicing of articles
Opus Clip - vertical video shorts from long form horizontal video
Access, analysis and visualization of public data
Pulitzer Center´s Rainforest Investigations Network and Earthrise Media �
DISCUSSION
What are you seeing in your area?��How are you using it in your work?
http://bit.ly/48ccxIo
Tim Olson tolson@kqed.org