CS 162: Natural Language Processing
Saadia Gabriel
Lecture 10:
Information Retrieval & Extraction Part 1
Announcements
Q & A
Before large-scale language modeling, BERT was better for discriminative tasks like MCQA and text classification. Now we can have massive generative models with large-scale pretraining that can perform most tasks well and are more flexible than a bidirectional model. BERT still often used when a lightweight encoder is needed.
The numbers that define the weights of pretrained models update based on information from the training data, so they implicitly encode semantics, world knowledge, etc.
Q & A
It depends on the complexity of the language task and test data, some are easier to model based on presence of keywords (e.g. sentiment analysis).
Yes! Next lecture we’ll learn about hidden Markov models (HMMs) and conditional random fields (CRFs), which are particularly useful for sequence tagging problems.
Goals of This Lecture
Recap:
LLMs make factual errors known as “hallucinations”
and they do this often.
Recap: Prompting Solution
Later on we’ll talk about retrieval-augmented generation as another method to improve accuracy!
Today is later…
Origins of Information Retrieval
- Manning, Schutze and Raghavan (2008)
- Manning, Schutze and Raghavan (2008)
Term-document boolean incidence matrix
Query: Which Shakespeare plays contain the words Brutus AND Caesar AND NOT Calpurnia?
Logical bitwise operations over word vectors:
Complement for NOT
Types of Retrieval Methods
Dense vector retrieval
Sparse retrieval, e.g. BM25
Hybrid retrieval
Look for semantically similar embeddings
Bag-of-words approach that rank document relevance based on keyword presence
A combination of dense and sparse approaches
What type of retrieval is the boolean example?
Types of Retrieval Methods
Dense vector retrieval
Sparse retrieval, e.g. BM25
Hybrid retrieval
Look for semantically similar embeddings
Bag-of-words approach that rank document relevance based on keyword presence
A combination of dense and sparse approaches
Retrieval-augmented Generation
Figure courtesy of Holt Skinner, Google I/O
Retrieval-augmented Generation
Query prompt
LLM
Knowledge cut-off: January 2025
Who is the president of UC?
🧑
Output
Michael Drake
As of August 2025:
Retrieval-augmented Generation
Query prompt
LLM
Knowledge cut-off: January 2025
Who is the president of UC?
🧑
Output
Michael Drake
As of August 2025:
We can fix this without making any changes to the underlying model!
Query prompt
Retrieval-augmented Generation
We can fix this without making any changes to the underlying model!
LLM
Who is the president of UC?
🧑
Knowledge cut-off: January 2025
Document Corpus
Embed query
Embed docs
Dense vector retrieval
Retrieve relevant doc based on a metric like cosine similarity
Output
James Milliken,
according to source
Example of retrieved passages from your text
Retrieval-Augmented Generation
Slides courtesy of Graham Neubig
Retrieval-Augmented Generation
Recall there are many types of retrieval methods
Slides courtesy of Graham Neubig
Retrieval-Augmented Generation
Dense retrieval
Slides courtesy of Graham Neubig
Retrieval-Augmented Generation
Learning retrieval-oriented embeddings
Slides courtesy of Graham Neubig
Retrieval-Augmented Generation
Simple retrieval + reading approach
Slides courtesy of Graham Neubig
Retrieval-Augmented Generation
Slides courtesy of Graham Neubig
Agentic RAG
RAG frameworks are now more adaptive due to integration with Agentic AI frameworks
LLM agents can work together to route queries to specialized external data sources
Agents can also refine queries and generate plans for how to optimally use retrieved evidence
QA Recap
Courtesy of Diyi Yang
Protosynthex system, which retrieved candidate answers based on term overlap
Dependency parsers were then used to see which answer mostly closely matched the question.
Textual QA
(Reading Comprehension)
Courtesy of Diyi Yang
Textual QA
(Reading Comprehension)
Courtesy of Diyi Yang
Open-domain QA
V
V
Courtesy of Diyi Yang
Courtesy of Diyi Yang
Where RAG Fits In
Where RAG Fits In
Courtesy of Diyi Yang
However, we still face challenges of how many documents to retrieve (context constraints) and inaccurate or hallucinated citations.
Information Extraction
A PubMed Article
What is this saying?
A PubMed Article
There is a lot of jargon.
Perhaps if we can automatically extract these domain-specific terms,
we can look them up and find an easier to understand description.
A PubMed Article
We refer to these technical terms as “entities.”
We are transitioning from unstructured knowledge…
We can assign types to the entities, like organizations (ORG)
We can also classify relationships between entities, like PART-WHOLE
… to structured knowledge
Structured Information Comes in Many Forms
Knowledge Graph
Database Table
We Can Use Entity Extraction for Classification
We Can Use Entity Extraction for Classification
Named Entities
Named Entity Recognition
Example of NER Output
One commonly used NER recognizer is Stanford NER (https://nlp.stanford.edu/software/CRF-NER.html), which is based on a CRF (another probabilistic model we’ll learn about these later).
Example Applications
Challenges of NER
What do you think the ambiguities are?
Is Washington a person, a sports team, a place, a political entity?
How do we know to segment “James Burroughs” as a single named entity?
Supervised Learning Setting
BIO Tagging
NER with BIO Tagging
https://www.google.com/url?sa=i&url=https%3A%2F%2Fwww.kaggle.com%2Fcode%2Fdevashishprasad%2Fbio-labeled-dataset%2Fnotebook&psig=AOvVaw2ImnUI01nLjBrguSeXuqba&ust=1747159874974000&source=images&cd=vfe&opi=89978449&ved=0CBcQjhxqFwoTCPDEqvHDno0DFQAAAAAdAAAAABAb
Hands-on Exercise
Pretrained language model for NER
https://cogcomp.seas.upenn.edu/page/demo_view/NEREnglish
Let’s check out a NER demo
What are the NER tags for the following sentences:
1 ) Ta-Nehisi Coates’ first book was The Beautiful Struggle.
2 ) On my way to Apple I dropped my iPhone in an apple pie.
Next Time
Student Presentations