CN408/SF323 AI Engineer

Lecture 4: Information Retrieval and Retrieval Augmented Generation

​

Nutchanon Yongsatianchot

​

News

https://blog.dynamicslab.ai/

News

https://x.com/cohere/status/1958542682890047511

News

https://www.reuters.com/business/meta-partners-with-midjourney-license-ai-tech-future-products-2025-08-22/

News: nano-banana is out!

https://developers.googleblog.com/en/introducing-gemini-2-5-flash-image/

Retrieval Augmented Generation

Many graphics are from https://www.deeplearning.ai/courses/retrieval-augmented-generation-rag/

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Motivation

• To answer questions, LLMs must have necessary information or knowledge
• LLMs only know what in their training data
• Many knowledge are outside of their training data
• Recent knowledge, private knowledge, novel knowledge
• LLMs will perform better with contexts or relevant information in the prompt.
• LLMs have limited memory (context window)

Solution: Providing Additional Information

Prompt

​

​

​

​

​

​

​

​

​

​

​

​

User’s Query

Additional Information

Instruction

We will need to retrieve!!

Many applications involve specialized knowledge

Legal or Medical use cases

Company chatbot

The Big Picture: Retrieval Augmented Generation (RAG)

Index

The center question of RAG:

How can we get all the right information to LLM?

Information Retrieval

Two board ways of retrieving information

• Sparse: Keyword Search
• Keyword Matching
• Bag-of-word Search

• Dense: Semantic Search
• Vector Database

Sparse Search

Keyword Matching

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Keyword Matching

• Idea: Database = A dictionary. You can simply look up the meaning of words.
• database = {‘CN101’: “วิชาโปรแกรมพื้นฐาน”,

‘CN240’: “วิชา data science”}

• query = “CN101 คือวิชาอะไร?”
• augmented_query = “CN101 คือวิชาอะไร? � <context>CN101 = วิชาโปรแกรมพื้นฐาน</context>”
• Good for searching document titles or article names
• Good for special words, unique words and novel words such as definitions, locations, etc.

Keyword Matching Demo

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_Lecture_4_RAG_Demo.ipynb

Metadata filtering

Uses rigid criteria to narrow down documents based on metadata like title, author,

creation date, access privileges, and more.

Metadata Filtering In RAG

Sparse Search

Bag-of-word search - TF-IDF

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Keyword Search

Bag of Words

Word order is ignored, only word presence and frequency matter

Bag of words

Sparse Vectors

Most words aren’t used. The bag of words is sparse, with few non-zero entries.

Bag of words view of document

Words appearing in each document

​

Bag of words search

Frequency Based vs. Term Frequency (TF) Based Scoring

Example Query: pizza oven

Document 1

​

Homemade pizza in oven is better

than frozen pizza

​

Contains: pizza (2x) oven (1x)

Document 2

​

Wood-fired oven is a better oven than a stone oven for cooking pizza

​

Contains: pizza (1x) oven ( 3x)

Simple Scoring = 2 points

Simple Scoring = 2 points

TF Scoring = 3 points

TF Scoring = 4 points

Normalized TF Scoring

Longer documents may contain keywords many times simply because they are longer.

​

Solution: Normalize by document length

Score = (Number of keyword occurrences) / (Total words in document)

Term Frequency-Inverse Document Freqeuncy (TF-IDF)

Basic TF scoring treats all words equally, whether they're common filler words or rare, meaningful terms.

​

​

Solution: Weight terms using “inverse document frequency” (IDF).

Score = TF(word, doc) × log(Total docs / Docs containing word)

TF vs. TF-IDF

​

TF-IDF

Documents with rare keywords score higher than documents with common words

Modern systems use a slightly refined version called BM25

Sparse Search

BM25

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

BM25 Scoring

BM25 (Best Matching 25) was named as the 25th variant in a series of scoring functions proposed by its creators.

​

​

​

​

• This gives the score for a single keyword
• Sum scores across all keywords for total relevance score for a document

BM25 Tunable Parameters

k₁ - Term Frequency Saturation

• Controls: How much term frequency influences the score.
• Range: Typically between 1.2 and 2.0.
• Effect: Higher values increase the impact of term frequency; lower values reduce it.

​

​

​

b - Length Normalization

• Controls: The degree of normalization for document length.
• Range: Between 0 (no normalization) and 1 (full normalization).
• Effect: Balances favoring shorter vs. longer documents.

​

​

Sparse/Keyword Search Summary

• Keyword Search: Match documents by keyword frequency
• TF-IDF
• Keyword rarity
• Term frequency
• Document length
• BM25: Most commonly used
• Document length normalization
• Term Frequency Saturation

BM25 Demo

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_Lecture_4_RAG_Demo.ipynb

Semantic Search

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Keyword search does not capture meaning of words

Semantic Search vs. Keyword Search

• Prompt and documents each get a vector
• Vectors compared to generate scores
• The main difference is how vectors are assigned
• Keyword Search: count words
• Semantic Search: use embedding model

Sparse vs. Dense Search

https://www.pinecone.io/learn/series/nlp/dense-vector-embeddings-nlp/

Vector Space

Similar words are closer in the vector space

Sentence Embedding Example

Measuring Vector Distance

Measuring Vector Distance

​

Measuring Vector Distance

Semantic Search Summary

Our Current RAG

Index

RAG with Vector Database

Query

Retrieve

Index

embedding

embedding

Embedding Choices

https://huggingface.co/spaces/mteb/leaderboard

Embedding Search High Level

• Embedding model:
• Indexing: convert documents into an embedding using the embedding model
• Searching: convert the query into an embedding using the same embedding model used during indexing
• Retriever: fetch k data chunks whose embeddings are closest to the query embedding, as determined by the retriever.

Vector Database

• Vector Database: The database where the generated embeddings of documents are stored.
• The key part is the vector search
• Naive Solution (K-Nearest Neighbour Search):
1. Compute the similarity across scores between the query embedding and all vectors in the database, using metrics such as cosine similarity
2. Rank all vectors by their similarity scores
3. Return k vectors with the highest similarity scores

Embedding Search without Vector Database Demo

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_Lecture_4_RAG_Demo.ipynb

Semantic Search

With Vector Database

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

You technically don’t need Vector Database to do RAG

RAG != Semantic Search with Vector Database

Vector Database Choices

https://www.graft.com/blog/top-vector-databases-for-ai-projects

Pinecone Database

https://www.pinecone.io/

Pinecone Database - Setup

RAG with Pinecone Database - Coding

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_Lecture_4_RAG_Demo.ipynb

*Pinecone has its own proprietary Approximate Nearest Neighbour (ANN) search

Break

Improving RAG

RAG with Vector Database

Retrieve

Index

embedding

Query

embedding

RAG with Vector Database

Query

Retrieve

embedding

embedding

Index

Hybrid Search

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Hybrid Search

Query

Retrieve

Index

embedding

embedding

Hybrid Search

Reciprocal Rank Fusion

• Rewards documents for being highly ranked on each list
• Control weight of keyword vs. semantic ranking
• Score points equal to reciprocal of ranking

1st = 1 point, 2nd = 0.5 points, etc.

• Total points from all ranked list used to perform final ranking

Reciprocal Rank Fusion

​

RRF only cares about ranks, not scores

Beta: Weighting Semantic vs. Keyword

If exact keyword matching is important, set a lower beta

Hybrid Search

Reranking

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Reranking

Retrieve

Index

embedding

Query

embedding

Overview of Reranking

Reranker

• Rerank models sort text inputs by semantic relevance to a specified query.
• Reranking models are more accurate but more costly than simple cosine similarity retriever
• Therefore, you do it after selecting top-k documents from vector database
• With reranker, you will retrieve more documents (higher k)
• Example of reranker: Cohere, Voyage
• You can also use LLM to rerank - scoring relevance

Hybrid Search + Reranker Demo

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_Lecture_4_RAG_Demo.ipynb

Chunking Strategies

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Chunking

Retrieve

Index

embedding

Query

embedding

Why Chunk Documents?

Indexing without chunking

• Knowledge base contains 1,000 books
• Each book is vectorized by an embedding model
• Result: 1,000 vectors

The problems with this approach

• Compresses entire book meaning into single vector
• Can't sharply represent specific topics, chapters or pages
• Creates "averaged" and “noisy” representation across all content
• Results in poor search relevance
• Retrieves entire books, quickly filling LLM context window

Chunking your content

Chunking Considerations

​

1. What is the nature of the content being indexed? Are you working with long documents, such as articles or books, or shorter content, like tweets or instant messages?
2. Which embedding model are you using, and what chunk sizes does it perform optimally on? For instance, sentence-transformer models work well on individual sentences, but a model like text-embedding-ada-002 performs better on chunks containing 256 or 512 tokens.
3. What are your expectations for the length and complexity of user queries? Will they be short and specific or long and complex?
4. How will the retrieved results be utilized within your specific application?
1. Will they be used for semantic search, question answering, summarization, or other purposes?
2. If your results need to be fed into another LLM with a token limit, you’ll have to take that into consideration.

https://www.pinecone.io/learn/chunking-strategies/

Fixed Size Chunking

Fixed Size Chunking: Overlapping Chunking

​

Fixed Size Chunking

​

​

Chunk based on the number of token and (optionally) any overlap between chunks.

Sentence splitting

​

​

​

• Split based on sentence
• Many libraries provide a way to do sentence splitting such as NLTK and spaCy.

Recursive Chunking

​

​

​

• Recursive chunking divides the input text into smaller chunks in a hierarchical and iterative manner using a set of separators.
• If the initial attempt at splitting the text doesn’t produce chunks of the desired size or structure, the method recursively calls itself on the resulting chunks with a different separator or criterion until the desired chunk size or structure is achieved

Chunking Strategy Demo

https://chunkviz.up.railway.app/

Specialized chunking

​

​

• Splitting based on Markdown
• Splitting based on Latex

Specialized chunking

​

Semantic Chunking

​

Groups sentences together based on similar meanings rather than arbitrary character limits

Semantic Chunking

​

​

​

1. Break up the document into sentences. Process the document sentence by sentence.
2. Vectorize & Compare. Convert chunk and next sentence to vectors, calculate cosine distance
3. Check Threshold. If distance is below threshold, add sentence to chunk.
1. If the theme is the same, the distance between chunk before and start adding a new sentence will be low.
4. Split when Different. When distance crosses threshold, start new chunk
1. Higher semantic distance indicates that the theme has changed.

https://medium.com/@danushidk507/chunking-strategies-f93dbdec7634

Language Based Chunking

• Prompt LLM to create chunks from a document
• Include instructions on types of chunks, like keeping concepts together, adding breaks when new topic starts
• Performs well, increasingly more economically viable

Contextual Retrieval

https://www.anthropic.com/news/contextual-retrieval

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Contextual Retrieval

Retrieve

Index

embedding

Query

embedding

Problem

• A chunk may not contain all information to answer the question
• Example:
• Q: "What was the revenue growth for ACME Corp in Q2 2023?"
• A relevant chunk might contain the text: "The company's revenue grew by 3% over the previous quarter."
• However, this chunk on its own doesn't specify which company it's referring to or the relevant time period, making it difficult to retrieve the right information or use the information effectively.

Contextual Retrieval

• Contextual Retrieval solves the problem by prepending chunk-specific explanatory context to each chunk before embedding.

original_chunk = "The company's revenue grew by 3% over the previous quarter."

​

contextualized_chunk = "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter."

Implementing Contextual Retrieval

• Use LLM to create contextual Retrieval
• Example Prompt:

<document>

{{WHOLE_DOCUMENT}}

</document>

Here is the chunk we want to situate within the whole document

<chunk>

{{CHUNK_CONTENT}}

</chunk>

Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

Choosing a Chunking Approach

• Fixed Width and Recursive Character Splitting: good defaults
• Semantic and LLM Chunking: can yield higher performance, but more complex. Experiment to see if it’s worth it
• Contextual Retrieval: improves any chunking technique at some cost. A good “first improvement” to explore

Query Rewriting

This section is from LangChain: https://www.youtube.com/playlist?list=PLfaIDFEXuae2LXbO1_PKyVJiQ23ZztA0x

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

Query Rewriting

Retrieve

Index

embedding

Query

embedding

Queries != documents

https://twitter.com/abacaj/status/1652011519863926786?s=20

Query Rewriting

• Query Rewriting = Query reformulation, query normalization, and sometimes query expansion.
• Example:
• User: “When was the last time John Doe bought something from us?”
• AI: “John last bought a Fruity Fedora hat from us two weeks ago, on Jan 3, 2023.”
• User: “How about Emily Doe?”
• Rewriting: “When was the last time Emily Doe bought something from us?”
• Another common issue is multiple questions in one query → need to expand

AI Engineering: Building Applications with Foundation Models

Query Rewriting

Use an LLM to rewrite the query before it’s submitted to the retriever.

Transform a question into multiple perspectives

Intuition: Improve search

​

​

Use this with parallelized retrieval

​

​

Multi-Query

• Prompt LLMs to generate multiple versions of the question

HyDE: Hypothetical Document Embeddings

​

• Normally a retriever is matching prompts to documents
• Instead, uses generated “hypothetical documents” that would be ideal search results to help with the search process
• HyDE means the retriever is matching documents to documents

https://arxiv.org/pdf/2212.10496.pdf

HyDE

RAG Evaluations

• Retrieval Augmented Generation
• Sparse Search
• keyword search
• TF-IDF
• BM25
• Semantic Search
• Semantic Search with Vector DB
• Hybrid Search
• Reranking
• Chunking Strategies
• Contextual Retrieval
• Query Rewriting
• RAG Evaluations

RAG Evaluations

Retrieve

Index

embedding

Query

embedding

RAG Evaluations

​

RAG systems have three core components:

• A question (Q)
• Retrieved context (C)
• An answer (A)

https://jxnl.co/writing/2025/05/19/there-are-only-6-rag-evals/#tier-2-primary-rag-relationships

Part 1: Context Relevance (C|Q)

Def: How well do the retrieved chunks address the question information needs? Does your retriever find passages that contain information relevant to answering the user's question.

Bad

Question: "What are the health benefits of meditation?"

Context: "Meditation practices vary widely across different traditions. Mindfulness meditation, which originated in Buddhist practices, focuses on present-moment awareness, while transcendental meditation uses mantras to achieve deeper states of consciousness."

Reasoning: Despite being factually correct about meditation, this context did not discuss health benefits.

Good

​

Question: "What are the health benefits of meditation?"

Context: "Regular meditation has been shown to reduce stress hormones like cortisol. A 2018 study in the Journal of Cognitive Enhancement found meditation improves attention and working memory."

Reasoning: Strong relevance. The context directly addresses multiple health benefits with specific details.

Q: What source of information would you need to answer multiple-choice questions on financial knowledges?

Finance textbooks

Retrieval Quality Metrics

Common ingredients to most retriever quality metrics:

If you want to evaluate your retriever you need to know the correct answers

The Question

The specific question being evaluated

Ranked Results

Documents returned in ranked order

Ground Truth All documents labeled as relevant or irrelevant

Recall and Precision

Recall penalizes for leaving out relevant documents

Precision penalizes for returning irrelevant documents

Top k

• Retrieval metrics are influenced by how many documents the retriever returns
• Metrics are discussed in terms of top-k documents

Example

Evaluate RAG using Synthetic Data and LLM-as-a-judge

Ques- tions

Score

Only use good questions

Evaluate RAG using Synthetic Data and LLM-as-a-judge

• Get chunks from our knowledge base
• Ask an LLM to generate questions based on these chunks
• Check that questions are good (using LLM-as-a-judge):
• Groundedness: Can the question be answered from the given context?
• Relevance: Is the question relevant to users?
• Stand-alone: Is the question understandable free of any context, for someone with domain knowledge/Internet access?
• Select good questions to be in the test set

https://huggingface.co/learn/cookbook/en/rag_evaluation#rag-evaluation

Part 2: Faithfulness/Groundedness (A|C)

Def: To what extent does the answer restrict itself only to claims that can be verified from the retrieved context?

Bad

Context: "The Great Barrier Reef is the world's largest coral reef system. It stretches for over 2,300 kilometers along the coast of Queensland, Australia."

Answer: "The Great Barrier Reef, the world's largest coral reef system, stretches for over 2,300 kilometers along Australia's eastern coast and is home to about 10% of the world's fish species."

Reasoning: The first part is supported, but the claim about "10% of the world's fish species" isn't in the provided context.

Good

​

Context: "The Great Barrier Reef is the world's largest coral reef system."

Answer: "The Great Barrier Reef is the largest coral reef system in the world."

Reasoning: Perfect faithfulness. The answer only states what's in the context.

Part 3: Answer Relevance (A|Q)

Def: How directly does the answer address the specific information need expressed in the question? This evaluates the end-to-end system performance.

Bad

Question: "How does compound interest work in investing?"

Answer: "Interest in investing can be simple or compound. Compound interest is more powerful than simple interest and is an important concept in finance."

Reasoning: Low relevance. The answer doesn't actually explain the mechanism of how it works.

Good

Question: "How does compound interest work in investing?"

Answer: "Compound interest works by adding the interest earned back to your principal investment, so that future interest is calculated on the new, larger amount."

Reasoning: High relevance. The answer directly explains the concept asked about.

Advanced RAG Relationships

• Context Support Coverage (C|A)
• Definition: Does the retrieved context contain all the information needed to fully support every claim in the answer? This measures whether the context is both sufficient and focused.
• Question Answerability (Q|C)
• Definition: Given the context provided, is it actually possible to formulate a satisfactory answer to the question? This evaluates whether the question is reasonable given the available information.
• Self-Containment (Q|A)
• Definition: Can the original question be inferred from the answer alone? This measures whether the answer provides enough context to stand on its own.

Other considerations

• Latency
• Cost

HW 4

https://colab.research.google.com/github/yongsa-nut/SF323_CN408_AIEngineer/blob/main/SF323_CN408_HW4_RAG.ipynb

Project

Don’t forget about your project!

Extra

ColBert: Similarity at Token Levels

https://github.com/stanford-futuredata/ColBERT

Score = Sum of max similarity of each query embedding to any document embedding

ColBert

https://twitter.com/marktenenholtz/status/1751415287709102088

ColBert

https://til.simonwillison.net/llms/colbert-ragatouille