1 of 131

RAG

GenAI Enterprise Accelerator Master Class

LAVANYA.AI

2 of 131

Fine-Tuning

Updating the weights of some or all layers of the pre-trained model to adapt it to the new task

Prompt Engineering

Prompt engineering is temporary learning during inference.

It lets us communicate with an LLM to steer its behavior without updating the model weights

RAG

For tasks that require latest knowledge after the model pretraining time cutoff or access to private knowledge base

LAVANYA.AI

3 of 131

RAG

LAVANYA.AI

4 of 131

RAG

Personalized context to the query.

Overcome the models’ context limitations

Having access to relevant information reduces hallucinations.

5 of 131

RAG Overview

LAVANYA.AI

6 of 131

Components of a RAG System

LAVANYA.AI

A RAG system has two components:

A retriever that

gets info from external data sources

A generator that

generates a response based on the retrieved information

A RAG system has two components: a retriever that retrieves information from external memory sources and a generator that generates a response based on the retrieved information. A high-level architecture of a RAG system

In the original RAG paper, Lewis et al. trained the retriever and the generative model together. In today’s RAG systems, these two components are often trained separately. Many teams build their RAG systems using off-the-shelf retrievers and models. However, fine tuning the whole RAG system end-to-end can improve its performance significantly.

The success of a RAG system depends on the quality of its retriever.

A retriever has two main functions: indexing and querying. Indexing involves processing data so that it can be quickly retrieved later. Sending a query to retrieve data relevant to it is called querying. How to index data depends on how you want to retrieve it later on.

For simplicity, let’s assume that the external memory is a database of documents, such as a company’s memos, contracts, and meeting notes. A document can be 10 tokens or 1 million tokens. Naively retrieving whole documents can cause your context to be arbitrarily long. To avoid this, you can split each document into more manageable chunks. Chunking strategies will be discussed later in this chapter. For now, let’s assume that all documents have been split into workable chunks. For each query, our goal is to retrieve the data chunks most relevant to this query. Minor post-processing is often needed to join the retrieved data chunks with the user prompt to generate the final prompt to input into the generative model.

Retrieval vs. search

Retrieval is typically limited to one database or system, whereas search involves retrieval across various systems. This chapter uses retrieval and search interchangeably.

7 of 131

LAVANYA.AI

8 of 131

?

embed

1

110

1

LAVANYA.AI

9 of 131

Building a RAG Pipeline

LAVANYA.AI

10 of 131

You have a dataset, and you want to build an LLM to do interesting things with it.

LAVANYA.AI

11 of 131

Dataset

12 of 131

Building an

Evaluation Dataset

LAVANYA.AI

13 of 131

13

Comb through the corpus and come up with questions.

Disadvantages

Super cumbersome and time consuming

Nearly impossible to come up with questions that comprehensively cover your dataset

Manually

LAVANYA.AI

14 of 131

14

Use ChatGPT

LAVANYA.AI

15 of 131

15

Advantages

Generate a lot of diverse questions quickly�
Much faster than doing it manually

Disadvantages

Not the most evocative – they’re quite simple�
If your corpus is big (e.g. years of research data) then ChatGPT’s context window isn’t big enough to hold them all�
Our questions to be comprehensive, generate questions spanning most of the document, be of varying levels of difficulty etc

Use ChatGPT

LAVANYA.AI

16 of 131

16

Lets you systematically generate complex questions and tweak what types of questions you want to optimize for

Use an

LLM Framework

Reasoning

Craft the question so the LLM needs to reason to answer it

Multi-Context

Create questions that need information from multiple sections for the LLM to answer them

Conditioning

Modify the question to introduce a conditional element�

If X, then → Y

LAVANYA.AI

17 of 131

17

18 of 131

Building a RAG Pipeline

LAVANYA.AI

19 of 131

LAVANYA.AI

20 of 131

Load

Dataset

20

21 of 131

21

LAVANYA.AI

22 of 131

Advanced chunking techniques that take the meaning into account

Sentence based

chunking

Advanced chunking techniques that take the meaning into account

Chunking

LAVANYA.AI

23 of 131

23

LAVANYA.AI

24 of 131

LAVANYA.AI

Vector Embedding

25 of 131

LAVANYA.AI

26 of 131

26

Embed your Chunked Documents

LAVANYA.AI

27 of 131

When we get a user question…

LAVANYA.AI

28 of 131

28

LAVANYA.AI

29 of 131

LAVANYA.AI

Vector Embedding

30 of 131

Retrieve Context

30

LAVANYA.AI

31 of 131

31

LAVANYA.AI

32 of 131

Generate Answer

32

LAVANYA.AI

33 of 131

Evaluate Answers

LAVANYA.AI

34 of 131

Evaluation Is Key

34

LAVANYA.AI

35 of 131

35

A

Facts

A

?

LAVANYA.AI

36 of 131

Red Teaming

36

Red Teaming

probing, testing, and attacking of AI systems�
By a diverse group of people – AI, social sciences, security

Goal

understand the risk surface, and�
develop the list of harms… that can inform what needs to be measured and mitigated

LAVANYA.AI

37 of 131

37

LLM As

A Judge

Define Our Own Rubric

LAVANYA.AI

38 of 131

38

LLM as a Judge

Hamel Husain Blog

LAVANYA.AI

39 of 131

SPADE

39

Automatically write useful evaluation functions for custom LLM chains

Prompt

Assertion

Code

LLM�as a judge

LAVANYA.AI

40 of 131

40

Evaluate

One Answer

LAVANYA.AI

41 of 131

LAVANYA.AI

Evaluate

All Answers

42 of 131

42

Evaluate All Answers

43 of 131

43

Learn from What we got Wrong

LAVANYA.AI

44 of 131

Challenge

LAVANYA.AI

Beat The Leaderboard

45 of 131

Challenge

(30 mins)

45

1. Understand the dataset wandb.me/explore-dataset

2. Get the code wandb.me/class-rag

Beat The Leaderboard

3. Beat the leaderboard

wandb.me/leaderboard

Setup

Get keys

Get OpenAI key
Get Together.ai key
Get W&B key�

In Colab: Runtime > Run All to run code�
Improve performance� look for # 🛠️LEVER�
Tip: Change eval set to 5 to iterate fast

look for # 🔵 NOTE

46 of 131

Feedback Survey

46

LAVANYA.AI

wandb.me/feedback

47 of 131

47

Community

wandb.me/discord

LAVANYA.AI

48 of 131

Thank You

Learn more at lavanya.ai

3 RAG

49 of 131

50 of 131

GenAI Enterprise Accelerator Master Class

RAG

51 of 131

ACCELERATOR.CC

Fine-Tuning

Updating the weights of some or all layers of the pre-trained model to adapt it to the new task

Prompt Engineering

Prompt engineering is temporary learning during inference.

It lets us communicate with an LLM to steer its behavior without updating the model weights

RAG

For tasks that require latest knowledge after the model pretraining time cutoff or access to private knowledge base

52 of 131

ACCELERATOR.CC

RAG

53 of 131

ACCELERATOR.CC

RAG

Personalized context to the query.

Overcome the models’ context limitations

Having access to relevant information reduces hallucinations.

54 of 131

ACCELERATOR.CC

RAG Overview

55 of 131

ACCELERATOR.CC

Components of

a RAG system

A RAG system has two components:

A retriever that

gets info from external data sources

A generator that

generates a response based on the retrieved information

A RAG system has two components: a retriever that retrieves information from external memory sources and a generator that generates a response based on the retrieved information. A high-level architecture of a RAG system

In the original RAG paper, Lewis et al. trained the retriever and the generative model together. In today’s RAG systems, these two components are often trained separately. Many teams build their RAG systems using off-the-shelf retrievers and models. However, fine tuning the whole RAG system end-to-end can improve its performance significantly.

The success of a RAG system depends on the quality of its retriever.

A retriever has two main functions: indexing and querying. Indexing involves processing data so that it can be quickly retrieved later. Sending a query to retrieve data relevant to it is called querying. How to index data depends on how you want to retrieve it later on.

For simplicity, let’s assume that the external memory is a database of documents, such as a company’s memos, contracts, and meeting notes. A document can be 10 tokens or 1 million tokens. Naively retrieving whole documents can cause your context to be arbitrarily long. To avoid this, you can split each document into more manageable chunks. Chunking strategies will be discussed later in this chapter. For now, let’s assume that all documents have been split into workable chunks. For each query, our goal is to retrieve the data chunks most relevant to this query. Minor post-processing is often needed to join the retrieved data chunks with the user prompt to generate the final prompt to input into the generative model.

Retrieval vs. search

Retrieval is typically limited to one database or system, whereas search involves retrieval across various systems. This chapter uses retrieval and search interchangeably.

56 of 131

ACCELERATOR.CC

57 of 131

?

embed

1

110

1

ACCELERATOR.CC

58 of 131

ACCELERATOR.CC

Building a

RAG Pipeline

59 of 131

You have a dataset, and you want to build an LLM to do interesting things with it.

ACCELERATOR.CC

60 of 131

Dataset

61 of 131

Building an

Evaluation Dataset

ACCELERATOR.CC

62 of 131

62

Comb through the corpus and come up with questions.

Disadvantages

Super cumbersome and time consuming

Nearly impossible to come up with questions that comprehensively cover your dataset

Manually

ACCELERATOR.CC

63 of 131

63

ACCELERATOR.CC

Use ChatGPT

64 of 131

64

Advantages

Generate a lot of diverse questions quickly�
Much faster than doing it manually

Use ChatGPT

Disadvantages

Not the most evocative – they’re quite simple�
If your corpus is big (e.g. years of research data) then ChatGPT’s context window isn’t big enough to hold them all�
Our questions to be comprehensive, generate questions spanning most of the document, be of varying levels of difficulty etc

ACCELERATOR.CC

65 of 131

65

Lets you systematically generate complex questions and tweak what types of questions you want to optimize for

Use an LLM Framework

Reasoning

Craft the question so the LLM needs to reason to answer it

Multi-Context

Create questions that need information from multiple sections for the LLM to answer them

Conditioning

Modify the question to introduce a conditional element�

If X, then → Y

ACCELERATOR.CC

66 of 131

66

67 of 131

ACCELERATOR.CC

Build A RAG Pipeline

68 of 131

ACCELERATOR.CC

69 of 131

Load

Dataset

69

70 of 131

70

ACCELERATOR.CC

71 of 131

Advanced chunking techniques that take the meaning into account

Sentence based chunking

Advanced chunking techniques that take the meaning into account

ACCELERATOR.CC

Chunking

72 of 131

72

ACCELERATOR.CC

73 of 131

ACCELERATOR.CC

Vector Embedding

74 of 131

ACCELERATOR.CC

75 of 131

75

Embed your Chunked Documents

ACCELERATOR.CC

76 of 131

ACCELERATOR.CC

When we get a user question…

77 of 131

77

ACCELERATOR.CC

78 of 131

ACCELERATOR.CC

Vector Embedding

79 of 131

Retrieve Context

79

ACCELERATOR.CC

80 of 131

80

ACCELERATOR.CC

81 of 131

Generate Answer

81

ACCELERATOR.CC

82 of 131

ACCELERATOR.CC

Evaluate Answers

83 of 131

Evaluation Is Key

83

ACCELERATOR.CC

84 of 131

84

A

Facts

A

?

ACCELERATOR.CC

85 of 131

Red Teaming

85

Red Teaming

probing, testing, and attacking of AI systems�
By a diverse group of people – AI, social sciences, security

Goal

understand the risk surface, and�
develop the list of harms… that can inform what needs to be measured and mitigated

ACCELERATOR.CC

86 of 131

86

LLM As

A Judge

Define Our Own Rubric

87 of 131

87

LLM as a Judge

Hamel Husain Blog

88 of 131

SPADE

88

Automatically write useful evaluation functions for custom LLM chains

Prompt

Assertion

Code

LLM�as a judge

ACCELERATOR.CC

89 of 131

89

Evaluate

One Answer

ACCELERATOR.CC

90 of 131

ACCELERATOR.CC

Evaluate

All Answers

91 of 131

91

Evaluate All Answers

92 of 131

92

Learn from What we got Wrong

93 of 131

93

Beat The Leaderboard

CHALLENGE

94 of 131

Challenge

(30 mins)

94

1. Understand the dataset wandb.me/explore-dataset

2. Get the code wandb.me/class-rag

Beat The Leaderboard

3. Beat the leaderboard

wandb.me/leaderboard

Setup

Get keys

Get OpenAI key
Get Together.ai key
Get W&B key�

In Colab: Runtime > Run All to run code�
Improve performance� look for # 🛠️LEVER�
Tip: Change eval set to 5 to iterate fast

look for # 🔵 NOTE

95 of 131

Feedback Survey

wandb.me/feedback

95

ACCELERATOR.CC

96 of 131

Community

96

wandb.me/discord

ACCELERATOR.CC

97 of 131

Thank You

lavanya@wandb.com

@lavanyai

98 of 131

Challenge

(30 mins)

98

Beat The Leaderboard

Parameter	Good	Default
chunksize	2048	10000
embedding	text-embedding-3-small	text-embedding-3-small
chunking strategy	simple
temperature	0.5	1.0
top_k	1	1000
k=2	2	1
prompt
models

99 of 131

ACCELERATOR.CC

RAG

RAG is a solution for knowledge-intensive tasks where all the available knowledge can’t be input into the model directly.

With RAG, only the information most relevant to the query, is retrieved and input into the model.
Having access to relevant information in the context can help the model generate more detailed responses while reducing hallucinations.
Its main purpose was to overcome the models’ context limitations.

RAG, Retrieval-Augmented Generation, is a technique that enhances a model’s generation by retrieving the relevant information from external memory sources. An external memory source can be an internal database, a user’s previous chat sessions, or the Internet.

The retrieve-then-generate pattern was first introduced in Reading Wikipedia to Answer Open-Domain Questions (Chen et al., 2017). In this work, the system first retrieves five Wikipedia pages most relevant to a question, then a model1 uses (reads) the information from these pages to generate an answer, as visualized in Figure 6-1.

RAG is a solution for knowledge-intensive tasks where all the available knowledge can’t be input into the model directly.

With RAG, only the information most relevant to the query, as determined by the RAG system, is retrieved and input into the model.

Having access to relevant information in the context can help the model generate more detailed responses while reducing hallucinations.

For example, given the query “Will Acme’s fancy-printer-A300 print 100pps?”, the model will be able to respond better if it’s given the specifications of fancy-printer-A300.

100 of 131

Its main purpose was to overcome the models’ context limitations.

ACCELERATOR.CC

Will a sufficiently long context will be the end of RAG?

Overcoming context limitations

Its main purpose was to overcome the models’ context limitations.

Many people think that a sufficiently long context will be the end of RAG. I don’t think so. First, no matter how long a model’s context length is, there will be applications that require context longer than that. The amount of available data only grows over time. People generate and add new data, but rarely delete data. Context length is expanding fast, but not fast enough for the data needs of arbitrary applications.

Second, a model that can process long context doesn’t necessarily use that context well, as discussed in the section Context length and context efficiency. The longer the context, the more likely the model is to focus on the wrong part of the context. Every extra context token incurs extra cost and potentially extra latency. RAG allows a model to use only the most relevant information for each query, reducing the number of input tokens while potentially increasing the model’s performance.

Efforts to expand context length are happening in parallel with efforts to make models use context more effectively.

101 of 131

You can think of RAG as a technique to construct context specific to each query, instead of using the same context for all queries.

Context construction for foundation models is equivalent to feature engineering for classical ML models.

ACCELERATOR.CC

Personalizing Context

102 of 131

LLM

Information Retriever

LLM

BM25

Medical Literature

Question2Query

Query

Relevance

Classifier

Relevant

Articles

Summarization

Synthesis

Question

Answer + Citations

110

?

1

embed

ACCELERATOR.CC

103 of 131

ACCELERATOR.CC

Retrieval Algorithms

Term-based Retrieval

Embedding Based

Term-based retrieval

Given a query, the most straightforward way to find relevant documents is by keywords. Some people call this approach lexical retrieval. For example, given the query “AI engineering,” retrieve all the documents that contain “AI engineering.” However, this approach has two problems.

Many documents might contain the given term, and your model might not have sufficient context space to include all of them as context. A heuristic is to include the documents that contain the term the most times. The assumption is that the more a term appears in a document, the more relevant this document is to this term. The number of times a term appears in a document is called term frequency (TF).
A prompt can be long and contain many terms. Some are more important than others. For example, the prompt “Easy-to-follow recipes for Vietnamese food to cook at home” contains nine terms: easy-to-follow, recipes, for, vietnamese, food, to, cook, at, home. You don’t want to focus on more informative terms like vietnamese and recipes, instead of for and at. You need a way to identify important terms.
An intuition is that the more documents contain a term, the less informative this term is. “For” and “at” are likely to appear in most documents, hence they are less informative. So a term’s importance is inversely proportional to the number of documents it appears in. This metric is called Inverse Document Frequency (IDF). To compute IDF for a term, count all the documents that contain this term, then divide the total number of documents by this count. If there are 10 documents and 5 of them contain a given term, then the IDF of this term is 10 / 5 = 2. The higher a term’s IDF, the more important it is.

TF-IDF is an algorithm that combines these two metrics. Mathematically, the TF-IDF score of document D for the query Q is computed as follows.

Let be the terms in the query Q.
Given a term , the term frequency of this term in the document D is f(t, D).
Let N be the total number of documents, and C(t) be the number of documents that contain t. The IDF value of the term t can be written as .
Naively, the TF-IDF score of a document D with respect to Q is defined as:

Two common term-based retrieval solutions are Elasticsearch and BM25. Elasticsearch (Shay Banon, 2010), built on top of Lucene, uses a data structure called an inverted index. It’s a dictionary that maps from terms to documents that contain them. This dictionary allows for fast retrieval of documents given a term. The index might also store additional information such as the term frequency and the document count (how many documents contain this term), which are helpful for computing TF-IDF scores. Table 6-1 illustrates an inverted index.

104 of 131

ACCELERATOR.CC

Compare retrievers

	Term-based retrieval	Semantic retrieval
Querying speed	Much faster than semantic retrieval	Query embedding generation and vector search can be slow
Performance	Typically strong performance out-of-the-box, but hard to improve Can retrieve wrong documents due to term ambiguity	Can outperform term-based retrieval with fine tuning Allows using more natural queries, as it focuses on semantics instead of terms
Cost	Much cheaper than semantic retrieval	Embedding, vector storage, and vector search solutions can be expensive

Compare retrieval algorithms

Due to the long history of retrieval, its many mature solutions make both term-based and embedding-based retrieval relatively easy to start. Each approach has its pros and cons.

Term-based retrieval is generally much faster than semantic retrieval both during indexing and query. Term extraction is faster than embedding generation, and mapping from a term to documents that contain it can be less computationally expensive than a nearest neighbor search.

Term-based retrieval also works well out-of-the-box. Solutions like Elasticsearch and BM25 have successfully powered many search and retrieval applications. However, its simplicity also means that it has fewer components you can tweak to improve its performance.

Embedding-based retrieval, on the other hand, can be significantly improved over time to outperform term-based retrieval. You can finetune the embedding model and the retriever, either separately, together, or in conjunction with the generative model.

The quality of a retriever can be evaluated based on the quality of the data it retrieves. Two metrics often used by RAG evaluation frameworks are context precision and context recall, or precision and recall for short. Context precision is sometimes called context relevance.

Context precision: out of all the documents retrieved, what percentage is relevant to the query?
Context recall: out of all the documents that are relevant to the query, what percentage is retrieved?

To compute these metrics, you curate an evaluation set with a list of test queries and a set of documents. For each test query, you annotate each test document to be relevant or not. The annotation can either be done by humans or AI judges. You then compute the precision and recall score of the retriever on this evaluation set.

In production, some RAG frameworks only support context precision, not context recall. To compute context recall, you need to annotate all documents in your database. Context precision is much simpler to compute. You only need to compare the retrieved documents to the query. This comparison can be handled by an AI judge.

If you care about the ranking of the retrieved documents, such as more relevant documents should be ranked first, you can use metrics such as NDCG (Normalized Discounted Cumulative Gain), MAP (Mean Average Precision), and MRR (Mean Reciprocal Rank).

For semantic retrieval, you need to also evaluate the quality of your embeddings. As discussed in Chapter 3, embeddings can be evaluated either dependently or in the context of a task. Independently, embeddings are considered good if more similar documents have closer embeddings. Embeddings are also evaluated by how well they work for specific tasks. The MTEB benchmark (Muennighoff et al., 2023) evaluates embeddings for a broad range of tasks including retrievals, classification, and clustering.

The quality of a retriever should also be evaluated in the context of the whole RAG system. Ultimately, does the retriever help the system generate high-quality answers? Evaluating outputs of generative models was already discussed in Chapters 3 and 4.

Whether the performance promise of a semantic retrieval system is worth pursuing depends on how much you care about cost and latency, especially during the querying phase. Given that a large chunk of RAG latency comes from output generation, especially for long outputs, the added latency by query embedding generation and vector search might be minimal compared to the total RAG latency. However, the added latency can still affect user experience.

With retrieval systems, there’s a certain amount of tradeoffs you can do between indexing and querying. The more detailed the index is, the more accurate the retrieval process will be, but the indexing process will be slower and more memory-consuming. Imagine building an index of potential customers. Adding more details (name, company, email, phone, interests) makes it easier to find relevant people but takes longer to build and requires more storage.

In general, a detailed index like HNSW provides high accuracy and fast query times but requires significant time and memory to build. In contrast, a simpler index like LSH is quicker and less memory-intensive to create but results in slower and less accurate queries.

The ANN-benchmarks website compares different ANN algorithms on multiple datasets using four main metrics, taking into account the tradeoffs between indexing and querying.

Recall: the fraction of the nearest neighbors found by the algorithm.
Query per second (QPS): the number of queries the algorithm can handle per second. This is crucial for high-traffic applications.
Build time: the time required to build the index. This metric is important especially if you need to frequently update your index (e.g. because your data changes).
Index size: the size of the index created by the algorithm, which is crucial for assessing its scalability and storage requirements.

BEIR (Benchmarking IR) (Thakur et al., 2021) is an evaluation harness for retrieval. It supports retrieval systems across 14 common retrieval benchmarks.

To summarize, the quality of a RAG system should be evaluated both component-by-component and end-to-end.

Evaluate the retrieval quality.
Evaluate the final RAG outputs.
For embedding-based retrieval, evaluate the embeddings.

Another concern is cost. Generating embeddings costs money. This is especially an issue if your data changes frequently and requires frequent embedding regeneration. Imagine having to generate embeddings for 100M documents every day! Depending on what vector databases you use, vector storage and vector search queries can be expensive too. It’s not uncommon to see a company’s vector database spending to be ⅕ or even ½ of their spending on model APIs.

105 of 131

ACCELERATOR.CC

Retrieval Optimization

106 of 131

ACCELERATOR.CC

107 of 131

ACCELERATOR.CC

add graphic

Query by

108 of 131

ACCELERATOR.CC

Key factors to keep in mind when evaluating a retrieval solution.
What retrieval mechanisms does it support? Does it support hybrid search?
If it’s a vector database, what embedding models and vector search algorithms does it support?
How scalable is it, both in terms of data storage and query traffic? Does it work for your traffic patterns?
How long does it take to index your data? How much data can you process (such as add/delete) in bulk at once?
What’s its query latency for different retrieval algorithms?
If it’s a managed solution, what’s its pricing structure? Is it based on the document/vector volume or on the query volume?

Evaluation

109 of 131

ACCELERATOR.CC

RAG++

110 of 131

ACCELERATOR.CC

Multi-modal RAG

If your generator is multimodal, its contexts might be augmented with not only text documents but also images, videos, audio, etc. from external sources. I’ll use images in the examples to keep the writing concise, but you can replace images with any other modality. Given a query, the retriever fetches both texts and images relevant to it. For example, given “What’s the color of the house in the Pixar movie Up?”, the retriever can fetch a picture of the house in Up to help the model answer, as shown in Figure 6-6.

If the images have metadata--such as titles, tags, and captions--they can be retrieved using the metadata. For example, an image is retrieved if its caption is considered relevant to the query. If you want to retrieve images based on their content, you’ll need to have a way to compare images to queries. If queries are texts, you’ll need a multimodal embedding model that can generate embeddings for both images and texts. Let’s say you use CLIP (Radford et al., 2021) as the multimodal embedding model. The retriever works as follows:

Generate CLIP embeddings for all your data, both texts and images, and store them in a vector database.
Given a query, generate its CLIP embedding.
Query in the vector database for all images and texts whose embeddings are close to the query embedding.

111 of 131

ACCELERATOR.CC

RAGs with tabular data

Order ID	Timestamp	Product ID	Product	Unit price ($)	Units	Total
1	…	2044	Meow Mix Seasoning	10.99	1	10.99
2	…	3492	Purr & Shake	25	2	50
3	…	2045	Fruity Fedora	18	1	18
…	…	…	…	…	…	…

Most applications work not only with unstructured data like texts and images, but also with tabular data. Many queries might need information from data tables to answer. The workflow for augmenting a context using tabular data is significantly different from the classic RAG workflow.

Imagine you work for an e-commerce site that specializes in cat fashion called Kitty Vogue. This store has an order table named Sales, as shown in Table 6-3.

Image 1

To generate a response to the question: “How many units of Fruity Fedora were sold in the last 7 days?”, your system needs to query this table for all orders involving Fruity Fedora and sum the number of units across all orders. Assume that this table can be queried using SQL. The SQL query might look like this:

Image 2

The workflow is as follows, visualized in Figure 6-7. To run this workflow, your system must have the ability to generate and execute the SQL query.

Text-to-SQL: Based on the user query and the provided table schemas, determine what SQL query is needed. Text-to-SQL is an example of semantic parsing, as discussed in Chapter 2.
SQL execution: Execute the SQL query.
Generation: Generate a response based on the SQL result and the original user query.

Image 3

For the text-to-SQL step, if there are many available tables whose schemas can’t all fit into the model context, you might need an intermediate step to predict what tables to use for each query. Text-to-SQL can be done by the same generator that is used to generate the final response or one of many specialized text-to-SQL models.

112 of 131

SpaceRAG Demo

Code + Weave

112

ACCELERATOR.CC

113 of 131

ACCELERATOR.CC

RAG From Scratch

114 of 131

ACCELERATOR.CC

Level 1

The Basics

Add visuals to these slides

115 of 131

ACCELERATOR.CC

Level 2

More Structured Processing

116 of 131

ACCELERATOR.CC

Level 3

Observability

117 of 131

ACCELERATOR.CC

Level 4

Evaluations

118 of 131

ACCELERATOR.CC

Level 5

Understanding Short Comings

119 of 131

ACCELERATOR.CC

Level 6

Level 7

Level 8

Level 9

Finding Segments and Routing: Techniques for identifying and directing data segments efficiently.

Processing Tables: Strategies for interpreting and manipulating tabular data.

Processing Images: Methods for incorporating image data into RAG applications.

Advanced Data Handling

Building Up Timeline Queries: Crafting queries that span across different timeframes for dynamic data analysis.

Adding Additional Metadata: Leveraging metadata to enrich query context and improve response accuracy.

Query Enhancement

Summarization and Summary Indices: Developing concise summaries from extensive datasets to aid quick insights.

Summarization Techniques

Modeling Business Outcomes: Applying RAG techniques to predict and model business outcomes, facilitating strategic decision-making.

Outcome

Modeling

More Levels

120 of 131

ACCELERATOR.CC

Embedding based Retrieval

Term-based retrieval computes relevance at a lexical level rather than a semantic level. As mentioned in Chapter 3, the appearance of a text doesn’t necessarily capture its meaning. This can result in returning documents irrelevant to your intent. For example, querying “transformer architecture” might return documents about the electric device or the movie Transformers. On the other hand, embedding-based retrievers aim to rank documents based on how closely their meanings align with the query. This approach is also known as semantic retrieval.

With embedding-based retrieval, indexing has an extra function: converting the original data chunks into embeddings. The database where the generated embeddings are stored is called a vector database. Querying then consists of two steps, as shown in Figure 6-3.

Convert the query into an embedding using the same embedding model used during indexing.
Fetch k data chunks whose embeddings are closest to the query embedding, as determined by the retriever. The number of data chunks, k, depends on the use case, the generative model, and the query.

The semantic retrieval workflow shown in Figure 5-9 is simplified. Real-world semantic retrieval systems might contain other components such as a reranker to rerank all retrieved candidates, and cache to reduce latency.⁹

With semantic retrieval, we again encounter embeddings, which were discussed in Chapter 3. As a reminder, an embedding is typically a vector that aims to preserve the important properties of the original data. A semantic retriever doesn’t work if the embedding model is bad.

Semantic retrieval also introduces a new component: vector databases. A vector database stores vectors. However, storing is the easy part of a vector database. The hard part is vector search. Given a query embedding, a vector database is responsible for finding vectors in the database close to the query and returning them. Vectors have to be indexed and stored in a way that makes vector search fast and efficient.

Like many other mechanisms that generative AI applications depend on, vector search isn’t generative AI. Vector search is common in any application that uses embeddings: search, recommendation, data organization, information retrieval, clustering, fraud detection, and more.

Vector search is typically framed as a nearest-neighbor search problem. Given a query, find the k nearest vectors. The naive solution is k-Nearest Neighbors (k-NN), which works as follows:

Compute the similarity scores between the query embedding and all vectors in the database, using metrics such as cosine similarity.
Rank all vectors by their similarity scores.
Return k vectors with the highest similarity scores.

This naive solution ensures that the results are precise but it’s computationally heavy and slow. It should only be used for small datasets.

For large datasets, vector search is typically done using an approximate nearest neighbor (ANN) algorithm. Due to the importance of vector search, many algorithms and libraries have been developed for it. Among popular vector search libraries are FAISS (Facebook AI Similarity Search) (Johnson et al., 2017), Google’s ScaNN (Sun et al., 2020), Spotify’s ANNOY (Bernhardsson, 2013), and hnswlib (Hierarchical Navigable Small World) (Malkov et al., 2016).

Most application developers won’t implement vector search themselves, so I’ll give only a quick overview of different approaches. This overview might be helpful as you evaluate solutions.

In general, vector databases organize vectors into buckets, trees, or graphs. Vector search algorithms differ based on the heuristics they use to increase the likelihood that similar vectors are close to each other. Vectors can also be quantized (reduced precision) or made sparse. The idea is that simplified vectors are less computationally intensive to work with. For those wanting to learn more about vector search, Zilliz has an excellent series on it.

LSH (Locality-Sensitive Hashing) (Indyk and Motwani, 1999): This is a powerful and versatile algorithm that works with more than just vectors. This involves hashing similar vectors into the same buckets to speed up similarity search, trading off some accuracy for efficiency. It’s implemented in FAISS and ANNOY.
HNSW (Hierarchical Navigable Small World) (Malkov et al., 2016): HNSW constructs a multi-layer graph where nodes represent vectors, and edges connect similar vectors, allowing nearest neighbor searches by traversing graph edges. Its implementation by the authors is open source, and it’s also implemented in FAISS and Milvus.
Product Quantization (Jégou et al., 2011) works by reducing each vector into a much simpler, lower-dimensional representation by decomposing each vector into multiple sub-vectors. The distances are then computed using the lower-dimensional representations, which are much faster to work with. Product quantization is a key component of FAISS and is supported by almost all popular vector search libraries.
IVF (Inverted File Index) (Sivic and Zisserman, 2003): IVF uses K-means clustering to organize similar vectors into the same cluster. Depending on the number of vectors in the database, it’s typically to set the number of clusters so that on average, there should be 100 to 10,000 vectors in each cluster. During querying, IVF finds the cluster centroids closest to the query embedding, and the vectors in these clusters become candidate neighbors. Together with product quantization, IVF forms the backbone of FAISS.
Annoy (Approximate Nearest Neighbors Oh Yeah) (Bernhardsson, 2013) is a tree-based approach. It builds multiple binary trees, each tree splits the vectors into clusters using random criteria, such as randomly drawing a line and splitting the vectors into two branches using this line. During a search, it traverses these trees to gather candidate neighbors. Spotify has open sourced its implementation.

There are other algorithms such as Microsoft’s SPTAG (Space Partition Tree And Graph), and FLANN (Fast Library for Approximate Nearest Neighbors).

Even though vector databases emerged as their own category with the rise of RAG, any database that can store vectors can be called a vector database. Many traditional databases have extended or will extend to support vector storage and vector search.

121 of 131

ACCELERATOR.CC

Chunking

Sentence based chunking

Advanced chunking techniques that take the meaning into account

E.g. The 2 red chunks on the bottom left are about math

How your data should be indexed depends on how you intend to retrieve it later. The last section covers different retrieval algorithms and their respective indexing strategies. The discussion was based on the assumption that documents have already been split into manageable chunks. This section discusses different chunking strategies. The chunking strategy used can significantly impact the performance of your retrieval system.

The simplest strategy is to chunk documents into chunks of equal length based on a certain unit. Common units are characters, words, sentences, and paragraphs. For example, you can split each document into chunks of 2048 characters or 512 words. You can also split each document so that each chunk can contain a fixed number of sentences (such as 20 sentences) or paragraphs (such as each paragraph is its own chunk).

You can also split documents recursively using increasingly smaller units until each chunk fits within your maximum chunk size. For example, you can start by splitting a document into sections. If a section is too long, split it into paragraphs. If a paragraph is still too long, split it into sentences. This reduces the chance of related texts being arbitrarily broken off.

Specific documents might also support creative chunking strategies. For example, there are splitters developed especially for different programming languages. Q&A documents can be splitted by question, answer pair, where each pair makes up a chunk. Chinese texts might need to be split differently from English texts.

When a document is split into chunks without overlap, the chunks might be cut off in the middle of important context, leading to the loss of critical information. Consider the text “I left my wife a note”. If it’s split into “I left my wife” and “a note”, neither of these two chunks conveys the key information of the original text. Overlapping ensures that important boundary information is included in at least one chunk. If you set the chunk size to be 2048 characters, you can perhaps set the overlapping size to be 20 characters.

The chunk size shouldn’t exceed the maximum context length of the generative model. For the embedding-based approach, the chunk size also shouldn’t exceed the embedding model’s context limit.

You can also chunk documents using tokens, determined by the generative model’s tokenizer, as a unit. Let’s say that you want to use Llama 3 as your generative model. You then first tokenize documents using Llama 3’s tokenizer. You can then split documents into chunks using tokens as the boundaries. Chunking by tokens makes it easier to work with downstream models. However, the downside of this approach is that if you switch to another generative model with a different tokenizer, you’d need to reindex your data.

Chunk sizes matter. A smaller chunk size allows for more diverse information. Smaller chunks mean that you can fit more chunks into the model’s context. If you halve the chunk size, you can fit twice as many chunks. More chunks can provide a model with a wider range of information, which can enable the model to produce a better answer.

Small chunk sizes, however, can cause the loss of important information. Imagine a document that contains important information about the topic X throughout the document, but X is only mentioned in the first half. If you split this document into two chunks, the second half of the document might not be retrieved, and the model won’t be able to use its information.

Smaller chunk sizes can also increase computational overhead. This is especially an issue for embedding-based retrieval. Halving the chunk size means that you have twice as many chunks to index and twice as many embedding vectors to generate and store. Your vector search space will be twice as big, which can reduce the query speed.

There is no universal best chunk size or overlap size. You have to experiment to find what works best for you.

122 of 131

Add graphic

ACCELERATOR.CC

Reranking

The initial document rankings generated by the retriever can be further reranked to be more accurate. Reranking is especially useful when you need to reduce the number of retrieved documents, either to fit them into your model’s context or to reduce the amount of input tokens.

One common pattern for reranking was discussed in the section Combine retrieval algorithms. A cheap but less precise retriever fetches candidates, then a more precise but more expensive mechanism reranks these candidates.

Documents can also be reranked based on time, giving higher weight to more recent data. This is useful for time-sensitive applications such as news aggregation, chat with your emails, or stock market analysis.

Context reranking differs from traditional search reranking in that the exact position of items is less critical. In search, the rank (e.g., first or fifth) is crucial. In context reranking, the order of documents still matters because it affects how well a model can process them. Models might better understand documents at the beginning and end of the context, as discussed earlier in the section “Context length and context efficiency”. However, as long as a document is included, the impact of its order is less significant compared to in search ranking.

123 of 131

ACCELERATOR.CC

Query Rewriting

Query rewriting is also known as query reformulation, query normalization, and sometimes query expansion. Consider the following conversation.

User: When was the last time John Doe bought something from us?

AI: John last bought a Fruity Fedora hat from us two weeks ago, on January 3, 2030.

User: How about Emily Doe?

The last question, “How about Emily Doe?”, is ambiguous without context. If you use this query verbatim to retrieve documents, you’ll likely get irrelevant results. You need to rewrite this query to reflect what the user is actually asking. The new query should make sense on its own. In this case, the query should be rewritten to “When was the last time Emily Doe bought something from us?”

Query rewriting is typically done using other AI models, using a prompt similar to “Given the following conversation, rewrite the last user input to reflect what the user is actually asking.” Figure 6-5 shows how ChatGPT rewrote the query using this prompt.

Image 1

You can use other generative models to rewrite queries

Query rewriting can get complicated, especially if you need to do identity resolution or incorporate other knowledge. For example, if the user asks “How about his wife?”, you will first need to query your database to find out who his wife is. If you don’t have this information, the rewriting model should acknowledge that this query isn’t solvable instead of hallucinating a name, leading to a wrong answer.

124 of 131

ACCELERATOR.CC

Start with Synthetic Data

warns against using scores on a 1-5 scale, instead promoting an alternative he calls "Critique Shadowing". Find a domain expert (one is better than many, because you want to keep their scores consistent) and have them answer the yes/no question "Did the AI achieve the desired outcome?" - providing a critique explaining their reasoning for each of their answers.

This gives you a reliable score to optimize against, and the critiques mean you can capture nuance and improve the system based on that captured knowledge.

Most importantly, the critique should be detailed enough so that you can use it in a few-shot prompt for a LLM judge. In other words, it should be detailed enough that a new employee could understand it.

Once you've gathered this expert data system you can switch to using an LLM-as-a-judge. You can then iterate on the prompt you use for it in order to converge its "opinions" with those of your domain expert.

Hamel concludes:

The real value of this process is looking at your data and doing careful analysis. Even though an AI judge can be a helpful tool, going through this process is what drives results. I would go as far as saying that creating a LLM judge is a nice “hack” I use to trick people into carefully looking at their data!

125 of 131

ACCELERATOR.CC

Continuous RAG Improvement

126 of 131

ACCELERATOR.CC

Initial Implementation
Synthetic Data Generation
Fast Evaluations
Real-World Data Collection
Classification and Analysis
System Improvements
Production Monitoring
User Feedback Integration
Iteration

127 of 131

ACCELERATOR.CC

Start with Synthetic Data

The biggest mistake I see teams make is spending too much time on complex generation before understanding if their retrieval even works. Synthetic data is your secret weapon here.

Generate synthetic questions for each chunk of text in your database. Use these to test your retrieval system and calculate precision and recall scores. This gives you a baseline to work from and helps identify low-hanging fruit for improvement.

Why is this so powerful?

It helps you select the right embedding models and methods

Enables lightning-fast evaluations (milliseconds vs. seconds per question)

Allows rapid iteration and testing of ideas

Can be done before you have any real user data

Forces clarity on product goals and non-goals

When you have concrete metrics like precision and recall, your stand-ups become far more productive. Instead of vague progress reports, you can say things like: "We improved recall by 5% by tweaking our chunking strategy." This focuses the team and gives leadership clear indicators of progress.

128 of 131

ACCELERATOR.CC

Focus on Leading Metrics

129 of 131

ACCELERATOR.CC

Fast, Unit Test-Like Evaluations

130 of 131

ACCELERATOR.CC

Real-World Data and Clustering

Once you have some real user data, things get interesting. You'll quickly realize that real-world questions are often stranger and more idiosyncratic than your synthetic ones. They may not even have clear answers within your system.

This is where clustering becomes powerful:

Use unsupervised learning to identify question topics and patterns

Work with domain experts to refine and label these clusters

Build few-shot classifiers to generate topic distributions for new questions

Now you can analyze:

Types and frequency of questions per topic

Cosine similarity scores within clusters

Customer satisfaction and feedback per topic

This segmentation is crucial. Just like Google eventually specialized into Maps, Images, and Shopping, you'll likely need to build targeted solutions for different question types.

131 of 131

ACCELERATOR.CC

Continuous Improvement Loop