1 of 39

Advanced RAG patterns �with AWS and LlamaIndex

Simon Suo, Cofounder / CTO @ LlamaIndex

Shreyas Subramanian, Principal Data Scientist @ AWS

© 2024, Amazon Web Services, Inc. or its affiliates.

© 2024, Amazon Web Services, Inc. or its affiliates.

2 of 39

Agenda

  • Advanced RAG
    • Complex-document RAG
    • Agentic RAG
  • LlamaIndex + AWS
    • 5 example patterns
  • Notebook walkthrough
    • Llamaindex & LlamaParse + Bedrock KB

© 2024, Amazon Web Services, Inc. or its affiliates.

3 of 39

LlamaIndex Advanced RAG

3

© 2024, Amazon Web Services, Inc. or its affiliates.

© 2024, Amazon Web Services, Inc. or its affiliates.

4 of 39

Optimizing: From Simple to Advanced

Less Expressive

Easier to Implement

Lower Latency/Cost

More Expressive

Harder to Implement

Higher Latency/Cost

Table Stakes

Better Parsers

Chunk Sizes

Hybrid Search

Metadata Filters

​

🛠️

​

Advanced Retrieval

Reranking

Recursive Retrieval

Embedded Tables

Small-to-big Retrieval

​

🔎

Agentic Behavior

Routing

Query Planning

Multi-document Agents

​

🤖

Fine-tuning

Embedding fine-tuning

LLM fine-tuning

​

⚙️

© 2024, Amazon Web Services, Inc. or its affiliates.

5 of 39

RAG in the wild

  • Complex layout
  • Tables, charts, schematics
  • Scanned, handwritten

© 2024, Amazon Web Services, Inc. or its affiliates.

6 of 39

Introducing LlamaParse

Capabilities

✅ Extracts tables / charts

✅ Input natural language parsing instructions

✅JSON mode

✅Image Extraction

✅Support for ~10+ document types (.pdf, .pptx, .docx, .xml)

​

​

​

© 2024, Amazon Web Services, Inc. or its affiliates.

7 of 39

Current PDFReader

Llama Parse

© 2024, Amazon Web Services, Inc. or its affiliates.

8 of 39

Advanced Table Understanding

© 2024, Amazon Web Services, Inc. or its affiliates.

9 of 39

Parsing Instructions

© 2024, Amazon Web Services, Inc. or its affiliates.

10 of 39

JSON Mode

© 2024, Amazon Web Services, Inc. or its affiliates.

11 of 39

RAG over Powerpoints

© 2024, Amazon Web Services, Inc. or its affiliates.

12 of 39

From RAG to Agents

RAG

Query

Response

© 2024, Amazon Web Services, Inc. or its affiliates.

13 of 39

From RAG to Agents

Agents?

RAG

Query

Response

Agents?

Agents?

© 2024, Amazon Web Services, Inc. or its affiliates.

14 of 39

From RAG to Agents

Agents?

RAG

Query

Response

Agents?

Agent Definition: Using LLMs for automated reasoning and tool selection

​

RAG is just one Tool: Agents can decide to use RAG with other tools

Agents?

© 2024, Amazon Web Services, Inc. or its affiliates.

15 of 39

From Simple to Advanced Agents

Simple

Lower Cost

Lower Latency

Advanced

Higher Cost

Higher Latency

Routing

One-Shot Query Planning

Tool Use

ReAct

Dynamic Planning + Execution

© 2024, Amazon Web Services, Inc. or its affiliates.

16 of 39

Routing

Simplest form of agentic reasoning.

Given user query and set of choices, output subset of choices to route query to.

​

​

Use Case: Joint QA and Summarization

Guide

© 2024, Amazon Web Services, Inc. or its affiliates.

17 of 39

Query Planning

Break down query into parallelizable sub-queries.

Each subquery can be executed against any set of RAG pipelines

​

Use Case: multidoc reasoning

​

Example: Compare revenue of Uber and Lyft in 2021

Query Planning Guide

Uber 10-K chunk 4

top-2

Uber 10-K chunk 8

Lyft 10-K chunk 4

Lyft 10-K chunk 8

Compare revenue growth of Uber and Lyft in 2021

Uber 10-K

Lyft 10-K

Describe revenue growth of Uber in 2021

Describe revenue growth of Lyft in 2021

top-2

© 2024, Amazon Web Services, Inc. or its affiliates.

18 of 39

Tool Use

Use LLM to infer parameter for different API interfaces

​

Examples:

  • Auto-retrieval -> VectorDB
  • text-to-SQL -> SQL DB
  • etc

© 2024, Amazon Web Services, Inc. or its affiliates.

19 of 39

This is cool but

  • How can an agent tackle sequential multi-part problems?
  • How can an agent maintain state over time?

© 2024, Amazon Web Services, Inc. or its affiliates.

20 of 39

This is cool but

  • How can an agent tackle sequential multi-part problems?
    • Let’s make it loop
  • How can an agent maintain state over time?
    • Let’s add basic memory

© 2024, Amazon Web Services, Inc. or its affiliates.

21 of 39

Data Agents - Core Components

Agent Reasoning Loop

​

  • ReAct Agent (any LLM)
  • OpenAI Agent (only OAI)

Tools

Query Engine Tools (RAG pipeline)

​

LlamaHub Tools (30+ tools to external services)

© 2024, Amazon Web Services, Inc. or its affiliates.

22 of 39

ReAct: Reasoning + Acting with LLMs

© 2024, Amazon Web Services, Inc. or its affiliates.

23 of 39

ReAct: Reasoning + Acting with LLMs

Add a loop around query decomposition + tool use

© 2024, Amazon Web Services, Inc. or its affiliates.

24 of 39

ReAct: Reasoning + Acting with LLMs

Superset of query planning + routing capabilities.

​

ReAct + RAG Guide

© 2024, Amazon Web Services, Inc. or its affiliates.

25 of 39

Can we make this even better?

  • Stop being so short-sighted - plan ahead at each step
  • Parallelize execution where we can

© 2024, Amazon Web Services, Inc. or its affiliates.

26 of 39

LLMCompiler

Kim et al. 2023

An agent compiler for parallel multi-function planning + execution.

© 2024, Amazon Web Services, Inc. or its affiliates.

27 of 39

LLMCompiler

Plan out steps beforehand, and replan as necessary

LLMCompiler Agent

© 2024, Amazon Web Services, Inc. or its affiliates.

28 of 39

Tree-based Planning

Tree of Thoughts (Yao et al. 2023)

Reasoning via Planning (Hao et al. 2023)

Language Agent Tree Search (Zhou et al. 2023)

© 2024, Amazon Web Services, Inc. or its affiliates.

29 of 39

Additional Considerations

  • Observability: see the full trace of the agent
    • Observability Guide
  • Control: Be able to guide the intermediate steps of an agent step-by-step
  • Customizability: Define your own agentic logic around any set of tools.

© 2024, Amazon Web Services, Inc. or its affiliates.

30 of 39

LlamaIndex + AWS�Patterns

30

© 2024, Amazon Web Services, Inc. or its affiliates.

© 2024, Amazon Web Services, Inc. or its affiliates.

31 of 39

Examples available

on github

© 2024, Amazon Web Services, Inc. or its affiliates.

32 of 39

Pattern 1 – Use Llamaparse and Llamacloud APIs

  • Async trigger Lambda to look at documents dropped to S3
  • Use LlamaParse to convert to Markdown
  • Ingest into Llamaindex

© 2024, Amazon Web Services, Inc. or its affiliates.

33 of 39

Pattern 2 – Use LlamaIndex with Bedrock models

  • Use Secrets manager to store API key
  • Lambda retrieves documents from LlamaIndex
  • Use context with Claude 3 Haiku on Bedrock

© 2024, Amazon Web Services, Inc. or its affiliates.

34 of 39

Pattern 3 – Router Query Engine

  • Create two or more indices and corresponding tools
  • LLM routes to the right index based on the query
  • For example, tree_summarize for summary questions, and vector index for detailed questions

© 2024, Amazon Web Services, Inc. or its affiliates.

35 of 39

Pattern 4 - Agentic RAG

  • Bedrock Agents can use use Lambda “action groups” as tools

​

  • Connect to Bedrock Knowledge Base as a data source (see demo)

​

​

© 2024, Amazon Web Services, Inc. or its affiliates.

36 of 39

Pattern 5 – Connect to �other AWS services

  • Central Lambda can have multiple API paths – Llamaindex, AWS Glue, Amazon Athena
  • Bedrock agent uses this as an action group

​

​

© 2024, Amazon Web Services, Inc. or its affiliates.

37 of 39

Let’s see some code

© 2024, Amazon Web Services, Inc. or its affiliates.

38 of 39

© 2024, Amazon Web Services, Inc. or its affiliates.

39 of 39

Simon Suo

Co-founder / CTO

simon@runllama.ai

Shreyas Subramanian, PhD Principal Data Scientist

subshrey@amazon.com

© 2024, Amazon Web Services, Inc. or its affiliates.

Thank you!

© 2024, Amazon Web Services, Inc. or its affiliates.