Advanced RAG patterns �with AWS and LlamaIndex
Simon Suo, Cofounder / CTO @ LlamaIndex
Shreyas Subramanian, Principal Data Scientist @ AWS
© 2024, Amazon Web Services, Inc. or its affiliates.
© 2024, Amazon Web Services, Inc. or its affiliates.
Agenda
© 2024, Amazon Web Services, Inc. or its affiliates.
LlamaIndex Advanced RAG
3
© 2024, Amazon Web Services, Inc. or its affiliates.
© 2024, Amazon Web Services, Inc. or its affiliates.
Optimizing: From Simple to Advanced
Less Expressive
Easier to Implement
Lower Latency/Cost
More Expressive
Harder to Implement
Higher Latency/Cost
Table Stakes
Better Parsers
Chunk Sizes
Hybrid Search
Metadata Filters
🛠️
Advanced Retrieval
Reranking
Recursive Retrieval
Embedded Tables
Small-to-big Retrieval
🔎
Agentic Behavior
Routing
Query Planning
Multi-document Agents
🤖
Fine-tuning
Embedding fine-tuning
LLM fine-tuning
⚙️
© 2024, Amazon Web Services, Inc. or its affiliates.
RAG in the wild
© 2024, Amazon Web Services, Inc. or its affiliates.
Introducing LlamaParse
Capabilities
✅ Extracts tables / charts
✅ Input natural language parsing instructions
✅JSON mode
✅Image Extraction
✅Support for ~10+ document types (.pdf, .pptx, .docx, .xml)
© 2024, Amazon Web Services, Inc. or its affiliates.
Current PDFReader
Llama Parse
© 2024, Amazon Web Services, Inc. or its affiliates.
Advanced Table Understanding
© 2024, Amazon Web Services, Inc. or its affiliates.
Parsing Instructions
© 2024, Amazon Web Services, Inc. or its affiliates.
JSON Mode
© 2024, Amazon Web Services, Inc. or its affiliates.
RAG over Powerpoints
© 2024, Amazon Web Services, Inc. or its affiliates.
From RAG to Agents
RAG
Query
Response
© 2024, Amazon Web Services, Inc. or its affiliates.
From RAG to Agents
Agents?
RAG
Query
Response
Agents?
Agents?
© 2024, Amazon Web Services, Inc. or its affiliates.
From RAG to Agents
Agents?
RAG
Query
Response
Agents?
Agent Definition: Using LLMs for automated reasoning and tool selection
RAG is just one Tool: Agents can decide to use RAG with other tools
Agents?
© 2024, Amazon Web Services, Inc. or its affiliates.
From Simple to Advanced Agents
Simple
Lower Cost
Lower Latency
Advanced
Higher Cost
Higher Latency
Routing
One-Shot Query Planning
Tool Use
ReAct
Dynamic Planning + Execution
© 2024, Amazon Web Services, Inc. or its affiliates.
Routing
Simplest form of agentic reasoning.
Given user query and set of choices, output subset of choices to route query to.
Use Case: Joint QA and Summarization
© 2024, Amazon Web Services, Inc. or its affiliates.
Query Planning
Break down query into parallelizable sub-queries.
Each subquery can be executed against any set of RAG pipelines
Use Case: multidoc reasoning
Example: Compare revenue of Uber and Lyft in 2021
Uber 10-K chunk 4
top-2
Uber 10-K chunk 8
Lyft 10-K chunk 4
Lyft 10-K chunk 8
Compare revenue growth of Uber and Lyft in 2021
Uber 10-K
Lyft 10-K
Describe revenue growth of Uber in 2021
Describe revenue growth of Lyft in 2021
top-2
© 2024, Amazon Web Services, Inc. or its affiliates.
Tool Use
Use LLM to infer parameter for different API interfaces
Examples:
© 2024, Amazon Web Services, Inc. or its affiliates.
This is cool but
© 2024, Amazon Web Services, Inc. or its affiliates.
This is cool but
© 2024, Amazon Web Services, Inc. or its affiliates.
Data Agents - Core Components
© 2024, Amazon Web Services, Inc. or its affiliates.
ReAct: Reasoning + Acting with LLMs
Source: https://react-lm.github.io/
© 2024, Amazon Web Services, Inc. or its affiliates.
ReAct: Reasoning + Acting with LLMs
Add a loop around query decomposition + tool use
© 2024, Amazon Web Services, Inc. or its affiliates.
ReAct: Reasoning + Acting with LLMs
© 2024, Amazon Web Services, Inc. or its affiliates.
Can we make this even better?
© 2024, Amazon Web Services, Inc. or its affiliates.
LLMCompiler
Kim et al. 2023
An agent compiler for parallel multi-function planning + execution.
© 2024, Amazon Web Services, Inc. or its affiliates.
LLMCompiler
Plan out steps beforehand, and replan as necessary
© 2024, Amazon Web Services, Inc. or its affiliates.
Tree-based Planning
Tree of Thoughts (Yao et al. 2023)
Reasoning via Planning (Hao et al. 2023)
Language Agent Tree Search (Zhou et al. 2023)
© 2024, Amazon Web Services, Inc. or its affiliates.
Additional Considerations
© 2024, Amazon Web Services, Inc. or its affiliates.
LlamaIndex + AWS�Patterns
30
© 2024, Amazon Web Services, Inc. or its affiliates.
© 2024, Amazon Web Services, Inc. or its affiliates.
Examples available
on github
© 2024, Amazon Web Services, Inc. or its affiliates.
Pattern 1 – Use Llamaparse and Llamacloud APIs
© 2024, Amazon Web Services, Inc. or its affiliates.
Pattern 2 – Use LlamaIndex with Bedrock models
© 2024, Amazon Web Services, Inc. or its affiliates.
Pattern 3 – Router Query Engine
© 2024, Amazon Web Services, Inc. or its affiliates.
Pattern 4 - Agentic RAG
© 2024, Amazon Web Services, Inc. or its affiliates.
Pattern 5 – Connect to �other AWS services
© 2024, Amazon Web Services, Inc. or its affiliates.
Let’s see some code
© 2024, Amazon Web Services, Inc. or its affiliates.
© 2024, Amazon Web Services, Inc. or its affiliates.
Simon Suo
Co-founder / CTO
simon@runllama.ai
Shreyas Subramanian, PhD Principal Data Scientist
subshrey@amazon.com
© 2024, Amazon Web Services, Inc. or its affiliates.
Thank you!
© 2024, Amazon Web Services, Inc. or its affiliates.