The Future of Knowledge Assistants
Jerry Liu
July 9th, 2024
Building a Knowledge Assistant
2
Agent: …
Human: …
Agent: …
Knowledge Base
Answer:
Sources: …
Human: <Question>
Human: …
Goal: Build an interface that can take in any task as input and give back an output.
Input forms: simple questions, complex questions, research tasks
Output forms: short answer, structured output, research report
The LLM Application Ecosystem
Enterprise developer teams are the best equipped to build a new category of AI-powered software.
But ⚠️
Developers need tooling and infrastructure to efficiently build production-quality LLM applications.
Data
Foundation Models
Knowledge-Intensive LLM Applications
Sales
Dev
Legal
Finance
…
🛠️ Tooling + Infra
🧑💻 Developers
LlamaIndex: Build Production LLM Apps over Enterprise Data
LlamaIndex helps any developer build context-augmented LLM apps from prototype to production.
Open-Source: Leading developer toolkit for building production LLM apps over data.
1M+ monthly downloads
170K+ LinkedIn followers
30k+ stars
LlamaCloud: A centralized knowledge interface for your production LLM application.
RAG was just the beginning
🧠 🛠️
Knowledge Assistant with Basic RAG
⚠️ Naive data processing, primitive retrieval interface
⚠️ Poor query understanding/planning
⚠️ No function calling or tool use
⚠️ Stateless, no memory
6
Data Processing and Indexing
Index
Data
Basic Text Splitting
Top-k = 5
Simple QA Prompt
Response
Basic Retrieval and Prompting
Problem: Productionizing LLM Applications at Scale
Pain Points
Accuracy Issues:
Bad retrievals, hallucinations
Hard to Improve:
Too many parameters, deep-tech expertise required
Hard to Scale:
every new data source requires eng hours, custom parsing, tuning
Prototype
Productionize
Small set of simple documents
Low quality bar
Large set of complex, heterogeneous documents
High quality bar
Data Silos Exacerbate this Problem
Structured data sits in Snowflake, BigQuery, and Redshift clusters
Unstructured data sits in your S3 buckets, filesystems
APIs
Raw Files
SQL DBs
Vector Stores
Production LLM Apps
Q&A
Chat
Copilot
Autonomous Agents
Structured Extraction
?
?
?
?
Can we do more?
There’s many questions/tasks that naive RAG can’t give an answer to.
🚫 Hallucinations
🚫 Limited time savings
🚫 Limited decision-making enhancement
9
💡 How do we aim to build a production-ready knowledge assistant?
A Better Knowledge Assistant
10
Advanced Data and Retrieval Tool
Index
Data
Data Processing
Agent
Response
A Better Knowledge Assistant
11
Advanced RAG and Retrieval Tool
Response
Other tools
Other tools
Agent
Memory
Tool Use
Query Planning
Reflection
A Better Knowledge Assistant
12
Agent 1
E.g. Orchestrator
Agent 2
E.g. RAG
Agent 3
E.g. SQL
Task
Agent 4
E.g. Reflection
Response
Advanced Data and Retrieval
🧠 🛠️
Any LLM App is only as Good as your Data
Garbage in = garbage out
Good data quality is a necessary component of any production LLM app.
14
Raw Data
Data Processing
Clean Data
Production LLM Apps
Q&A
Chat
Copilot
Autonomous Agents
Structured Extraction
Any LLM App is only as Good as your Data
15
ETL for LLMs
Index
Data
Data Processing
Agent
Response
LlamaCloud: Parsing (LlamaParse)
“As an AI Applied Data Scientist who was granted one of the first ML patents in the U.S., and who is building cutting-edge AI capabilities at one of the world’s largest Private Equity Funds, I can confidently say that LlamaParse from LlamaIndex is currently the best technology I have seen for parsing complex document structures for Enterprise RAG pipelines. Its ability to preserve nested tables, extract challenging spatial layouts, and images is key to maintaining data integrity in advanced RAG and agentic model building.”
Dean Barr, Applied AI Lead at Carlyle
LlamaCloud: Advanced ETL
Centralize and enhance your data for your production LLM application.
“LlamaCloud has really sped up our development timelines. Getting to technical prototypes quickly allows us to show tangible value instantly, improving our sales outcomes. When needed, switching from the LlamaCloud UI to code has been really seamless. The configurable parsing and retrieval features have significantly improved our response accuracy. We've also seen great results with LlamaParse and found it outperforming GPT-4 vision on some OCR tasks!”
Teemu Lahdenpera, CTO at Scaleport.ai
Use Cases
Customers (pending name approval):
Select Verticals
financial services, consulting, healthcare, tech
Select Use Cases
Financial Analyst Assistant
Centralized Internal Search (Organization + User-level)
Analytics Dashboard for Sensor Data
Internal LLM Application Development Platform
Advanced Single-Agent Flows
🧠 🛠️
From Simple to Advanced Agents
20
Full Agents
Agent Ingredients
Simple
Lower Cost
Lower Latency
Advanced
Higher Cost
Higher Latency
Routing
One-Shot Query Planning
Tool Use
ReAct
Dynamic Planning + Execution
Conversation Memory
Agentic RAG
Every data interface is a tool
Use agent reasoning loops (sequential, DAG, tree) to tackle complex tasks
End Result: Build personalized QA systems capable of handling complex questions!
21
21
Advanced RAG and Retrieval Tool
Response
Other tools
Other tools
Agent
Memory
Tool Use
Query Planning
Reflection
Remaining Gaps
A single agent cannot solve an infinite set of tasks - specialist agents do better!
Agents are increasingly interfacing with services that may be other agents
22
Multi-Agent Task Solvers
🧠 🛠️
Multi-Agents: Motivation and Use Case
Why Multi-Agents?
Challenges:
24
Llama Agents (Preview)
Agents as microservices.
25
Llama Agents: Architecture
26
LlamaCloud: Opening up a Waitlist
Advanced unstructured data processing for LLMs
27
Advanced ETL
Advanced Parsing
Scan here