We make AI work
Vespa: Combining AI and data, online
Search, organization and machine-learned model inference
Data items can contain any number of
Vespa works seamlessly at any scale
Use cases: Search, RAG, recommendation, personalization, ad targeting
Some key users: Perplexity, Spotify, Yahoo!, Wix
One-page summary
About Vespa
Vespa is a separate company
2023
Vespa (a division in Yahoo) becomes company standard for Search & Personalization
2011
2021
Vespa Cloud is launched
(On-boarding External Customers)
Vespa is Open Sourced
2017
2014
Vespa adopts Vectors & Tensors
Powering the world’s largest companies in e-commerce, banking, retail, technology
You need more than a vector database, but Vespa is also a leader among vector databases
Download the full report at https://content.vespa.ai/gigaom-report-2024
Vespa: Combining AI and data, online
To achieve quality, leverage all types of data relevant to your problem
Most real-world problems need all of vectors, text, and structured data
Vespa lets you define and index data items with fields that are:
All fields can be used together, to
Vespa: Combining AI and data, online
Typical online serving requirements:
Low latency + high throughput is physically impossible to achieve unless:
Requires a platform handling all data types and computation.
Model all your data
Document type: research-article
Document id: 101.1029/2023GB007789
Category | Global biogeochemical cycles |
Title | Coral Reef Carbonate Chemistry Reveals Interannual Seasonal and Spatial Impacts on Ocean Acidification Off Florida |
Title embedding | |
Authors | |
Publish time | 1701903600 |
Content | Abstract. Ocean acidification (OA) threatens … |
Content chunk embeddings | … |
0.98 | 1.58 | 13.7 | 11.1 | 7.89 | 14.8 |
12.3 | 0.23 | 0.01 | 5.22 | 17.0 | 0.56 |
51.9 | 0.45 | 0.11 | 8.38 | 11.9 | 12.2 |
45.6 | 43.6 | 9.77 | 1.43 | 7.90 | 19.2 |
Palacio-Castro | Enochs | Besemer | … |
Use signals from all your data (using ML)
Document type: research-article
Document id: 101.1029/2023GB007789
Category | Global biogeochemical cycles |
Title | Coral Reef Carbonate Chemistry Reveals Interannual Seasonal and Soatial Impacts on Ocean Acidification Off Florida |
Title embedding | |
Authors | |
Publish time | 1701903600 |
Content | Abstract. Ocean acidification (OA) threatens … |
Content chunk embeddings | … |
0.98 | 1.58 | 13.7 | 11.1 | 7.89 | 14.8 |
12.3 | 0.23 | 0.01 | 5.22 | 17.0 | 0.56 |
51.9 | 0.45 | 0.11 | 8.38 | 11.9 | 12.2 |
45.6 | 43.6 | 9.77 | 1.43 | 7.90 | 19.2 |
Palacio-Castro | Enochs | Besemer | … |
signals
Vespa machine-learned scoring and inference
Combine:
(neural nets etc.)
In multiple ranking phases
Scale it by distribution and local inference
Content partition node 1
Content partition node 2
Content partition node 3
Query container node
When do you need Vespa?
You want to provide users with great search results.
You want to create a high quality GenAI+data (RAG) application.
You want to recommend users the best possible personalized content.
You want to provide all of recommendation, search and navigation in data.
You want to do something new combining AI and data, online.
Example: Semantic search at Spotify
Selecting and serving all the content on Yahoo pages/apps
Query by personalized profile, score using machine-learned models.
A mixture of many “streams” using various approaches
Example: Yahoo personalized content
Example: Ravenpack
(Bigdata.com)
RAG for finance analytics
Index, vectorize and extract knowledge from many types of finance data
Provide a chat interface to this data
SPANN indexes, hybrid search, machine-learned ranking
Example: Global 10 Bank
Vespa for
Deploying Vespa
Vespa’s managed service provides
Available in AWS and GCP, optionally in your accounts/projects
Open source: You can always also deploy and operate Vespa applications yourself using our Docker images/RPMs
Vespa, summarized
Store & index data with any number of vectors, tensors, text and structured fields.
Run queries specifying any condition over any collection of fields.
Use signals from any field to make data-local tensor computation/ML inference
Update individual fields in true real time, up to 100k+ updates/sec per node.
Embed data inside Vespa, using local or remote models.
Group and aggregate all data matching a query.
Serve data and dynamic snippets.
Run logic that intercepts queries, responses, writes etc, and integrates with LLMs
Use any number of schemas, clusters, ML models, and rankings in one application.
Scale to any amount of data and traffic, and rescale easily while serving.
Managed system: One to hundreds of nodes, scalable in any dimension, CD, autoscaling, auto-ops.
Architecture and capabilities
Vespa architecture overview
Scalable low-latency computation
Query
Application Package
Admin &
Config
Content node
Deploy
- Configuration
- Components
- ML models
Scatter-gather
Core sharding
models
models
models
Container node
models
Data in Vespa
Data in Vespa is represented by a schema
The data of a schema is managed on a content cluster
Fields can be indexed:
schema music {
document music {
field artist type string {
indexing: summary | index
enable-bm25
}
field embedding type tensor<int8>(x[128]) {
indexing: index | attribute
attribute {
distance-metric: euclidean
}
index {
hnsw
}
}
field popularity type int {
indexing: summary | attribute
attribute: fast-search
}
}
rank-profile myScoring inherits default {
first-phase {
expression {
0.5 * bm25(artist) +
0.3 * closeness(embedding) +
0.2 * attribute(popularity)
}
}
}
}
Queries in Vespa
Queries can match on any combination of fields:
select * from sources * where popularity > 0.1 and
(
(title contains “foo” and body contains “bar”) or
nearestNeighbor(titleEmbedding, myQueryEmbedding) or
nearestNeighbor(bodyEmbeddings, myQueryEmbedding)
)
Works on all indexes and all in-memory (attribute) fields.
Approximate nearest neighbor vector queries
ANN: Millions of vectors/node, of thousands of numbers, in milliseconds
Nearest neighbor searches are just items in query trees:
Efficient when combined with filter and text search items
Post-/pre-filtering/brute force automatically chosen in query planning
Vectors can be updated in real time, thousands of writes/second per node
Multiple vector fields per document, and multiple vectors per field: tensor(x{}, y[512])
Achieved by Vespa embedding its own in-process HNSW implementation in C++
Writes in Vespa
Writes are fully realtime and immediately persisted
Add/replace/delete documents, or update certain fields
End-to-end async write pipeline -> high throughput + realtime
Updates to in-memory fields are > 100k+/second per node
Data is automatically linguistically processed/embedded as appropriate, and reprocessed when needed
Data distribution
Vespa auto-distributes data over
Changes to nodes/configuration -> Automatic online data redistribution
No need to manually partition data or manage partition placement
Distribution based on the CRUSH algorithm: Minimal data movement without registry
<nodes count=’12’ groups=’4’>
<resources vcpu=’8’
memory=’30Gb’
disk=’300Gb’/>
</nodes>
Ranking and inference in Vespa
Any set of functions over tensors can be computed during ranking
Inputs are any query features, document attributes, and match features
Functions can be handwritten or machine learned (ONNX, GBDT)
Computation is local on data nodes
Multiple ranking phases: First, second, global
schema music {
rank-profile myScoring inherits default {
function scale(val) {
expression: 2*atan(val/4)/(3.14159)
}
first-phase {
expression {
0.5 * scale(bm25(artist))*completeness(artist))
0.3 * closeness(embedding) +
0.2 * attribute(popularity)
}
}
second-phase {
expression: onnx(my_onnx_model) + xgboost(my_gbdt)
}
global-phase {
expression: sum(onnx(my_large_onnx_model))
rerank-count: 100
}
Inference in Vespa: tensors
Tensor data model: Multidimensional collections of numbers: In queries, documents, and models |
Tensor math operations express all common machine-learned models with join, map, reduce etc. |
Tensor dimensions may be sparse (mapped) or dense (indexed): tensor<float>(key{}, x[384]) Math operations work the same over both. |
Makes it possible to evaluate any large and complex ML model on Vespa: Neural nets, collaborative filtering over all data, custom doc/query interactions etc, |
Inference in Vespa: Signals
Any attribute in documents can be used as rank features (scalars and tensors)
Fast partial updates -> Signals can track behaviors in real time
Text ranking is much more than bm25
Linguistic processing in Vespa
Linguistic processing is crucial for good quality text matching and ranking
Happens automatically
Match modes can be set by fields:
lexical with/without stemming/prefix/fuzzy/n-gram/regexp/exact
Embedding in Vespa
Embeddings can (optionally) be created inside Vespa
Any model, provided by Vespa, or your own
Embedders can produce multiple vectors: Short/long, high/low precision
Queries:
Documents:
<container id="myEmbedder" version="1.0">
<component id="hf-embedder" type="hugging-face-embedder">
<transformer-model path="my-models/model.onnx"/>
<tokenizer-model path="my-models/tokenizer.json"/>
</component>
...
</container>
field embeddings type tensor<bfloat16>(p{},x[1024]) {
indexing: input chunks | embed myEmbedder | attribute | index
index: hnsw
}
input.query(q)=embed(myEmbedder, "Hello world")
Vector/tensor support
Vectors values:
double, float, bfloat16, int8, binary
Sparse and dense dimensions
Any dimension size, any number of dimensions (tensors)
Multiple vectors, and multiple vector values per document
Vectors can be used in queries/ranking whether indexed or not
Binary vectors: 8-30 times cheaper and faster
Vectors for ranking can be paged to disk
Grouping and aggregation
Organize data at query time
(yql query) | all(group(merchant) each( output(count()) max(3) each(output(summary()))))
Example use:
Operates on all matches to a query, over all nodes involved.
Orchestration logic and LLMs in Vespa
Applications can add any logic operating efficiently on queries, results and documents
Logic can access local or remote LLMs
Personal content search
When searching personal data, indexes are not cost effective
Vespa supports streaming search for such cases
About 20x cheaper than regular indexing
Retrieves all nearby documents with vector search (not approximate)
Supports the same features and APIs as the regular index mode
<content id="mycluster" version="1.0">
<documents>
<document type="email" mode="streaming" />
Additional notable features
Innovation at scale
Vespa is designed to deliver sustainable fast development at scale
Vespa operations
Vespa systems are designed for resiliency
Cloud resources are automatically provisioned
Changes to resources are automatically and safely managed
Autoscaling allows resources to follow load and data sizes
The Vespa Console makes it easy to get insight into your applications
<content id=myCluster version='1.0'>
<redundancy>2</redundancy>
<documents>
<document mode='index' type='music'/>
</documents>
<nodes count='[10, 200]' groups='[1, 10]'>
<resources vcpu='10' memory='30Gb' disk='[50Gb, 1Tb]'/>
</nodes>
</content>
Releases and deployment
New verified production releases of Vespa are released once per day (Monday-Thursday)
All versions are backwards compatible (except bi-yearly major version changes)
Applications are automatically upgraded (Vespa and OS) without downtime
Applications can run in Vespa’s AWS account/Gcp project, or yours (Enclave)
Vespa is fully open source (Apache 2.0):
Resources
Vespa documentation: https://docs.vespa.ai
The Vespa blog: https:// blog.vespa.ai
Get started in minutes: https://cloud.vespa.ai/en/getting-started
Questions?