1 of 38

Generative AI

and LangStream.ai

December, 2023

Stream Meet @Bigbasket

Jauneet Singh

2 of 38

About myself

2

Jauneet Singh

Open Source Enthusiast
Working at DataStax on distributed databases, streaming as senior Data architect.
Working on on Generative AI (LangStream.ai + LangChain + RAGStack)
Technology consultant to startup’s
Trainer
Food, Drive and Drive to new places for food

3 of 38

GenAI is changing the industry landscape

3

4 of 38

Prompting

4

Using GenAI is just about building a prompt

The problem to solve is not anymore in the model, it’s in the prompt.
The quality of the description of the problem directly impacts the accuracy of the result.
The LLM is static:

The live data must be included into the prompt
Information not present in the LLM training data must be included into the prompt

The size of the prompt is limited

5 of 38

Anatomy of a prompt

5

Problem to Solve:�Eg: “Should we propose a reduction coupon to the user ?”

6 of 38

Anatomy of a prompt

6

Problem to Solve:�Eg: “Should we propose a reduction coupon to the user ?”

Dynamic data:

Eg: “user payment history, item price, item stock, …”

Aggregate structured data from multiple sources in real-time.

Event-Driven Architectures are a perfect match for this.

7 of 38

Anatomy of a prompt

7

Problem to Solve:�Eg: “Should we propose a reduction coupon to the user ?”

Dynamic data (RAG):

Eg: “user payment history, item price, item stock, …”

Domain specific rules:

Eg: “rules to propose a reduction”

Data to solve the problem that was not used during the training.

8 of 38

Anatomy of a prompt

8

Problem to Solve:�Eg: “Should we propose a reduction coupon to the user ?”

Dynamic data:

Eg: “user payment history, item price, item stock, …”

Domain specific data:

Eg: “rules to propose a reduction”

Limited size.�For OpenAI’s gpt-3.5-turbo: 4097 tokens (token ~= word)

For OpenAI’s gpt-3.5-turbo-16k: 16385 tokens

gpt-4-32k: 32,768

gpt-4-turbo: 128,000

9 of 38

Problem: the domain specific data is huge and doesn’t fit into the LLM prompt.

9

10 of 38

What is a vector/embedding ?

10

v1

v2

v3

x

y

0

1

An embedding model transforms a text into a vector called an embedding.
The embedding can be N dimensions. For instance OpenAI’s embeddings are 1536 dimensions.
Similarity: v1 is more similar to v2 than v3. This is a simple mathematical formula.

2 dimensions normalised vectors

11 of 38

What is a vector/embedding ?

11

cat

dog

house

x

y

0

The vector captures the essence of a word or a block of text within its context.
The dimensions are the result of the LLM training.

12 of 38

Vector Search

12

Vector stores / vector databases

Embeddings storage (with or without metadata depending on the DB)
Built-in algorithms for fast retrieval of so-called “nearest-neighbors” embeddings (eg. HNSW, JVector, …)
Vectors are a new type of data supported in established databases (DataStax AstraDB, Cassandra, …) and new specialized databases (Pinecone, Milvius, …)

13 of 38

Let’s go back to the problem: our domain specific data is huge and doesn’t fit into the LLM prompt.

13

14 of 38

Retrieval Augmented Generation (RAG)

14

Domain specific data

Vector

Database

(eg Astra, Pinecone)

Embeddings model

(eg OpenAI)

Compute embeddings

Prompt question

Compute embedding

Similarity search

Build a prompt

Store embeddings

Query the LLM

LLM

(eg OpenAI)

15 of 38

Realtime Generative AI

15

Data

Streaming system (Kafka, Pulsar, …)

Web site crawling

Database

S3 / Block storage

Micro-service

16 of 38

Realtime Generative AI

16

Unstructured data

Extract text (eg. from PDF, HTML, MS doc)

Normalise (lower case, trim spaces)

Detect/filter/tag language

Text splitting

Data

Streaming system (Kafka, Pulsar, …)

Web site crawling

Database

S3 / Block storage

Micro-service

17 of 38

Realtime Generative AI

17

Structured data

Data manipulation

Conversion

Filtering

Flow control

Unstructured data

Extract text (eg. from PDF, HTML, ,MS doc)

Normalise (lower case, trim spaces)

Detect/filter/tag language

Text splitting

Data

Streaming system (Kafka, Pulsar, …)

Web site crawling

Database

S3 / Block storage

Micro-service

18 of 38

Realtime Generative AI

18

Structured data

Data manipulation

Conversion

Filtering

Flow control

Unstructured data

Extract text (eg. from PDF, HTML, ,MS doc)

Normalise (lower case, trim spaces)

Detect/filter/tag language

Text splitting

AI Operations

Compute embeddings

Store embeddings

Similarity search

Re-rank (MMR)

Chat/text completions (LLM)

Data

Streaming system (Kafka, Pulsar, …)

Web site crawling

Database

S3 / Block storage

Micro-service

19 of 38

Streaming

19

Streaming and Event Driven Architectures

System integration
Scalability
Back-pressure
Fault-tolerance
Replaying your data

20 of 38

Here comes LangStream

20

21 of 38

21

LLMs

Vector stores

AI libraries

Connectors

Messaging

22 of 38

What is LangStream?

22

Open source

Apache 2.0 licensed
Open community discussions and roadmap
GitHub: https://github.com/LangStream/langstream
Slack or Linen

23 of 38

What is LangStream?

23

Not just a library

Framework for developing generative AI applications
Runtime environment for generative AI applications
Data integration platform to bring relevant and recent data to gen AI
Powered by proven technology: Kubernetes, Kafka, Kafka Connect

24 of 38

What is LangStream?

24

Rapid development and deployment

Configuration-driven, no-code agents for rapid application prototyping
Supports custom agents written in Python using popular libraries like LangChain and LlamaIndex
Developer friendly tooling for deploying applications to multiple environments
Support for unstructured (PDF, HTML) and structured data

25 of 38

What is LangStream?

25

Event-driven, real-time architecture

Event-driven architecture for scaling, robustness, extensibility
Asynchronous processing and micro-batching (“pipelining”) for performance
Agent-level observability with Prometheus metrics, Kubernetes logging
Enables building of responsive/reactive Gen AI applications

26 of 38

High-Level Architecture

26

Kubernetes

Kafka

Agent�chat

completion

Vector DB

AI Services

Agent�OSS�embedding

LangStream

LangStream Operator

DB

Control Plane and API

WebSocket Gateway

27 of 38

Agents

27

Source

Processor

KC

Sink

KC

built-in

Sources:

Kafka
Pulsar
Kafka-Connect
Apache Camel
Python custom
Web crawler
S3/Azure
…

Sinks:

Kafka
Pulsar
Kafka-Connect
Python custom
Vector DB
…

1 pod

Processors:

Python custom
Compute embeddings
Chat completions
Compute fields
Drop fields
Extract text
Normalise text
Detect language
Split into chunks
Query Vector DB
Re-rank documents
…

28 of 38

Declarative low-code

28

topics:

- name: "input-topic"

creation-mode: create-if-not-exists

- name: "output-topic"

creation-mode: create-if-not-exists

pipeline:

- type: "document-to-json"

input: "input-topic"

configuration:

text-field: "question"

- type: "ai-chat-completions"

configuration:

model: "${secrets.open-ai.chat-completions-model}"

completion-field: "value.answer"

log-field: "value.prompt"

stream-to-topic: "output-topic"

stream-response-completion-field: "value"

min-chunks-per-message: 10

messages:

- role: user

content: "You are a helpful assistant. Below you can find a question from the user. Please try to help them the best way you can.\n\n{{% value.question}}"

Example application:

Reads user input
Asks OpenAI for chat completion
Writes answer to the stream

29 of 38

Web Gateway

29

WebSocket interface
Authentication (OAuth2, Google, GitHub, custom…)
Easy to use in Frontend applications
Streaming responses

gateways:

- id: "user-input"

type: produce

topic: "${globals.questionsTopic}"

parameters:

- sessionId

authentication:

provider: github

configuration:

clientId: "${secrets.github.client-id}"

30 of 38

Agenda

30

ChatBot with LangStream

E-Commerce recommendation service (LangChain)

Async e-Commerce recommendation service (LangChain + LangStream)

31 of 38

Build a ChatBot with LangStream

31

Crawl and index LangStream documentation
Answer questions about LangStream documentation

brew install LangStream/langstream/langstream

export OPEN_AI_ACCESS_KEY=<your-openai-api-key>

langstream docker run test -app https://github.com/LangStream/langstream/tree/main/examples/applications/docker-chatbot -s https://github.com/LangStream/langstream/blob/main/examples/secrets/secrets.yaml