ALL-Sort

improved retrieval augmented generation

Trelis Research

All-sort

(assisted large language sorting)

Text chunks

Database

--------

--------

--------

--------

​

Helper LLM (Smaug 34B)

Sorted Chunks

​

Text chunks

Rate Relevance

​

all-sort

Prompt

=

{ context = high relevance chunks }

+

{ Query }

Sorted Chunks

​

overview

1. Preparing the test questions.
2. Full context prompt setup.
3. “standard” RAG setup.
4. All-sort setup:
1. Enforcing Regex for responses.
2. Classifying chunks.
3. Sorting chunks by relevance.
4. Deploying an api endpoint (SMAUG 34B).
5. Live demo running all-sort.
6. Costing + latency
7. Helper model Ablations: openchat_3.5 7b, mixtral, yi.

costing

Assumptions:

• 100k tokens of data.
• GPT-4-Turbo costs $1 per 100k.
• ALL-SORT Self-hosted GPU (e.g. runpod):
• A100 ADA to run yi costs $2.25/hr
• Takes ~75 seconds for prompt prep (unoptimised)
• => ~$0.05 for prompt prep
• ALL-SORT Hosted gpu:
• 100k tokens of input x $0.5/M-token = $0.05

Costing

Latency (very unoptimised)