DOU AI Meetup

​

​

​

​

​

​

Roman Kyslyi,

Applied Data Scientist

AI HOUSE IS THE LARGEST AND MOST POWERFUL AI COMMUNITY IN UKRAINE

Roman Kyslyi

LinkedIn

Applied Data Scientist

Host of AI HOUSE Podcast

<Large Language Models (LLM)>

Overview, challenges and future

Language model

1. A language model encodes statistical information about language.
2. Fluent speakers of a language subconsciously have statistical knowledge of that language. For example, given the context My favorite color is __, if you speak English, you know that the word in the blank is much more likely to be green than car.
3. You can think of a language model as a “completion machine”: given a text (prompt), it can generate a response to complete that text.
4. Completion turned out to be incredibly powerful, as many tasks can be framed as completion tasks: translation, summarization, writing code, doing math, etc.

Ways of selecting tokens

1. A model outputs a set of probabilities to an array where each value in that array represents the probability of a specific word/token.
2. When we are selecting a token in machine-generated text, we have a few alternative methods for performing this decode — and options for modifying the exact behavior too.

Greedy Decoding�

The simplest option we have is greedy decoding. This takes our list of potential outputs and the probability distribution already calculated — and chooses the option with the highest probability (argmax).

Random Sampling

Random sampling chooses the next word based on these probabilities — so in our example, we might have the following words and probability distribution:

Beam Search

So whereas greedy decoding and random sampling calculate the best option based on the very next word/token only — beam search checks for multiple word/tokens into the future and assesses the quality of all of these tokens combined.

Yann LeCun perspective

​

​

Unsupervised learning: data scraped from the Internet: think clickbait, misinformation, propaganda, conspiracy theories, or attacks against certain demographics.

Supervised learning: higher quality data – think StackOverflow, Quora, or human annotations – which makes it somewhat socially acceptable

RLHF: polished using RL to make it customer-appropriate

Data bottleneck

​

Today, a language model like GPT-4 uses so much data that there’s a realistic concern that we’ll run out of Internet data in the next few years. It sounds crazy, but it’s happening. To get a sense of how big a trillion token is: a book contains around 50,000 words or 67,000 tokens. 1 trillion tokens are equivalent to 15 million books.

Paper published on 28th of November of 2023 claim that training data can be extracted from the models like GPT: [link]

​

LLM scaling laws

Performance of LLMs is a smooth, well-behaved, predictable function of:

- N, the number of parameters in the network

- D, the amount of text we train on

�=> We can expect more intelligence “for free” by scaling

Supervised fine tuning (SFT)

​

How to do that? We know that a model mimics its training data. During SFT, we show our language model examples of how to appropriately respond to prompts of different use cases (e.g. question answering, summarization, translation). The examples follow the format (prompt, response) and are called demonstration data.

​

OpenAI calls supervised fine tuning behavior cloning: you demonstrate how the model should behave, and the model clones this behavior.

OpenAI’s 40 labelers created around 13,000 (prompt, response) pairs for InstructGPT. Here are a few examples:

​

Supervised fine tuning (SFT)

​

1. ML task: language modeling
2. Training data: high-quality data in the format of (prompt, response)
3. Data scale: 10,000 - 100,000 (prompt, response) pairs
1. InstructGPT: ~14,500 pairs (13,000 from labelers + 1,500 from customers)
2. Alpaca: 52K ChatGPT instructions
3. Databricks’ Dolly-15k: ~15k pairs, created by Databricks employees
4. OpenAssistant: 161,000 messages in 10,000 conversations -> approximately 88,000 pairs
5. Dialogue-finetuned Gopher: ~5 billion tokens, which I estimate to be in the order of 10M messages. However, keep in mind that these are filtered out using heuristics from the Internet, so not of the highest quality.
4. Model input and output
• Input: prompt
• Output: response for this prompt
5. Loss function to minimize during the training process: cross entropy, but only the tokens in the response are counted towards the loss.
6. ​

Fine Tuning with distillation

https://crfm.stanford.edu/2023/03/13/alpaca.html

Reinforcement learning

​

Key components of reinforcement learning:

1. Agent: The learner or decision-maker that interacts with the environment.
2. Environment: The external system or context in which the agent operates.
3. State (s): A representation of the current situation or configuration of the environment.
4. Action (a): The set of possible moves or decisions that the agent can take in a given state.
5. Reward (r): A numerical signal provided by the environment to indicate the immediate benefit or cost associated with an action taken in a particular state.
6. Policy (p): The strategy or mapping that defines how the agent selects actions in different states.

RLHF

The idea: what if we have a scoring function that, if given a prompt and a response, outputs a score for how good that response is? Then we use this scoring function to further train our LLMs towards giving responses with high scores. That’s exactly what RLHF does. RLHF consists of two parts:

1. Train a reward model to act as a scoring function.
2. Optimize LLM to generate responses for which the reward model will give high scores.

Paper: https://arxiv.org/pdf/2203.02155.pdf

Reward model (RM)

1. Training data: high-quality data in the format of (prompt, winning_response, losing_response)
2. Data scale: 100K - 1M examples
3. rθ:: the reward model being trained, parameterized by θ. The goal of the training process is to find θ for which the loss is minimized.
4. Training data format: x- prompt; yw - winning response; yl - losing response
5. For each training sample:
1. reward model’s score for the winning response: sw=rθ(x,yw)
2. reward model’s score for the losing response: sl=rθ(x,yl)
6. Goal: find θ to minimize the expected loss for all training samples

​

Building LLM applications for production

​

1. Prompt: 10k tokens ($0.06/1k tokens)
2. Output: 200 tokens ($0.12/1k tokens)
3. If you use GPT-4 with 10k tokens in input and 200 tokens in output, it’ll be $0.624 / prediction.
4. If you use GPT-3.5-turbo with 4k tokens for both input and output, it’ll be $0.004 / prediction or $4 / 1k predictions.
5. As a thought exercise, in 2021, Booking.com models made 10 billion predictions a day. If each prediction costs $0.004, that’d be $40 million a day!
6. By comparison, AWS personalization costs about $0.0417 / 1k predictions and AWS fraud detection costs about $7.5 / 1k predictions [for over 100,000 predictions a month]. AWS services are usually considered prohibitively expensive (and less flexible) for any company of a moderate scale.

Latency

Additional output

​

• It’s easy to make something cool with LLMs

​

• It is hard to make production-ready application

Inconsistency

Hallucinations

​

Chain of Thoughts

​

Chain of Thoughts: https://arxiv.org/abs/2201.11903

RAG

1. Retrieval Augmented Generation
2. MuRAG: Multimodal Retrieval-Augmented Generator
3. Ensemble of RAG
4. HyDE: (Hypothetical Document Embeddings)

Hallucinations

Compressing information (why gzip will not work)

LLM

RDBMS

Data Lake

Keyword search

Vector Search

Prompt

In computer science, ACID (atomicity, consistency, isolation, durability) is a set of properties of database transactions intended to guarantee data validity despite errors, power failures, and other mishaps.

In computer science, atomicity, consistency, isolation, durability are a set of properties of database transactions intended to guarantee data validity despite errors, power failures, and other mishaps.

Keyword: ACID

No Keyword: ACID

Source: https://drive.google.com/file/d/1pxx_ZI7O-Nwl7ZLNk5hI3WzAsTLwvNU7/view

<Q&A>