How Large Language Model Works Under the Hood

Cindy Sheng

Mingdao School

Linear Regression

y = wx +b

Linear Regression

y = w1*x1 + w2*x2 + b

Neutral Network

Weights

( 13 * 5 ) +

( 5 * 4 ) +

( 4 * 3 ) = 97.

The total number of biases

5 + 4 + 3 = 12.

Total number of parameters 97 + 12 = 109

Putting things in perspective

https://informationisbeautiful.net/visualizations/the-rise-of-generative-ai-large-language-models-llms-like-chatgpt/

What does Large Mean?

Powerful, Expensive, Slow

Powerful

MMLU (Massive Multitask Language Understanding)

Paper is here. Github source here.

The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more specialized areas like law and ethics. The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots.

​

https://informationisbeautiful.net/visualizations/the-rise-of-generative-ai-large-language-models-llms-like-chatgpt/

Expensive

GPU has thousands of cores for parallel processing

Optimized for Tensor operations acceleration (CUDA architecture)

Much higher memory bandwidth

How much does it cost?

LLAMA 405B technical report says:

​

• GPU: 16,000 x H100 = $500M, $2.23 per card per hour
• Electricity: Each GPUs running at 700W TDP with 80GB HBM3, using Meta’s Grand Teton AI server platform. Each server is equipped with eight GPUs and two CPUs.
• Storage: 240 PB of storage out of 7,500 servers equipped with SSDs, and supports a sustainable throughput of 2 TB/s and a peak throughput of 7 TB/s
• Network: RoCE-based AI cluster comprises 24,000 GPUs
• Time: weeks to months per train

​

How is Large Language Model Trained

LLM Training is just like any other ML model training

Data: The Pile: 825GB, 22 subsets paper link

Prepare Data - Turn Text into Floating Numbers

Tokenization

There are many tokenization algorithms. GPT2 used BPE, to create 50,257 tokens.

Embeddings

GPT2 used 1024 dimensions for embedding, making its embedding matrix 50,257 x 1024�LLaMA 405B embedding matrix is 32000 token x 16384 embedding dimensions

Embedding Example

King - Man + Woman = Queen

Paris - France + Italy = Rome�

Use two-dimensional PCA projection of 1000 dimensional skip gram vectors for visualization

Model Training Using Transformer Architecture

Next Token Prediction

Transformer paper

Tokenizer

Transformer Block

Self-attention Layer

Activation Functions

Output

Limitations of the Pre-trained Model

Pre-trained Model

• Trained up to a certain time
• Hallucinations
• Domain shift

what’s the youngest age to get a driver’s license.

• Task shift

Tell me about the elephant on the moon!

• Resource constraint
• No access to private data

Retrieval Augmented Generation

RAG Architecture paper

LLM Agents

Agents paper

Future

• Pre-training as we know has ended.
• AI applications are built with RAG & agents
• Emotional companions are being prepared for everyone of us.

Ten years ago, Facebook said, we know you are in love before you know it. Now LLM knows more and knows better. Go to the gym humans, after losing in intelligence, you can only count on excelling in six packs.

Q & A