A NON-TECHNICAL GUIDE
How Large Language
Models Work
Understanding AI That Writes — From Next-Word Prediction to Human-Like Conversation
From Andrej Karpathy's Introduction to LLMs
The Foundation
What Is a Large Language Model?
Just Two Files
Fundamentally, an LLM is remarkably simple: just two files. No internet connection, no complex infrastructure—just compressed knowledge and a program to access it.
File 1: Parameters
~140GB of numbers that encode patterns from training data
File 2: Run Script
Simple code (can be written in C) that processes the parameters
The Librarian Analogy
Imagine downloading the entire internet and having an expert librarian who has read it all. When you ask a question, the librarian consults their vast knowledge to craft a response.
"The complexity isn't in running the model—it's in creating those parameters through training."
The Magic Trick
The Core Mechanism: Next-Word Prediction
At their heart, LLMs do one thing remarkably well: predict what comes next.
How It Works
1
Given text: "The cat sat on the"
2
Calculate probabilities for every possible next word
3
Select the most likely: "mat" (60% probability)
4
Add to sequence, repeat billions of times
The Illusion of Intelligence
This simple task—predicting one word at a time—creates the appearance of understanding, reasoning, and creativity. But it's all just statistical pattern matching. Clawbot example
Probability Distribution
mat
60%
chair
20%
floor
10%
roof
5%
others
5%
"The model doesn't 'know' the answer—it calculates what word is most likely to come next based on patterns it learned."
Preprocessing
Tokenization: Breaking Text into Pieces
Before processing, text must be broken into tokens—the fundamental units LLMs actually see.
What Are Tokens?
Tokens are subword units—not whole words, not individual letters. They're the building blocks that balance efficiency with meaning.
Word → Tokens
unhappiness
un
happiness
Word → Tokens
ChatGPT
Chat
G
PT
Common word → Single token
the
the
Why This Matters
LLMs process tokens, not characters. This is why they struggle with spelling tasks or counting letters—they don't see individual letters!
Token Count Example
"Large language models are fascinating technologies that learn from vast amounts of text data."
15
tokens
12
words
The Tokenization Trade-off
More tokens = more precise meaning but slower processing. Fewer tokens = faster but less nuanced. Modern LLMs use ~50,000-100,000 unique tokens in their vocabulary. KEY POINT: ANYTHING (WITH A STRUCTURE) CAN BE TOKENIZED
Representation
How do computers understand meaning? By converting words into coordinates in a multi-dimensional space.
The Geographic Analogy
Just as cities have coordinates in physical space (latitude, longitude), words have coordinates in meaning space—vectors of hundreds or thousands of numbers.
City Coordinates
Paris: [48.9, 2.4]
London: [51.5, 0.1]
Word Coordinates (simplified)
cat: [0.2, -0.5, 0.8...]
dog: [0.3, -0.4, 0.7...]
Vector Arithmetic = Reasoning
Because words are numbers, we can do math with meaning:
king
−
man
+
woman
≈
queen
Paris
−
France
+
Italy
≈
Rome
Similar Words Cluster Together
Context Matters
Words with multiple meanings get different vectors based on context. "Bank" (financial) ≠ "bank" (river)—the model learns to distinguish them.
Phase 1: Pretraining
The Training Process: Compressing the Internet
Creating an LLM requires ingesting vast amounts of text and compressing it into billions of parameters.
The Scale of Training
Training Data
Internet text, books, articles
10TB+
terabytes
Compute Power
GPU clusters
6,000
GPUs
Training Time
Continuous processing
~12
days
Cost
Cloud computing
$2M
approximate
The Compression Miracle
10TB
Input
100x
compression
140GB
Output
The model learns to compress patterns, not memorize text. It captures grammar, facts, reasoning patterns, and world knowledge in a fraction of the original size.
The Learning Process
Like a student reading every book in a library, the model sees billions of examples and learns: "Given these words, what typically comes next?" It adjusts billions of internal knobs to minimize prediction errors.
The Transformer
Attention: The Focus Mechanism
How does the model understand context? By learning which words to pay attention to.
What Is Attention?
Attention is like reading while highlighting relevant words. When processing each word, the model looks at all other words to understand context.
"I went to the bank to deposit money."
deposit
money
went
the
High attention Low attention
The model learns that "deposit" and "money" strongly indicate "bank" means financial institution, not river bank.
Multi-Head Attention
Modern LLMs use multiple attention heads working in parallel—some focus on grammar, others on meaning, others on relationships. It's like having several readers with different expertise.
Attention Visualization
"Attention is all you need" — this breakthrough (2017) enabled LLMs to understand long-range dependencies and context, revolutionizing natural language processing.
Generation
Inference: How LLMs Generate Text
When you chat with an LLM, here's what's happening behind the scenes—one token at a time.
The Inference Loop
1
Receive Prompt
User input is tokenized and fed to the model
2
Calculate Probabilities
Model computes likelihood for every possible next token (50,000+ options)
3
Sample Next Token
Select one token based on probability distribution (not always the highest)
4
Append & Repeat
Add token to sequence, feed back to model, continue until completion
The Autocomplete Analogy
Think of it as a sophisticated autocomplete that builds sentences word by word. Each word choice influences the next, creating coherent (but not always correct) responses. Bender and Gerbu (2022) call it a “stochastic parrot”
Live Generation Example
Prompt:
"Explain quantum computing"
Generation step by step:
→
Quantum
(85%)
→
computing
(92%)
→
is
(78%)
→
a
(65%)
→
revolutionary
(43%)
... and so on
Temperature & Creativity
The temperature setting controls randomness. Low = predictable, high = creative. It's like choosing between a careful writer and an imaginative storyteller.
Phase 2: Fine-Tuning
Fine-Tuning: From Predictor to Assistant
Pretrained models predict text. Fine-tuning teaches them to be helpful, harmless, and honest.
The Transformation
Pretrained Model
Raw text predictor trained on internet data
Example: Given "How do I bake bread?" might continue with random internet text rather than answering helpfully.
Fine-Tuned Assistant
Helpful AI trained on quality conversations
Example: Given "How do I bake bread?" provides step-by-step instructions, tips, and encouragement.
The Student Analogy
Pretraining is like a student reading every book in the library. Fine-tuning is like that student being taught social skills, helpfulness, and how to have productive conversations.
Fine-Tuning Data
Human: How do I stay productive?
Here are some evidence-based strategies for productivity: First, use time-blocking to schedule focused work periods. Second, take regular breaks using the Pomodoro Technique. Third, minimize distractions by turning off notifications. Remember, productivity is personal—experiment to find what works for you!
Human: Can you explain photosynthesis?
Absolutely! Photosynthesis is how plants convert sunlight into energy. Here's a simple breakdown: Plants absorb sunlight through chlorophyll, take in CO₂ from air, and water from roots. They combine these to create glucose (food) and release oxygen. It's nature's way of powering life on Earth!
Thousands of such high-quality conversations
Safety & Alignment
Fine-tuning also teaches the model to refuse harmful requests, acknowledge uncertainty, and avoid biased responses—aligning it with human values. But you still get “persona drift”....
Phase 3: RLHF
Reinforcement Learning from Human Feedback—the secret sauce that makes ChatGPT so effective.
How RLHF Works
1
Generate Multiple Responses
Model produces several different answers to the same prompt
2
Human Labelers Rank Them
People compare responses: "A is better than B because..."
3
Train a Reward Model
Learn to predict human preferences from rankings
4
Optimize LLM for Higher Rewards
Fine-tune model to maximize reward model scores
The Teacher Analogy
Instead of memorizing "correct" answers, the model learns what makes a good answer through feedback. Like a student improving essays based on teacher comments rather than just copying examples.
Preference Ranking Example
Response A (Preferred)
"Here's a simple cake recipe: Mix 2 cups flour, 1 cup sugar, 3 eggs, and 1/2 cup butter. Bake at 350°F for 30 minutes. Would you like variations?"
Clear, helpful, asks follow-up
Response B
"Cake recipes vary. Some use different ingredients. You can find many online."
Vague, unhelpful
Iterative Improvement
RLHF is repeated many times. Each iteration makes the model more aligned with what humans actually want—helpful, harmless, honest responses.
Reality Check
Limitations and Quirks
LLMs are powerful tools, but they're not perfect. Understanding their limitations helps us use them wisely.
Hallucinations
LLMs can confidently generate false information. They don't know what's true—they predict what sounds plausible.
Example: "Who invented the lightbulb?" → "Thomas Edison in 1879" (oversimplified, ignores earlier inventors)
Jagged Intelligence
LLMs can write complex essays but fail at simple tasks like basic arithmetic or counting letters in words.
Example: "How many 'r's in 'strawberry'?" → Often wrong because they process tokens, not characters.
Token-Level Reasoning
LLMs struggle with character-level tasks because they see tokens, not letters. Spelling, reversing words, or precise counting are challenging.
Why: "strawberry" might be 2-3 tokens (straw + berry), not 10 characters.
No True Understanding
LLMs are statistical pattern matchers, not thinking beings. They don't have beliefs, desires, or consciousness—they simulate human-like text.
Implication: They're tools to augment human capabilities, not replacements for human judgment.
THE BIG PICTURE
Understanding LLMs Empowers Us
What They Are
Statistical pattern matchers trained on human text, compressing internet-scale data into billions of parameters
How They Work
Predict tokens one at a time, attend to context, learn from feedback—building responses word by word
How to Use Them
Powerful tools that augment human capabilities—use wisely, verify facts, and recognize limitations
The key insight: LLMs are remarkable technological achievements, but they're not magic. They're sophisticated statistical models that learn patterns from data. Understanding their mechanics helps us harness their power while maintaining appropriate skepticism and human oversight.
Based on Andrej Karpathy's "Intro to Large Language Models" • 2023