LLM architecture, training, and mechanisms are advanced topics. Have a positive attitude and embrace the challenge.
Taking notes by hand helps you learn more and remember better. Lecture notes are not available.
Having experience with coding, linear algebra, maching learning, and deep learning will help you excel in this course, although these topics are introduced in the course as they are necessary.
You can choose which lectures to watch and to skip, but keep in mind that knowledge and skills are cumulative.

You can access GPUs for free on Google Colab, though compute time and RAM are limited.
Most of this course can be done on the CPU and limited GPU (free colab plan), but will be slower.
Paying to upgrade to Colab Pro will be convenient for many lectures but is not necessary.
You can upgrade to Pro and downgrade after the course.

You are the master of your education, and you should engage with this course in a way that best suits your skills and goals.
Use different difficulty levels for different videos.
Watch the code demos, even if you aren’t coding yourself.

Text must be transformed into numbers before LLM.
A chunk of text is a “token” and can be a character, subword, or full word.
Embeddings are dense representations of tokens.
Tokenization and embeddings are learned from data, and there are many ways to create these schemes.

Text can be split into words via spaces, although this is not done in real tokenization.
Encoder and decoder functions are simple look-up tables.
Tokenization (encoding text using integers) is conceptually straightforward.

Encoders and decoders are created using dictionary comprehension and functions.
The “context” of a token is its neighbors (before and possibly after); “context window” is the number of neighbors.

“One-hot encoding” is a sparse tokenization, with one row per token and one column per vocab item.

Real text from the web is easy to import but a pain to clean…
Creating a tokenizing scheme is tricky and involves many choices with few clear optimal decisions.
Encoders and decoders are easy to create and use.

Randomly generated tokens are usually nonsensical.
There is a mathematical relationship between word length and frequency (more on this later!), which has implications for LLM performance.
Tokenizers need special characters to deal with unknown tokens.

Every tokenization scheme has advantages and limitations.
Current best tokenizers use a combination of characters, subwords, and words.
The vocab is learned based on statistical characteristics of human-written text.

The BPE algorithm is based on replacing frequent token sequences with new tokens.
The basic BPE algorithm is simple and easy to implement.
Production-level tokenizers add several more steps to ensure accuracy, efficiency, and speed.

Even simple byte-pair encoding on a tiny datset creates tokens with preceeding spaces, just like professional tokenizers.

OpenAI’s tokenizer is available, but is model-specific (e.g., GPT2 vs. GPT4).
Tokenizers use character- subword- and word-level tokens. Preceding spaces are part of tokens.
No text preprocessing is necessary! Just feed all the text into the tokenizer.

Words and tokens differ in several ways, though they can overlap.
Words vary in their encoding efficiency, which is partly related to how often they appear in texts.

Words and tokens differ in several ways, though they can overlap.
Words vary in their encoding efficiency, which is partly related to how often they appear in texts.

ChatGPT has difficulties with letter-based calculations because it represents words as tokens.
Asking ChatGPT to implement its calculations in python increases accuracy (same for math problems).

Different tokenizers are optimized for different purposes.
BERT tokenizer has more words (vs. subwords) than GPT, and is therefore more human-interpretable.
BERT tokenizer by default adds special tokens before and after text. Don’t forget about this!

Professional tokenizers are easy to work with once you get used to them.
Tokenizers contain “special tokens” that you might want to filter out of analyses.

The more familiar you are with tokenization, the easier it will be to understand embeddings and LLM mechanisms.

Importing text data from the web is really easy.
Tokenization = compression? The primary goal of a tokenizer is to make text more efficient for LMs, but compression is a common byproduct due to redundancies in (some) written languages.
Token compression ratios are stable across different texts, with higher variability for text that includes code.

Tokenization ≠ compression (but it often is).
Languages that have more complex written forms (e.g., morphemes in Chinese, richer morphology in Tamil) may require more tokens that characters.
Tokenization is less effective in languages they have less training data on.

Zipf’s law, a.k.a. power-law scaling, a.k.a. scale-free organization, a.k.a. fractal-like, a.k.a. self-similarity is a pervasive characteristic of biological and physical systems, and is taken as evidence of complex systems.
Modern computing tools and accessible digitized datasets allow you to explore nature in ways that were unthinkable until very recently.

Spaces are meaningful to humans, but are treated just like any other character to tokenizers.
Lots of subwords and words have preceding spaces.
Language models need a huge amount of data to learn all the ambiguities, errors, and varieties in human text.

There are several word embeddings matrices that have different goals and applications, and are created in different ways. It is not trivial to compare them (more on this in the mech.interp section). The embeddings used in LLMs are not fixed, but instead are adjusted by the model based on context.

The GloVe embeddings can be used to study texts and relations between words.
The embeddings vectors are fixed once trained.
GloVe is not used in LLMs, partly because of the large and word-based vocabularies.
Cosine similarity has many applications in language modeling and computational linguistics.

A lot of diversity across word embeddings matrices.
It is difficult or impossible to compare different embeddings matrices directly, but sets of relationships can be compared (see RSA in next section!).
Visualizing embeddings vectors often helps with interpretation.

A lot of diversity across word embeddings matrices.
It is difficult or impossible to compare different embeddings matrices directly, but sets of relationships can be compared (see RSA in next section!).
Visualizing embeddings vectors often helps with interpretation.

All model parameters of publicly available LLMs are easily accessible.
GPT2 and BERT embeddings are not directly comparable, though model comparisons are possible (see next section).
LLM embeddings are not fixed! They are adjusted by attention and MLP layer as they pass through the LM.

Token indices are arbitrarily mapped to meaningful subwords (numbers).
“Token math” makes no sense (but can be fun :P ).
Embeddings vectors, however, can be manipulated mathematically — this is how LLM attention works!
Use LLMs for math theory/explanations, have it solve math problems by writing code to solve the problems.

Cosine similarity is one of the most commonly used measures of a relationship between two variables when working with embeddings and LLMs.
Cosine similarity is related to correlation: Both variance-normalize; Pearson additionally mean-centers.
Use the correlation to quantify a linear relationship in data that have different scales or mean offsets.
But when the variables are in the same scale and offsets are meaningful, use cosine similarity. This is often the case in LLM investigations.
When learning a new technical topic, try to code the math yourself; in applications, use established libraries if available and if it’s easier.

Multitoken words present several challenges in LLM investigations (hint: give the model context and only analyze the final token).
Understanding the math of analyses makes you a better and more flexible data scientist.

An “unembeddings matrix” is the conceptual inverse of the embeddings matrix (but not a literal inverse matrix).
The token with the largest unembeddings value is the next token in a generated sequence.
Generated text can quickly lose meaning, which was a major hurdle for language models to overcome.
The broad strokes mechanisms of next-token generation is simple conceptually and mathematically.

Language models use position embeddings to focus on tokens from various locations (“time points”) in the token sequence.
Predefined position embeddings are probably good enough for smaller models; modern architectures use learned embeddings.
Cosine similarity matrices can be tricky to interpret, but convey a lot of information.

The position embeddings matrix is complicated and impacts token processing in ways that are difficult to predict a priori.
Visual appearances (especially of apparent null effects) should be statistically evaluated using shuffled data before making strong interpretations.
The shuffling method here was too liberal; circular shifting and spectral phase scrambling are better.

Embeddings matrices in LLMs are trained with the rest of the model. In the next several videos, however, we will train only and embeddings layer, to focus on the mechanisms of learning embeddings.

Preparing data to train models can become complicated (a common experience in many data fields…).
Simpler data organization methods are possible, but can be suboptimal for professional-grade model training.

Embeddings matrices are learned from text data using gradient descent and dimension-squeezing deep learning models.
The embedding dimension is preserved throughout the entire language model.
Language models generate text by concatenating one new token onto an existing token sequence.

Negative log likelihood is the main loss function used in language model training.
Log softmax increases sensitivity at small probabilities, and gives a stronger penalty for errors.
Most loss functions (and other mathematical bases of DL) are simple and well-defined. Difficulties arise from the explosive dimensionality of models.

Language models are trained on a small number of epochs, because each epoch has a huge amount of data (batches).
GPU access will be increasingly important as the course progresses…

The embeddings vectors expanded to fill up more of the embeddings space, reflecting encoding of high-dimensional information in a lower-dimensional space.
Interpretable semantic relationships (as measured through Sc) emerge even with a small amount of targeted training data, though the strength is weak.

Embeddings vectors of the same token appear to be completely unrelated across repeated training runs.
Cosine similarity appears to be more consistent for related tokens.
Relative embeddings within a matrix are more interpetable than absolute vectors or across matrices.
The loss profile is relevant, but is just a tiny glimpse at what happens during training.

Building a model from scratch is a fantastic way to learn how LLMs are created and trained.
Please don’t ever build an LLM from scratch again (except for more education)

Saying that language models “use only the final token" for next-token prediction is not accurate. They use all tokens; the final token contains the most information.
Next-token selection is probabilistic.
LLMs get complicated quickly; it’s good to learn about them one step at a time.

nn.Embedding is a convenient wrapper for nn.Parameter. Use it to create embeddings matrices.

GELU is the most common activation function in LLMs. It is more complicated than ReLU but smoother.
Many PyTorch functions are available as functions and classes; using one or the other is sometimes by necessity and sometimes a personal choice.
Measuring computation time is not trivial, because of optimized implementations, GPU fusing, overhead, etc.

Softmax transforms LLM outputs from “raw” values (logits) into a probability distribution.
Temperature increases stochasticity, which helps language models produce more “creative” text. Typical temperatures are .5—1.5.
You’ll learn more about nuances of softmax theory and implementation in later videos.

torch.multinomial directly maps token probability values onto selection probabilities.
Many PyTorch functions are sensitive to input data type — something to check if (when) you get errors!
Numpy and pytorch functions can sometimes produce identical results given proper input arguments, but often not by default. Be careful when translating between libraries.

There is no right or wrong token selection method.
Random sampling can increase response variability, which might be good for social chatting but bad for coding or legal documents.
The sampling method interacts with softmax temperature: Higher temperatures have stronger boosts of fewer tokens.

Softmax is mathematically simple, but some aspects of the transformation are apparent only for some numerical ranges.
The number of data values (e.g., vocab size) impacts the probability values, because they must all sum to 1.

Layernorm is simple and critical.
“Set it and forget it”: The learned parameters have little theoretical relevance other than preserving numerical stability throughout the model.

Position embeddings are added to the token embeddings, and help the model learn temporal patterns.
Embeddings are trained inside the model, not separately as with tokenization (c.f. previous section).
Tokens in a sequence are processed simultaneously, not in a for-loop (causality from attention mechanism).

Time-causal attention can be implemented using a time vector in which the future is weighted zero while the past is weighted non-zero.
Causality can be implemented using matrices and softmax-probability, which is very computationally efficient on GPUs.
Causal attention is not necessary for LLMs, but improves next-token generation (good for chatbots).

Matrix multiplication with masks are an efficient way to avoid for-loops.
PyTorch has optimized functions that fuse the attention algorithm, including the causal mask (next lecture!).

The “attention” mechanism is a clever way of pooling information across different embeddings vectors that allows for surrounding context to modify the current token transformation.
All analogies break down; some are useful (or at least entertaining).
The full LLM Transformer architecture is more complicated than just attention, but attention is a key aspect.

Q, K, and V matrices are trainable weights that are not changed during inference (applications).
Q, K, and V are the activations resulting from multiplication with token embeddings vectors.
“Highly optimized” functions can be hardware-specific.
Very powerful LLMs require specialized hardware, the sale of which is regulated by governments.

Attention adjusts (not replaces) the embeddings vectors as they pass through the model.
Different tokenizers (and also different pretrained models) use different terms and variable names. Understanding how LLMs work will help you identify features in a model.

The Transformer block contains an attention sublayer and an MLP sublayer. Together, they calculate an adjustment to the token embedding to point towards an appropriate next-token embedding.
Expansion-nonlinearity-contraction is a typical MLP architecture for feature extraction and linear separability.
LLMs comprise dozens of Transformer blocks that learn different features and timescales of texts.

You can now implement a GPT-style Transformer :)
Separating modules into callable classes helps keep code neat and organized.
Both Transformer sublayers comprise the operations copy → normalize → adjustment → add back to copy.

Use nn.Sequential to create repeated instances of the same component of a model (with different weights matrices).
Specialization of Transformer blocks is not imposed by the architecture, but is thought to be an emergent property. We’ll discuss this more in the mechanistic interpretability sections.

Multihead Attention (MHA) involves applying the attention equation to different slices of the QKV matrices.
MHA is thought to increase the richness and complexity of feature isolation and context-sensitivity, without increasing the number of trainable parameters.
Data from all heads are linearly combined via W0.

GPUs are great at number-crunching, and can save a lot of time in LLM training and applications.
On the other hand, the CPU can be sufficient for a lot of explorations and investigations of LLMs.
Accessing and using a GPU might be expensive and requires additional code, so use it only when it's really beneficial.

We just built a GPT2-small :) although the weights are all random, so it’s not functional.
Commercial models, e.g., GPT4, have the same architecture and computations, but have more layers and parameters. But, quantitative increases can have qualitative impacts.

LLMs are basically worthless without a large number of dedicated high-end GPUs.
Regulating GPU access/sales is part of AI safety.

Publicly available pretrained models are really easy to use, explore, experiment with, and learn from.

There are many ways to understand a deep learning model; a combination of perspectives (diagrams, explanations, code, equations) is most beneficial.
Rewriting equations and checking matrix sizes can help you understand the flow and order of calculations.
Neither Q nor K directly contribute to the embedding vector adjustment; their combination determines which vectors in V guide next-token selection.

Biases are a tiny fraction of model parameters, which is why many deep learning models (especially models with layernorm) ignore them.
MLP layers are dense and can have 2-3x as many parameters as attention layers.
The number of parameters does not equate to importance in the model (cf layernorm).
Examining and dissecting models are useful skills.

Biases are a tiny fraction of model parameters, which is why many deep learning models (especially models with layernorm) ignore them.
MLP layers are dense and can have 2-3x as many parameters as attention layers.
The number of parameters does not equate to importance in the model (cf layernorm).
Examining and dissecting models are useful skills.

Characteristics of model weights often have smooth transitions across layers, reflecting shifts in representations and calculations.
In histograms, use counts for equal sample sizes, and use density (or other scaling) for unequal sample sizes.

Causal manipulations are easy to implement, although knowing what to manipulate is challenging and non-trivial.
Model evaluations are tricky. There are quantitative evaluation methods, but many evaluations are based on qualitative inspection of generated text.

Pretraining is a necessary first step for any LLM. A pretrained model understands the structure and patterns of written language, and can generate text.
Pretraining to create a useful modern base model is prohibitively expensive for most individuals and companies.
You should learn how pretraining works, but don’t try it at home ;)

HuggingFace provides resources for downloading pretrained LLMs (and other models), training datasets, and more. We will some some of the free resources in this course. You do not need a HuggingFace login to access materials for this course.

L2 regularization in Adam involves updating and regularizing in one step (i.e., regularizing the update).
AdamW updates the weights first, then regularizes (i.e., regularizing the weights).
Without regularization, Adam==AdamW.
AdamW implements “constant shrinkage” instead of “adaptive shrinkage,” and empirically better in large models.

Simple mechanisms work better in simple models.
Adam is adaptive and more likely to stabilize.
Gradient accumulation speeds learning, but risks over-generalizing by pooling across more training data.
Gradient accumulation is only used for training very large models, and you probably will always want to reset the gradients.

Code to train LLMs has the same basic organization as code to train any deep learning model.
Even very simple models with little and limited training quickly learn text structure such as punctuation and line breaks.

Creating and evaluating a test set is not so difficult, but requires extra code.
Train/test splits are less important for pretraining LLMs, but is still good practice.
Additional subtleties about devsets, model vs. researcher overfitting, etc., are not discussed here.

Freezing weights is a common technique in deep learning for transfer learning or when re-training on limited data.
Freezing weights is not necessarily advantageous, especially in simple models or if the new training data differ from the previously trained data.

There are several ways to sample data, depending on how meticulous you want the procedure. Some overlap or skipping is less consequential with limitless training data.
Models learn to produce “language-looking” text very quickly.

Loss functions are extremely important for training deep learning models.
Loss functions should be as simple as possible, both mathematically and in code implementation.
Knowing how to create your own loss function gives you more control and flexibility over precise model designs and outcomes.

KL divergence is a useful loss function for training distributions instead of individual tokens.
It is easy to train biases into models. This has major implications for fairness, misuse, persuasion or manipulation, cultural or political biases, marketing, and other AI safety topics.

Each matrix multiplication (dot products) increases the variance and numerical range of numbers.
This can have a negative impact on softmax probabilities by flattening the distribution.
Repeated normalization is very important for the stability of deep learning models including LLMs.

Weight initialization is not important in small models, but is crucial for training large models including LLMs.
Weights should be initialized to small values, often proportional to the matrix sizes.
Bias terms are typically ignored in initializations, because there are so few bias terms.
Any initialization is important; the exact details and distribution shape seems to be less relevant.

Now you know how to initialize weights :)
Examining changes in the model during learning is an approaching in mechanistic interpretability. Weights distributions tend to widen as the models learn more diverse patterns and representations.

Dropout involves “switching off” units during training, and is thought to promote distributed representations.
LLM pretraining sets are so large and diverse that dropout is less important compared to, e.g., CNNs.
Dropout is more useful in fine-tuning when datasets are smaller and overfitting risk is higher.

Calculating log-softmax inside the LLM is often fine and convenient during training and classification, but outputting the raw values allows for more flexibility in subsequent applications.

HuggingFace provides several high-quality datasets for LLM training, including FineWeb (and several more specific variants).

Dropout is conceptually simple, but adding it to code can be tricky.
There are many moving parts and parameters in an LLM; make sure to track variables and transformations.
Models can learn very fast on small homogeneous datasets, but take longer to train on datasets with more variability.

Dropout is conceptually simple, but adding it to code can be tricky.
There are many moving parts and parameters in an LLM; make sure to track variables and transformations.
Models can learn very fast on small homogeneous datasets, but take longer to train on datasets with more variability.

Because softmax affects all token logits, not just the ones currently being processed, all tokens embeddings are trained inside the model, not just the ones that appear in the current token sequence.
This has implications for model output coherence and accuracy for tokens that are less commonly used in training datasets (e.g., obscure topics, new coding languages).

There are many ways to decrease computation time during LLM training.
Different strategies focus on the data, the model, or the hardware.
LLMs take so long to pretrain that even a few milliseconds improvement per batch can be significant.

Fine-tuning means making targeted adjustments to an existing pretrained model. It takes days or weeks instead of months, and has much smaller computational requirements.
There are myriad choices in fine-tuning; the challenge is from knowing what you want to do with the model.

Shortcuts are great, but make sure you understand the mechanisms before oversimplifying your code.
Fine-tuning is easy to implement; the challenge comes from choosing the appropriate datasets.
The learning rate should generally be lower.
Model evaluation should rely on multiple qualitative and quantitative metrics.