Machine Learning�Week 14 – Large Language Models

​

Seungtaek Choi

Division of Language & AI at HUFS

seungtaek.choi@hufs.ac.kr

Deep Learning

Attention

Attention

https://alex.smola.org/talks/ICML19-attention.pdf

Attention in Cognitive Science

• Behavioral and cognitive process of selectively concentrating on a discrete aspect of information while ignoring other perceivable information
• State of arousal
• It is the taking possession by the mind in clear and vivid form of one out of what seem several simultaneous objects or trains of thought.
• Focalization, the concentration of consciousness
• The allocation of limited cognitive processing resources

https://www.youtube.com/watch?v=CIQ354Y_gOc

Issues with Recurrent Models (1)

• Linear interaction distance
• RNNs are unrolled “left-to-right”.
• This encodes linear locality: a useful heuristic!
• Nearby words often affect each other’s meanings

​

• Problem: RNNs take O(sequence length) steps for distant word pairs to interact.
• Hard to learn long-distance dependencies (because gradient problems!)

​

short-term dependency�: enough with RNN

long-term dependency�: NOT enough with RNN

O(sequence length)

https://web.stanford.edu/class/cs224n/

Issues with Recurrent Models (2)

• Lack of parallelizability
• Forward and backward passes have O(sequence length) unparallelizable operations
• GPUs can perform a bunch of independent computations at once!
• But future RNN hidden states can’t be computed in full before past RNN hidden states have been computed.
• Inhibits training on very large datasets!

https://web.stanford.edu/class/cs224n/

1

2

3

0

1

2

n

Numbers indicate min # of steps before a state can be computed

Issues with Recurrent Models (3)

• Encoding bottleneck
• It’s like a one-shot compression (sequence of features 🡪 a vector).
• RNN + classification is like judging a book by its summary.

BiLSTM Encoder

LSTM Decoder

https://web.stanford.edu/class/cs224n/

Solution: Attention

Solution: Attention

• We can think of attention as performing fuzzy lookup in a key-value store.

https://web.stanford.edu/class/cs224n/

In a lookup table, we have a table of keys that map to values. The query matches one of the keys, returning its value.

In attention, the query matches all keys softly, to a weight between 0 and 1. The keys’ values are multiplied by the weights and summed.

query

d

keys

a

values

v1

b

v2

c

v3

d

v4

e

v5

output

v4

query

q

keys

k1

values

v1

k2

v2

k3

v3

k4

v4

k5

v5

output

weighted �sum

Attention for Machine Translation

• Sequence-to-sequence (seq2seq)
• Encode source sequence to latent representation
• Decode to target sequence one character at a time
• Need memory for long sequences

https://alex.smola.org/talks/ICML19-attention.pdf

Sequence to Sequence Learning with Neural Networks (https://arxiv.org/pdf/1409.3215)

Attention for Machine Translation

https://alex.smola.org/talks/ICML19-attention.pdf

Neural Machine Translation by Jointly Learning to Align and Translate (https://arxiv.org/pdf/1409.0473)

Effective Approaches to Attention-based Neural Machine Translation (https://arxiv.org/pdf/1508.04025)

Image generated by Gemini 3 Pro

Attention for Machine Translation

• With attention, translation quality remains much more stable for long sentences.

https://alex.smola.org/talks/ICML19-attention.pdf

Neural Machine Translation by Jointly Learning to Align and Translate (https://arxiv.org/pdf/1409.0473)

Attention for Machine Translation

Neural Machine Translation by Jointly Learning to Align and Translate (https://arxiv.org/pdf/1409.0473)

Attention for Image Captioning

• In computer vision, attention learns alignment between image region and word.

​

Show, Attend, and Tell: Neural Image Caption Generation with Visual Attention (https://arxiv.org/pdf/1502.03044)

Self-Attention as a New Building Block

• Prev: Attention on recurrent models
• Todays: Attention only
• Number of unparallelizable operations does not increase with sequence length.
• Maximum interaction distance: O(1), since all words interact at every layer!
• How? Treat each word’s representation as a query to access and incorporate information from a set of values.

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

All words attend to all words in previous layer

Self-Attention as a New Building Block

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

Limitations of Self-Attention (1)

• 1^st challenge: self-attention doesn’t have an inherent notion of order!

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

q

went

to

HUFS

LAI

at

2025

and

learned

No order information!

There is just summation

over the set

Position Representation Vectors

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

In deep self-attention networks, we do this at the first layer! You could concatenate them as well, but people mostly just add…

Position Representation Vectors

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (https://arxiv.org/pdf/1810.04805)

Limitations of Self-Attention (2)

• 2^nd challenge: there is no non-linearities. It’s all just weighted averages.

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

query

q

keys

k1

values

v1

k2

v2

k3

v3

k4

v4

k5

v5

output

Even stacking multiple layers cannot introduce any non-linearity because it’s still a summation of value vectors

output

Adding nonlinearities in self-attention

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

Limitations of Self-Attention (3)

• 3^rd challenge: need to ensure we don’t “look at the future” when predicting a sequence

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

Masking the Future in Self-Attention

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

“Transformer”

• Self-attention:
• The basis of the method.
• Position representations:
• Specify the sequence order, since self-attention is an unordered function of its inputs.
• Nonlinearities:
• At the output of the self-attention block
• Frequently implemented as a simple feed-forward network.
• Masking:
• In order to parallelize operations while not looking at the future.
• Keeps information about the future from “leaking” to the past.

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

Understanding Transformer Step-by-Step

https://www.youtube.com/watch?v=GvezxUdLrEk

Transformer Decoder

• A Transformer decoder is how we’ll build systems like language models.
• It’s a lot like our minimal self-attention architecture, but with a few more components.
• The embeddings and position embeddings are identical.
• We’ll next replace our self-attention with multi-head self-attention.

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

Multi-Head Attention

https://web.stanford.edu/class/cs224n/

Attention head 1 �attends to entities

Attention head 2 attends to syntactically relevant words

q

went

to

HUFS

LAI

at

2025

and

learned

q

went

to

HUFS

LAI

at

2025

and

learned

Multi-Head Attention

https://web.stanford.edu/class/cs224n/

Attention Is All You Need (https://arxiv.org/pdf/1706.03762)

https://www.youtube.com/watch?v=nfti9lEs8k8

Multi-head attention = stacking multiple self-attention

Transformer …

• Today, we will focus more on “attention”, rather than “Transformer”.
• However, it’s recommended to study the full details of Transformer, such as …
• Low-rank computation in multi-head attention
• Scaled Dot-Product Attention (dot product between large vectors is bad)
• Residual connections
• Layer normalization
• Cross-attention (decoder attends to encoder states)
• Quadratically growing computation by sequence length
• …

Analysis of Transformer Attention

What Does BERT Look At? An Analysis of BERT’s Attention (https://arxiv.org/pdf/1906.04341)

Could We Trust Attention? (1)

• Is attention an explanation for its prediction?
• If attention were a faithful explanation for its prediction, then perturbing attention should significantly change the prediction.
• In that sense, (Jain & Wallace, 2019) claims attention weights are NOT reliably faithful explanations.
• They constructed shuffled / adversarial attention distributions on a fixed model, but the predictions stay almost the same.

​

Attention is not Explanation (https://arxiv.org/pdf/1902.10186)

Could We Trust Attention? (2)

• Is a token with high attention really one that matters?
• (Serrano & Smith, 2019) remove words in order of attention vs. gradients, and find that deleting high-attention words often barely affects the prediction.
• => Attention weights are not a reliable importance ranking, so we should be careful using them as explanations.

Is Attention Interpretable? (https://arxiv.org/pdf/1906.03731)

Advanced Topics

Large Language Models

Today’s Topic

• What is language model?
• What is “large” language model?
• How to train large language model?
• How to evaluate large language model?
• How to use large language model?

What is Language Model?

• A language model (LM) is a probabilistic model over sequences of symbols (words, subwords, characters).
• It assigns a probability to each possible sentence or text sequence.
• It captures what “sounds natural” in a language.
• Core capability: predicting the next token given previous tokens.

What is Language Model?

Example: Smartphone Keyboard

https://jalammar.github.io/illustrated-gpt2/

What is Language Model?

What is Language Model?

Autoregressive Language Models

Autoregressive Language Models

• Predict next word -> predict next word -> predict next word -> …

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• Generic language model is thus a next word predictor.
• Input: a sequence of tokens
• Output: the most likely next token

​

Google Cloud Tech -Introduction to large language models (https://www.youtube.com/watch?v=zizonToFXDs)

Autoregressive Language Models

• Transformer decoders take the token embeddings and positional encodings.

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• In the decoder, self-attention with causal attention mask computes the (left-to-right) representations.

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• In the decoder, we could get the probability distribution over the vocabularies by multiplying the last hidden representation and token embeddings.

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• Each row in the embedding matrix corresponds to the embedding of a word in the model’s vocabulary. The results of this multiplication is interpreted as a score for each word in the model’s vocabulary.

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• We can simply select the token with the highest score (top_k = 1).

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• Example of machine translation with decoder-only transformer:

https://jalammar.github.io/illustrated-gpt2/

Autoregressive Language Models

• Example of summarization with decoder-only transformer:

https://jalammar.github.io/illustrated-gpt2/

How to Train Language Models?

How to Train Language Models?

How to Train Language Models?

How to Train Language Models?

• Maximum likelihood training
• Language model defines:

​

​

​

​

• Maximum likelihood objective:

​

​

​

​

• Equivalent minimization of cross-entropy loss:

​

​

​

How to Train Language Models?

How to Train Language Models?

• What is teacher forcing?
• At every step, the context tokens come from the dataset, not from the model’s own predictions.

Generated by Gemini 3 Pro (Nano Banana Pro)

How to Train Language Models?

• Training loop (w/ teacher forcing)
• input_tokens are always ground-truth tokens from the dataset.

How to Train Language Models?

How to Train Language Models?

• Exposure bias
• During training, the model is only exposed to correct prefixes from the dataset.
• During inference, the model is exposed to its own mistakes:
• If it predicts a wrong token early, all future contexts are “off-distribution”.
• Consequences:
• The model may behave well when the prefix is correct,�but poorly when earlier errors happen.
• Errors can snowball and produce unnatural text.

Generated by Gemini 3 Pro (Nano Banana Pro)

Rethinking Next Token Prediction

• Training objective:
• Next token prediction
• But we can also view it as:
• Learning to solve (multiple) tasks
• Injecting knowledge into the model
• Same objective, �different perspectives

Generated by Gemini 3 Pro (Nano Banana Pro)

Next Token Prediction = Learning to Solve Tasks

• Many texts already contain implicit tasks:
• “Translate to French: The cat is sleeping. 🡪 Le chat dort.”
• “Question: Where is the Eiffel Tower? Answer: In Paris, France.”
• “Summary: … In conclusion, …”
• “def add(a, b): return a + b”
• For the model, each of these is just one long token sequence.
• To predict the next token correctly, the model must:
• Understand the instruction in the prefix.
• Understand the input (sentence, question, code, …).
• Produce the correct output tokens.

Multitask Learning Emerges from Raw Text

• A single next-token objective is trained on:
• Dialogues (conversation modeling)
• Articles and headlines (summarization)
• Q&A forums (question answering)
• Code repositories (program synthesis, debugging)
• Tables, lists, bullet points (information extraction)
• The model sees many task formats mixed together.
• Result:
• The model becomes a general task solver conditioned on the input text (the prompt).

​

Next Token Prediction = Knowledge Injection

• To assign high probability to real text, the model must capture:
• Linguistic knowledge:
• Grammar, syntax, collocations, style.
• World knowledge:
• “Paris is the capital of France.”
• “Water boils at 100°C at sea level.”
• Procedural knowledge:
• “2 + 2 = 4”
• “for i in range(n): …”
• Each gradient update injects tiny pieces of knowledge into the parameters.
• After large-scale training, the model stores a compressed representation of the knowledge in the corpus.

Pre-training with Next Token Prediction

• Pre-training:
• Train a large neural network on massive text corpora.
• Use the next-token prediction objective on billions or trillions of tokens.
• Objective:

​

​

​

• No explicit task labels are needed:
• The supervision signal comes from the text itself.
• Result:
• A general-purpose pre-trained language model.

From Language Model to “Large” Language Model

• Language model (LM):
• Any model trained with the next-token objective.
• Large language model (LLM):
• Same objective, but:
• Much larger model (billions of parameters).
• Much larger and more diverse data (web, books, code, dialogues, …).
• Longer training, often on specialized hardware.
• Often followed by:
• Instruction fine-tuning.
• Preference optimization (e.g., RLHF).
• The core remains the same:
• Next-token prediction as task learning and knowledge injection.

Generated by Gemini 3 Pro (Nano Banana Pro)

What Makes a Language Model “Large”?

• Model size has exploded
• Models grew from hundreds of millions to hundreds of billions of parameters.
• Larger capacity allows storing more patterns, tasks, and knowledge.
• But it requires much more data, compute, and engineering.

https://www.youtube.com/watch?v=W0TcVrI_vRg

What Makes a Language Model “Large”?

• Max effective context length
• Same next-token objective, but Transformers can attend to thousands of tokens at once.
• This enables long-range dependencies and complex prompts that describe a task, give examples, and provide background.

https://www.youtube.com/watch?v=W0TcVrI_vRg

What Makes a Language Model “Large”?

Language Models are Few-Shot Learners (https://arxiv.org/pdf/2005.14165)

What Makes a Language Model “Large”?

• Zero-shot, one-shot, and few-shot in-context learning
• Zero-shot
• Input: natural language task description only.
• The model must infer the task from the description.
• One-shot
• Input: Task description + one example.
• Few-shot
• Input: Task description + several examples.
• No gradient updates; the model uses in-context information only.

Language Models are Few-Shot Learners (https://arxiv.org/pdf/2005.14165)

What Makes a Language Model “Large”?

• Larger models use context more efficiently.
• All models improve with more in-context examples.
• Larger models show steeper in-context learning curves.
• With a natural language prompt, large models can often perform tasks zero-shot or few-shot.
• Scaling up improves the model’s ability to read, understand, and exploit the prompt as training data.

Language Models are Few-Shot Learners (https://arxiv.org/pdf/2005.14165)

What Makes a Language Model “Large”?

• More recently: increased context size x more examples (shots)

Many-Shot In-Context Learning (https://arxiv.org/pdf/2404.11018)

What Makes a Language Model “Large”?

• Scaling law

https://www.youtube.com/watch?v=W0TcVrI_vRg

What Makes a Language Model “Large”?

• Scaling law

https://www.youtube.com/watch?v=W0TcVrI_vRg

From Pre-trained LLMs to Practical Systems

• Pre-trained LLM:
• Trained with next-token prediction at scale.
• Already capable of many tasks via prompting.
• To build real applications, we often add:
• Instruction fine-tuning on human-written prompts and responses.
• Preference optimization (e.g., RLHF) for helpful and safe behavior.
• Retrieval-augmented generation (RAG) to access up-to-date external knowledge.
• Tool use and APIs (search, code execution, databases, …).
• These layers turn a general LLM into:
• A practical assistant, coding partner, or domain-specific agent.

How to Train Large Language Models?

• Several paradigms exist
• General pre-training -> Domain pre-training -> Task fine-tuning
• Pre-training -> Mid-training -> Post-training
• Pre-training -> Instruction tuning -> Alignment tuning -> Reinforcement learning

Pre-training

Post-training

General �Pre-training

Domain �Pre-training

Mid-training

Instruction Tuning

Preference Tuning

Reinforcement Learning

Pre-training

• Pre-training is usually considered as a step for injecting knowledge.
• Usually, LLMs are trained in the order of specificity (general to specific).
• By the specificity of corpus, the model could be specialized.
• General corpus (news, wiki, book, paper, code, …)
• Specific corpus (finance, medical, legal, game, …)

https://www.youtube.com/watch?v=_HfdncCbMOE

Pre-training

• Foundation model in Korea

https://www.msit.go.kr/bbs/view.do?sCode=user&mId=307&mPid=208&pageIndex=&bbsSeqNo=94&nttSeqNo=3186073

Pre-training

• Domain-specific foundation model in Korea

https://www.msit.go.kr/bbs/view.do;?sCode=user&mPid=208&mId=307&bbsSeqNo=94&nttSeqNo=3186445

Post-training

• Language modeling != assisting users
• Language models are not aligned with user intent.
• Post-training is required as a step for making LLMs follow human instructions.

​

https://openai.com/ko-KR/index/instruction-following/

Post-training

• Instruction tuning
• Collect examples of (instruction, output) pairs across many tasks and finetune an LM
• (and evaluate on unseen tasks)

Scaling Instruction-Finetuned Language Models (https://arxiv.org/pdf/2210.11416)

Post-training

• Instruction tuning with chat (conversational) template.

https://huggingface.co/datasets/KORMo-Team/preference-dataset-qwen3/viewer/lose8B_win80B-A3B

https://huggingface.co/docs/transformers/main/chat_templating

User view

Data view

Model view

Post-training

• However, instruction tuning is not enough to make LLMs aligned. �(seeing only good examples)
• The training objective is still next-token prediction (w/ cross-entropy).
• It aims to increase the probability of correct response, rather than calibrating it. �(over/under-confidence issue)
• Thus, instruction-tuned LLMs often fail to NOT generate wrong responses.
• Unsafe responses …

​

Iterative Reasoning Preference Optimization (https://arxiv.org/pdf/2404.19733)

Post-training

• Preference tuning
• LLMs learn to generate chosen (preferred) response over rejected response.
• Higher likelihood on the chosen response, lower likelihood on the rejected response.

https://tech.kakao.com/posts/662

Post-training

• A recipe of ChatGPT: Reinforcement Learning from Human Feedback (RLHF)

Training language models to follow instructions with human feedback (https://arxiv.org/pdf/2203.02155)

Post-training

• However, collecting labels from human experts is very expensive.
• Alternative: Reinforcement Learning from AI Feedback (RLAIF)

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (https://arxiv.org/pdf/2309.00267)

Post-training

• Also, handling multiple models (policy, reward, reference) is very complex and difficult to optimize.
• Alternative: Direct Preference Optimization (DPO)

MIT 6.S191 (Liquid AI): Large Language Models (https://www.youtube.com/watch?v=_HfdncCbMOE)

Direct Preference Optimization: Your Language Model is Secretly a Reward Model (https://arxiv.org/pdf/2305.18290)

LLM-related Framework

• Do it yourself!
• Model definition framework
• Hugging Face transformers (https://github.com/huggingface/transformers)
• LLM training frameworks
• Hugging Face PEFT (https://github.com/huggingface/peft)
• Hugging Face TRL (https://github.com/huggingface/trl)
• Unsloth (https://github.com/unslothai/unsloth)
• Axolotl (https://github.com/axolotl-ai-cloud/axolotl)
• LLaMA-Factory (https://github.com/hiyouga/LLaMA-Factory)

Masked Language Models

Masked Language Models

https://jalammar.github.io/illustrated-bert/

Conditional Language Models

• Sometimes we want to model output text given an input:
• Example tasks: translation, summarization, question answering
• A conditional language model defines a distribution����where x is some input (source sentence, document, prompt, etc.)
• With the chain rule:

​

​

​

• Many modern “sequence-to-sequence” models are conditional autoregressive language models.

​

How to Use Large Language Model?

• Prompt engineering
• Write your task as a natural language description as clearly as possible.
• You’ve learned few-shot & in-context learning. So, add more examples in the prompt!
• Ultimately, it’s about thinking “which texts are more likely by the LLMs?”

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

How to Use Large Language Model?

• Chain-of-thought prompting
• Add reasoning steps before answering the question.
• Trigger words: “Let’s think step by step.”

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (https://arxiv.org/pdf/2201.11903)

Large Language Models are Zero-Shot Reasoners (https://arxiv.org/pdf/2205.11916)

How to Use Large Language Model?

• Retrieval-Augmented Generation (RAG)
• Augment the input context with relevant documents.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (https://arxiv.org/pdf/2005.11401)

How to Use Large Language Model?

• Context engineering
• Ultimately, it’s about engineering the input “context”.
• system prompt, dialogue history, relevant documents, tools, personal memory, …

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

How to Use Large Language Model?

• But, please equip your own evaluation first.
• Generation largely depends on randomness.
• It’s not guaranteed that your prompt works for every scenario you want.

Stanford CS230 | Autumn 2025 | Lecture 7: Agents, Prompts, and RAG.

How to Use Large Language Model?

• But, please equip your own evaluation first.
• Generation largely depends on randomness.
• It’s not guaranteed that your prompt works for every scenario you want.

Stanford CS230 | Autumn 2025 | Lecture 7: Agents, Prompts, and RAG.

How to Use Large Language Model?

• Be aware of that LLMs are not perfect.
• Sometimes they say incorrect answers.
• Sometimes they say incorrect references.
• Sometimes they say unsafe answers.

Stanford CS230 | Autumn 2025 | Lecture 7: Agents, Prompts, and RAG.

How to Use Large Language Model?

• 1. Local LLM
• transformers & generate()
• serving engine (vLLM)
• 2. Cloud LLM
• OpenAI API
• Structured Outputs
• Classification example

Local LLM – Basics

• 1. Load the tokenizer and model from huggingface.

https://huggingface.co/LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct

🡪 MODEL_NAME = “LGAI-EXAONE/EXAONE-3.5-2.4B-Instruct”

Local LLM – Basics

• 2. Tokenize the prompt

Local LLM – Basics

• 3. Compute logits over vocabulary

Local LLM – Basics

• 4. Select the most likely token among the vocabulary (greedy decoding)

Local LLM – Basics

• 5. Select the most likely token among the vocabulary (greedy decoding)

Local LLM – Basics

• 6. Append the token to the inputs (generated_ids = “Hello, my name is Alex”)

Local LLM – Basics

• 7. Repeat until generating {max_new_tokens} tokens

Local LLM – Basics

• 7. Repeat until generating {max_new_tokens} tokens

Local LLM – Basics

• 7. Repeat until generating {max_new_tokens} tokens

Local LLM – Basics

• 7. Repeat until generating {max_new_tokens} tokens

Local LLM – Basics

• Limitation
• The model should recomptue the entire tokens (input + generated) every time generating each token.

​

KV Caching Explained: Optimizing Transformer Inference Efficiency (https://huggingface.co/blog/not-lain/kv-caching)

Local LLM – Basics

• Solution
• In transformers, the key-value vectors are cached.

​

KV Caching Explained: Optimizing Transformer Inference Efficiency (https://huggingface.co/blog/not-lain/kv-caching)

Local LLM – Basics

• Solution
• In transformers, the key-value vectors are cached.

​

KV Caching Explained: Optimizing Transformer Inference Efficiency (https://huggingface.co/blog/not-lain/kv-caching)

KV Caching

Standard Inference

Local LLM – transformers & generate()

• Generation with KV caching
• Pass the “past_key_values”

​

By Me!

Next

• How to use LLMs (in short)
• Project Presentation Day

LLMs

• tokenization
• decoding
• training
• vLLM
• API usage
• json response
• structured outputs (schema-guided decoding)
• Recommend: https://lmsys.org/blog/2024-02-05-compressed-fsm/

​

Today’s Topic

• Large Language Models (LLMs)
• What is Language Model?
• Data preparation
• Tokenization
• Data Deduplication
• Quality Filtering
• Neural Architecture
• Encoder-only (BERT)
• Encoder-Decoder (T5)
• Decoder-only (GPT)
• Scaling law
• Training Paradigm
• General PT -> Domain PT -> Task Fine-tuning
• Pre-training -> Mid-training -> Post-training
• Reinforcement Learning
• Training Objective
• Maximum likelihood estimation (MLE)
• Attention mask for supervised fine-tuning (SFT), a.k.a. instruction fine-tuning
• Chat template
• Reinforcement Learning from Human Feedback (RLHF) PPO
• Direct preference optimization (DPO)
• Group Relative Preference Optimization (GRPO)

• Large Language Models (LLMs)
• Evaluation
• MMLU
• LMArena
• LLM-as-a-Judge
• Inference
• Decoding
• Greedy decoding
• Beam search
• Schema-guided decoding (a.k.a. structured outputs)
• Speculative decoding
• Test-time scaling
• Prompt engineering
• Majority voting
• Chain-of-thought
• Reasoning
• Tool use (a.k.a. function calling)
• Retrieval-Augmented Generation (RAG)
• Context Engineering
• Memory
• API
• OpenAI-compatible API

Further Topics

• Architectures: Encoder-only, Encoder-Decoder, Decoder-only LMs
• Data preparation: Tokenization, Dataset Construction
• Scaling Law
• Training paradigm: Scaling Law, Domain pre-training, Task fine-tuning, Post-training, Preference Learning, Reinforcement learning, …
• Evaluation: Metrics, Benchmarks, LLM-as-a-Judge, …
• Inference: Decoding, Test-time scaling, Retrieval-augmented generation (RAG), Reasoning, Tool Use, …

References

• [1]: Stanford CS230 | Autumn 2025 | Lecture 7: Agents, Prompts, and RAG.
• Link: https://www.youtube.com/watch?v=k1njvbBmfsw
• Material: https://cs230.stanford.edu/syllabus/fall_2024/rag_agents.pdf
• [2]: MIT 6.S191: Language Models and New Frontiers
• Link: https://www.youtube.com/watch?v=HLKo4fJx_7k
• [3]: MIT 6.S191 (Google): Large Language Models
• Link: https://www.youtube.com/watch?v=ZNodOsz94cc
• [4]: MIT 6.S191 (Liquid AI): Large Language Models
• Link: https://www.youtube.com/watch?v=_HfdncCbMOE
• [5]: The Illustrated Transformer
• Link: https://jalammar.github.io/illustrated-transformer/
• [6]: The Illustrated GPT-2 (Visualizing Transformer Language Models)
• Link: https://jalammar.github.io/illustrated-gpt2/
• Korean Ver.: https://wikidocs.net/160343