Practical Guide to Pretrained Language Models

Taiwei Shi

LLM Training Paradigm

1. Pretraining
2. Post-training
1. Supervised Fine-Tuning (SFT)
2. Preference Optimization (e.g., Reinforcement Learning from Human Feedback - RLHF)

Pretraining

Pretraining is the act of training a model from scratch: the weights are randomly initialized, and the training starts without any prior knowledge.

Post-training (Finetuning + Preference Optimization)

Post-training, on the other hand, is the training done after a model has been pretrained.

• Transfer learning: the pretrained model was already trained on a dataset that has some similarities with the fine-tuning dataset.
• Efficiency: since the pretrained model was already trained on lots of data, the fine-tuning requires way less data and resources to get decent results.

Transformers for causal language modeling

Transformers for masked language modeling (MLM)

​

Transformers are big models

Encoder vs Decoder

• Encoder (left): The encoder receives an input and builds a representation of it (its features). This means that the model is optimized to acquire understanding from the input.
• Decoder (right): The decoder uses the encoder’s representation (features) along with other inputs to generate a target sequence. This means that the model is optimized for generating outputs.

Three Categories of Transformers

1. GPT-like (also called auto-regressive Transformer models)
2. BERT-like (also called auto-encoding Transformer models)
3. BART/T5-like (also called sequence-to-sequence Transformer models)

Examples of Encoder-only Models (MLM)

• BERT (Bidirectional Encoder Representations from Transformers)
• BERT is designed to understand the context of words in search queries. It has significantly improved tasks like question answering and sentiment analysis.
• RoBERTa (A Robustly Optimized BERT Pretraining Approach)
• RoBERTa is trained with more data and optimized techniques
• DistilBERT
• A distilled version of BERT that is smaller, faster, and lighter
• XLNet
• XLNet integrates bidirectional context into autoregressive models

Examples of Decoder-only Models

• Proprietary (only API access)
• ChatGPT, Gemini, Claude
• Open-weight (no data/code access, just the model weight)
• Meta LLaMA, Google Gemma, Mistral, Microsoft Phi, Alibaba Qwen, …
• Open-source (access to everything including data and code)
• AI 2 OLMo, EleutherAI GPT-Neo/Pythia/GPT-J

Can we get something that is even close to ChatGPT by just doing next-token prediction on the web???

Can we get something that is even close to ChatGPT by just doing next-token prediction on the web???

NO

Original GPT-3

GPT-3 after Instruction Tuning and RLHF

Technical Evolution of GPT-series Models

LLM Training Paradigm

• Pretraining
• Supervised Fine-Tuning (SFT)
• Preference Optimization (e.g., Reinforcement Learning from Human Feedback - RLHF)

HuggingFace

• One-stop shop for modern NLP
• Datasets
• Pretrained models
• Training scripts
• Evaluation metrics
• Efficient training
• Parameter Efficient Finetuning → PEFT
• Multi-gpu or multi-node training → accelerate

HuggingFace Datasets → https://huggingface.co/datasets

HuggingFace Models → https://huggingface.co/models

LLM Inference Framework

vLLM (https://github.com/vllm-project/vllm)

24 times faster than using vanilla pytorch!!!

LLM Training Framework

LLaMA Factory (https://github.com/hiyouga/LLaMA-Factory)

OpenRLHF (https://github.com/OpenRLHF/OpenRLHF)

LLM Hardware Requirement

Scaling Laws for Train-Time Compute

The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters N, the dataset size D, or the optimally allocated compute budget Cmin

Scaling Laws for Test-Time Compute

The performance of transformers consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).