CS 162: Natural Language Processing

Saadia Gabriel

Lecture 8:

Language Modeling Part 5

Announcements

• Reminder that homework #1 is due this Friday!
• I’ve posted a sign-up sheet under week 4 for group presentations of final project papers (early sign-ups welcome)

Quiz #3

https://forms.gle/MxUjCaDb3rEeceau5

The weights are the same (same transformation of the input), but the resulting query, word and value vectors are word-specific.

Increasing the complexity of the network usually improves performance because more information can be captured (same rationale for why we use multiple, varying numbers of attention heads).

Yes, efficient NLP is a whole area of research which you can explore for your final project! Often components or entire layers can be removed for certain tasks (pruning), or a smaller model can be trained to mimic the behavior of a larger model (knowledge distillation).

You can also reduce the precision of numbers used to store the model’s parameters (quantization) to save memory.

Computational Limitations of Self-attention

• The computational cost of self-attention is O(n²d), where n is the sequence length and d is the embedding dimension size.

• This limits the length of sequences we can feasibly process!

Good question. BPE is used in most modern models, though there are now a number of tokenization variations and some may use a variation.

On a side note, you can use the tiktoken package to see how GPT models will tokenize an input prompt.

Saadia is teaching a class right now.

Sa

adia

is

teaching

a

class

right

now

.

Sparse attention, key-value caching and other modifications allow us to have longer context sizes.

Goals of This Lecture

• How do we handle limitations of self-attention? (information loss, inefficiency)

​

• We’ll motivate model pretraining from word embeddings

​

• We’ll dive into pretraining model architectures
• Unidirectional
• Independently bidirectional
• Bidirectional

​

• We’ll discuss scaling laws for language models

​

• We’ll introduce large language models

Following slides courtesy of Violet Peng and Jia-Chen Gu

Residual Connections

• Issue of information loss
• Self-attention can decide not to attend to itself

• Add input back again in each sublayer!

Residual Connections

Say we want to do movie review classification with our transformer.

​

Should we train from scratch on our labeled movie review dataset?

Language Model

The Illustrated GPT-2, https://jalammar.github.io/illustrated-gpt2/

Word Embeddings

Contextualized Representations

The Pre-train

Fine-tune Paradigm

Image from Gu et al. (2020)

Why Pre-training? 

The Pretraining Revolution

• Pretraining has had a significant and tangible impact on how well NLP systems work

Scale Unsupervised Learning and Generalize

Parametric knowledge

acquisition

Pre-training Model Architectures

Jacob Devlin, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2018.

Unidirectional Pre-training

• Decoder-only and cannot condition on future words: Processes input sequences in a left-to-right manner, so only considers the preceding context when predicting a word (well-suited to text generation)

• See Improving Language Understanding by Generative Pre-Training (GPT), OpenAI (2018)

GPT Pre-training and Fine-tuning

GPT Architecture

https://jalammar.github.io/illustrated-gpt2/

GPT Technical Details

Fine-tuning GPT for Downstream Tasks

GPT for Natural Language Inference (Entailment)

GPT Results on NLI

Effect of Pre-training in GPT

Effect of Pre-training in GPT

You don’t even have to fine-tune! Tasks can be performed “zero-shot” meaning the task instructions are specified in natural language at test-time in a prompt:

Effect of Pre-training in GPT

Stronger GPT-2

GPT-2 Results

Stronger GPT-3

https://ourworldindata.org/grapher/artificial-intelligence-training-computation

GPT-3 with In-context Learning

GPT-3 with In-context Learning

Independently Bidirectional Pre-training

ELMo Technical Details and Results

Limitations of ELMo and GPT series

Bidirectional Pre-training

Masked Language Models

Next Sentence Prediction

BERT was also trained on a next sentence prediction objective to improve its capabilities for capturing long-range dependencies.

Next Sentence Prediction

BERT performance with/without next sentence prediction (NSP)

Devlin et al. (2019)

BERT Input Representation

BERT Technical Details

BERT Fine-tuning

BERT Results

Extension of BERT

Domain-specific BERTs

Extension of Pre-training Encoder-Decoders

Resources for Coding

Scaling Laws

Scaling Laws

*** Discuss ***

Does it matter how we scale our LLMs?

Scaling Laws

Large Language Models and Beyond

Image from https://blogs.cfainstitute.org/investor/2023/05/26/chatgpt-and-large-language-models-six-evolutionary-steps/

Many components of LLM development have been scaling up….

Training LLMs: Data

Training LLMs: Context Size

Next Time…

Post-training

LLM Hallucination

Why does this happen and what can we do about it?

In general, LLM Safety

Lecture Take-aways