CS 263:
Advanced NLP
Saadia Gabriel
Lecture 6
Announcements
Last time…
What is the best approach for continuing training to improve general model performance on aspects like safety or usability?
Today
We will explore various sampling or search based approaches for constructing actual text sequences from our language model’s token probability distributions
fun?
Quiz #2
https://forms.gle/ZUcdgV2aWjmsPCvs7
So far we’ve mostly talked about training models…
The following slides are partially adapted from Graham Neubig’s and Amanda Bertsch’s slides for CMU 11-664/763
How do we sample from our trained models?
Introduction to Inference Algorithms
Our model:
How do we select each yj from this distribution?
At each time step, select the most likely token under the model’s probability distribution:
Greedy decoding:
Issues with Greedy Decoding
This ignores the long-tail of the model’s distribution, and leads to generic outputs:
Other Sampling Methods
Ancestral sampling:
Random based on underlying probabilities
Now we can end up with very improbable sequences
Other Sampling Methods
Top-k:
At each time step you sample from the renormalized distribution from the top k set
Other Sampling Methods
Nucleus (or top-p sampling):
Here top-k with k=3 would cover most of the probability mass
Many likely options are discarded under top-k
Distribution Temperature
Distribution Temperature
Scale the logits by a hyperparameter T > 0
What happens if T is large (e.g 1.5)?
What happens if T is small (e.g .2)?
https://medium.com/@imisri1/how-to-set-sampling-temperature-for-gpt-models-762887df1fac
HuggingFace Model Generation
Prompt: We were sitting in CS263 at UCLA…
Continuation: , learning about the basics of computer science…
Continuation (k=10): , a class on computer security, where I…
Continuation (p=.9): in 1989, where we learned…
Continuation (k=10, T=2.0): in January of 21 when I had what I considered to be…
Issues with sampling-based methods
While this is mostly likely at each timestep, it may not generate an optimal sequence.
But is it always true that a “good” token selection at timestep t will lead to a “good” token selection at timestep t+1, t+2, etc?
Beam Search
The basic idea is that we explore more options for our partial sequence before making a decision about which tokens to keep.
This is a breadth-first search:
Beam Search
Beam Search
Beam Search
Beam Search
Beam Search
Beam Search
Beam Search
Why?
Beam Search Summary
Beam Search Variants
Diverse Beam Search
We can modify this step
Diverse Beam Search
Diverse Beam Search
Diverse Beam Search
Diverse Beam Search
Diverse Beam Search
Stochastic Beam Search
(Kool et al., 2019)
Improves diversity by adding randomness to selection process.
Gumbel Distribution
(often used to model the distribution for sample extremes (min, max))
Stochastic Beam Search
(Kool et al., 2019)
Stochastic Beam Search
(Kool et al., 2019)
Selecting a Beam Size
Trade-offs of Inference Algorithms
Greedy
Pros: Fast and deterministic
Cons: Can be suboptimal (particularly repetitive outputs)
Sampling (top-k, top-p)
Pros: More creative and diverse outputs
Cons: Can introduce too much variance and incoherent or unrealistic outputs
Beam Search
Pros: Explores possible subsequences, improving coherence and overall quality
Cons: Computationally expensive and suffers from diversity issues
Advanced Inference Methods
Constrained Decoding
Fudge (Yang and Klein, 2021)
Constrained Decoding
Fudge (Yang and Klein, 2021)
Constrained Decoding
Fudge (Yang and Klein, 2021)
Contrastive Decoding
Li et al. (2023)
Contrastive Decoding
Li et al. (2023)
Self-correcting Decoding
Madaan et al. (2023)
What is test-time scaling?
Instead of retraining the model, improve task performance through additional inference-time compute (e.g. generating additional tokens to explain step-by-step process of problem-solving)
Chain-of-thought prompting
(Wei et al., 2022)
Implicit Chain-of-thought
(Kojima et al., 2022)
Next time:
Evaluation!