CS 162: Natural Language Processing
Saadia Gabriel
Lecture 5:
Language Modeling Part 2
Announcements
The midterm will be on 5/4,
with a make-up midterm available on 5/8.
Quiz Recap
Quiz Recap
Semantics concerns the literal meaning of words and sentences, while pragmatics concerns how meaning is interpreted in context and use
Quiz Recap
Statistical associations are what language models learn rather than true semantic understanding. That being said, to what degree learning the statistical associations leads to implied semantic understanding is not clear…
No, because model can learn and generate according to patterns in the corpus without understanding what they mean
Maybe???
Yes, because it should be generated by not just arranging words, but understanding the context and intent of language to make a sensible sentence
This is debatable
Quiz Recap
Q & A
We don’t like negative numbers
When you’re first developing a LM, perplexity is a good check to make sure it’s sensible. Then you can further tune that model to perform well on downstream tasks.
Q & A
They are limited in terms of capturing discourse
coherence.
This limited context window can also lead to coreference resolution issues...
Q & A
We’re going to do a quick recap!
Smoothing Recap
Add-0.001 Smoothing
High variance solution: this overfits to the observed data
“Novel event” = 0-count event (never happened in training data).
“Novel event” = 0-count event (never happened in training data).
Add-1000 Smoothing
High bias solution: this underfits to the observed data
“Novel event” = 0-count event (never happened in training data).
“Novel event” = 0-count event (never happened in training data).
Add-Lambda Smoothing
Setting Smoothing Parameters
Setting Smoothing Parameters
Setting Smoothing Parameters
5-fold Cross-Validation
N-fold Cross-Validation
But is Add-Lambda the best strategy we could come up with to handle unseen events?
*** Short Discussion ***
Consider for a moment…
Backoff Smoothing
Backoff Smoothing
Solution:
Backoff Smoothing
We won’t cover Good-Turing, but if curious you can read the paper “Good-Turing smoothing without tears”
Backoff Smoothing
*** Short Discussion ***
Tokenization
Example from Yoav Artzi
Tokenization
*** Short Discussion ***
Tokenization
Example from Yoav Artzi
Tokenization
Slide from Yoav Artzi
https://www.cs.cornell.edu/courses/cs5740/2025sp/
We’ll revisit this later when we talk about transformers and byte-pair encodings
Back to smoothing:
Model averaging
Backoff Smoothing
Log-linear Language Models
Recall Logistic Regression
Input features
Output classes
Just linear scoring!
Sigmoid converts to class probability
Linear Scoring for NLP
Word probs are conditioned on some context
Linear Scoring
Example from Graham Neubig
Special case for sparsity:
Linear Scoring
Example from Graham Neubig
Special case for sparsity:
Log-Linear Conditional Probability�(interpret score as a log-prob)
Log-Linear Conditional Probability�(interpret score as a log-prob)
This is softmax!
Training
Gradient-based training
Recall:
Recall:
Apply log rules: (1) log(a/b) = log(a) - log(b), (2) log(e^a) = a
Observed features
Expected features
Recap: optimizing language models
We want to minimize the difference between our predicted and actual language distributions
We predict q
We observe p
Neural Language Models
From https://research.gatech.edu/new-neural-network-makes-decisions-human-would
Early work on neural networks began in the 40s and 50s.
Earliest neural network (the perceptron) implemented in 1958.
(Roughly inspired by natural brain function)
Downsides of Prior Models
We’re still relying on very simple features and narrow context windows but human language is extremely complex…
Ideally, we should have models that can learn appropriate features on their own and capture richer patterns in language data for modeling linguistic phenomena like entity disambiguation…
Neural language model
How to model word similarities?
Neural language model
We embed text
We apply linear and non-linear transformations to embedded text
We output a probability distribution
We apply linear and non-linear transformations to embedded text
Neural Network Basics
How Neural Networks Make Predictions
z1 = xW1 + b1
a1 = σ(z1)
z2 = a1W2 + b2
a2 = softmax(z2)
activation units
z1
x
a1
z2
a2
Learning the Parameters
Guess?
Backpropagation
Backpropagation
Backpropagation
Backpropagation
Update the Parameters
Use Case:
Text Classification
Positive or negative review?
Positive or negative review?
Text Classification
Sentiment Analysis
Why sentiment analysis?
Recall our old solution:
Recall our old solution:
Supervised Classification
We can build a neural classifier!
We’ll also talk about attention mechanisms next time, these are central to transformers!
In the next lecture we’ll discuss a specific kind of RNN design, the long-short term memory model or LSTM
Example of RNN classifier (courtesy of MDPI)
Next time…