Large Language Models

Lecture 2

Introduction to Language Model:

Statistical Language Modeling

Krishnendu Ghosh

What is a Language Model (LM)?

Language Model gives the probability distribution over a sequence of tokens.

What is a Language Model (LM)?

Language Model gives the probability distribution over a sequence of tokens.

What is a Language Model (LM)?

Language Model gives the probability distribution over a sequence of tokens.

Next Word Prediction

Language Models: Applications

Speech Recognition

• P(I bought fresh mangoes from the market) >> P(I bot fresh man goes from the mar kit)
• P(I love eating spicy samosas) >> P(eye love eat tin spy sea some o says)

Machine Translation

• P(Heavy rainfall) >> P(Big rainfall)
• P(The festival of lights) >> P(the festival of lamps)
• P(Family gatherings) > P(Family meetings)

Context Sensitive Spelling Correction

• …

Natural Language Generation

• …

Probabilistic Language Models

​

Goal: Calculate the probability of a sentence or sequence consisting of n words P(W) = P(w1, w2, w3, ... , wn)

​

or

​

Related Task: Calculate the probability of the next word conditioned on the preceding words P(w6 | w1, w2, w3, w4, w5)

Probability of a Sentence

Let’s consider the following sentence:

The monsoon season has begun

​

How to compute the probability of the sentence?

P(W)

= P(“The monsoon season has begun”)

= P(The, monsoon, season, has, begun)

​

We compute the above joint probability by using the principles of

Chain Rule of Probability

Chain Rule of Probability

​

Definition of conditional probability: P( A | B ) = P( A, B ) / P( B )

​

Rewriting: P( A, B ) = P( A | B ) P( B )

​

More variables: P(A, B, C, D) = P(A) . P(B | A) . P(C | A, B) . P(D| A, B, C)

​

The Chain Rule in general:

​

P(x1, x2, x3, … , xn) = P(x1) P(x2 | x1) P( x3 | x1,x2) … P(xn| x1,…,xn-1)

Probability of a Sequence

​

​

​

​

P(W) = P(“The monsoon season has begun”)

= P(The, monsoon, season, has, begun)

= P(The) x P(monsoon | The) x P(season | The monsoon) x P(has | The monsoon season) x P(begun | The monsoon season has)

Estimate Conditional Probabilities

​

​

​

​

Problem: Enough data is not available to get an accurate estimate of the above quantities.

​

Solution: Markov Assumption

​

Every next state depends only the previous k states

Markov Assumption

Every next state depends only the previous k states

​

​

N-gram Language Models

Let’s consider the following conditional probability:

​

P(begun | the monsoon season has)

​

An N-gram model considers only the preceding N −1 words.

• Unigram: P(begun)

• Bigram: P(begun | the)

• Trigram: P(begun | the monsoon)

N-gram Language Models

N-gram Language Models

Limitation: N-gram LMs

An insufficient model of language since they are not effective in capturing long-range dependencies present in language.

​

Example:

The project, which he had been working on for months, was finally approved by the committee.

​

The above example highlights the long-distance dependency between “project” and “approved”, where the context provided by earlier words affects the interpretation of later parts of the sentence.

​

Estimate N-gram Probabilities

Maximum Likelihood Estimate (MLE):

• Used to estimate the parameters of a statistical model
• Determine the most likely values of the parameters that would make the observed data most probable

Limitations with MLE Estimation

Problem: N-grams only work well for word prediction if the test corpus looks like the training corpus. It is often not the case in real scenarios.

​

Training set:

• ... enjoyed the movie

• ... enjoyed the food

• ... enjoyed the game

• ... enjoyed the vacation

​

Zero probability n-grams:

P(concert | enjoyed the) = P(festival | enjoyed the) = P(walk | enjoyed the) = 0

​

​

​

Test set:

• ... enjoyed the concert

• ... enjoyed the festival

• ... enjoyed the walk

​

Limitations with MLE Estimation

Limitations with MLE Estimation

Laplace Smoothing

Imagine that we encountered each word (N-gram) one more time than its actual occurrence.

​

Simply increase all the counts by one!

Before and After Smoothing

Before and After Smoothing

Before and After Smoothing

Before and After Smoothing

Laplace Smoothing

More Smoothing Techniques

Advanced Smoothing Algorithms

Naïve smoothing algorithms have limited usage and are not very effective. Not frequently used for N-grams.

​

However, they can be used in domains where the number of zeros isn’t so huge.

​

Popular Algorithms:

• Good-Turing

• Kneser-Ney

Good Turing Smoothing

N_C = Frequency of frequency of c

​

Rohan I am I am Rohan I like to play

​

I 3

Rohan 2

Am 2

like 1

to 1

play 1

Good Turing Smoothing

N_C = Frequency of frequency of c

​

Rohan I am I am Rohan I like to play

​

I 3

Rohan 2

Am 2

like 1

to 1

play 1

Good Turing Smoothing

You are birdwatching in the Jim Corbett National Park and you have observed the following birds: 10 Flamingos, 3 Kingfishers, 2 Indian Rollers, 1 Woodpecker, 1 Peacock, 1 Crane = 18 birds

​

How likely is it that the next bird you see is a woodpecker?

• 1/18

Good Turing Smoothing

You are birdwatching in the Jim Corbett National Park and you have observed the following birds: 10 Flamingos, 3 Kingfishers, 2 Indian Rollers, 1 Woodpecker, 1 Peacock, 1 Crane = 18 birds

​

How likely is it that the next bird you see is a new species -- Purple Heron or Painted Stork?

• We will use our estimate of things we saw once to estimate the new things.
• 3/18 (because N1 = 3)

Good Turing Smoothing

You are birdwatching in the Jim Corbett National Park and you have observed the following birds: 10 Flamingos, 3 Kingfishers, 2 Indian Rollers, 1 Woodpecker, 1 Peacock, 1 Crane = 18 birds

​

Assuming so, how likely it is that the new species is Woodpecker?

• Must be less than 1/18

Limitations with MLE Estimation

Good Turing Calculations

Good Turing Calculations

Limitations with MLE Estimation

Good Turing Estimation

• Numbers from Church and Gale (1991)
• 22 million words of AP Newswire

​

​

​

It looks like c* = (c - 0.75)

Kneser-Ney Smoothing

Back-off and Interpolation

​

As N grows larger, the N-gram model becomes more powerful. However, its capability to accurately estimate parameters decreases due to data sparsity problem.

​

​

When we have limited knowledge about larger contexts, it can be helpful to consider less context.

Back-off and Interpolation

Back-off:

​

• Opt for a trigram when there is sufficient evidence, otherwise use bigram, otherwise unigram

​

Interpolation:

​

• Mix unigram, bigram, trigram
• Interpolation generally results in improved performance

Back-off and Interpolation

Back-off and Interpolation

Interpolation

Absolute Discounting Interpolation

Adjusts the probability estimates for n-grams by discounting each count by a fixed amount (usually a small constant) before computing probabilities

​

​

​

​

​

But considering the regular unigram probability has some limitations, as we will see in the upcoming slides.

Evaluation of a Language Model

Does our language model prefer good sentences over bad ones?

• Assign higher probability to “real” or “frequently observed” sentences than “ungrammatical” or “rarely observed” sentences

​

Terminologies:

• We optimize the parameters of our model based on data from a training set.
• We assess the model's performance on unseen test data that is disjoint from the training data.
• An evaluation metric provides a measure of the performance of our model on the test set.

Extrinsic Evaluation

Measure the effectiveness of a language model by testing their performance on different downstream NLP tasks, such as machine translation, text classification, speech recognition.

​

Let us consider two different language models: A and B

• Select a suitable evaluation metric to assess the performance of the language models based on the chosen task.
• Obtain the evaluation scores for A and B
• Compare the evaluation scores for A and B

Intrinsic Evaluation: Perplexity

Intuition: The Shannon Game

​

How well can we predict the next word?

• I always order pizza with cheese and ...
• The president of India is ...
• I wrote a …

​

Observation: The more context we consider, the better the prediction.

Perplexity

The best language model is one that best predicts an unseen test set.

​

Perplexity is the inverse probability of the test data, normalized by the number of words.

​

Given a sentence W consisting of n words, the perplexity is calculated as follows:

Perplexity

Perplexity

Problems of Statistical LMs

• N-gram LMs suffer from data sparsity and limited context.
• Predicting the next word using a fixed window of previous words.
• Fixed Context Size: Limited to a fixed window of previous words.

​

• Smoothing techniques address data sparsity.
• But even with smoothing, rare n-grams are hard to predict.

​

• Large vocabulary leads to high memory requirements.

​

• High computational cost for large n-grams.

​

• Lack of generalization to unseen word combinations.

Need for Richer Representations

Requirements:

​

• Contextual Understanding: Need for models that understand context beyond fixed windows.

​

• Semantic Similarity: Ability to capture relationships between words (e.g., synonyms).

​

• Scalability: Models that can scale to large datasets and handle vast vocabularies efficiently.

Need for Richer Representations

Moving towards

Word Embeddings & Neural LM

In the successive lectures, we will see how representing words (actually, tokens) as vectors and transition to neural LMs solve many of those problems.

​

• Move from discrete to continuous representations.

• Capture richer semantic information.

• Enable generalization to unseen data.

• Scale to large datasets.

Moving towards

Word Embeddings & Neural LM

Moving towards

Word Embeddings & Neural LM

Timeline in Language Modelling

Source: https://lcs2-iitd.github.io/ELL881-AIL821-2401/lectures/