Large Language Models
Lecture 6
Neural LMs: Seq-to-Seq and Attention
Krishnendu Ghosh
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
LSTM
Attention based Models
Attention based Models
Attention based Models
Attention based Models
Attention based Models
Neural Machine Translation?
• Neural Machine Translation (NMT) is a way to do Machine Translation with a single neural network.
• The neural network architecture is called sequence-to-sequence (aka seq2seq) and it involves two RNNs.
Neural Machine Translation (NMT)
Neural Machine Translation (NMT)
Sequence-to-Sequence is Versatile!
• The general notion here is an encoder-decoder model
• One neural network takes input and produces a neural representation
• Another network produces output based on that neural representation
• If the input and output are sequences, we call it a seq2seq model
• Sequence-to-sequence is useful for more than just MT
• Many NLP tasks can be phrased as sequence-to-sequence:
• Summarization (long text → short text)
• Dialogue (previous utterances → next utterance)
• Parsing (input text → output parse as sequence)
• Code generation (natural language → Python code)
Neural Machine Translation (NMT)
Training an NMT System
Greedy Decoding
Problems: Greedy Decoding
• Greedy decoding has no way to undo decisions!
• Input: il a m’entarté
• → he
• → he hit
• → he hit a
(he hit me with a pie)
(whoops! no going back now...)
How to fix this?
Exhaustive Search Decoding
Beam Search Decoding
Beam Search Decoding:
Stopping Criterion
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding: Example
Beam Search Decoding:
Stopping Criterion
Beam Search Decoding:
Finishing Up
Sequence-to-Sequence:
The Bottleneck Problem
• Linear interaction distance
• Bottleneck problem
• Lack of parallelizability
Attention
• Attention provides a solution to the bottleneck problem.
• Core idea: on each step of the decoder, use direct connection to the encoder to focus on a particular part of the source sequence
• Let’s start with the visualization of the attention mechanism.
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Sequence-to-Sequence
with Attention
Attention: In Equations
Attention is Great
Attention significantly improves NMT performance
• It’s very useful to allow decoder to focus on certain parts of the source
• Attention solves the bottleneck problem
• Attention allows decoder to look directly at source; bypass bottleneck
• Attention helps with vanishing gradient problem
• Provides shortcut to faraway states
• Attention provides some interpretability
• By inspecting attention distribution, we can see what the decoder was focusing on
• We get (soft) alignment for free!
• This is cool because we never explicitly trained an alignment system
• The network just learned alignment by itself
Seq2Seq+Attention for LM
Attention is General Deep Learning
• We’ve seen that attention is a great way to improve the sequence-to-sequence model for Machine Translation.
• However: You can use attention in many architectures (not just seq2seq) and many tasks (not just MT)
More general definition of attention:
• Given a set of vector values, and a vector query, attention is a technique to compute a weighted sum of the values, dependent on the query.
• We sometimes say that the query attends to the values.
• For example, in the seq2seq + attention model, each decoder hidden state (query) attends to all the encoder hidden states (values).
Intuition:
• The weighted sum is a selective summary of the information contained in the values, where the query determines which values to focus on.
• Attention is a way to obtain a fixed-size representation of an arbitrary set of representations (the values), dependent on some other representation (the query).
Attention
Attention
Attention
Attention
Attention
Attention: Examples
Attention: Examples
Attention: Examples
Variants of Attention