Encoder Decoder / Attention/ Transformers /

1

10-Apr-23

Today Nov 4

• Encoder Decoder
• Attention
• Transformers

2

10-Apr-23

Encoder-Decoder

• RNN: input sequence is transformed into output sequence in a one-to-one fashion.

​

​

​

• Goal: Develop an architecture capable of generating contextually appropriate, arbitrary length, output sequences
• Applications:
• Machine translation,
• Summarization,
• Question answering,
• Dialogue modeling.

Simple recurrent neural network illustrated as a feed-forward network

Most significant change: new set of weights, U

• connect the hidden layer from the previous time step to the current hidden layer.
• determine how the network should make use of past context in calculating the output for the current input.

​

Simple-RNN abstraction

y₂

y₁

y₃

RNN Applications

• Language Modeling

​

​

• Sequence Classification (Sentiment, Topic)

​

​

• Sequence to Sequence

​

​

Sentence Completion using an RNN

​

• Trained Neural Language Model can be used to generate novel sequences
• Or to complete a given sequence (until end of sentence token <\s> is generated)

Extending (autoregressive) generation to Machine Translation

• Build an RNN language model on the concatenation of source and target

• Training data are parallel text e.g., English / French

there lived a hobbit vivait un hobbit

……..

there lived a hobbit <\s> vivait un hobbit <\s>

……..

word generated at each time step is conditioned on word from previous step.

Extending (autoregressive) generation to Machine Translation

• Translation as Sentence Completion !

(simple) Encoder Decoder Networks

• Encoder generates a contextualized representation of the input (last state).
• Decoder takes that state and autoregressively generates a sequence of outputs

Limiting design choices

• E and D assumed to have the same internal structure (here RNNs)
• Final state of the E is the only context available to D
• this context is only available to D as its initial hidden state.

General Encoder Decoder Networks

Abstracting away from these choices

1. Encoder: accepts an input sequence, x_1:n and generates a corresponding sequence of contextualized representations, h_1:n
2. Context vector c: function of h_1:n and conveys the essence of the input to the decoder.
3. Decoder: accepts c as input and generates an arbitrary length sequence of hidden states h_1:m from which a corresponding sequence of output states y_1:m can be obtained.

Popular architectural choices: Encoder

Widely used encoder design: stacked Bi-LSTMs

• Contextualized representations for each time step: hidden states from top layers from the forward and backward passes

Decoder Basic Design

• produce an output sequence an element at a time

Decoder Design�Enhancement

Context available at each step of decoding

Decoder: How output y is chosen

• Sample soft-max distribution (OK for generating novel output, not OK for e.g. MT or Summ)
• Most likely output (doesn’t guarantee individual choices being made make sense together)

For sequence labeling we used Viterbi – here not possible ☹

• 4 most likely “words” decoded from initial state
• Feed each of those in decoder and keep most likely 4 sequences of two words
• Feed most recent word in decoder and keep most likely 4 sequences of three words …….
• When EOS is generated. Stop sequence and reduce Beam by 1

Today Nov 4

• Encoder Decoder
• Attention
• Transformers

17

10-Apr-23

Sequence to Sequence Learning

• An encoder processes the input sequence and compresses the information into a context vector (also known as sentence embedding or “thought” vector) of a fixed length. This representation is expected to be a good summary of the meaning of the whole source sequence.�
• A decoder is initialized with the context vector to emit the transformed output. The early work only used the last state of the encoder network as the decoder initial state.

​

• Both the encoder and decoder are recurrent neural networks, i.e. using LSTM or GRU units.

​

• A critical and apparent disadvantage of this fixed-length context vector design is incapability of remembering long sentences. Often it has forgotten the first part once it completes processing the whole input. The attention mechanism was born (Bahdanau et al., 2015) to resolve this problem.

Attention Model

• The attention mechanism was born to help memorize long source sentences in neural machine translation (NMT). Rather than building a single context vector out of the encoder’s last hidden state.

​

• The last state can not remember all the information of previous states. So they introduced attention model to overcome this problem.

​

• The secret sauce invented by attention is to create shortcuts between the context vector and the entire source input. The weights of these shortcut connections are customizable for each output element.

​

• Context vector has access to the entire input sequence, we don’t need to worry about forgetting. The alignment between the source and target is learned and controlled by the context vector.

​

​

​

Essentially the context vector consumes t pieces of information:

• encoder hidden states;
• decoder hidden states;
• alignment between source and target.

​

Flexible context: Attention

Context vector c: function of h_1:n and conveys the essence of the input to the decoder.

h₁

h₁

h₂

h₂

h_n

h_m

Flexible?

• Different for each h_i
• Flexibly combining the h_j

Attention (1): dynamically derived context

• Replace static context vector with dynamic c_i
• derived from the encoder hidden states at each point i during decoding

Ideas:

• should be a linear combination of those states

​

​

​

• should depend on ?

Attention (2): computing c_i

• Compute a vector of scores that capture the relevance of each encoder hidden state to the decoder state

• Just the similarity

• Give network the ability to learn which aspects of similarity between the decoder and encoder states are important to the current application.

Attention (3): computing c_i�From scores to weights

• Create vector of weights by normalizing scores

• Goal achieved: compute a fixed-length context vector for the current decoder state by taking a weighted average over all the encoder hidden states.

Attention: Summary

Encoder

Decoder

Explain Y. Goldberg different notation

Intro to Encoder-Decoder and Attention (Goldberg’s notation)

Encoder

Decoder