STATS / DATA SCI 315

​

Transformers

​

slides credit: Dr. Simon J. D. Prince

https://udlbook.github.io/udlbook/

​

Natural language processing (NLP)

• Translation
• Question answering
• Summarizing
• Generating new text
• Correcting spelling and grammar
• Finding entities
• Classifying bodies of text
• Changing style etc.

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Motivation

Design neural network to encode and process text:

Motivation

Design neural network to encode and process text:

Word embeddings convert words into fixed dimensional vectors

word2vec is a popular family of word embeddings

Motivation

Design neural network to encode and process text:

x N

Standard fully-connected layer

Standard fully-connected layer

Standard fully-connected layer

Problem:�

• A very large number of parameters
• Can’t cope with text of different lengths

�

Conclusion:

• We need a model where parameters don’t increase with input length

Motivation

Design neural network to encode and process text:

The word their must “attend to” the word restaurant.

​

​

Motivation

The word their must “attend to” the word restaurant.

​

Conclusions:�

• There must be connections between the words.
• The strength of these connections will depend on the words themselves.

Design neural network to encode and process text:

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Dot-product self attention

• Takes N inputs of size Dx1 and returns N inputs of size Dx1
• Computes N values (no ReLU)

​

​

• N outputs are weighted sums of these values

​

​

​

• Weights depend on the inputs themselves

​

​

Dot-product self attention

• Takes N inputs of size Dx1 and returns N inputs of size Dx1
• Computes N values (no ReLU)

​

​

• N outputs are weighted sums of these values

​

​

​

• Weights depend on the inputs themselves

​

​

Dot-product self attention

• Takes N inputs of size Dx1 and returns N inputs of size Dx1
• Computes N values (no ReLU)

​

​

• N outputs are weighted sums of these values

​

​

​

• Weights depend on the inputs themselves

​

​

Attention as routing

Attention as routing

Attention as routing

Attention weights

• Compute N “queries” and N “keys” from input

​

​

​

• Calculate similarity and pass through softmax:

​

​

• Weights depend on the inputs themselves

​

​

Attention weights

• Compute N “queries” and N “keys” from input

​

​

​

• Take dot products and pass through softmax:

​

​

• Weights depend on the inputs themselves

​

​

Dot product = measure of similarity

Motivation

Conclusions:�

• We need a model where parameters don’t increase with input length

​

​

• There must be connections between the words.
• The strength of these connections will depend on the words themselves.

Design neural network to encode and process text:

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Matrix form

• Store N input vectors in matrix X

​

​

​

​

• Compute values, queries and keys (all dims are (D x 1) x (1 x N) + D x D x (D x N)):�

​

​

​

​

• Combine self-attentions (D x N x ColSoftmax( N x D x D x N ))

Matrix form

Positional encoding

Self-attention is equivariant to permuting word order

�But word order is important in language:

​

The man ate the fish

vs.

The fish ate the man

​

Positional encoding

Positional encoding

Multi-head self-attention

Why do we learn attention weights?

• Without learning attention weights, a self-attention layer is closely related to Nadaraya-Watson kernel regression
• https://d2l.ai/chapter_attention-mechanisms-and-transformers/attention-pooling.html
• https://en.wikipedia.org/wiki/Kernel_regression
• Self-attention layer is more powerful because it can learn the attention weights as opposed to using a fixed hard-coded function (like the kernel in NW kernel regression)

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Encoders, decoders, and encoder-decoders
• Transformers for vision

​

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

The transformer

In Keras: https://keras.io/api/layers/normalization_layers/layer_normalization/

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Tokenizer

Goal: Tokenizer chooses input “units”

​

• Inevitably, some words (e.g., names) will not be in the vocabulary.
• It’s not clear how to handle punctuation
• The vocabulary would need different tokens for versions of the same word with different suffixes (e.g., walk, walks, walked, walking) and there is no way to clarify that these variations are related

​

Solution: Sub-word tokenization�One particular algorithm: Byte pair encoding

Learning vocabulary

”One hot encoding”

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Three types of transformer layer

• Encoder (BERT)
• Decoder (GPT3) ← will only look at this today
• Encoder-decoder (Translation)

Decoder model

Decoder model: GPT3

• One job: predict the next word in a sequence
• More formally builds an autoregressive probability model

Decoder model: GPT3

• Builds autoregressive probability model
• E.g. “It takes great courage to let yourself appear weak”

Predicting all next words simultaneously

Masked self-attention

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

GPT3 (Brown et al. 2020)

• Sequence lengths are 2048 tokens long
• Batch size is 3.2 million tokens.
• 96 transformer layers (some of which implement a sparse�version of attention), each of which processes a word embedding of size 12288.
• 96 heads in the self-attention layers and the value, query, and key dimension is 128.
• 300 billion tokens
• 175 billion parameters

try some out at

https://umgpt.umich.edu/

What does it learn?

• Syntax

​

“Tomorrow, let’s <VERB>...”

​

• General knowledge:

​

“The train pulled into the …”

​

​

Text completion

Understanding Deep Learning is a new textbook from MIT Press by Simon Prince that's designed to offer an accessible, broad introduction to the field. Deep learning is a branch of machine learning that is concerned with algorithms that learn from data that is unstructured or unlabeled. The book is divided into four sections:

​

• Introduction to deep learning
• Deep learning architecture
• Deep learning algorithms
• Applications of deep learning

​

The first section offers an introduction to deep learning, including its history and origins. The second section covers deep learning architecture, discussing various types of neural networks and their applications. The third section dives into deep learning algorithms, including supervised and unsupervised learning, reinforcement learning, and more. The fourth section applies deep learning to various domains, such as computer vision, natural language processing, and robotics.

​

Few shot learning:

ChatGPT

• GPT3.5 fine-tuned with human annotations
• Trained to predict the next word + be “helpful, honest, harmless”

​

ChatGPT

• GPT3.5 fine-tuned with human annotations
• Trained to predict the next word + be “helpful, honest, harmless”

​