Deep Learning (DEEP-0001)�

Prof. André E. Lazzaretti

lazzaretti@utfpr.edu.br

https://sites.google.com/site/andrelazzaretti/graduate-courses/deep-learning-cpgei/2025

13 – Transformers

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Motivation

Design neural network to encode and process text:

The word their must “attend to” the word restaurant.

​

​

Motivation

The word their must “attend to” the word restaurant.

​

Conclusions:�

• There must be connections between the words.
• The strength of these connections will depend on the words themselves.

Design neural network to encode and process text:

Motivation

• RNN models become critical at longer sequence lengths, as memory constraints limit batching across examples.

​

Architecture

Input Embedding

Positional encoding

Order is important in language:

​

The man ate the fish

vs.

The fish ate the man

​

Positional Encoding

Positional Encoding

Problem: the scale of these numbers may limit optimization!

Problem: for length 5, 0.8=4/5, meaning it would be the 4th element. For sequence length 20, 0.8=16/20 means 0.8 represents the 16th element!

Positional Encoding

Positional Encoding

Positional Encoding

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Dot product = measure of similarity

Self Attention

Query, Key, and Value

Query, Key, and Value

Search request

Possible similar videos (e.g., same title)

Content of a video

Similarity: proxy to attention!

Derive key, query, and value

Attention scores (filter)

All the inputs

Multi-head

Instead of one single attention head, Q, K, and V are split into multiple heads. It allows the model to jointly attend to information from different representation subspaces at different positions.

As we encode the word "it", one attention head is focusing most on "the animal", while another is focusing on "tired" - in a sense, the model's representation of the word "it" bakes in some of the representation of both "animal" and "tired".

General overview

Residual Connections

Add and Norm

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Matrix form

• Store N input vectors in matrix X

​

​

​

​

• Compute values, queries and keys:�

​

​

​

​

• Combine self-attentions

Matrix form

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

Encoder/Decoder

Encoder/Decoder

• Decoder has two inputs:
• Output from encoder;
• Input text.

Decoder

Output from encoder:

split into two copies

​

Decoder

Logits

​

Training

Training

Training

Training

softmax

Masked Attention

Masked Attention

Masked Attention

Masked Attention

Transformers

• Motivation
• Dot-product self-attention
• Matrix form
• The transformer
• NLP pipeline
• Decoders
• Large Language models

GPT3 (Brown et al. 2020)

• Sequence lengths are 2048 tokens long
• Batch size is 3.2 million tokens.
• 96 transformer layers (some of which implement a sparse�version of attention), each of which processes a word embedding of size 12288.
• 96 heads in the self-attention layers and the value, query, and key dimension is 128.
• 300 billion tokens
• 175 billion parameters