1 of 47

Transformers

Dr. Dinesh Kumar Vishwakarma

PROFESSOR, DEPARTMENT OF INFORMATION TECHNOLOGY

DELHI TECHNOLOGICAL UNIVERSITY, DELHI.

Webpage: http://www.dtu.ac.in/Web/Departments/InformationTechnology/faculty/dkvishwakarma.php

Email: dinesh@dtu.ac.in

2 of 47

Introduction

2

11/11/2026

Encoder–decoder architecture

An encoder reads the input and a decoder writes the output. Introduced in 2017 by Vaswani et al.

Fixes RNN and LSTM limits

Built to overcome the slow, word-by-word processing of RNNs and LSTMs in tasks like translation.

Self-attention at the core

Every word looks at every other word to work out what it means in this sentence.

Whole sequence at once

Reads all words in parallel, not one by one, so it trains much faster than an RNN.

In plain terms: a Transformer is like a translator who reads the whole sentence first, checking how each word relates to the others, before writing anything.

The Transformer is the architecture behind today's large language models such as GPT and BERT. It was introduced in the 2017 paper 'Attention Is All You Need' by Vaswani et al.

Like earlier translation models, it has an encoder (reads and understands the input) and a decoder (generates the output).

What is new: it drops the step-by-step recurrence of RNNs and LSTMs. Instead it uses self-attention, where every word directly compares itself with every other word in the sentence.

Because nothing has to wait for the previous word, the whole sentence is processed in parallel. This makes training much faster on GPUs and lets the model handle long-range relationships.

Roadmap for the next slides: embeddings (how words become numbers), attention (how context changes a word's meaning), and multi-head attention (looking at a sentence in several ways at once).

3 of 47

Overview of Transformer Model

3

11/11/2026

How to read this figure

1 Embedding

Each word becomes a list of numbers, like a point on a map.

2 Attention

Each word looks at the other words and shifts toward the meaning they suggest.

3 Stack and predict

Attention + feedforward repeat a few times; Softmax picks the output word.

In plain terms: a Transformer turns words into numbers, lets every word look at every other word, and repeats this a few times before producing the output.

4 of 47

Embedding

4

11/11/2026

An embedding turns each word into numbers: similar meanings get similar numbers.

In plain terms: embedding is the bridge between language and maths. Once words are numbers, the model can measure how close two meanings are.

5 of 47

Embedding…

5

11/11/2026

In plain terms: each word is placed by its coordinates. Fruits sit together and vehicles sit together, so apple belongs at (5, 5), next to banana and strawberry.

6 of 47

Embedding…

6

11/11/2026

Where do we put the apple?

Problem

In plain terms: apple has two meanings. One fixed point cannot sit next to both banana and Android, so a single embedding is not enough. Attention fixes this.

7 of 47

Attention

7

11/11/2026

In plain terms: Attention captures the context. Next to orange, apple moves toward fruit (8, 8.5); next to phone, it moves toward technology (4, 5).

8 of 47

What about the other words?

8

11/11/2026

Attention

In plain terms: every word can pull on apple, but how hard it pulls depends on how related it is. Orange pulls hard; please, buy, am, and sit far away and barely pull.

9 of 47

Multi-Head Attention

Is one embedding enough? Each plot puts the same three words in different places.

9

11/11/2026

Which one is good? We cannot know in advance which arrangement is most useful.
Learning many from scratch is slow, so we build new ones by modifying the existing one.

In plain terms: multi-head attention looks at the words through several embeddings at once, each one a cheap transformation of the original.

10 of 47

Linear Transformation

Stretch, rotate or skew one embedding to get a new one

10

11/11/2026

Okay

The original embedding

Bad

Skewed: related words pulled apart

Good

Apple moves closer to orange

In plain terms: a linear transformation is a matrix multiplication. It reshapes the whole map at once, and some new maps place related words better than others.

11 of 47

Score

Each new embedding gets a score: higher when related words sit close

11

11/11/2026

In plain terms: the score tells the model which embedding to trust. The Good map (score 4) brings apple and orange together, so it counts 40 times more than the Bad one (0.1).

12 of 47

Multi-head Attention

12

11/11/2026

In plain terms: each head reads the sentence in its own way (one may track meaning, another grammar). Concat joins all views into one embedding.

Input: Q (query), K (key), V (value) for every word

Head 1

Linear

Scaled dot-product attention

Head 2

Linear

Scaled dot-product attention

Head h

Linear

Scaled dot-product attention

…

Concat: place all head outputs side by side

Linear: mix them back to one embedding size

Output: one embedding per word, now with context

How it works

1 Q, K, V: each word asks (Query), advertises (Key) and carries content (Value)

2 Linear: each head projects Q, K, V its own way, giving one new view of the words

3 Attention: each head scores every Query against every Key and mixes the Values: softmax(QKᵀ/√dₖ)·V

4 Concat + Linear: join the h views and blend them into one result

Multi-head attention is the core building block of the Transformer.

Step 1, Q, K, V: every word is turned into three vectors. The Query is what the word is looking for, the Key is what it offers to others, and the Value is the information it passes on. In self-attention all three come from the same sentence.

Step 2, Linear: each head has its own learned weight matrices, so it projects Q, K and V differently. This is the 'new embedding from an existing one' idea from the earlier slides: each head is one transformed embedding.

Step 3, Scaled dot-product attention: inside each head, Q·K gives a score for every pair of words, we divide by √dₖ so the numbers do not get too large, softmax turns the scores into weights that add up to 1, and those weights mix the Values. Example: in 'please buy an apple and an orange', the word apple gives a high weight to orange.

Why several heads? One head might focus on meaning (apple ↔ orange), another on grammar (buy ↔ apple). In the original paper h = 8 heads, each of size 64, so 8 × 64 = 512, the model size.

Step 4, Concat + Linear: the outputs of all heads are placed side by side and a final linear layer blends them into one vector per word. The result is an embedding that carries context from the whole sentence.

Check for understanding: ask the class what would be lost if we used only one head.

13 of 47

Why we need?

13

11/11/2026

RNN

Reads one word at a time

Problems

Vanishing gradient: forgets early words in long sentences
Sequential: word 2 must wait for word 1

LSTM

Better memory, same order

Problems

Slow to train
Still sequential: reads and writes one word at a time
Hard to parallelise; no explicit model of long- and short-range links

Transformer

Reads the whole sentence at once

How it fixes this

Parallel: all words processed together, so training is fast
Attention: links any two words directly, near or far

In plain terms: RNNs and LSTMs read a sentence the way we read aloud, one word after another. A Transformer sees every word at the same time.

Why did we need a new architecture? Because the older sequence models, RNN and LSTM, share two weaknesses.

RNN: information from early words fades as it is passed along step by step (the vanishing gradient problem), so long sentences are hard. And each step depends on the previous one, so it cannot be parallelised.

LSTM: adds gates to remember things longer, which helps memory, but it is still sequential. Training is slow on modern GPUs because the words cannot be processed at the same time, and distant words are still only connected through a long chain of steps.

Transformer: drops recurrence completely. All words are processed in parallel, and attention connects any word to any other word in a single step, whether they are neighbours or 50 words apart.

Analogy for students: an RNN is like passing a message along a line of people (it gets distorted); a Transformer is like everyone in a room being able to talk to everyone else directly.

14 of 47

Applications

Machine Translation: English sentence in, Hindi sentence out
Document Summarization: a 10-page report shrunk to a few lines
Document Generation: writing new text, as chatbots do
Named Entity Recognition: finding names, places and dates in text
Biological sequence analysis: reading DNA and protein sequences

14

11/11/2026

In plain terms: if the data is a sequence where order and context matter (words, DNA, code), a Transformer can learn from it.

15 of 47

Recap: the RNN encoder–decoder

Encoder (an RNN) reads the input one word at a time and squeezes it into one context vector.
Decoder (another RNN) turns that vector into the output, again one word at a time.
Problem: one fixed-size vector must hold the whole sentence, so long sentences lose detail.

15

11/11/2026

16 of 47

Recap: RNN with attention

Attention (Bahdanau 2014, Luong 2015): the encoder passes all its hidden states, not just the last one.
At each output step the decoder scores every input word and focuses on the best match.
Heatmap: for "European Economic Area", attention follows the reversed French word order.
Still sequential: words are processed one after another. The Transformer drops the RNN and keeps only attention.

16

11/11/2026

17 of 47

Transformer

The Transformer was proposed in the paper Attention is All You Need.
A TensorFlow implementation of it is available as a part of the Tensor2Tensor package.
Harvard’s NLP group created a guide annotating the paper with PyTorch implementation.

17

11/11/2026

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017).

18 of 47

The Transformer architecture

18

11/11/2026

Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." Advances in neural information processing systems 30 (2017).

19 of 47

From input text to output word

19

11/11/2026

20 of 47

A High Level Look

In a machine translation application, transformer would take a sentence in one language, and output its translation in another. A black box manner.

20

11/11/2026

21 of 47

A High Level Look…

Transformer as:

Encoder and Decoder

21

11/11/2026

22 of 47

A High Level Look…

The encoding component is a stack of encoders (the paper stacks six of them on top of each other – there’s nothing magical about the number six, one can definitely experiment with other arrangements).
The decoding component is a stack of decoders of the same number.

22

11/11/2026

23 of 47

Conclusion

Attention helps in improving neural machine translation.
The Transformer – a model that uses attention to boost the speed with which these models can be trained.

23

11/11/2026

24 of 47

Self Attention Mechanism

A self-attention module takes in n inputs and returns n outputs.
The self-attention mechanism allows the inputs to interact with each other (“self”) and find out who they should pay more attention to (“attention”).
The outputs are aggregates of these interactions and attention scores.

24

11/11/2026

25 of 47

Step involved in Self Attention

Prepare inputs
Initialise weights
Derive key, query and value
Calculate attention scores for Input 1
Calculate softmax
Multiply scores with values
Sum weighted values to get Output 1
Repeat steps 4–7 for Input 2 & Input 3

25

11/11/2026

Consider 3 inputs

26 of 47

Step involved in Self Attention…

Step1: Consider 3 inputs with dimension 4

26

11/11/2026

Input 1: [1, 0, 1, 0] �Input 2: [0, 2, 0, 2]�Input 3: [1, 1, 1, 1]

27 of 47

Step involved in Self Attention…

Step 2: Initialise weights

Every input must have three representations & these are called key (orange), query (red), and value (purple).
For this example, let’s take that we want these representations to have a dimension of 3.
Because every input has a dimension of 4, each set of the weights must have a shape of 4×3.

27

11/11/2026

28 of 47

Step involved in Self Attention…

Step 2: Initialise weights

To obtain these representations, every input (green) is multiplied with a set of weights for keys, a set of weights for querys (incorrect spelling), and a set of weights for values.
In our example, we initialise the three sets of weights as follows:

28

11/11/2026

[[0, 0, 1],�[1, 1, 0],�[0, 1, 0],�[1, 1, 0]]

Weights for key:

Weights for query:

[[1, 0, 1],�[1, 0, 0],�[0, 0, 1],�[0, 1, 1]]

Weights for value:

[[0, 2, 0],�[0, 3, 0],�[1, 0, 3],�[1, 1, 0]]

29 of 47

Step involved in Self Attention…

Step 3: Derive key, query and value

Now that we have the three sets of weights, let’s obtain the key, query and value representations for every input.
Key representation for Inputs:

29

11/11/2026

[0, 0, 1]�[1, 0, 1, 0] x [1, 1, 0] = [0, 1, 1]� [0, 1, 0]� [1, 1, 0]

[0, 0, 1]�[0, 2, 0, 2] x [1, 1, 0] = [4, 4, 0]� [0, 1, 0]� [1, 1, 0]

Input-1

Input-2

Input-3

[0, 0, 1]�[1, 1, 1, 1] x [1, 1, 0] = [2, 3, 1]� [0, 1, 0]� [1, 1, 0]

[0, 0, 1]�[1, 0, 1, 0] [1, 1, 0] [0, 1, 1]�[0, 2, 0, 2] x [0, 1, 0] = [4, 4, 0]�[1, 1, 1, 1] [1, 1, 0] [2, 3, 1]

A faster way is to vectorise the above operations:

30 of 47

Step involved in Self Attention…

Step 3: Derive key, query and value

30

11/11/2026

Derive key representations from each input

31 of 47

Step involved in Self Attention…

Step 3: Derive key, query and value

Let’s do the same to obtain the value representations for every input:

31

11/11/2026

[0, 2, 0]�[1, 0, 1, 0] [0, 3, 0] [1, 2, 3] �[0, 2, 0, 2] x [1, 0, 3] = [2, 8, 0]�[1, 1, 1, 1] [1, 1, 0] [2, 6, 3]

Derive value representations from each input

32 of 47

Step involved in Self Attention…

Step 3: Derive key, query and value

Let’s do the same to obtain the Query representations for every input:

32

11/11/2026

[1, 0, 1]�[1, 0, 1, 0] [1, 0, 0] [1, 0, 2]�[0, 2, 0, 2] x [0, 0, 1] = [2, 2, 2]�[1, 1, 1, 1] [0, 1, 1] [2, 1, 3]

Derive query representations from each input

33 of 47

Step involved in Self Attention…

Step 4: Attention Score for Input 1

33

11/11/2026

To obtain attention scores, we take dot product between Input 1’s query (red) with all keys (orange), including itself.

[0, 4, 2]�[1, 0, 2] x [1, 4, 3] = [2, 4, 4]�[1, 0, 1]

34 of 47

Step involved in Self Attention…

Step 4: Attention Score for Input 2

Step 4: Attention Score for Input 3

34

11/11/2026

[0, 4, 2]�[2, 2, 2] x [1, 4, 3] = [4, 8, 12]�[1, 0, 1]

[0, 4, 2]� [2, 1, 3] x [1, 4, 3] = [4, 12, 10]�[1, 0, 1]

35 of 47

Step involved in Self Attention…

Step 5: Calculate Softmax Query 1

35

11/11/2026

softmax([2, 4, 4]) = [0.0, 0.5, 0.5]

36 of 47

Step involved in Self Attention…

Step 6: Multiply scores with values for Query 1

36

11/11/2026

The softmaxed attention scores for each input (blue) is multiplied by its corresponding value (purple). This results in 3 alignment vectors (yellow). In this tutorial, we’ll refer to them as weighted values.

1: 0.0 * [1, 2, 3] = [0.0, 0.0, 0.0]�2: 0.5 * [2, 8, 0] = [1.0, 4.0, 0.0]�3: 0.5 * [2, 6, 3] = [1.0, 3.0, 1.5]

37 of 47

Step involved in Self Attention…

Step 7: Sum weighted values to get Output 1

37

11/11/2026

The resulting vector [2.0, 7.0, 1.5] (dark green) is Output 1, which is based on the query representation from Input 1 interacting with all other keys, including itself.

38 of 47

Step involved in Self Attention…

Step 8: Repeat for Input 2 & Input 3

38

11/11/2026

We repeat Steps 4 to 7 for Output 2 and Output 3

39 of 47

Example

Consider the sentence: “I am going to university” Assume that the Transformer model represents each word with a 2-dimensional embedding as follows:

We will compute self-attention for the word “going”, using simplified Query (Q), Key (K), and Value (V) matrices (assumed) defined as follows:

39

11/11/2026

Word	Embedding (vector)
I	[1, 0]
am	[0, 1]
going	[1, 1]
to	[0, 2]
university	[2, 1]

40 of 47

Example…

Query, key and value are computed using:

where X is the matrix of all word embeddings.

Compute the attention output for the word “going” using the scaled dot-product attention formula:

Q, K and V will be same because metrices considered are same.

40

11/11/2026

41 of 47

Example…

41

11/11/2026

Word	Q (same as K, V)
I	[1, 0]
am	[0, 1]
going	[1, 1]
to	[0, 2]
university	[2, 1]

Word	Kᵢ	Dot product (Qgoing · Kᵢ)
I	[1, 0]	(1.1 + 1.0 = 1)
am	[0, 1]	(1.0 + 1.1 = 1)
going	[1, 1]	(1.1 + 1.1 = 2)
to	[0, 2]	(1.0 + 1.2 = 2)
university	[2, 1]	(1.2 + 1.1 = 3)

S=[1,1,2,2,3]

42 of 47

Example …

42

11/11/2026

Word
I	0.707	2.028
am	0.707	2.028
going	1.414	4.113
to	1.414	4.113
university	2.121	8.340

Word
I	0.0984
am	0.0984
going	0.1995
to	0.1995
university	0.4042

43 of 47

Example …

Step 6 : Compute final attention output for “going”

43

11/11/2026

44 of 47

Example…

Final Attention Outputs

44

11/11/2026

Word	Output (Ox, Oy)	Interpretation
I	(1.21, 0.90)	Learns a mix of itself and “university” — subject linked to goal.
am	(0.63, 1.28)	Emphasizes grammatical information (high y-axis).
going	(1.11, 1.10)	Balanced between action and context.
to	(0.44, 1.53)	Grammatical connector (high y weight).
university	(1.53, 1.00)	Semantically rich — conceptually dominant.

45 of 47

Summary

The dimension of query and key must always be the same because of the dot product score function.
However, the dimension of value may be different from query and key. The resulting output will consequently follow the dimension of value.

45

11/11/2026

46 of 47

Reference

46

11/11/2026

47 of 47

Thank You�Contact: dinesh@dtu.ac.in �Mobile: +91-9971339840

47

11/11/2026