BERT and GPT-2

Improvements of Transformer

1. Google’s BERT
• Bidirectional Encoder Representations from Transformers
• BERT is an improvement of GPT
• Key technical innovation is bidirectional training of Transformer
2. Open AI’s Generative Pre-Training (GPT)
• Combines supervised and unsupervised learning to improve word vectors
• Performs Generative Pre-Training
1. State-of-the art for following tasks:
• Textual entailment, semantic similarity, reading comprehension, common sense reasoning, linguistic acceptability, multi-task benchmark

Bidirectional Encoder Representations from Transformers (BERT)

Google’s BERT

(Bidirectional Encoder Representations from Transformers)

1. Uses Transformer attention mechanism to learn contextual relations between words
• Transformer includes two mechanisms
1. An Encoder that reads text input
2. A Decoder that produces a prediction for the task
• Since BERT’s goal is to generate a language model, only the encoder mechanism is necessary
2. Uses Bidirectional training
• Deeper sense of context/flow than single-direction
1. Masked LM (MLM) allows bidirectional training

Structure of BERT

The BERT Encoder block implements the base version of the BERT network

​

It is composed of 12 successive transformer layers, each having 12 attention heads

​

The total number of parameters is 110 million.

https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/

blocks/bert-encoder?fbclid=IwAR2rt4ObbHCMvuGsfrNNOnotfW_Dfc-5y3QorqW1nQUbw4XzuwjjCbnQDy0

BERT Architecture

• BERT architecture builds on top of Transformer
• There are two variants:
• BERT Base:
• 12 layers (transformer blocks), 12 attention heads, and 110 million parameters
• BERT Large:
• 24 layers (transformer blocks), 16 attention heads and, 340 million parameters

Model input dimension 512

Input and output vector size

Text Pre-processing

Position Embeddings:

BERT learns and uses positional embeddings to express the position of words

in a sentence. These are added to overcome the limitation of Transformer which, unlike an RNN, is not able to capture “sequence” or “order” information

Segment Embeddings:

BERT can also take sentence pairs as inputs for tasks (Question-Answering).

That’s why it learns a unique embedding for the first and the second sentences to help the model distinguish between them. In the above example, all the tokens marked as EA belong to sentence A (and similarly for EB)

Token Embeddings:

These are the embeddings learned for the specific token from the WordPiece token vocabulary

BERT pretraining

• ULM-FiT (2018): Pre-training ideas, transfer learning in NLP.
• ELMo: Bidirectional training (LSTM)
• Transformer: Although used things from left, but still missing from the right.
• GPT: Use Transformer Decoder half.
• BERT: Switches from Decoder to Encoder, so that it can use both sides in training and invented corresponding training tasks: masked language model

Pre-training BERT

• BERT is pre-trained on two NLP tasks:
1. Masked Language Modeling(For Bidrectionality)
2. Next Sentence Prediction

BERT Pretraining Task 1: masked words

• Approach 1:

Masked LM

BERT

……

……

[MASK]

Linear Multi-class

Classifier

Predicting the masked word

vocabulary size

BERT Pretraining Task 1: masked words

Out of this 15%,

80% are [Mask],

10% random words

10% original words

Before feeding word sequences 15% of words in each sequence replaced with a [MASK] token

Model attempts to predict original value of masked words, based on context provided by non-masked words in sequence

Prediction of the output words requires

• Adding a classifier layer on top of encoder output
• Multiplying output vectors by the embedding matrix, transforming them into the vocabulary dimension.
• Calculating probability of each word in the vocabulary with softmax

Word prediction using masking

BERT Pretraining Task 2: two sentences

• A text dataset of 100,000 sentences has 50,000 training examples or pairs of sentences as training data.
• For 50% of pairs, the second sentence would be the next sentence to the first sentence
• For the remaining 50% of pairs, second sentence would be a random sentence from corpus
• The labels for the first case would

be ‘IsNext’ and ‘NotNext’ for the second case

BERT

BERT Pretraining Task 2: Next Sentence Prediction

Linear Binary

Classifier

yes

[CLS]: the position that outputs classification results

[SEP]: the boundary of two sentences

Approaches 1 and 2 are used at the same time.

BERT

Linear Binary

Classifier

No

[CLS]: the position that outputs classification results

[SEP]: the boundary of two sentences

Approaches 1 and 2 are used at the same time.

BERT Pretraining Task 2: Next Sentence Prediction

Fine-tuning BERT for other specific tasks

SST (Stanford sentiment treebank): 215k phrases with fine-grained sentiment labels in the parse trees of 11k sentences.

MNLI

QQP (Quaro Question Pairs)

Semantic equivalence)

QNLI (NL inference dataset)

STS-B (texture similarity)

MRPC (paraphrase, Microsoft)

RTE (textual entailment)

SWAG (commonsense inference)

SST-2 (sentiment)

CoLA (linguistic acceptability

SQuAD (question and answer)

How to use BERT – Case 1

BERT

[CLS]

w₁

w₂

w₃

Linear Classifier

class

Input: single sentence,

output: class

sentence

Example:

Sentiment analysis (our HW),

Document Classification

Trained from Scratch

Fine-tune

How to use BERT – Case 2

BERT

[CLS]

w₁

w₂

w₃

Input: single sentence,

output: class of each word

sentence

Example: Slot filling

How to use BERT – Case 3

Linear Classifier

BERT

Class

Sentence 1

Sentence 2

Input: two sentences, output: class

Example: Natural Language Inference

Given a “premise”, determining whether a “hypothesis” is T/F/ unknown.

How to use BERT – Case 4

• Extraction-based Question Answering (QA) (E.g. SQuAD)

QA

Model

Document:

Query:

Answer:

17

77

79

How to use BERT – Case 4

BERT

question

document

dot product

Softmax

0.5

0.3

0.2

The answer is “d₂ d₃”.

s = 2, e = 3

How to use BERT – Case 4

BERT

question

document

dot product

Softmax

0.2

0.1

0.7

The answer is “d₂ d₃”.

s = 2, e = 3

Enhanced Representation through Knowledge Integration (ERNIE)

• Designed for Chinese

https://arxiv.org/abs/1904.09223

BERT

ERNIE

Source of image:

https://zhuanlan.zhihu.com/p/59436589

What does BERT learn?

https://arxiv.org/abs/1905.05950

https://openreview.net/pdf?id=SJzSgnRcKX

Multilingual BERT

https://arxiv.org/abs/1904.09077

Trained on 104 languages

Task specific testing data for Chinese

GPT-2 generates human-like output

• Trained to predict next word (40GB of internet text)

• While trained to predict the next word it surprisingly learned basic competence in tasks like MT and QA.

– That's without ever being told that it would be evaluated on those tasks.

GPT-2 in action

not

injure

injure

a

a

human

human

being

being

GPT-2 generated synthetic text

• System Prompt (human-written)
• In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
• Model Completion (machine-written, 10 tries)
• The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.
• Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
• Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.
• Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.
• Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns

Architecture of GPT

1. Text and position will be transformed to a vectors
2. Pass to multi-head self-attention
3. Combining result from step 1 and step 2 and performing a normalization
4. Pass to a fully-connected feed-forward network
5. Combining result from step 3 and 4 and performing a normalization
6. Finally, combing multi-head (total 12 self-attention block) to together for computing vectors.

Byte Pair Encoding (BPE)

Word embedding sometimes is too high level, pure character embedding too low level. For example, if we have learned

old older oldest

We might also wish the computer to infer

smart smarter smartest

​

But at the whole word level, this might not be so direct. Thus the idea is to break the words up into pieces like er, est, and embed frequent fragments of words.

GPT adapts this BPE scheme.

Byte Pair Encoding (BPE)

GPT uses BPE scheme. The subwords are calculated by:

1. Split word to sequence of characters (add </w> char)
2. Joining the highest frequency pattern.
3. Keep doing step 2, until it hits the pre-defined maximum number of sub-words or iterations.

​

Example (5, 2, 6, 3 are number of occurrences)

{‘l o w </w>’: 5, ‘l o w e r </w>’: 2, ‘n e w e s t </w>’: 6, ‘w i d e s t </w>’: 3 }

{‘l o w </w>’: 5, ‘l o w e r </w>’: 2, ‘n e w es t </w>’: 6, ‘w i d es t </w>’: 3 }

{‘l o w </w>’: 5, ‘l o w e r </w>’: 2, ‘n e w est </w>’: 6, ‘w i d est </w>’: 3 } (est freq. 9)

{‘lo w </w>’: 5, ‘lo w e r </w>’: 2, ‘n e w est</w>’: 6, ‘w i d est</w>’: 3 } (lo freq 7)

…..

Masked Self-Attention (to compute more efficiently)

Masked Self-Attention

Masked Self-Attention Calculation

Re-use previous computation results: at any step, only need to results of q, k , v related to the new output word, no need to re-compute the others. Additional computation is linear, instead of quadratic.

Input Formatting for GPT

GPT-2 Application: Translation

GPT-2 Application: Summarization