Lecture 9: Transformers, Pretrained language models, Finetuning, Python tools

Practical Natural Language Processing

Rasika Bhalerao

r.bhalerao@northeastern.edu

Reminders

• Assignment 6 due today
• Paper draft due Mar 30
• Assignment 7 due Apr 6
• Prof. Kenneth Church’s guest talk is today at 10am
• Please look at https://github.com/kwchurch/ACL2022_deepnets_tutorial beforehand

Common missed point in hw 5: lexicalized probabilities

Sentence: “We eat bread with butter.”

Question: Would it help to condition the expansion of vp on the head of the vp? In other words, if we use lexicalized probabilities just for expansion of vp? Why or why not? You do not need to calculate the probabilities for this question, just provide a sentence or two of reasoning for why it would (not) help.

Answer: Yes, because “eat” appears more often in the correct sentence structure: we eat 2 things together more often than we eat in a specific way (like “with gusto”).�VP -> V NP is more common than VP -> V NP PP for the verb “eat”, and this is not the case for most other verbs.

Resources for today’s material

• Paper and slides by BERT authors
• Transformers & Pretrained Language Models (J&M chapter 10)
• Fine-tuning (J&M chapter 11)
• Pytorch intro notebook (also linked in Pytorch slide)
• BERT intro notebook (also linked in hw slide)

BTW this is one of their slides (2018)

Pretraining motivation from the textbook

• “estimates of the vocabulary size of young adult speakers of American English range from 30,000 to 100,000”
• To reach that vocab size, children have to learn 7 - 10 new words every day
• Schools do not do this
• Even reading books doesn’t introduce that many new words
• How does it work? Distributional Hypothesis (and other context)
• We learn meanings from pure context, and then use those words in specific tasks later
• Pretraining: “learning some sort of representation of meaning for words or sentences by processing very large amounts of text”

Review: Encoder-Decoder Models

• Also called Sequence to Sequence Models
• Two RNNs: one that “encodes” by taking input from the first sequence, and the other that “decodes” by taking the last hidden state from the encoder as input and generating a sequence
• Used for any task where the input and output are sequences
• Machine translation
• Summarization
• Question answering
• Dialogue

https://d2l.ai/chapter_recurrent-modern/seq2seq.html

Review: Attention in Encoder-Decoder Models

• Pass all the hidden states from the encoder RNN to the decoder RNN (not just the last hidden state)
• The decoder uses attention to figure out which encoder hidden states are relevant at each step
• Attention is just another learned parameter, a number for each encoder hidden layer for each decoding step, softmaxed to be between 0 and 1
• At each decoding step, concatenate the current hidden layer with a combination of the attention-multiplied encoder hidden states, and pass this vector into a feedforward layer to get the output word at this step

https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/

Self-attention

Self-attention = for each word, how important is each other word (in the context)?

Multiple attention heads = learn self-attention multiple times

Review: transformer = self-attention + feedforward

Positional embeddings

It’s another learned embedding

This one is to encode the relative location (encodes “position 3” instead of the word)

Refresher on BERT

• Masked Language Model: select a random 15% of the tokens
• Keep 10%, replace 10%, mask 80% (objective = predict the masked ones)
• BERT Base is a stack of 12 encoder transformers
• “Attention heads” = the number of times it learns each attention (combines them into one output)

BERT’s other objective function: next sentence prediction

To help it learn relationships between sentences (not just subwords)

Transfer learning

1. Pretrain a BERT-like model using a large corpus
1. (Huggingface already trained BERT for you)
2. Input is length n (number of tokens)
3. Output is also length n
2. Add layer(s) for performing your task
• Input is length n (output of BERT)
3. Continue the training with task-specific data
• It benefits from having learned the words from the large corpus

Finetune it to do any task you want!

Bias in Language Models

• Language models learn implicit biases from the text they are trained on
• Crowdsourced Stereotype Pairs (CrowS-Pairs): a dataset to quantify bias in masked language models
• 1508 crowdsourced (and validated) pairs of sentences like this:
• Shane lifted the lumber and swung his ax.
• Jenny lifted the lumber and swung her ax.
• For each word in common between the sentences, mask each word and have the model predict it, and get the probability it gives to the correct word

Bias in Language Models

More language model harms are in the textbook ch. 10

• The data that popular language models were trained on includes things like banned Reddit communities
• These models can generate toxic language (even with harmless prompts)
• These models can amplify the biases found in their training set, making them worse than in real life (like we saw with word embeddings)
• The internet tends to amplify misinfo, radicalization, extremist movements, … (these models are trained on the internet)
• The training data also include personal info

Common Python Toolkits

• For dealing with Tensors
• Pytorch
• Tensorflow
• NLP Toolkits
• Huggingface
• AllenNLP
• ParlAI
• Text preprocessing
• SpaCy
• NLTK

Pytorch

• Open source Python package for machine learning
• Easy to use with GPU (or CPU)
• Computation is done with tensors
• You can think of it as Numpy with extra capabilities
• Pytorch intro notebook

Assignment 7

• Assignment instructions
• BERT intro notebook