1

1 Decoding

Domain Adaptation &

Adapting Pretrained Models 1

with wing.nus

Deep transfer RL �for text summarization

Reference: Keneshloo, Yaser & Ramakrishnan, Naren & Reddy, Chandan. (2018). Deep Transfer Reinforcement Learning for Text Summarization.

Text summarization and ROUGE scores

The text summarization task

long text → short text containing the essential information

That’s about all there is to it.

What is a good summary ?

• Syntactically / semantically correct
• Contains all the relevant information while maintaining brevity
• Subjective human judgement is obviously central to properly assess those.

⇒ automatically scoring summaries is really hard.

The ROUGE-n scores

Given a reference summary R and a machine generated summary M,

• Let O be the collection of all overlapping n-grams between M and R (one match suffices for repetitions in R)
• Let |R|, |M|, |O| be the number of words in R, M, O

​

Lin, Chin-Yew. 2004. ROUGE: a Package for Automatic Evaluation of Summaries. In Proceedings of the Workshop on Text Summarization Branches Out (WAS 2004), Barcelona, Spain, July 25 - 26, 2004.

Extractive and abstractive summarization

Extractive

• Pick important spans from the source text and reuse them in the summary.

⊕ If the spans are sentences, the result will be syntactically correct.

⊕ Can’t totally muck up the ROUGE scores.

⊖ It “only” needs to be intelligent enough to pay attention to the right spans.

⊖ Can’t improve on bland, verbose input�

Abstractive

• Using a combination of LSTM/GRU, transformers and intelligent tricks, create an abstract representation of the input text, and output a short sequence that paraphrases the important information.

⊕ A better approach towards truly modelling semantics than the extractive one

⊖ Very hard. ROUGE is too primitive to capture the subtleties of paraphrasing

​

State of the art of summarization models

Basic sequence to sequence

• DNNs : powerful but typically require fixed input and output dimensionality.
• But NLP tasks require processing sequences of varying length�
• Seq2seq is an LSTM-based model, that can handle such sequences

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. “Sequence to sequence learning with neural networks”. In Neural Information Processing Systems, 2014.

Timestep 1 2 3 4 5 6 7 8

Pointer networks

• This model is not specific to summarization but we’ll use the pointer concept
• Seq2seq limitation : trained with a fixed vocabulary size
• In this example we find the convex hull of a set of points on a 2d plane
• Attention mechanism generates a probability distribution over input
• Model was shown successful on convex hull, travelling salesman problem, delaunay triangulation

​

​

Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. "Pointer networks." Advances in neural information processing systems. 2015.

Pointer-generator model

​

• Ability to handle out-of-vocabulary words : important for transfer learning
• Soft switch mechanism σ_j ∈ [0, 1]
• Generate word from�vocabulary distribution p_vocab
• or copy from source, based on attention distribution α_j = Σ_i α_ij

Nallapati, Ramesh, et al. "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond." Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. 2016.

Pointer-generator model

​

• σ_j is learned, at each decoder step j, (step 2 in the exemple figure), based on :
• current input word (y₂)
• decoder state (s₂)
• context vector (c₂)
• Context vector c_j = �is the attention-weighted sum of the encoder states

Nallapati, Ramesh, et al. "Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond." Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. 2016.

Pointer-generator model - Coverage mechanism

​

• Repetition is a common problem of seq2seq models, particularly problematic for summarization, a repetition is “useless” information.
• To reduce repetitions, a “coverage vector” c^t = ∑ a^{[0 to t-1]} is added to the attention mechanism (where t is the decoder timestep)
• Intuitively, it is a distribution that represents the degree of coverage received by the source words so far.
• The loss function is modified to penalize the overlap between each attention distribution and previous coverage : λ ∑_i min(a_i^t, c_i^t)

​

A. See, P. J. Liu, and C. D. Manning. “Get to the point: Summarization with pointer-generator networks”. In ACL, volume 1, pages 1073–1083, 2017

Classical transfer learning

Knowledge distillation

Teacher (trained) model

Student model

• Trained model treated as ground truth and probed (maybe with generated data?) to create training data
• Student model learns from this

Generalized models

• Chimera that attempts many tasks (many outputs)
• Forced common architecture
• Implicit knowledge transfer between tasks

Transfer network layers and fine-tune

Trained neural net

Copy layers

Other model

Transfer network used as baseline in the paper

• Copy full architecture (all LSTM hidden layers weights) from network trained on source dataset S
• Fine-tune all layers without any clamps on target dataset G

Transfer Reinforcement Learning

The Cross-Entropy loss

• Similar to the usual seq2seq formulation — we are building a conditional language model; here, the “language of summarization” given an input text

The above equation reads as follows: “Given some input text, the CE loss for a model is none other than the probability that this model produces all the words in the reference summary.” — Note that this implies teacher forcing.

Exposure bias and inflexibility in Cross-Entropy loss

① The model is directly trained from the ground truth, which follows the source distribution ⇒ poor generalization to other target distributions

② The ground truth is not an absolute, unique correct summary, yet this score takes it very literally ⇒ further unnecessary hindrance to generalization

RL objective

• Generally, RL tries to maximize the expected reward over time.
• Here, luckily, we are in a fully observed setting (so we deal with states and not observations ⇒ FOMDP), and we don’t need fiddling with our reward function: r(state) = ROUGE

The above equation reads as follows: “the objective of the reinforcement learning problem here is to maximize the expected ROUGE score when we sample the summary from the proposed hybrid (pointer-generator / seq2seq) network parametrized by Θ (where Θ = all weights and biases in the model)”

REINFORCE algorithm for policy gradient descent

• To minimize this loss, we ascent its gradient along Θ,
• We use samples of the output of the network to estimate this gradient and to adjust its parameters:
• We add a baseline term b to compensate for the fact that the gradient is sensitive to a constant shift in samples (common instability problem w/ PG)

Ŷ, the greedy output from the model, is subtracted from the trajectory rewards so as to make trajectories that are less rewarding than average less likely, and those more rewarding more likely.

RL objective revisited for transfer learning

It’s simpler than it looks, it’s mostly about sampling from both datasets and striking a balance ζ between the loss on one and on the other.

Final RL loss function

The actual loss function used for RL is another parametrized mix of CE and TRL loss.

Note that this this mix is used only for fine-tuning after normal MLE training on the source dataset.

Paulus, Romain, Caiming Xiong, and Richard Socher. "A deep reinforced model for abstractive summarization." arXiv preprint arXiv:1705.04304 (2017). ← useful paper to understand the one we’re presenting

Experimental results

ROUGE-{1,2,L} F1 scores

Universal Language Model Fine-tuning for Text Classification

Domain adaptation vs Transfer Learning

In domain adaptation, the source and target domains all have the same feature space (but different distributions); in contrast, transfer learning includes cases where the target domain's feature space is different from the source feature space or spaces.

The standard classification setting is a input distribution p(X) and a label distribution p(Y|X). Domain adaptation: when p(X) changes between training and test. Transfer learning: when p(Y|X) changes between training and test.

​

In other words, in Domain Adaptation the input distribution changes but the labels remain the same; in Transfer Learning, the input distributions stays the same, but the labels change.

​

Universal Language Model Fine-tuning (ULMFiT)

1. Universal Language Model Fine-tuning (ULMFiT), a method that can be used to achieve CV-like transfer learning for any task for NLP.
2. discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing, novel techniques to retain previous knowledge and avoid catastrophic forgetting during fine-tuning.
3. significantly outperform the state-of-the-art on six representative text classification datasets, with an error reduction of 18-24% on the majority of datasets.

Universal Language Model Finetuning (ULMFiT)

Pretrains a language model (LM) on a large general-domain corpus and fine-tunes it on the target task. The method is universal in the sense that it meets these practical criteria:

1. works across tasks varying in document size, number, and label type;
2. uses a single architecture and training process;
3. requires no custom feature engineering or preprocessing; and
4. does not require additional in-domain documents or labels.

ULMFit: Three approaches

• General domain LM pre-training
1. Wikitext-103
2. 28,595 preprocessed Wikipedia articles and 103 million words

​

ULMFit: Three approaches (cont)

• Target task LM fine-tuning
1. fine-tune the LM on data of the target task
2. Discriminative fine-tuning (separate learning rates for each layer)
3. x = Last layer learning rates , x/2.6 = lower layer learning rates
4. Slanted triangular learning rates, a modification of TLR (Smith, 2017)
• Short increase of initial learning rate, then long decrease period

ULMFit: Three approaches (cont)

• Target task classifier fine-tuning
1. Two additional learning blocks
2. Recipes - batch normalization, dropout and ReLu activation
3. Gradual Unfreezing - gradually unfreeze the model starting from the last layer
• This is similar to ‘chain-thaw’ (Felbo et al., 2017), except that we add a layer at a time to the set of ‘thawed’ layers, rather than only training a single layer at a time