1 of 14

Copy mechanisms in Neural Nets

2 of 14

In theory

3 of 14

4 of 14

Pointer Networks

Oriol Vinyals, Meire Fortunato, Navdeep Jaitly. 2015

Goal: Learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. (Eg: sorting variable sized sequences, combinatorial optimization problems)

Issue: The number of target classes in each step of the output depends on the length of the input, which is variable.

Best idea so far: sequence to sequence.

Solution in the paper: Using the attention weights as the decoder’s proposed distribution.

5 of 14

Naïve solution to unknown words

Let the network generate UNK token during training. For metric evaluation, replace the <unk> token by the token with maximum attention weight at this time step.

Issue: not differentiable.

Use placeholders.

Issue: Done by hand, not possible in every setting

6 of 14

Get To The Point: Summarization with Pointer-Generator Networks

Abigail See, Peter J. Liu, Christopher D. Manning. 2017

Use a binary switch !

Issue: Still not differentiable.

Solution: use a soft switch in [0,1] and mix both distribution

Pointing the Unknown Words. Gulcehre et al. - 2016

Pointer Sentinel Mixture Models. Merity et al. - 2016

Incorporating Copying Mechanism in Sequence-to-Sequence Learning. Gu et al. - 2016

7 of 14

8 of 14

In practice

9 of 14

Separating copy and generation

We consider that if a token is in both the input and the target, it then was the result of a copy.

​

Using something akin to teacher forcing, we then compute the loss on either the attention distribution or the vocabulary distribution, following if the true token should be the result of a copy or not.

​

Issues: teacher forcing, strong assumption of copy

We can remove teacher forcing and still keep the copy assumption by computing the loss on the concatenation of the distributions. The final distribution will have shape V + X where V is the known vocabulary size and X the input sentence size.

​

Issues: strong assumption of copy, need to keep custom target vector to account for the position of the tokens in the sentence (Until now, each token had a definitive id. Now, because of attention, unknown token have for id their position in the sentence).

10 of 14

Extend attention weights to full distribution 1/2

Issue: not the same shape (most tokens are lacking), some token in attn_weights might even appear several times !

​

​

Solution: OneHotEncoding.

11 of 14

Extend attention weights to full distribution 2/2

12 of 14

Dealing with unknown words

Issue: How can we copy unknown words ? They won’t be in the OneHotEncoding.

​

Naïve solution: Use BPE or char-tokenization and use the copy mechanism to copy entire sequences of tokens. It still outperforms classic method using BPE without copy.

Solution: Keep a censored version of (input, target) as well as an uncensored version.

  • The censored version replace all token below a certain frequency threshold by <unk>, while the uncensored keeps all token.
  • The encoder and decoder are fed the censored version (and thus have to learn adequate representation for <unk>)
  • A custom loss module takes the attention weights as well as the distribution on the vocabulary and is fed the uncensored version. The OneHotEncoder knows all tokens (even rare ones) and is able to extend the weights correctly

​

  • After training, we can be less efficient and deal with truely unknown words using loops an’ stuff

13 of 14

Going further

14 of 14

Decoder needs to encode <unk>

Issue: We need to feed the previous token to the decoder. What if it’s <unk> ?

​

Naïve solution: Feed <unk>

Tur-fu Solution: use conditionnal embedding, like ELMo.

Problem: no one uses ELMo in decoding

  • we would have to recompute the encoding at each decoding step.
  • the target is uncomplete.

Solutions ?

  • Transformer does this and seems to work well.
  • Share embedding with encoder