1 of 29

A Dive into NLP

Sergei Averkiev

https://github.com/averkij/mstu-nlp-course

Attention, Transformers!

3

1

2 of 29

Seq2sec tasks

  • Machine translation
  • Language modeling
  • Speech recognition
  • Text summarization
  • ...

Tasks

2

3 of 29

RNN

Bush

​

3

4 of 29

RNN + Attention

Neural Machine Translation by Jointly Learning to Align and Translate

4

5 of 29

RNN + Attention

Image from Neural Machine Translation by Jointly Learning to Align and Translate

5

6 of 29

RNN + Attention

Image from Neural Machine Translation by Jointly Learning to Align and Translate

6

7 of 29

Transformer

Attention Is All You Need

7

8 of 29

Transformer

Image from Attention Is All You Need

8

9 of 29

Transformer

​

Image from The Illustrated Transformer

9

10 of 29

Transformer

Image from The Illustrated Transformer

10

11 of 29

Transformer

​

Image from The Illustrated Transformer

11

12 of 29

Transformer

​

Image from The Illustrated Transformer

12

13 of 29

Transformer

​

Image from The Illustrated Transformer

13

14 of 29

Transformer

Image from Rethinking Self-Attention in Transformer Models

14

15 of 29

Transformer

​

Image from The Illustrated Transformer

15

16 of 29

Transformer

​

Image from The Illustrated Transformer

16

17 of 29

Transformer

​

Image from The Illustrated Transformer

17

18 of 29

Transformer

​

Image from The Illustrated Transformer

18

19 of 29

Transformer

​

Image from The Illustrated Transformer

19

20 of 29

Transformer

​

Image from The Illustrated Transformer

20

21 of 29

Transformer

​

Image from The Illustrated Transformer

21

22 of 29

Transformer

​

Image from The Illustrated Transformer

22

23 of 29

Transformer

​

Image from The Illustrated Transformer

23

24 of 29

Transformer

​

Image from The Illustrated Transformer

24

25 of 29

Transformer

​

Image from The Illustrated Transformer

25

26 of 29

Transformers

​

Давид Дале Предобученные модели для русского языка

26

27 of 29

Transformers

​

Сергей Марков Посмотри в глаза чудовищ: генеративные нейросетевые текстовые модели

27

28 of 29

Huggingface

  • bert-base-multilingual-cased
  • sentence-transformer/LaBSE
  • sberbank-ai/rugpt3small_based_on_gpt2
  • google/mt5-base

Pretrained models

28

29 of 29

https://github.com/averkij/mstu-nlp-course

A Dive into NLP

29