learning for sequences
motivation, fundamentals and application
For SmartGateML
2018
language and its structure
applying machine learning to sequences
goo.gl/ryD4cv
deeplanguageclass.github.io/lab/fairseq-transliteration/
seq2seq lab:
translation + transliteration
The more interesting the message,
the less autocorrect helps you.
What is language?
Are natural languages like programming languages?
Language input/output can be text, images or audio.
It is has structure, but different than the structure of programming languages.
What is the structure of language?
Sequence of sequences?
layers of language
text/image/audio
tokens
morphology
syntax
semantics
dialogue
structures of language
Zipfian
token layer
tree-like
syntactic layer
unstructured
discourse layer
sequences
fundamentals
What are properties of natural language text input and output?
26 x 2 chars
To express most everything the Western world has thought since the Romans.
95 ^ 180
Most of the 9.8...e+355 permutations are invalid.
sparsity
www.tensorflow.org/tutorials/representation/word2vec
“The Pope’s Baby Steps on Syria”
“
“Boy paralyzed after tumor fights back to gain black belt”
“
ambiguity
web.stanford.edu/class/cs224n/lectures/lecture1.pdf
Владимир эту водку не п...
“
More people have been to Russia than I have.
“
More people have been to Russia than I have.
“
strict
http://languagelog.ldc.upenn.edu/nll/?p=39477
“Nice truck attack video”
“
“No forgetto!”
“
evolution
Tehre was a sutdy by Mcirosoft yares ago...
“
Ca n yo u rea d th is?
“
Ca n yo u rea d th is?
M@ch1nes C@n’t.
“
ვატ აბაუტ ზის...
“
ვატ აბაუტ ზის...
ор зис...
“
ვატ აბაუტ ზის...
ор зис...
բոտ նոտ զիս
“
noisy + dirty
www.mrc-cbu.cam.ac.uk/people/matt.davis/cmabridge/
docs.microsoft.com/en-us/typography/develop/word-recognition
wreck a nice beach
“
wreck a nice beach
recognize speech
“
arbitrary
en.wikipedia.org/wiki/Double_articulation
The car crashed because it...
“
The car crashed because it...
“
The car crashed because it had old tyres.
“
The car crashed because it had old tyres.
it == car
“
The car crashed because it was rainy.
“
The car crashed because it was rainy.
it == ?
“
The car crashed because it was rainy.
it == ?
“
hard
en.wikipedia.org/wiki/Winograd_Schema_Challenge
Humans are good at this.
Humans are good at this.
Even baby humans are good at this.
representations
for text sequences
How can we represent text sequences numerically?
word representations
What are the properties?
What are the limitations?
word representations
sub-word
character-level n-grams
supra-word
word-level n-grams
applications
text input/output
Which tasks can we frame as sequence tasks?
sequence input
unsupervised word representations, text classification...
sequence output
article generation (e.g. financial news)
sequence input + output
translation, transliteration, spelling + grammar correction, dialogue systems (e.g. Q&A, "chat bots"), summarisation, style transfer
mixed medium
image captioning
(e.g. CLEVR, @picdescbot)
beyond language
DNA sequences
seq2seq
INPUT : OUTPUT
| NUMBERS | AUDIO | TEXT | TEXT+ |
NUMBERS | regression | |||
LABEL | classification | |||
TEXT | sequence | |||
For which tasks can we use seq2seq?
seq2seq tasks + applications
How do we take as a sequence as input?
How do we generate a sequence?
Generating a sequence
ENCODER + DECODER
github.com/google/seq2seq
Other ways?
To generate a sequence
What if tokens are not 1:1?
How do we represent each token context?
Token context
ATTENTION
What if a token depends on what comes after it?
Token context
BI-DIRECTIONAL
seq2seq lab
The tasks of translation and transliteration tasks contain essentially all the challenges of natural language understanding (NLU) -- ambiguity, agreement, evolution, mixed language... -- which can require full AI.
approach
Find a dataset or generate the dataset from a monolingual corpus
WMT, Wikipedia
Find a seq2seq implementation that is easy to use and productive-strength
Fairseq from Facebook AI Research
Convert from word-level to char-level
Choose parameters, train, test and iterate
1
2
3
4
deeplanguageclass.github.io/lab/fairseq-transliteration/
goo.gl/ryD4cv
Questions?
ML EVN
Yerevan machine learning community
mlevn.org