Machine Translation: Introduction

​

​

​

​

Slides from Dan Jurafsky

Human Language Technologies

Dipartimento di Informatica

Università di Pisa

Outline

• Introduction and a little history
• Language Similarities and Divergences
• Three classic MT Approaches
• Transfer
• Interlingua
• Direct
• Modern Statistical MT
• Neural MT
• Evaluation

What is MT?

Translating a text from one language (the source language) to another (the target language) automatically

Criticism of Altavista, Umberto Eco. 2007

Google 2007

Babelfish 2004

Google Translate

• The translation

http://translate.google.com/translate?hl=en&sl=es&tl=en&u=http%3A%2F%2Fwww.cocinadominicana.com%2Facompanamientos-ensaladas-pastelones%2F1907-tostones.html

• The original recipe for tostones

http://www.cocinadominicana.com/acompanamientos-ensaladas-pastelones/1907-tostones.html

Tostones are green plantain (or male) slices, fried, flattened, and then fried again.

Los tostones son rodajas de plátanos verdes (o machos), fritas, aplanadas y luego fritas nuevamente.

Google Translate

French recipe

http://translate.google.com/translate?hl=en&sl=fr&tl=en&u=https%3A%2F%2Fcuisine.journaldesfemmes.fr%2Frecette%2F1000577-tarte-tatin

Machine Translation

The Story of the Stone (“The Dream of the Red Chamber”)

• Cao Xueqin 1792

Chinese gloss: Dai-yu alone on bed top think-of-with-gratitude Bao-chai again listen to window outside bamboo tip plantain leaf of on-top rain sound sigh drop clear cold penetrate curtain not feeling again fall down tears come

Hawkes translation: As she lay there alone, Dai-yu’s thoughts turned to Bao-chai. Then she listened to the insistent rustle of the rain on the bamboos and plantains outside her window. The coldness penetrated the curtains of her bed. Almost without noticing it she had begun to cry.

​

Machine Translation

• Issues:
• Sentence segmentation: 4 English sentences to 1 Chinese
• Grammatical differences
• Chinese rarely marks tense:
• As, turned to, had begun
• tou ⇨ penetrated
• No pronouns or articles in Chinese
• Stylistic and cultural differences
• Bamboo tip plantain leaf ⇨ bamboos and plantains
• Ma ‘curtain’ ⇨ curtains of her bed
• Rain sound sigh drop ⇨ insistent rustle of the rain

Alignment in Machine Translation

Not just literature

Hansards: Canadian parliamentary proceedings

What is MT already good enough for?

• Tasks for which a rough translation is fine
• Extracting information (finding recipes!)
• Web pages
• email
• Tasks for which MT can be post-edited
• MT as first pass
• Computer-aided human translation
• Tasks in sublanguage domains where high-quality MT is possible
• FAHQT (Fully Automatic High Quality Translation)

​

What is MT less good for?

• Really hard stuff
• Literature
• Natural spoken speech (meetings, court reporting)

​

• Really important stuff
• Medical translation in hospitals
• Emergency phone calls

MT Early History

1946 Booth and Weaver discuss MT at Rockefeller Foundation in New York

1947-48 idea of dictionary-based direct translation

1949 Weaver memorandum popularized idea

1952 all 18 MT researchers in world meet at MIT

1954 IBM/Georgetown Demo Russian-English MT

1955-65 lots of labs take up MT

IBM?Georhetown Thinking Machine

Warren Weaver memo

• http://www.stanford.edu/class/linguist289/weaver001.pdf
• “There are certain invariant properties which are… to some statistically useful degree, common to all languages.”
• On March 4, 1947, “having considerable exposure to computer design problems during the war, and being aware of the speed, capacity, and logical flexibility possible in modern electronic computers”, Weaver suggested that computers to be used for translation

Early Research

• MT research began in the early 1950s on machines less powerful than today’s calculators
• Concurrent with foundational work on automata, formal languages, probabilities, and information theory
• MT heavily funded by military, but basically just simple rule-based systems doing word substitution
• Human language is more complicated than that, and varies more across languages!
• Little understanding of natural language syntax, semantics, pragmatics
• Problem soon appeared intractable

History of MT: Pessimism

1959/1960: Bar-Hillel “Report on the state of MT in US and GB”

• Argued FAHQT too hard (semantic ambiguity, etc.)
• Should work on semi-automatic instead of automatic
• His argument:�Little John was looking for his toy box. Finally, he found it. The box was in the pen. John was very happy.
• Only human knowledge lets us know that ‘playpens’ are bigger than boxes, but ‘writing pens’ are smaller
• His claim: we would have to encode all of human knowledge

​

History of MT: Pessimism

The ALPAC report

• Headed by John R. Pierce of Bell Labs
• Conclusions:
• Supply of human translators exceeds demand
• All the Soviet literature is already being translated
• MT has been a failure: all current MT work had to be post-edited
• Sponsored evaluations which showed that intelligibility and informativeness was worse than human translations
• Results:
• MT research suffered
• Funding loss
• Number of research labs declined
• Association for Machine Translation and Computational Linguistics dropped MT from its name

​

​

History of MT: Revival

1976 Meteo, weather forecasts from English to French

Systran (Babelfish) in use for 50 years

1970’s

European focus in MT; mainly ignored in US

1980’s

ideas of using early AI techniques in MT (KBMT, CMU)

Focus on “interlingua” systems, especially in Japan

1990’s

Commercial MT systems

Statistical MT

Speech-to-speech translation

2000’s

Statistical MT takes off

Google Translate

2015

Neural MT takes off

Language Similarities and Divergences

• Some aspects of human language are universal or near-universal, others diverge greatly
• Typology: the study of systematic cross-linguistic similarities and differences
• What are the dimensions along with human languages vary?

Morphology

Morpheme

• Minimal meaningful unit of language

Word = Morpheme+Morpheme+Morpheme+…

Stems: root plus derivational morphemes

• hope+ing 🡪 hoping hop 🡪 hopping

Lemma: also called base form, root, lexeme

• hoping 🡪 hope hopping 🡪 hop

Affixes

• Prefixes: Antidisestablishmentarianism
• Suffixes: Antidisestablishmentarianism
• Infixes: hingi (borrow) – humingi (borrower) in Tagalog
• Circumfixes: sagen (say) – gesagt (said) in German

Morphological Variation

Isolating languages

• Cantonese, Vietnamese: each word generally has one morpheme

Agglutinative languages

• Turkish: morphemes have clean boundaries

Polysynthetic languages

• Siberian Yupik (‘Eskimo’): single word may have many morphemes

Fusion languages

• Russian: single affix may have many morphemes

​

vs

​

​

​

vs

One word one phrase

• Turkish

uygarlaştıramadıklarımızdanmışsınızcasına

uygar+laş+tır+ama+dık+lar+ımız+dan+mış+sınız+casına

Behaving as if you are among those whom we could not cause to become civilized

​

• German

Donaudampfschiffahrtselektrizitätenhauptbetriebswerkbauunterbeamtengesellschaft

Danube steam shipping electricity main plant construction subordinate company

​

Donaudampfschifffahrtsgesellschaftskapitän

Donau+dampf+Schiff+Fahrt+s+gesellschafts+kapitän

Danube steam shipping company captain

​

Index of synthesis

Slide from Holger Diessel

isolating

synthetic

Vietnamese

English

Russian

Oneida

Isolating language

Vietnamese

Khi tôi đến nhà bạn, chúng tôi bắt đầu làm bài.

When I come house friend PL I begin do lesson

​

When I came to my friend’s house, we began to do lessons.

​

Cantonese

keui wa chyuhn gwok jeui daaih gaan nguk haih li gaan

he say entire country most big building house is this building

Slide from Holger Diessel

Synthetic language

Kirundi

Y-a-bi-gur-i-ye abâna

CL1-PST-CL8.them-buy-APPL-ASP CL2.children

​

He bought them for the children.

Slide from Holger Diessel

Polysynthetic language

Noun-incorporation (cf. fox-hunting, bird-watching)

Mohawk

​

a. r-ukwe’t-í:yo

he-person-nice

He is a nice person

​

b. wa-hi-‘sereth-óhare-‘se

PST-he/me-car-wash-for

He car-wash for me (= He washed my car)

​

c. kvtsyu v-kuwa-nya’t-ó:’ase

fish FUT-they/her-throat-slit

They will throat-slit a fish

Slide from Holger Diessel

Index of fusion

agglutinative

fusional

Swahili

Russian

Oneida

Slide from Holger Diessel

Agglutinative language

Words are formed by stringing together morphemes without changing them

Turkish

SG PL

​

Nominative adam adam-lar

Accusative adam-K adam-lar-K

Genitive adam-Kn adam-lar-Kn

Dative adam-a adam-lar-a

Locative adam-da adam-lar-da

Ablative adam-dan adam-lar-dan

​

Slide from Holger Diessel

Fusional language

A single inflectional morpheme to denote multiple grammatical feature, e.g. both tense and person

Russian

​

SG PL SG PL

​

Nominative stol stol-y lip-a lip-y

Accusative stol stol-y lip-u lip-y

Genitive stol-a stol-ov lip-y lip

Dative stol-u stol-am lip-e lip-am

Instrumental stol-om stol-ami lip-oj lip-ami

Prepositional stol-e stol-ax lip-e lip-ax

​

Slide from Holger Diessel

Word Order

• SVO (Subject-Verb-Object) languages
• English, German, French, Mandarin
• SOV Languages
• Japanese, Hindi

​

​

​

• VSO languages
• Irish, Classical Arabic
• SVO languages generally use prepositions: to Yuriko
• VSO languages generally use postpositions: Yuriko ni

Segmentation Variation

• Not every writing system has word boundaries marked
• Chinese, Japanese, Thai, Vietnamese
• Some languages tend to have sentences that are quite long, closer to English paragraphs than sentences:
• Modern Standard Arabic, Chinese

​

Inferential Load: cold vs. hot langs

• Some ‘cold’ languages require the hearer to do more “figuring out” of who the various actors in the various events are:
• Japanese, Chinese
• Other ‘hot’ languages are pretty explicit about saying who did what to whom:
• English

Inferential Load (2)

All noun phrases in

blue do not appear

in the Chinese text …

But they are needed

for a good translation

Lexical Divergences

• Word to phrases:
• English “computer science” = French “informatique”
• POS divergences
• English: ‘she likes/VERB to sing’
• German: Sie singt gerne/ADV
• English: ‘I’m hungry/ADJ
• Spamish: ‘tengo hambre/NOUN

Lexical Divergences: Specificity

Grammatical constraints

• English has gender on pronouns, Mandarin not.
• So translating “3rd person” from Chinese to English, need to figure out gender of the person!
• Similarly from English “they” to French “ils/elles”

Semantic constraints

• English: ‘brother’
• Mandarin: ‘gege’ (older) versus ‘didi’ (younger)
• English: ‘wall’
• German: ‘Wand’ (inside) ‘Mauer’ (outside)
• German: ‘Berg’
• English: ‘hill’ or ‘mountain’

Lexical Divergence: many-to-many

Lexical Divergence: lexical gaps

• Japanese: no word for privacy
• English: no word for Cantonese ‘haauseun’ or Japanese ‘oyakoko’ (something like `filial piety’)

​

• English ‘cow’ vs. ‘beef’, Cantonese ‘ngau’
• English “fish”, Spanish “pez” vs. “pescado”
• Italian ”maiale”, English “pork” vs. “mouton”

​

Event-to-argument divergences

• English
• The bottle floated out.
• Spanish
• La botella salió flotando.
• The bottle exited floating

​

• Verb-framed lang: mark direction of motion on verb
• Spanish, French, Arabic, Hebrew, Japanese, Tamil, Polynesian, Mayan, Bantu families
• Satellite-framed lang: mark direction of motion on satellite
• Crawl out, float off, jump down, walk over to, run after
• Rest of Indo-European, Hungarian, Finnish, Chinese

Structural divergences

• German: Wir treffen uns am Mittwoch
• English: We’ll meet on Wednesday

​

Head Swapping

• English: X swim across Y
• Spanish: X crucar Y nadando

​

• English: I like to eat
• German: Ich esse gern

​

• English: I’d prefer vanilla
• German: Mir wäre Vanille lieber

Thematic divergence

• Spanish: Y me gusto
• English: I like Y

​

• German: Mir fällt der Termin ein
• English: I forget the date

Divergence counts from Bonnie Dorr

• 32% of sentences in UN Spanish/English Corpus (5K)

3 “Classical” methods for MT

• Direct
• Transfer
• Interlingua

Three MT Approaches: Direct, Transfer, Interlingual

Direct Translation

• Proceed word-by-word through text
• Translating each word
• No intermediate structures except morphology
• Knowledge is in the form of
• Huge bilingual dictionary
• word-to-word translation information
• After word translation, can do simple reordering
• Adjective ordering English -> French/Spanish

Direct MT Dictionary entry

Direct MT

Problems with direct MT

• German

​

​

​

​

​

• Chinese

The Transfer Model

• Idea: apply contrastive knowledge, i.e., knowledge about the difference between two languages
• Steps:
• Analysis: Syntactically parse Source language
• Transfer: Rules to turn this parse into parse for Target language
• Generation: Generate Target sentence from parse tree

English to French

• Generally
• English: Adjective Noun
• French: Noun Adjective
• Note: not always true
• ‘Route mauvaise’ -> ‘bad road, badly-paved road’
• ‘Mauvaise route’ ‘wrong road’
• but is a reasonable first approximation
• Rule:

Transfer rules

Japanese

Lexical transfer

• Transfer-based systems also need lexical transfer rules
• Bilingual dictionary (like for direct MT)

English German

home nach Hause (going home)

Heim (home game)

Heimat (homeland, home country)

zu Hause (at home)

• Can list “at home <-> zu Hause”
• Or do Word Sense Disambiguation

Systran: combining direct and transfer

• Analysis
• Morphological analysis, POS tagging
• Chunking of NPs, PPs, phrases
• Shallow dependency parsing
• Transfer
• Translation of idioms
• Word sense disambiguation
• Assigning prepositions based on governing verbs
• Synthesis
• Apply rich bilingual dictionary
• Deal with reordering
• Morphological generation

Transfer: some problems

• N² sets of transfer rules!
• Grammar and lexicon full of language-specific stuff
• Hard to build, hard to maintain

Interlingua

• Intuition: Instead of lang-lang knowledge rules, use the meaning of the sentence to help
• Steps:
1. translate source sentence into meaning representation
2. generate target sentence from meaning.

Interlingua

Mary did not slap the green witch

Interlingua

• Idea is that some of the MT work that we need to do is part of other NLP tasks
• E.g., disambiguating En:book Es:‘libro’ from En:book Es:‘reservar’
• So we could have concepts like BOOKVOLUME and RESERVE and solve this problem once for each language

Direct MT: pros and cons (Bonnie Dorr)

• Fast
• Simple
• Cheap
• No translation rules hidden in lexicon

​

​

• Unreliable
• Not powerful
• Rule proliferation
• Requires lots of context
• Major restructuring after lexical substitution

​

Cons

Pros

Interlingual MT: pros and cons (B. Dorr)

• Avoids the N² problem
• Easier to write rules

• Semantics is HARD
• Useful information lost (paraphrase)

​

Cons

Pros

Moving toward Statistical MT

Warren Weaver (1947)

When I look at an article in Russian, I say to myself: This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode.

Kevin Knight slide

Rosetta Stone

Carved in 196 BC

Found in 1799

Decoded in 1822

Egyptian hieroglyphs

Egyptian Demotic

Greek

Kevin Knight slide

Centauri/Arcturan [Knight, 1997]

Your assignment, translate this to Arcturan:

farok crrrok hihok yorok clok kantok ok-yurp

Kevin Knight slide

Centauri/Arcturan Parallel Corpus

Slide from Kevin Knight

Centauri to Arcturan Traslation

Slide from Kevin Knight

Translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Centauri/Arcturan Alignment

(

(

(

(

(

(

Translating this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

1a. ok-voon ororok sprok .

​

1b. at-voon bichat dat .

​

7a. lalok farok ororok lalok sprok izok enemok .

​

7b. wat jjat bichat wat dat vat eneat .

​

(

2a. ok-drubel ok-voon anok plok sprok .

​

2b. at-drubel at-voon pippat rrat dat .

​

8a. lalok brok anok plok nok .

​

8b. iat lat pippat rrat nnat .

​

(

3a. erok sprok izok hihok ghirok .

​

3b. totat dat arrat vat hilat .

​

9a. wiwok nok izok kantok ok-yurp .

​

9b. totat nnat quat oloat at-yurp .

​

(

4a. ok-voon anok drok brok jok .

​

4b. at-voon krat pippat sat lat .

​

​

10a. lalok mok nok yorok ghirok clok .

​

10b. wat nnat gat mat bat hilat .

​

(

5a. wiwok farok izok stok .

​

5b. totat jjat quat cat .

​

11a. lalok nok crrrok hihok yorok zanzanok .

​

11b. wat nnat arrat mat zanzanat .

​

(

6a. lalok sprok izok jok stok .

​

6b. wat dat krat quat cat .

​

12a. lalok rarok nok izok hihok mok .

​

12b. wat nnat forat arrat vat gat .

​

(

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

???

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

???

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

process of

elimination

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, translate this to Arcturan: farok crrrok hihok yorok clok kantok ok-yurp

cognate?

Slide from Kevin Knight

Centauri/Arcturan Alignment

Your assignment, put these words in order: { jjat, arrat, mat, bat, oloat, at-yurp }

zero

fertility

Slide from Kevin Knight

It’s Really Spanish/English

Clients do not sell pharmaceuticals in Europe => Clientes no venden medicinas en Europa

​

Slide from Kevin Knight

Summary

• Intro and a little history
• Language Similarities and Divergences
• Three classic MT Approaches
• Transfer
• Interlingua
• Direct