Evaluation Scores of Google Translate in 107 Languages

	A	B	C	D	E	F	G	H	I	K	L	M	O	P	Q	S	T	U	V	W	X
1	Source data for Teach You Backwards: An In-Depth Study of Google Translate for 108 Languages												http://teachyoubackwards.com
2	The tables in this file report the scores of translations of the same 20 English phrases to all 107 languages in Google Translate, as rated by native or highly-competent speakers.
3	The data in this file is free to use according to the terms of the CC-by license at https://creativecommons.org/licenses/by/4.0/legalcode, with credit to Martin Benjamin, Kamusi Project International, 2018
4	Evaluators rated each translation on a scale of A, B, or C. (A = good translation, B = not correct, but someone could understand the idea, C = completely wrong)																original English phrase	English explanation shown to evaluators	Bard	Tarzan	Fail
5	The original English phrases given to Google Translate, and the English explanations given to the evaluators, are shown in the light purple table on the right.											Click here for itemized translations and scores in all languages				1	fly out of London	take an airplane from London	35.20%	79.00%	21.00%
6	Methodology is discussed in the black boxes on the right.					Click here for language-to-language confidence scores (Tarzan-to-Tarzan intelligibility ratings between non-English pairs)										2	like a bat out of hell	escaping as quickly as possible	2.40%	23.80%	76.20%
7																3	out cold	unconscious	3.80%	9.50%	90.50%
8	Bard is a weighted ranking that indicates the proportion of translations that were judged close to human quality. Responses scored A = 5, B = 2.5, C = 0. Maximum score = 100.															4	out of bounds	unacceptable	6.70%	41.00%	59.00%
9	Tarzan indicates the percentage of times that translations could be understood, regardless of whether they were judged as human quality. Responses scored A = 5, B = 5, C= 0. Maximum score = 100.															5	out of breath	gasping for air (for example, after running)	34.80%	64.80%	35.20%
10	Fail is the percentage of times that translations were judged as completely wrong. Responses scored A = 0, B = 0, C = 5. Maximum score = 100.															6	out of curiousity	because a person is casually interested in something	33.30%	65.70%	34.30%
11																7	out of focus	not clear to see (blurry)	21.90%	49.50%	50.50%
12	Languages in blue are those that this page says run on the Neural Machine Translation Model: https://cloud.google.com/translate/docs/languages											White = Phrase-Based Machine Translation Model.				8	out of his mind	crazy	4.80%	28.60%	71.40%
13	Languages in orange are those that this page says run on the Neural Machine Translation Model: https://translate.google.com/intl/en/about/languages/															9	out of milk	the supply of milk is finished	4.30%	19.00%	81.00%
14	Languages in bold are among world's top 100 (https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers)															10	out of order	does not function (broken)	31.90%	50.50%	49.50%
15	Green-headed table shows GT results for top 100 languages by number of native speakers								Dark-blue-headed table shows GT results for languages not in top 100							11	out of pocket	paid for something from personal money	15.20%	50.50%	49.50%
16	Spanish was evaluated separately for SPain and Latin America. Portuguese was evaluated separately for PorTugal, Cape Verde, and BRazil.Traditional and Simplified Chinese were evaluated as one.															12	out of steam	no more energy (exhausted)	3.30%	10.50%	89.50%
17																13	out of style	unfashionable	21.90%	64.80%	35.20%
18	Alphabetical		Bard	Tarzan	Fail		Bard Ranking			Tarzan Ranking			% Fail Ranking			14	out of the closet	openly homosexual	9.50%	24.80%	75.20%
19	1	Afrikaans	67.5	87.5	12.5		1	Afrikaans	67.5	1	Afrikaans	87.5	1	Bengali	100	15	out of the game	no longer participating in a game	32.40%	68.60%	31.40%
20	2	Albanian	26.25	40	60		2	German	60	2	German	82.5	1	Haitian Creole	100	16	out of the office	away from the office	46.70%	82.90%	17.10%
21	3	Amharic	30	40	60		2	Portuguese-PT	60	3	Portuguese-BR	80	1	Tajik	100	17	out of this world	excellent	3.80%	17.10%	82.90%
22	4	Arabic	32.5	40	60		3	Spanish-SP	57.5	3	Portuguese-CV	80	2	Kurdish	95	18	out of time	a deadline has passed	14.80%	63.80%	36.20%
23	5	Armenian	25	40	60		4	Polish	56.25	4	Spanish-LA	75	2	Nepali	95	19	out of wedlock	between partners who are not married	30.00%	61.00%	39.00%
24	6	Azerbaijani	37.5	55	45		5	Chinese	55	4	Spanish-SP	75	3	Latin	90	20	out on the town	having a fun time going shopping or to bars/ restaurants (carousing)	4.30%	21.90%	78.10%
25	7	Basque	37.5	47.5	52.5		5	Croatian	55	5	Polish	72.5	3	Malaysian	90
26	8	Belarusian	40	55	45		5	Spanish-LA	55	6	Danish	70	3	Urdu	90	Methodology:
27	9	Bengali	0	0	100		6	Dutch	52.5	6	Greek	70	4	Maori	85
28	10	Bosnian	30	40	60		6	Galician	52.5	7	Chinese	65	5	Cebuano	80	We translated 20 common English phrases to 102 languages, and then sent a re-worded explanation of the phrase to independent evaluators. We did not send the original English phrase, because early testing showed that knowing the actual words in question influenced the way that evaluators understood the query. For example, if we showed "out of steam" along with its explanation, some evaluators would see that a word for "steam" was given and therefore judge the translation highly, whereas if they only saw the explanation "no more energy (exhausted)", they were able to judge whether the proposed translation captured the usual English meaning.
29	11	Bulgarian	40	60	40		6	Greek	52.5	7	Croatian	65	5	Georgian	80
30	12	Catalan	37.5	60	40		6	Portuguese-BR	52.5	7	Dutch	65	5	Kinyarwanda	80
31	13	Cebuano	12.5	20	80		6	Portuguese-CV	52.5	7	Finnish	65	5	Persian	80
32	14	Chichewa	17.5	30	70		7	Italian	50	7	Hungarian	65	5	Punjabi	80
33	15	Chinese	55	65	35		7	Latvian	50	7	Portuguese-PT	65	5	Uzbek	80
34	16	Corsican	22.5	35	65		8	Indonesian	47.5	8	Welsh	62.5	6	Xhosa	77.5
35	17	Croatian	55	65	35		8	Hungarian	47.5	9	Indonesian	60	7	Hawaiian	75
36	18	Czech	43.75	55	45		8	Igbo	47.5	9	Bulgarian	60	7	Javanese	75	We tested clusters of two or more words that often occur together, and that have a meaning that is generally discernable when they do (as can be readily seen in Twitter searches); for example, tweets with “out cold” almost always imply unconsciousness. We did not test single words, which can be highly ambiguous (for example, right = correct, legal entitlement, politically conservative, not left, etc.), and therefore too arbitrary in isolation. Although GT does not advertise itself as a dictionary, single-word lookups are a major proportion of real world uses of the service. Nor did we test full sentences, which add a lot of complexity to scoring, since translations might include both correct and incorrect elements; moreover, humans can translate the same sentence many ways, making it impossible to impose a gold standard for full sentences, especially across dozens of languages. It should be noted that GT alters its vocabulary choice on the fly, so the words chosen to translate a phrase may not be those selected in a longer sentence; for example, “run of the mill” is represented in French by “course du moulin” in isolation, and “course de l’usine” when translating a longer tweet (both completely wrong), and other instances might produce other results.
37	19	Danish	40	70	30		8	Serbian	47.5	9	Catalan	60	7	Lao	75
38	20	Dutch	52.5	65	35		9	Finnish	45	9	French	60	7	Myanmar/ Burmese	75
39	21	Esperanto	40	55	45		9	French	45	9	Galician	60	7	Pashto	75
40	22	Estonian	27.5	45	55		9	Hebrew	45	9	Igbo	60	7	Samoan	75
41	23	Filipino	25	35	65		9	Swedish	45	9	Italian	60	7	Swahili	75
42	24	Finnish	45	65	35		10	Czech	43.75	9	Kazakh	60	7	Thai	75
43	25	French	45	60	40		10	Welsh	43.75	9	Latvian	60	7	Yoruba	75
44	26	Frisian	30	40	60		11	Japanese	42.5	9	Macedonian	60	8	Chichewa	70
45	27	Galician	52.5	60	40		11	Kazakh	42.5	9	Maltese	60	8	Hindi	70
46	28	Georgian	10	20	80		11	Korean	42.5	9	Odia (Oriya)	60	8	Icelandic	70
47	29	German	60	82.5	17.5		11	Malaysian	42.5	9	Sesotho	60	8	Lithuanian	70	All of the clusters contained the word “out”. This word is ubiquitous in English texts. It is extremely ambiguous in isolation, but often occurs in clusters with unmistakable meanings. WordReference.com gives definitions and French equivalents for nearly 1700 composed expressions that include “out”, from “a fish out of water” to “zoom out” (http://www.wordreference.com/enfr/out?start=1600). We chose 20 formulations that are lexicalized in WordReference as composed forms such as “out of style”, or that, as queried on Twitter, usually reduce to defined meanings when matched with other particular words, such as “out of milk”. The premise is that all of these items have been translated in an electronic dictionary, so are thus similarly viable as units for machine translation.
48	30	Greek	52.5	70	30		11	Romanian	42.5	9	Slovenian	60	8	Marathi	70
49	31	Gujarati	30	45	55		12	Belarusian	40	9	Swedish	60	8	Shona	70
50	32	Haitian Creole	0	0	100		12	Bulgarian	40	10	Azerbaijani	55	8	Somali	70
51	33	Hausa	27.5	35	65		12	Danish	40	10	Belarusian	55	8	Zulu	70
52	34	Hawaiian	15	25	75		12	Esperanto	40	10	Czech	55	9	Kyrgyz	67.5
53	35	Hebrew	45	45	55		12	Mongolian	40	10	Esperanto	55	10	Corsican	65
54	36	Hindi	22.5	30	70		12	Russian	40	10	Japanese	55	10	Filipino	65
55	37	Hmong	27.5	40	60		12	Scots Gaelic	40	10	Korean	55	10	Hausa	65
56	38	Hungarian	47.5	65	35		12	Slovenian	40	10	Romanian	55	10	Luxembourgish	65	The expressions were not chosen to be especially simple or difficult, nor based on corpus frequency. Rather, they were chosen because they had clear meanings, and are broadly representative of the types of phrase that ordinary users are likely to seek to translate. The selection (see above) is therefore not rigidly scientific, and we leave it to the reader to decide whether the items provide a fair test of MT capability. The least-recognized phrase, “out cold”, resulted in just 3 “A” ratings (Hausa, Hindi, and Malaysian) and was only understandable to some extent 10 times, while the phrase “out of the office” produced an understandable result 83% of the time.
57	39	Icelandic	20	30	70		12	Turkish	40	10	Serbian	55	10	Tamil	65
58	40	Igbo	47.5	60	40		13	Azerbaijani	37.5	11	Malayalam	50	10	Tatar	65
59	41	Indonesian	47.5	60	40		13	Basque	37.5	11	Mongolian	50	10	Telugu	65
60	42	Irish	25	45	55		13	Catalan	37.5	11	Russian	50	10	Vietnamese	65
61	43	Italian	50	60	40		14	Macedonian	35	11	Scots Gaelic	50	11	Albanian	60
62	44	Japanese	42.5	55	45		14	Maltese	35	11	Turkish	50	11	Amharic	60
63	45	Javanese	17.5	25	75		14	Odia (Oriya)	35	11	Turkmen	50	11	Arabic	60	Scores are not absolute, for two reasons. First, the choice of expressions was arbitrary. A different selection of English expressions would generate different numerical results within each language; for example, scores would probably fall were a larger number of idioms to be included. However, relative results would likely remain the same; a language with high scores for our test set should perform highly with other data, while a language with low scores herein would have similarly low results with other input. Second, most language scores show the subjective opinion of a single reviewer. One could well argue that more reviewers per language would produce more reliable data. We had several languages with multiple reviewers, and found that inter-annotator disagreements were usually minor, with a handful of entries per language being judged good versus marginal, or marginal versus wrong. A single entry being ranked by different evaluators as good versus marginal does not change the Tarzan score and changes the Bard score by 2.5 points, and a disagreement between marginal and wrong changes both Tarzan and Bard by 2.5. We averaged the scores where annotators disagreed, and have kept disagreements visible in the full data release, in the "Itemized Translations and Scores" tab at the bottom of this page. Based on the inter-annotator disagreement values we discovered, the reader is advised to place mental error bars of ±10 around the score that is reported.
64	46	Kannada	20	40	60		14	Sinhala	35	11	Yiddish	50	11	Armenian	60
65	47	Kazakh	42.5	60	40		15	Arabic	32.5	12	Basque	47.5	11	Bosnian	60
66	48	Khmer	30	45	55		15	Sesotho	32.5	13	Estonian	45	11	Frisian	60
67	49	Kinyrwanda	12.5	20	80		16	Turkmen	31.25	13	Gujarati	45	11	Hmong	60
68	50	Korean	42.5	55	45		16	Uyghur	31.25	13	Hebrew	45	11	Kannada	60
69	51	Kurdish	3.75	5	95		17	Amharic	30	13	Irish	45	11	Malagasy	60
70	52	Kyrgyz	22.5	32.5	67.5		17	Bosnian	30	13	Khmer	45	11	Sindhi	60
71	53	Lao	21.25	35	65		17	Frisian	30	13	Norwegian	45	11	Sundanese	60
72	54	Latin	5	10	90		17	Gujarati	30	13	Sinhala	45	11	Ukrainian	60
73	55	Latvian	50	60	40		17	Khmer	30	13	Slovak	45	12	Estonian	55
74	56	Lithuanian	20	30	70		17	Sindhi	30	13	Uyghur	45	12	Gujarati	55
75	57	Luxembourgish	27.5	35	65		17	Ukrainian	30	14	Albanian	40	12	Hebrew	55
76	58	Macedonian	35	60	40		18	Estonian	27.5	14	Amharic	40	12	Irish	55
77	59	Malagasy	27.5	40	60		18	Hausa	27.5	14	Arabic	40	12	Khmer	55
78	60	Malayalam	42.5	50	50		18	Hmong	27.5	14	Armenian	40	12	Norwegian	55
79	61	Malaysian	5	10	90		18	Luxembourgish	27.5	14	Bosnian	40	12	Sinhala	55
80	62	Maltese	35	60	40		18	Malagasy	27.5	14	Frisian	40	12	Slovak	55
81	63	Maori	10	15	85		18	Norwegian	27.5	14	Hmong	40	12	Uyghur	50
82	64	Marathi	15	30	70		18	Slovak	27.5	14	Kannada	40	13	Basque	52.5
83	65	Mongolian	40	50	50		18	Tatar	27.5	14	Malagasy	40	14	Malayalam	50
84	66	Myanmar/ Burmese	12.5	25	75		18	Yiddish	27.5	14	Sindhi	40	14	Mongolian	50
85	67	Nepali	2.5	5	95		18	Albanian	26.25	14	Sundanese	40	14	Russian	50
86	68	Norwegian	27.5	45	55		20	Armenian	25	14	Ukrainian	40	14	Scots Gaelic	50
87	69	Odia (Oriya)	35	60	40		20	Filipino	25	15	Corsican	35	14	Turkish	50
88	70	Pashto	20	25	75		20	Irish	25	15	Filipino	35	14	Turkmen	50
89	71	Persian	10	20	80		21	Corsican	22.5	15	Hausa	35	14	Yiddish	50
90	72	Polish	56.25	72.5	27.5		21	Hindi	22.5	15	Lao	35	15	Azerbaijani	45
91	73	Portuguese-BR	60	80	20		21	Kyrgyz	22.5	15	Luxembourgish	35	15	Belarusian	45
92	74	Portuguese-CV	52.5	80	20		21	Sundanese	22.5	15	Tamil	35	15	Czech	45
93	75	Portuguese-PT	52.5	65	35		21	Telugu	22.5	15	Tatar	35	15	Esperanto	45
94	76	Punjabi	10	20	80		21	Lao	21.25	15	Telugu	35	15	Japanese	45
95	77	Romanian	42.5	55	45		23	Icelandic	20	15	Vietnamese	35	15	Korean	45
96	78	Russian	40	50	50		23	Kannada	20	16	Kyrgyz	32.5	15	Romanian	45
97	79	Samoan	15	25	75		23	Lithuanian	20	17	Chichewa	30	15	Serbian	45
98	80	Scots Gaelic	40	50	50		23	Pashto	20	17	Hindi	30	16	Bulgarian	40
99	81	Serbian	47.5	55	45		23	Tamil	20	17	Icelandic	30	16	Catalan	40
100	82	Sesotho	32.5	60	40		23	Vietnamese	20	17	Lithuanian	30	16	French	40