ABCDEFGHIJKLMNOPQRSTUVWXYZ
1
2
Instructions for the evaluation of the sample
3
4
Evaluate the sample in the sheet named with the suffix of your ParlaMint corpus (e.g., “AT” for ParlaMint-AT).
5
6
For each sentence, compare the MT outputs and write into the column “comparison” the name of the MT system that in your opinion provided the best output (the suffix after "translation-" in the header, e.g., write “de”, “gem” or “gmw” for "AT").
7
8
Decide which output is the best based on accuracy of the translation (whether the meaning in preserved), fluency, formal style and the number and severity of translation errors. You can ignore incorrect proper name translations, because they will be corrected as a post-processing step. Decide based on your opinion on which translation would be the most useful for the future end users of the translated corpus.
9
10
It is possible that none of the outputs will be completely correct - in that case, decide for the one where the meaning is the closest to the source meaning. If all MT outputs are completely incorrect (the meaning of the source sentence is lost in the translation), write "incorrect" to the column "comparison".
11
12
If all outputs are more or less correct and you cannot decide which one is better, leave the column "comparison" empty.
13
14
After we create the final translated corpus, we can extend the evaluation to more instances, more objective criteria and error typology, if there will be time and interest. For this preliminary analysis, we decided for a more pragmatic approach where the evaluation is based on which translation you prefer. If there are big differences between the quality of the MT translation systems, we believe that already such short evaluation will reveal that.
15
16
Please notify me via e-mail when you are finished. Thank you!
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100