Ptakopět Experiment 2: Instructions for Participants

Ptakopět tool entry point: http://www.statmt.org/bergamot/cgi/translationexperiment.php

Contact info: Mateo Obregon <mateo.obregon@ed.ac.uk>,

In this experiment, you are going to test the usefulness of machine translation of your sentences into a foreign language known as outbound machine translation. In the scenario of outbound translation, the sender usually has little or no knowledge of the target language but still needs a translation that is reliably adequate. It is the sender’s responsibility to ensure the quality of the translation.

A typical example of the outbound translation scenario is using the foreign language web forms. Imagine you want to buy a design product from a small Czech manufacturer. However, you do not know Czech. You visit the website of the manufacturer and find that it is only in Czech, though. You use an inbound machine translation tool integrated in your browser to translate the whole web page to English and find out that you can order the product online as the website also contains an e-shop. Since you are very cautious and have unpleasant experience with other e-shops, you would like to warn the seller to treat the product with care when packing it. The e-shop offers a text box to add any note to your order. This is the moment when the outbound machine translation service is enabled. By offering several kinds of clues, the outbound translation service assists you in formulating your query in English so that its automatic translation to Czech is fluent and, more importantly, adequate - the translation does not change the meaning and keeps the original intention of the query.

Ptakopět is the tool that simulates this scenario in order to empirically test various properties of the outbound machine translation service.

Ptakopět user interface

An example of Ptakopět simulating the case described above is shown in the next figure. Its user interface consists of a window vertically split into two panes:

left pane (from top to bottom)

brief instructions
navigation control
stimulus: a screenshot of a web form with a query to be translated highlighted in a green rectangle + a short description of the website’s domain

right pane (from top to bottom)

input box: contains the text in English. It is always present as the first box.
translation box: contains translated text in the target language (Czech or Estonian).
back-translation box: contains the backward translation in English.
paraphrase box: contains paraphrases of the text in the input box.

Stimulus shows the query to be translated and should give you necessary context to rephrase it if needed. It consists of a screenshot of the web page showing excerpts of the web form, including the query already filled in (and highlighted in a green rectangle). Apart from the query, the screenshot often tries to capture such elements of the webpage that should make it easier and faster for you to understand the query’s intention and its context. A short description of the website’s domain above the screenshot plays the same role.

Only the input box and translation box can be edited. The texts in the two boxes may be highlighted in red with different levels of brightness. In the translation box, the highlighting indicates that the translation is somehow problematic. This automatic estimate may be misleading, though. In the input box, the highlighted words are the ones that correspond to the problematic ones in the translation.

Along with the back-translation box and the paraphrase box, the translation box and red highlighting form so-called clue modules. They aid you with rephrasing the query given by the stimulus. Every stimulus may come with a different combination of available modules. That means that for some stimuli, you will see all boxes including the highlighting and for some only, for example, the source and target box. This can be seen in the next figure.

The navigation control shows an indicator of the experiment’s progress (in terms of stimuli grouped into blocks). Moreover, it contains buttons to submit the produced translation, to make a note on your experience with the stimulus, to skip the stimulus, or to show these instructions. To submit, you have to click one of the buttons 1 to 5 based on how much you trust the translation (5 being the most). By clicking on the Note button, you can put comments on your experience concerning the stimulus, which is completely voluntary, though. You may also decide to skip the stimulus, but this should be exceptional and in that case, please provide a reason.

Experiment workflow

The experiment is focused on participants, which do not speak Czech or Estonian. Before the experiment starts, you will be asked, which languages you can speak or understand and what is your proficiency level in these languages.

The experiment environment is available online. Therefore, only internet connection is required during the duration of the experiment. Please always access the experiment via the following link: http://www.statmt.org/bergamot/cgi/translationexperiment.php

The entire experiment consists of about 70 stimuli, which are distributed in 10 blocks. Your progress can be seen above the rating buttons. You may interrupt the experiment, but neither your browser nor your device can be changed, since your progress is stored in your current browser in your current device. Your browser and device can be turned off but don’t delete history/local storage/cookies for the duration of the experiment. After you reload the page after your interruption, the last stimuli you worked on should be displayed.

For each stimulus, your goal is to create the best possible translation (in terms of adequacy and fluency) of the query associated with the stimulus given the available tools. You may use the query as it is or rephrase it but in such a way that its original intention remains the same.

A typical experiment workflow is as follows:

you are presented with a scene including the stimulus and a combination of clue modules
read the query and try to understand its intention
either use the original query or try to rephrase it. Type the query to the input box.
use the clue modules to get assured that the translation in target language is correct, or to get inspired for another rephrasing of the query
if you feel satisfied with the translation, continue to (6); otherwise, go back to (3)
submit the translation by rating your confidence that the translation is correct (1 = lowest, 5 = highest) and voluntarily comment on your experience with the current stimulus
if no more stimuli remain, it is the end of the experiment; otherwise, go to (1)

In case of any problems, do not hesitate to contact the investigators (contacts are at the beginning of this document).