1 of 35

TV-APPARAT, Radiola Minett Typ 8410, Vänersborgs museum, Sweden, CC BY

Subtitling the Archive:�humans and machines�are better together

A look inside what

the machines do

2 of 35

Single Italian subtitle

3 of 35

Single Italian subtitle

An experience for few, inaccessible to:

Non-Italian-speaking people
Audio impaired users

What’s needed to widen the audience?

Translation…
…into proper subtitles

Ok, let’s translate!

However, in most of the cases the video alone is not enough…

…its mere watching, WITHOUT SUBTITLES, WOULD MAKE its content inaccessible TO A WIDE AUDIENCE including

non-italian speakers…and
audio impaired users…

This “experience for few” is something that we would ideally want to avoid.

Here’s where subtitles come into play, by providing users with a “textual view” of the video content.

But what about the subtitle itself? What can we say about the long, four-line Italian text you’ve just seen?

Well: it’s true that it may help audio-impaired users…

…but, still, it’s not enough for at least two reasons:

not only it is still in Italian…

…also, being presented as a long text, it’s far from being “pleasant”:

reading it, in fact, implies a considerable cognitive load that is somehow distracting from the video itself.

Also this cognitive load is something that we would ideally want to avoid.

So: what do we need to widen the audience, making the video not only accessible to a larger population of users but also pleasant to view?

We need two things:

translation…
…into proper subtitles @@@@@@@@

Let’s dig into the first problem: translation!

4 of 35

English subtitles, word by word

5 of 35

English subtitles, word by word

The video content is now accessible to a wider, English-speaking audience…

…but the experience is rather poor due to high cognitive load

Translation is not enough!

But what is a “proper subtitle”?

6 of 35

Human subtitling is an “art”

Governed by constraints* to reduce viewers’ cognitive load:

7 of 35

Human subtitling is an “art”

Governed by constraints* to reduce viewers’ cognitive load:

Length: max 2 lines, max 42 chars/line
Duration on screen: max reading speed of 21 chars/second
Style: syntactic constituents should not be broken

* Subtitling guidelines may (slightly) vary across:

iiiproviders / languages / target audience

The above ones are from Netflix guidelines

Besides the need to follow the rhythm of the speech

with synchronized in and out times…

such constraints typically relate to

the length of the so-called “subtitle block”

which should comprise at most two lines of maximum 42 characters each

the time a subtitle should appear on screen

which should require a maximum reading speed of 21 characters per second

and the readability of the subtitles

for instance according to syntactic criteria like not breaking constituents

Carefully defined guidelines formalizing such constraints exist…

…and may slightly vary across providers, languages and target audiences.

Those, for instance, are the ones adopted by Netflix

Let’s now look at the original subtitles of our video,

as they were created by the Italian Istituto Luce,

one of the partners of the Eurosub project. @@@@@@@@

8 of 35

Manual subtitles

Even the original - manually generated - subtitles released by Istituto Luce not necessarily stick to common subtitling guidelines

We know that it is a great structure whose

perfect functioning is linked to a great brain.

Too many characters (91)

Broken syntactic constituent

SPEZZARE!!!

@@@@@@@@

Interestingly, even such original subtitles are not optimal

according to the standard criteria set on length, readability and style. @@@@@@@@

THE LAST SUBTITLE OF THE VIDEO, for instance:

breaks the constraint on subtitles’ length
as well as the one about not breaking syntactic constituents

These issues, in this specific case are rather easy to solve…

…but wait: why did we define subtitling as an “art”

The reason is that it is not always as easy as it looks:

Satisfying ALL the constraints is often very difficult, especially when it conflicts with the most natural word choice

and, in general, with the criteria defining a “good translation”.

Therefore, what makes subtitling an art is the ability (the HUMAN ability)

to apply strategies to balance all these aspects finding the best trade-off…..

Keeping in mind such difficulties, let’s now have a look at the subtitles

automatically generated by the Eurosub pipeline…

watching our video for the last time. @@@@@@@@

9 of 35

Manual subtitles

10 of 35

Manual subtitles

Even the original - manually generated - subtitles released by Istituto Luce do not necessarily stick to common subtitling guidelines

11 of 35

Manual subtitles

Even the original - manually generated - subtitles released by Istituto Luce do not necessarily stick to common subtitling guidelines

We know that it is a great structure whose

perfect functioning is linked to a great brain.

Too many characters (91)

Broken syntactic constituent

for instance, THE LAST SUBTITLE OF THE VIDEO,

breaks the constraint on subtitles’ length

as well as the one about not breaking syntactic constituents

These issues, in this specific case are rather easy to solve…

…but wait: why did we define subtitling as an “art”

The reason is that it is not always AS EASY AS IT LOOKS:

Satisfying ALL the constraints is often very difficult, especially when it conflicts with the most natural word choice

and, in general, with the criteria defining a “good translation”.

Therefore, what makes subtitling an art is the ability (the HUMAN ability)

to apply strategies to balance all these aspects finding the best trade-off…..

Keeping in mind such difficulties, let’s now have a look at the subtitles

automatically generated by the Eurosub pipeline…

watching our video for the last time. @@@@@@@@

12 of 35

Eurosub automatic subtitles

The English subtitles automatically generated by the Eurosub processing chain (try to) be correct and satisfy captioning compliance

We know that it is a great organism.

whose perfect functioning

is linked to a large brain.

Non-compliant subtitle split in two

(max 2 lines, <42 characters each)

13 of 35

Eurosub automatic subtitles

The English subtitles automatically generated by the Eurosub processing chain (try to) be correct and satisfy captioning compliance

We know that it is a great organism.

whose perfect functioning

is linked to a large brain.

Non-compliant subtitle now split in two

(max 2 lines, <42 characters each)

14 of 35

Eurosub automatic subtitles

The English subtitles automatically generated by the Eurosub processing chain (try to) be correct and satisfy captioning compliance

We know that it is a great organism.

whose perfect functioning

is linked to a large brain.

Non-compliant subtitle now split in two

(max 2 lines, <42 characters each)

error

15 of 35

How is this achieved?

16 of 35

Automatic generation of subtitles

A 5-step process

Automatic transcription (ASR)
Segmentation into “blocks” (subtitles)
Automatic translation (MT) of each block
Segmentation of blocks into lines
Timing projection

17 of 35

ASR output

For each word:

For each word: text + timing

For each word: text

18 of 35

ASR output

casing

19 of 35

ASR output

punctuation

20 of 35

ASR output

pauses

21 of 35

ASR output

	Questa	Termini,	stazione		quale	cose.
ASR	è	una	moderna		conosciamo	<PAUSE>
	Roma	grande		della	tante

22 of 35

Automatic translation of blocks

Segmentation into blocks

	Questa	Termini,	stazione		quale	cose.
SEGMENTER	è	una	moderna		conosciamo
	Roma	grande	EOB	della	tante	EOB

	This	Termini,	modern		which	things.
MT	is	a	station		we know
	Roma	large		of	many

23 of 35

MT output

	This	Termini,	modern		which	things.
MT	is	EOL a	station		we know
	Roma	large		of	many

24 of 35

Segmentation into lines

Timing projection

	This	Termini,	modern		which	things.
SEGMENTER	is	EOL a	station		we know
	Roma	large		of	many

1

00:00:00,040 --> 00:00:03,280

This is Roma Termini,

a large modern station

2

00:00:03,360 --> 00:00:05,000

of which we know many things.

ASR TIMING

English subtitles (SRT)

25 of 35

Everything looks easy and magic but…

Photo by Pablo Arroyo on Unsplash

26 of 35

Everything looks easy and magic but…

ASR

Segmenter

MT

27 of 35

Everything looks easy and magic but…

ASR

MT

Segmenter

Deep neural networks, i.e. mathematical models

with hundreds of millions of parameters

trained on

thousands of hours of transcribed audio
millions of parallel sentences
thousands of segmented sentences

fine tuned on in-domain data

using high-end GPUs

All of them are deep neural networks… @@@@@@@ …a term that many of you probably know

Neural networks are mathematical models that are loosely inspired by the human brain and its billions of interconnected neurons. @@@@@@

These models contain tons of parameters that have to be properly set by learning from data. @@@@@@

But what are these data?

Data are examples from which machines should learn

In our case, they comprise

thousands of hours of transcribed audio for the speech recognizer
parallel corpora containing millions of sentences associated with their exact translation for the machine translation component
and thousands sentences properly segmented into subtitles, for the segmenter.

An important aspect about data is that not all of them are equally useful.

Indeed, generic data are usually enough to train general-purpose models…

…but what about their use in specific domains, characterized by their own peculiarities in terms of style and register?

In these conditions, general-purpose models are often inadequate to operate in specific domains…

think, for instance about the medical domain with its obscure jargon…

…or the legal one with its complex structures.

This is why, among all the data needed, the in-domain ones have a particular importance…

through a specific training step called FINE-TUNING… @@@@@@

…they help our systems to capture the peculiarities of the application domain in which they will be used.

Unfortunately, if finding general-purpose data is sometimes hard, finding the in-domain ones is often much harder…… @@@@

The last important aspects is the COMPUTATIONAL POWER:

as a matter of fact, the training of current deep neural networks cannot be done with consumer hardware and, in particular, with the CPU of any computer…

…it requires GRAPHICS PROCESSING UNITS, which are costly and energy-devouring processors…

designed for the parallel processing involved in the training and the use of our neural networks………….

This is all about the main INGREDIENTS…

…but what about the whole process?

…and how it translates into the actions needed to add a new language to those currently covered by EuroSub?

28 of 35

What about adding a new language?

in-domain ASR/MT available?

in-domain train DATA available?

generic train DATA available?

in-domain ada. DATA available?

collect DATA &

train from scratch generic ASR/MT

collect DATA & adapt generic ASR/MT

FAILURE

START

N

collect DATA &

train from scratch in-domain ASR/MT

Y

SUCCESS

generic ASR/MT available?

FAILURE

SUCCESS

Y

N

Y

N

29 of 35

What about adding a new language?

in-domain ASR/MT available?

in-domain train DATA available?

generic train DATA available?

in-domain ada. DATA available?

collect DATA &

train from scratch generic ASR/MT

collect DATA & adapt generic ASR/MT

FAILURE

START

N

collect DATA &

train from scratch in-domain ASR/MT

Y

SUCCESS

generic ASR/MT available?

FAILURE

SUCCESS

Y

N

Y

N

30 of 35

The devil is in the details

“train” and “adapt” imply the need of:

computer scientists and deep learning specialists
costly Graphics Processing Units (GPUs)

“collect” implies the need of:

computer scientists and language specialists
data that are not copyright-protected
an adequate data storage infrastructure

31 of 35

Take-home message

The Eurosub project has shown the power of AI to:

Automatize the all the steps of the subtitling process
so as to provide professionals with good suggestions, suitable for post-editing with reduced effort

Subtitling is an “art”: doing it well is still a prerogative of humans

32 of 35

Take-home message

We have come a long way

but we have a long way to go.

Thanks for watching!

Photo by Vidar Nordli-Mathisen on Unsplash

33 of 35

Take-home message

We have come a long way but we have a long way to go

Thanks for watching!

Photo by Vidar Nordli-Mathisen on Unsplash

34 of 35

Acknowledgments & links

Fondazione Bruno Kessler, Trento, Italy

Mauro Cettolo cettolo@fbk.eu

Matteo Negri negri@fbk.eu

https://pro.europeana.eu/project/europeana-subtitled

https://subbit.eu

35 of 35

Conclusions

Automatic subtitling pipeline of Eurosub:

relies on cutting-edge technology
generates compliant and high quality subtitles
nevertheless, is not free from errors

According to the application requirements:

imperfections are judged as acceptable
post-editing is deemed necessary and carried out