1

Understanding Encoders in Multilingual Neural Machine Translation:

From Strengths to Decoder-only Alternatives

Zhi Qu

​

D3 Student, Natural Language Processing Lab., NAIST;

Technical Researcher, Advanced Translation Technology Lab., NICT.

​

September 11, 2025

2

Find My Slides!

3

Contents

1. Background

​

• Understanding the Success of Encoders

​

Paper: Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation.

​

• From Strengths to Decoder-only Alternatives

​

Paper: Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation.

​

• Summarization

4

Background: MNMT

Multilingual Neural Machine Translation (MNMT)

​

A system enables arbitrary translations from multiple languages to multiple languages.

Benefits of implementing a single model for MNMT:

1. Low costs in deployment and convenient for use.

​

• Knowledge transfer, especially for low-resource languages.

​

For instance, zero/few-shot translation:

​

Although a translation direction is not seen in training or is trained with very small amount of data, the model still can translate this direction well.

5

Background: Paradigms

Traditional (M)NMT.

Training or fine-tuning a single (M)NMT-specific model by parallel data.

Encoder-decoder architecture. Extremely Successful!

Decoder-only architecture. Not Always Work, although it has a lot of advantages...

Gao et al. (2022) and Zhang et al. (2022) empirically show that decoder-only architecture is weak in MNMT.

​

LLM-driven Methods.

6

Background: Paradigms

LLM-driven Methods.

Fine-tuning or directly employing LLM to do multilingual translation.

Commercial LLMs (GPT, Claude, Gemini…), strong but expensive and closed-source.

Open-source LLMs, still limited:

• Tower 7B [Alves et al., 2024] supports 10 languages only.
• ALMA 13B [Xu et al., 2024] supports only 6 languages (English-centric).
• XALMA 13B (+ adapters) [Xu et al., 2025] supports 50 languages, however, the translation directions are not arbitrary.
• BigTranslate 13B [Yang et al., 2023], which supports more than 100 languages but not strong.

Why decoder-only models CANNOT simply replicate

the success of encoder-decoder models?

7

Part 1

Part 1, Understanding the Success of Encoders:

​

​

Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation.

​

Zhi Qu, Chenchen Ding, Taro Watanabe.

​

MT Summit 2025.

​

Springer EAMT 2025 Best Paper Award.

8

Part 1: Problem

For training a traditional model in MNMT:

​

Adding a tag specified to the target language at the beginning of input sequence.

MNMT Model

[de] Hello, world!

Hallo, welt!

[de] is an instruction of translating to German.

Note:

There are other strategies of instructions, but this is the optimal [Wu et al. 2021] and most popular one.

Enc

Dec

Dec-only

or

9

Part 1: Problem

Although the role of encoder in MNMT has been discussed a lot,

a discrepancy existed in previous research:

​

• An ideal encoder distinguishes representations by the target language, i.e., learning language-specific representations. (Kudugunta et al., 2019; Liu et al., 2021; Tan and Monz, 2023; Stap et al., 2023; Sun et al., 2024)

​

• An ideal encoder learns language-agnostic representations, which are transferable across languages. (Pan et al., 2021; Gu and Feng, 2022; Gao et al., 2023; Bu et al., 2024)

​

10

Part 1: Problem

Analysis based on sentence-level representations extracted from the encoder output:

1. shows representations are clustered by their target languages (analyzed by SVCCA scores);
2. shows representations from different source languages are aligned (analyzed by t-SNE).

Why? Can we reach a unified conclusion?

11

Part 1: Investigation Setup

Two English-centric datasets:

​

• Europarl-15 (2 × 14 translation pairs), each language has 189,310 instances, and all languages are semantically parallel.

​

• TED-19 (2 * 18 pairs), each translation pair contains 103,093~214,111 instances.

Models:

Transformer with 6/8/10 encoder layers and 6 decoder layers, size of 512 × 1024.

12

Part 1: Identity Pair

Our tool for analysis:

Identity pair refers to a pseudo and zero-shot pair translating a sentence to itself, which can present the optimal state of a language.

Example:

• A real pair of en → de:

[de] Hello, world. → Hallo, welt.

​

• The identity pair of en:

[en] Hello, world. → Hello, world.

​

• The identity pair of de:

[de] Hallo, welt. → Hallo, welt.

In fact, identity pairs have been intuitively used in previous works. (Tiedemann and Scherrer, 2019; Thompson and Post, 2020; Bu et al., 2024)

​

However, these works did not define, validate and utilize identity pairs.

Features:

• Uninvolving another language.

​

• Zero-shot, but the model can perfectly recover the source sentence at the output.

​

• It is indeed a proxy. However, it’s still a better solution than comparing real translation pairs.

13

Part 1: Sentence-level Analysis

Identity Pairs help us to analyze the encoder.

​

Given the language (1) and the language (2), we have 3 sentence-level cases:

• (i) Comparing (2) → (2) and (1) → (2) to shows target language features by SVCCA.

​

• (ii) Comparing (2) → (2) and (2) → (1) to shows source language features by SVCCA.

​

• (iii) Comparing (2) → (2) and (1) → (1) to shows language-agnostic features by SVCCA.

​

​

​

Note 1: Identity pair is underlined.

Note 2: SVCCA refers to the singular value canonical correlation analysis used to compare the similarity between two vectors. We follow Liu et al. (2021) to compare sentence-level representation by mean-pooling all tokens.

14

Part 1: Sentence-level Analysis

Layer-wise results of 6 encoder layers:

Layer-wise results of 8 and 10 encoder layers:

Higher scores indicate the corresponding features showing more in the representation.

• Target language
• Source language
• Language-agnostic

15

Part 1: Sentence-level Analysis

What phenomena that we can observe?

• The feature of target language is gradually increasing; source is gradually decreasing.

​

• At the output of the encoder:
• target language feature is superiority
• source language feature and language-agnostic feature are not much different.

We can conclude:

​

Translation representations are indeed transferred to the target language layer-by-layer.

​

This supports and supplements the 1^st opinion [Kudugunta et al. (2019)].

​

However, how to explain the semantic alignment?

16

Part 1: Token-level Analysis

Token-level analysis by t-SNE in TED-19:

4 real sentences translating to English, and the identity pair belongs to English.

Embedding Layer

var. of 1.45

Encoder Output

var. of 0.09

Language Transfer means:

Representations from different languages are semantically aligned at the representational subspace of the target language.

​

The discrepancy is two sides of the same thing!

17

Part 1: Significance

This analysis can help us to figure out the deficiency of zero-shot translation.

• Representations of identities are uniformly distributed.
• Representations of supervised translations are similar.
• Representations of zero-shot translations are entangled.

This phenomenon fits the performance scoring:

​

Identity pairs> supervised pairs > zero-shot pairs.

18

Part 1: Verification

We design a method as an engineering practice to verify our conclusions.

​

Low-rank Language-specific Embedding (LoLE)

• Initializing a set of embedding specified to each language.
• Biasing the token representation at the top-2 layer of encoder by the target language.
• Only the head portion is biased.

19

Part 1: Verification

Experiments on Europarl-15, where the performance (BLEU scores) and features (SVCCA scores) are shown at the language-family level.

The result of Pearson Correlation Analysis proves they are positively related.

20

Part 1: Verification

After

After

21

Part 1: Summarization

What role does the encoder play in MNMT?

​

​

Our answer: language transferred within the encoder.

​

• Representations are transferred into the target language subspace.

​

• Semantics are aligned at the target language subspace.

22

Part 2

Now, we understand the success of Encoders in MNMT, so how can we use it?

​

Part 2, From Strengths to Decoder-only Alternatives:

​

​

Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation.

​

Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Taro Watanabe.

​

ACL 2025 Oral.

23

Part 2: Problem

The deficiency of decoder-only architecture in traditional MNMT model is caused by lack of language transfer. [Qu et al., 2024]

• Higher than 0.5 means the representation has much more target language features;

​

• Lower means more source language features.

Previous works always focus on how to reinforce the signal of instruction.

​

• Expensive, such as language-specific modules;

​

• Cannot achieve a strict distinguishing among languages.

24

Part 2: Motivation

A contrary motivation: If the generation process do not see the source token, it would not be influenced by the source language!

​

How to achieve that? Inserting a set of registers to bridge source and target tokens.

​

The generation only relies on the register pointing to the target language.

Pointing to target language!

The attention from target tokens to source tokens is forbidden!

25

Part 2: Methodology

1. Create Registers:
• Duplicate the language tag multiple times to match length(source tokens), named registers.
2. Build Input:
• Insert registers in the input sequence between the source and target tokens.
3. Constrain by attention mask:
• Removing source tokens from the view of target tokens, the generation of target tokens solely relies on the activation of registers.

​

Registering: based on a decoder-only model.

26

Part 2: Experimental Setup

Comparing with related works on EC-40, a large-scale benchmark.

• EC-40 is English-centric for maximally measuring zero-shot translation ability.
• Two cases (12 and 24 layers) for showing scalability.
• Compared to prior SOTA in enc-dec and dec-only architectures, respectively.

Pre-training models: MITRE (Multilingual Translation with Registers)

• Two versions: 466M and 913M
• Features:
• Supporting arbitrary translations among 24 languages;
• Training on 80 V100 GPUs with data as little as possible;
• Better than M2M-100 (483M, 615M, 1.2B), NLLB-200 (615M, 1.3B, 3.3B), GPT-3.5 turbo and GPT-4o mini.

We have three sets of experiments

Fine-tuning MITRE

• Better than NLLB with full parameters and LoRA.
• Three scenarios consist of 5/25/100 random translation directions.
• Training data is the dev set of Flores++.

Details can be found in our paper!

27

Part 2: Experimental Setup

Evaluation:

• Inference on devtest of Flores+ with beam size of 5.

​

• 3 auto metrics:
• spBLEU, an improved version of SacreBLEU by unifying tokenization by an open-source SentencePiece model.
• chrF++, like BLEU but at the character-level.
• COMET, evaluating the semantics at the representation-level with considering reference and source sentence together.

​

• A secondary metric:
• Off-Target Ratio, indicating the ratio of inference failing to reach the expected target language.

28

Part 2: Experimental Result

A new state-of-the-art compared to related works!

On benchmark

29

Part 2: Experimental Result

MITRE-913M outperforms NLLB 3.3B (+1.14 scores on average) and performs competitively with GPT-4o mini.

Pre-trained models (MITRE vs M2M/NLLB/GPT)

30

Part 2: Experimental Result

Fine-tuning (MITRE vs NLLB)

MITRE series have a strong adaptability in fine-tuning!

31

Part 2: Ablation Study

Attention mask is the factor of improvements, instead of the additional computation costs.

32

Part 2: Ablation Study

Registering, len(x) / len(r) = 1, is the optimal setting.

​

When fixing the length of source tokens (x):

​

len(x) / len(r) > 1 → compression

​

len(x) / len(r) < 1 → augmentation

33

Part 2: Ablation Study

The success of registering, i.e., ratio of 1, comes from the higher utilization on source tokens.

​

1. For each register, we find the source tokens with top-1 and top-2 attention.

​

• We compute the distance between these two source tokens.

​

• Entropy is computed by showing the frequency of each one source token is selected as the top-1 attention.

34

Part 2: Mechanism

Attention weights of two translation instances from German to English. (token-level)

Translating to the same target tokens, but source tokens are different in their syntax.

The register mirrors the semantics of the positionally-aligned source token.

35

Part 2: Mechanism

2D distribution of token representations from 3 random translation directions.

• Random 100 instances, then reduce the dimension by t-SNE.
• Illustrate random 300 tokens for each class.

Observations:

• Source tokens, registers, and target tokens are divided into 3 spaces.
• Source tokens are language-agnostic.
• Registers and target tokens are divided into different subspaces corresponding to the target languages.
• (Appendix D shows the variation from Embedding layer to Top layer.)

The representation of registers is located in the different target language space.

36

Part 2: Mechanism

Sentence-level analysis can validate our statement at the token level.

Trends:

• At lower layers, source and target tokens have the highest similarity, i.e., lower layers primarily encode shared semantic content;
• As depth increases, source-target similarity drops noticeably, suggesting that upper layers capture more language-specific features;
• Source-register similarity remains relatively stable until the top layers, where it declines sharply. In contrast, register-target similarity gradually decreases in the middle layers but rises sharply in the top layers;
• In the final layer, registers are most similar to the target, followed by the source, while source-target similarity is lowest.

Registering source tokens to target language spaces!

37

Part 2: Enc-dec v.s. Dec-only

Answer is NO!

​

When implement Registering into MNMT-specific models on the benchmark.

​

Decoder-only indeed has better parameter-efficiency and shows higher improvement than Encoder-decoder.

Back to the first question!

Is the decoder-only architecture really weaker than encoder-decoder in MNMT?

38

Summarization

We investigate and understand the success of encoders in MNMT-specific models.

Based on the conclusion, decoder-only can be better in MNMT-specific models.

A remained challenge:

This is the era of LLMs.

​

Can we utilize the pattern (not only registering but also further methods) to benefit converting LLMs to MNMT?

39

Thank you for your listening!

Q&A