Neural Machine Translation English-Bengali

Rishi Dey Chowdhury

Ayush Bilkhiwal

Sujeet Kumar

Confidential

Customized for Lorem Ipsum LLC

Version 1.0

Hello! হ্যালো!

How are you? তুমি কেমন আছো?: 0.80, তুমি কেমন আছ?: 0.70 তুমি কেমন আছো?

(English Input) (Model generated scores) (Reference)

Confidential

Customized for Lorem Ipsum LLC

Version 1.0

Outline

Overview

Problems to solve

Project objective

Strategy

Model Architecture

Hyperparameters

Model Performance

Translation Results

Conclusion

Confidential

Customized for Lorem Ipsum LLC

Version 1.0

Overview

• Language being the heart of communication, with the increase in computational power and growing need for conversion of English content to Indic Languages to make it more accessible to local people, Neural Network based methods have taken over the existing statistical methods to generate better machine(automated) translations.
• We here explore one such direction of Neural Machine Translation to convert English Content to North-Eastern Language (in our case Bengali).

Project objective

• Creating Baseline Systems using different architectures for Language Pair.
• Identification of Features which can be used in Building Feature based MT systems.

Literature

Survey

• A detailed search for previous works related to the field of Neural Machine Translation Applications for English to Bengali and vice-versa translation, revealed very few approaches being experimented with these Language Pairs.
• The methods adopted in all these papers mostly included Simple RNN, LSTM and GRU [1].
• Other efforts involve using Back Translation to increase the amount of training examples to better train a model [2].
• As per our search only one paper used BiLSTM with attention and Self-Attention based Modern Transformer Based Architecture [3].
• This is mostly due to lack of availability of large parallel corpus.��

Literature Survey

Datasets

• English-Bangla (en-bn) dataset from Samanantar and  WAT Indic (Workshop on Asian Translation).
• The Samanantar dataset contains 92,51,702 parallel sentences. WAT Indic dataset contains 2390 parallel sentences.
• Various domains such as news, sports, tech, entertainment, lifestyle, education, business, and general.
• Training
• 1,54,836 parallel sentences from Samanatar for running our small model.
• 7,28,047 parallel sentences from Samanatar for running our large model.
• Number of unique words in English and Bengali data are 1,16,349 and 1,48,459 respectively. �

Datasets

• Validation
• 3694 sentences taken from Samanatar.
• 1194 sentences from Indic. WAT
• Testing
• 1194 sentences from Indic. WAT

�

Data Preprocessing

• Byte-Pair Encoding to tokenize both the train and test data.
• Vocabulary size of 32,000 for both English and Bengali language.
• Normalization
• Converting the sentences to lower case and then by using standard unicode normalization.
• Tokenization
• Adding the bos (beginning of sentence) token at the start and the eos (end of sentence) token and end of each encoded sentence in both the language.
• Adding unk (unknown) for the unknown subwords encountered and pad (padding) token for padding the sentences.

Output Sentence Selection

Neural Network is just the way to calculate the conditional probability of the next word that comes in the output sentence given the previous word generated in the output sequence till now and the input sentence in the case of Machine Translation. Now, we can opt for several ways, like Greedy Search, Beam Search, Minimizing Baye's Risk,etc. to find the output sequence with the highest joint probability. Looking for all possible combinations is computationally very expensive. Hence, we resort to heuristic and asymptotically best methods to generate the output sequence with highest probability. We resorted to two ways which are:

• Greedy Search: Picks the next id with the highest softmax probability. Though it works for short sentences it is not suitable for many cases.
• Minimum Baye's Risk(MBR): Generates multiple candidate translations. Compare each one of them with all other using a similarity score(in our case ROUGE). Then, choosing the one with the highest similarity score, gives us the candidate translation that is in consensus with all the generated samples.

​

ROUGE SCORE =

MBR Selection Criterion:

MODELS

Architecture

1 Layer Transformer Architecture

​

​

We changed the Hyperparameters of the Transformer and trained 2 of these models to compare their performance.

​

With Attention head 0

Reference: https://jalammar.github.io/illustrated-transformer/

heads(0-7)

Feed Forward Neural Network

Reference: https://jalammar.github.io/illustrated-transformer/

With Multi headed Self Attention

​

Here we are using 2 heads we get the importance of tired word as well as the importance of animal word.

If we add all the attention heads to the picture, however, things can be harder to interpret

Reference: https://jalammar.github.io/illustrated-transformer/

Architecture

Architecture

2 Layer Transformer Architecture

​

​

We changed the Hyperparameters of the Transformer and trained 4 of these models to compare their performance.

​

Architecture

Architecture

3 Layer Transformer Architecture

​

Hyperparameters:

​

Number of Layers = 2

Number of Heads = 4

Embedding Dimension = 256

Key Dimension = 32

Value Dimension = 32

Parallel Sentences = 1,54,836

Epochs = 10

Batch Size = 256

Architecture

4 Layer Transformer Architecture

​

​

We changed the Hyperparameters of the Transformer and trained 2 of these models to compare their performance.

​

Architecture

RESULTS

Results

• For 1 Layers we look at the comparison between 8 Heads and 8 Heads Big’s accuracy

​

For smaller dataset

For larger dataset

Results

• For 1 Layers we look at the comparison between 8 Heads and 8 Heads Big’s Loss

For smaller data

For larger data

Results

Manual Evaluation of 8 Heads on Test Data

Manual Evaluation of 8 Heads on Test Data

Results

Results

• For 2 Layers we look at the comparison between 4 Heads and 8 Heads’ Accuracy

Results

• For 2 Layers we look at the comparison between 4 Heads and 8 Heads’ Loss

Results

• For 2 Layers we look at the comparison between 8 Heads and 8 Heads Big’s Accuracy

Results

• For 2 Layers we look at the comparison between 8 Heads Big and 8 Heads Dim’s Accuracy

And the same trend holds for losses as well.

Results

Manual Evaluation of 8 Heads Big on Test Data

Manual Evaluation of 8 Heads Big on Test Data

Results

Results

• For 3 Layers & 4 Heads

Results

Manual Evaluation of all examples gives 0 Adequacy and 0 Fluency

Results

• For 4 Layers we look at the comparison between 4 Heads and 4 Heads Big’s Accuracy

Results

• For 4 Layers we look at the comparison between 4 Heads and 4 Heads Big’s Loss

Results

Manual Evaluation of all examples gives 0 Adequacy and 0 Fluency

Conclusion

• Model faces issues in fitting when it grows in complexity but has less data available to train all the parameters.
• Out of all the models trained it is quite starking that the one with the least complexity seems to perform the best that is the transformer with 1 Layer and 8 Heads. The reason might be due to not that big training data.
• The model exposed to larger data performs better than the one exposed to less data.
• Model trained on larger data is able to translate named entity better, even if it hasn't seen it before in the data. (Like my Name Rishi, Places name like Hyderabad)
• The Model learns associations between words quite well e.g. Modiji is converted to Narendra Modi in translations due to its appearance multiple times in dataset.
• Larger Models tend to robust to punctuation marks.
• Model faces issue in discriminating between spellings like ভাল and ভালো.

�

English Bengali Translation: MBR Score Reference Translation

​

​

Thank You! ধন্যবাদ!: 0.99, ধন্যবাদ তোমার!: 0.72 ধন্যবাদ!

​