SEQ2SEQ++: A MULTI-TASKING BASED SEQ2SEQ MODEL �TO GENERATE MEANINGFUL AND RELEVANT ANSWERS

PhD Candidate

Kulothunkan Palasundram (GS50783)

​

Supervisory Committee

Associate Prof. Dr. Nurfadhlina Mohd Sharef

Associate Prof. Dr. Azreen bin Azman

Dr. Khairul Azhar bin Kasmiran

​

University Putra Malaysia

Faculty of Computer Science & Information Technology

Outline

Introduction

Chatbot Types and Research Scope

Introduction

Seq2Seq is a transformation of a sequence of words (question) into another sequence of words (answer)

Seq2Seq Learning

Introduction

Seq2Seq Learning

Research Problem – Key Observations

• Seq2Seq – popular method for natural answer generation
• However, the generated answer may not be relevant to the question
• Result - conversations with chatbots can be meaningless, abruptly terminated by users, and eventually lowers the chatbot adoption rate

Research Objective

Propose an MTL-based Seq2Seq model that can generate meaningful and relevant answers

​

Main Objective

Research Scope

Question-answering as a single turn conversation task (a pair of question and answer) under the Multi-task Learning (MTL ) framework as defined in (Huang & Zhong, 2018) which is a key reference for this research.

Literature Review

• 26 articles reviewed
• 3 key issues identified
• 5 methods/ approaches found

Literature Review – Issues & Methods

Literature Review – Approaches

Research Objectives - Refined

Propose an MTL-based Seq2Seq model that can generate meaningful and relevant answers by addressing the three issues which are language model influence, answer generation overfit, and question encoder overfit issues.

​

Main Objective

Sub-objective 1

address

​

New attention mechanism

​

​

Language model influence issue

​

Sub-objective 2

address

Improved MTL loss calculation method

​

Answer generation overfit issue

​

Sub-objective 3

address

​

Auxiliary tasks for MTL

​

​

Question encoding issue

​

Research Problem Statements Refined

Research Problem Statements Refined

Research Problem Statements Refined

Research Methodology - Phases

• Initial literature review 
• Research scope
• Research objectives

• Seq2Seq Issues
• Existing approaches, strengths, limitations and gaps 
• Evaluation metrics

• Benchmark models (MTL-BC, STL, MTL-LTS) 
• Proposed methods (CAM, DL, TC, MFE)
• Final model (SEQ2SEQ++)
• Data for training and evaluation

• Experiments
• Analysis
• Thesis 

Phase 1 -  �Planning

Phase 2 -  Literature Review

Phase 4 - �Experiment & Analysis

Phase 3 - Design and Development

Research Methodology - Dataset

Research Methodology - Metrics

• Bilingual Evaluation Understudy (BLEU) (Papineni et al., 2002)
• to measure answer correctness
• It gives a score of 0 to 1
• The higher BLEU score indicates generating answers that are more relevant to the question

​

• Word Error Rate (WER) (Mikolov et al., 2010)
• to measure model error
• It gives a score of 0 to 1
• The lower the score means the fewer the errors in the generated answers

​

• Distinct-2 (Li, Galley, Brockett, Gao, et al., 2016)
• to measure the diversity of generated answers
• It gives a score of 0 to 1
• The higher the score means the more diverse are the answers

Research Methodology – Proposed Methods

Research Methodology – Experiments

Proposed Methods: CAM

• Comprehensive Attention Mechanism - An attention mechanism utilized during answer generation
• In this mechanism:-
• The attention weights are computed based on all encoder’s hidden states and the sum of all the decoder’s previous hidden states
• These computed attention weights are then used to compute the context vector.
• Subsequently, the decoder utilizes the context vector to eventually generate the answer.

​

• This ensures all the hidden states are continuously considered for the next word prediction

​

​

​

Comprehensive Attention Mechanism (CAM)

Models that implemented this method are STL-CAM, MTL-BC-CAM and SEQ2SEQ++

Comprehensive Attention Mechanism (CAM) vs Global Attention Mechanism

Proposed Method:

CAM

Existing Method:

Global Attention Mechanism

Proposed Methods: DL

• Dynamic Tasks Loss Weight Scheme - An algorithm to dynamically compute the task loss weight during multi-task learning

​

• In this algorithm:-
• The task loss weight (α) for each task will be updated during each epoch
• The new task loss weights for each task will be proportional to the total model loss.
• The new weights are then used for the next epoch to compute the MTL loss

​

​

​

​

• This means that the influence of each task in each epoch is dynamically determined by the task loss in the previous epoch

​

Models that implemented this method are MTL-BC-DL, MTL-MFE and SEQ2SEQ++

Proposed Methods: DL

​

Algorithm 1: SEQ2SEQ++ Training Algorithm

Input: {Question (X), Answer(Y), Label(L), First Word (FW), Last Word(LW)} quintuplets, Maximum answer length (T), Maximum Epoch (E)

Steps:

Initialize Multi-functional Encoder (MFE), Answer Encoder(AE), Answer Decoder (AD) and Ternary Classifier (TC)

Initialize each losses LMTL = Lag = Ltc = Lfw = Llw = 0

Initialize each task loss weights αag = αtc = αfw = αlw = 0.25

​

For epoch 1 to Number of Epochs

1. For batch 1 to Number of Batches Do

1.1. Perform question encoding

1.2. Predict First Word and compute the first-word prediction losses (Lfw)

1.3. Predict Last Word and compute the last-word prediction losses (Llw)

1.4. Perform answer encoding

1.5. Perform ternary classification and compute ternary classification loss (Ltc)

1.6. Perform answer generation using CAM and compute answer generation loss (Lag)

1.7. Compute multi-task loss: LMTL = αag*Lag + αtc*Ltc + αfw*Lfw + αlw*Llw

1.8. Update the model parameters (neural network weights)

1.9. For each task, tk ⊆ {ag, tc , fw , lw} : Calculate the average loss (Ltk-avg)

End for Batch Looping

2. Calculate the total average loss for all tasks: Lavg-total = Lag-avg + Ltc-avg + Lfw-avg + Llw-avg

3. For each task, tk ⊆ {ag, tc , fw , lw}

3.1. Calculate new task loss weight: αtk = Ltk-avg / ( Lavg-total)

End for Epoch Looping

​

Output: Trained SEQ2SEQ++ model

Proposed Methods: MFE

• MFE - Multi-Functional Encoder - Performs question encoding, first-word prediction and last word prediction based on the question encoder hidden states
• MFE will learn to produce hidden states to ensure all 3 tasks have minimal losses

​

• By sharing the question encoding with multiple tasks (answer generation, first-word prediction and last word predictions), the question encoding overfit can be reduced

​

Models that implemented this method are MTL-MFE and SEQ2SEQ++

Proposed Methods: TC

• TC - Ternary Classifier- Performs ternary classification of an answer based on the question and answer encodings
• Classifies as correct, partially-correct and wrong

​

• By sharing the question encoding with multiple tasks (answer generation and answer classification), the question encoding overfit can be reduced

​

Models that implemented this method are MTL-MFE and SEQ2SEQ++

Proposed Methods: TC

Proposed Methods:

Final Model – SEQ2SEQ++

Recap – Experiments Done

Experiment Result –STL vs STL-CAM

STL-CAM over STL - Percentage Improvements

• * Negative values indicate STL performed better than STL-CAM

Experiment Result –MTL-BC vs MTL-BC-CAM

MTL-BC-CAM over MTL-BC - Percentage Improvements

• * Negative values indicate MTL-BC performed better than MTL-BC-CAM

CAM vs Global attention mechanism conclusion

• In general, CAM-based models produced answers with :-
• higher correctness (higher BLEU scores)
• lower errors (lower WER scores)
• higher diversity (higher Distinct-2 scores)

than the global attention-based models in their respective experiments

• These performance improvements are due to the reduction in the generation of frequently occurring words in answers.
• CAM can capture the representation of the answer more precisely without loss of important information by utilizing all the decoder's previously generated hidden states as the decoding progresses.
• This means CAM is a more effective attention mechanism than the global attention mechanism to address the language model influence issue.
• Both outcomes also show that CAM can be utilized and is effective in both single-task and multi-task learning frameworks

Experiment Result –MTL-BC vs MTL-BC-DL

MTL-BC-DL over MTL-BC Percentage Improvements

• * Negative values indicate MTL-BC performed better than MTL-BC-DL

Dynamic vs Fixed Tasks Loss Weight Scheme conclusion

• The overall result shows that by utilizing dynamic tasks loss weight scheme, the performance of an MTL model can be improved by the reduction in the occurrence of high-frequency words in the generated answers.

​

• In terms of BLEU and WER scores, MTL-BC-DL performed consistently better than MTL-BC.

​

• This demonstrates that by utilizing DL, answer generation overfit can be effectively reduced and the MTL-BC-DL can produce answers with higher correctness (higher BLEU scores) and lower error rates (lower WER scores).

​

• By dynamically adjusting the task loss weight during each epoch and using it for the next epoch, the model can increase learning for the task that is lagging (having higher loss) thus improving overall model learning and eventually improving model performance in answer generation.

Experiment Result –MTL-LTS vs MTL-MFE

MTL-MFE over MTL-LTS Percentage Improvements

Parallel task learning (MTL-MFE) vs sequential learning to start (MTL-LTS) conclusion

• MTL-MFE performed significantly better than MTL-LTS on all the measurements (BLEU, WER and Distinct-2).

​

• This demonstrates that by training first-word and last word prediction tasks in parallel with answer generation tasks which is a slightly more complex task, question encoder overfit can be reduced as compared to performing the first-word prediction first and then performing answer generation next

​

​

Experiment Result –MTL-BC vs MTL-TC

MTL-TC over MTL-BC Percentage Improvements

• * Negative values indicate MTL-BC performed better than MTL-TC

Ternary classifier vs Binary classifier conclusion

• MTL-TC performed consistently better than MTL-BC on BLEU and WER and with mixed result on Distinct-2

​

• This demonstrates that by utilizing a ternary classification task instead of a binary classification task, which is a more complex task, question encoder overfit can also be reduced and the models can improve answer generation quality by reducing generating high-frequency words incorrectly

Experiment Result – SEQ2SEQ++ vs interim models

SEQ2SEQ++ over Interim Models Percentage Improvements

SEQ2SEQ++ vs interim models conclusion

• SEQ2SEQ++ shows better performance than the interim models

​

• SEQ2SEQ++ can capitalize on the strengths of the individual methods which are :-
• Comprehensive Attention Mechanism (CAM)
• Dynamic Tasks Loss Weighting Scheme (DL)
• Multifunctional Encoder (MFE)
• Ternary Classifier (TC).

Experiment Result – SEQ2SEQ++ vs benchmark models

SEQ2SEQ++ over Benchmark Models Percentage Improvements

SEQ2SEQ++ vs benchmark models conclusion

• SEQ2SEQ++ shows better performance than the benchmark models. It can generate answers with higher quality (highest BLEU score), lower error rate (lowest WER score), and higher diversity (highest Distinct-2 score) in comparison with all the other benchmark models.

​

• This result indicates that SEQ2SEQ++ can address all three issues (language model influence, answer generation overfit, question encoder overfit) more effectively than the benchmark models.

Summary

• MTL framework is one of the popular methods in Seq2Seq based answer generation as it provides a mechanism to introduce one or more auxiliary tasks that can be learned in parallel with the main task of answer generation to address the limitations and issues with Seq2Seq learning.

​

• In other words, several methods can be integrated into an MTL setting to address several issues at hand.

​

• This work capitalizes on this advantage by proposing SEQ2SEQ++ to address all the three issues encountered during answer generation

Summary

Summary

Summary

Summary

Future Works

Areas that have great potential for future works :-

• Multi-turn conversation: Investigation of SEQ2SEQ++ for multi-turn conversation

• Diversity: Further investigation can be performed to further improve this method to show a much more significant improvement in diversity in comparison with other existing methods.

• Pre-trained Language models: Investigation can also be conducted on how existing pre-trained language models such as Generative Pre-trained Transformer (GPT-3)

• Pre-trained embeddings: In this research, the embedding was jointly trained. There are also pre-trained embeddings that such as Word2Vec (Mikolov et al., 2013) and Glove (Pennington et al., 2014) that can be utilized with SEQ2SEQ++ to evaluate their effectiveness.

Publications

​

1. Palasundram, K., Mohd Sharef, N., Nasharuddin, N., Kasmiran, K. & Azman, A. (2019). Sequence to Sequence Model Performance for Education Chatbot. International Journal of Emerging Technologies in Learning (iJET), 14(24), 56-68. Kassel, Germany: International Journal of Emerging Technology in Learning.

​

1. K. Palasundram, N. Mohd Sharef, K. A. Kasmiran and A. Azman, "Enhancements to the Sequence-to-Sequence-Based Natural Answer Generation Models," in IEEE Access, vol. 8, pp. 45738-45752, 2020, doi: 10.1109/ACCESS.2020.2978551.

​

1. K. Palasundram, N. M. Sharef, K. A. Kasmiran and A. Azman, "SEQ2SEQ++: A Multitasking-Based Seq2seq Model to Generate Meaningful and Relevant Answers," in IEEE Access, doi: 10.1109/ACCESS.2021.3133495.

​

Backup

Proposed Methods: MFE

Proposed Methods: TC