Fact Aware Multi-Task Learning for Text Coherence Modeling

Tushar Abhishek, Daksh Rawat, Manish Gupta, Vasudeva Varma

tushar.abhishek@research.iiit.ac.in, daksh.rawat@students.iiit.ac.in, manish.gupta@iiit.ac.in, vv@iiit.ac.in

What is textual coherence?

• Information flow doesn’t break while transitioning from one sentence to the next sentence
• Helps the reader to understand the text/paragraph as a whole rather than a series of disparate sentences.
• Vital for success of NLG systems (summarization, question answering, question generation, …)

Coherent Text

1. John went to his favorite music store to buy a piano.
2. He had frequented the store for many years.
3. He was excited that he could finally buy a piano.
4. He arrived just as the store was closing for the day.

Incoherent Text

1. John went to his favorite music store to buy a piano.
2. It was a store John had frequented for many years.
3. He was excited that he could finally buy a piano.
4. It was closing just as John arrived.

​

Previous work

• EGRID: builds an entity grid which is a matrix that tracks entity mentions over sentences. Random forest classifier is trained over features extracted from entity grid.
• CNN-Egrid: local coherence model that employs a CNN that operates over the entity grid representation.
• PARSEQ: three stacked LSTMs to represent sentence, paragraph and document.
• Hierachical LSTM: Similar to PARSEQ. Uses attentional BiLSTMs
• Local Coherence Discriminator (LCD-L): uses max-pooling on the hidden state of the language model to get the sentence representation. Representations for two consecutive sentences are concatenated and fed to a dense layer
• LCD BERT: LCD-L but uses averaged BERT (instead of GloVe) embeddings as the sentence representations.
• LCD RoBERTa: Uses RoBERTa embeddings
• LC (Local coherence): sentences are encoded with a recurrent or recursive layer and a filter of weights is applied over each window of sentence vectors to extract scores that are aggregated to calculate overall document coherence score.

​

Previous work

• Coh+GR: extends Hierachical LSTM by training it to predict word-level labels indicating the predicted grammatical role (GR) type, along with the document-level coherence score.
• Coh+GR BERT: Similar to Coh+GR, BERT embeddings are used instead of GloVe as input to BiLSTMs.
• Coh+SOX: same as Coh+GR where, for each word, we only predict subject (S), object (O) and 'other' (X) roles.
• Seq2Seq: two LSTM generative language models and uses the difference between conditional log likelihood of a sentence given its preceding/succeeding context, and the marginal log likelihood of the current/next sentence.
• Unified: Uses a combination of LSTMs and CNNs.
• Inc-lex-Coh: extracts sentence representations using a pretrained language model and combines the semantic centroid vector with semantic similarity vector to obtain coherence output.
• Avg-XLNET-Doc: Encodes an text content at the document level and averages the encoded representations.
• Avg-RoBERTa-Doc: Same as Avg-XLNET-Doc. Uses RoBERTa embedding instead of XLNET.

​

Proposed Transformer based architectures

1. Vanilla Transformers
2. Fact-aware Transformers
3. Fact-aware Multi-Task Learning (MTL) Transformers

​

RoBERTa (Liu et al., 2019) for short sequences (less than 512 tokens)

Longformer (Beltagy et al., 2020) for very long sequences (up to 2048 tokens)

​

1. Vanilla Transformer

• Feed the input text directly to a Transformer model.
• Use the task specific output vector for training on different tasks. �

2. Fact-Aware transformers

• Input = text + facts
• Extract Facts (<subject, verb, object>) using MinIE (Gashteovski et al., 2017).
• 3 modules
• Document encoder: encodes text
• Fact encoder
• Encodes each fact separately.
• Shares weights with document encoder
• Fact-aware document encoder
• Encodes output from document and fact encoders
• Outputs final document representation.

3. Fact-Aware Multi-Task Learning (MTL) Transformers

• Extension of fact-aware Transformer-based method
• Multi-task learning
• Train for text coherence and Natural Language Inference (NLI) together

Combining coherence specific loss and NLI-based task loss

Text coherence evaluation tasks

1. Sentence ordering:
• Positive: Every original document is assumed to be coherent.
• Negatives: 20 random permutations (different from the original document) of sentences in the document.
• Rank the original document higher than the permuted ones.
2. 3-way classification:
• Classify document into one of the three different coherence levels (high, medium and low)
3. Essay Score Prediction:
• Assign an automatic score for a given essay

Text coherence datasets

• Wall Street Journal (WSJ) Dataset:
• Part of the Penn Treebank (Elsner and Charniak,2008; Nguyen and Joty, 2017)
• Contains long articles without any constraint on style.
• Used for sentence ordering task
• Used sections 00-13 for training and 14-24 for testing (documents with just 1 sentence are removed).

Text coherence datasets

• Grammarly Corpus of Discourse Coherence (GCDC) Dataset (Lai and Tetreault, 2018)
• Contains texts from four domains: Yahoo online forum posts, emails from Hillary Clinton’s office, emails from Enron and Yelp business reviews
• Each document is annotated with a coherence score.
• Used for 3-way classification

​

Text coherence datasets

• Automated Student Assessment Prize (ASAP) dataset
• Taken from the Kaggle competition
• The essays are associated with scores given by humans
• Essays are categorized in eight prompts based on essay topic and genre.
• Used for Essay Score Prediction task

​

​

Experimental setup

• For all tasks except sentence ordering, we pass the document representation obtained from proposed models to a�dense layer with ReLU activation which is then connected to a task-specific output layer.
• Reported results are the mean of 10 runs with different random seeds.
• For MTL based Transformers, categorical cross entropy loss was used for NLI.
• For Longformer, we fixed maximum sequence length to 2048. For RoBERTa, we fixed it to 512.
• For sentence ordering task, we apply Siamese network (Bromley et al., 1993) or twin neural network approach with each of our four proposed architectures. ���

Results: Sentence ordering on WSJ

• Pair-wise ranking accuracy (PRA)
• Fact aware transformer > vanilla transformer
• Fact-aware MTL model > other variants

Results: 3-way classification on GCDC

• 3-way classification accuracy
• Fact-aware model > vanilla model across all the domains
• transitions of facts is important
• Classifying documents of medium level coherence is hard
• Less samples.

Results: Automated Essay Scoring (ASAP)

• Quadratic weighted kappa (QWK) measure
• EASE
• ranked third amongst 154 participants at ASAP
• Uses features+SVR and Bayesian Linear Ridge Regression
• Constraint MTL: constrained multi-task pairwise preference learning approach
• Attention based RCNN: Uses hierarchical sentence-document model to represent essays, using the attention mechanism to learn the relative importance of words and sentences
• SkipFlow: Uses similarity between multiple states of an LSTM over time with a bounded window

Qualitative analysis

• Similar to (Li and Jurafsky, 2017), we examine the coherence scores assigned to some artificial miniature discourses.
• Score ranges from 1 to 3�

Lexical coherence

Qualitative analysis: Temporal order

Qualitative analysis: Centering/Referential coherence

Take-aways

• Proposed a fact-aware MTL model for text coherence assessment.
• Text+Facts
• Coherence+MTL
• Works for synthetic data (WSJ) as well as real-world data (GCDC).
• Improves automated essay scoring.

​

• Future: Text coherence in an open domain setting?

THANK YOU