1 of 22

XWikiGen: Cross-lingual Summarization for Encyclopedic Text

Generation in Low Resource Languages

Dhaval Taunk | Shivprasad Sagare | Anupam Patil

Shivansh Subramanian | Manish Gupta | Vasudeva Varma

WebConf 2023

2 of 22

Wikipedia for Indian Languages

Number of Wikipedia articles per language

Log of number of new articles created or edits made

Huge information divide Need to increase the human productivity

3 of 22

Encyclopedic generation

Neutral Point of View

Encyclopedic writing style

Factual correctness and grounding

Encyclopedic generation is not the same as general text generation

Wikipedia and encyclopedia specific requirements

4 of 22

Solution Approach

Generated article stubs

Use SOTA Text Generation Models

Relevant data source

Generate Wikipedia-style text in Indic languages using the created dataset with the help of multilingual language models.

For generating the articles, it is required to look for the relevant datasets. No such dataset existed.

For generating the articles, recent state of the art LLM’s can be used

These article stubs can then be reviewed and edited by humans

Created a dataset from Wikipedia articles by taking the Wikipedia article text and corresponding citation text.

5 of 22

Set of reference URLs

https://www.atptour.com/en/players/roger-federer/f324/bio
https://www.rediff.com/sports/2005/jul/04wimb1.htm
…
…
https://usatoday30.usatoday.com/sports/tennis/wimb/2005-07-03-Roddick-marvels_x.htm

Section title�<hindi> परिचय�<english> Introduction

…..�<tamil>அறிமுகம்

XWikiGen

<hindi> रॉजर फ़ेडरर (जन्म 8 अगस्त 1981) एक व्यवसायिक स्विस टेनिस खिलाड़ी हैं, जिनकी वर्तमान में एटीपी वरीयता 2 है। उनके नाम 2 फ़रवरी 2004 से 17 अगस्त 2008 तक 237 हफ़्तों तक प्रथम वरीयता पर रहने का रिकॉर्ड है। फ़ेडरर को व्यापक रूप से इस युग के महानतम एकल खिलाड़ी के रूप में जाना जाता है।

<english> Roger Federer (born 8 August 1981) is a Swiss former professional tennis player. He was ranked world No. 1 by the Association of Tennis Professionals (ATP) for 310 weeks, including a record 237 consecutive weeks, and finished as the year-end No. 1 five times.

ரோஜர் ஃபெடரர் (பிறப்பு - ஆகத்து 8, 1981) சுவிட்சர்லாந்தைச் சேர்ந்த டென்னிசு வீரர். 20 கிராண்`ட் சிலாம் எனப்படும் பெருவெற்றித் தொடர்களை வென்றுள்ளார். மேலும், மொத்தம் 302 வாரங்கள் தரவரிசைப் பட்டியலில் முதல் இடம் பிடித்தவராகவும், தொடர்ச்சியாக 237 வாரங்கள் தரவரிசைப் பட்டியலில் முதலிடம் பெற்றிருந்தமையும் இவரது முக்கிய சாதனைகளுள் ஒன்றாகும்

XWikiGen

6 of 22

Building the dataset

6

Wikidata API

Select Wikidata entities

Wikipedia dump

Extract Wikipedia articles using selected entities

Wikipedia

Article

Extract the section text

Preprocessing

Retrieve corresponding URL’s for each section

URL’s

Scrape the URL’s to get the citation text

We use Wikidata API to initially filter the domains of interest and further fetch the entities in it that have Wikipedia page in our set of languages.
Later, we use Wikipedia language-specific XML dumps to extract the Wikipedia pages of filtered entities.
We use custom regex to filter the sections and subsections till the depth of 2. More deep sections are merged in parent sections.
We use custom regex to extract the citation URLs which are present in each section in the form of wiki markup construct {{cite:...}}.
We use MediaWikiParserFromHell module in python to clean all the wiki markup in particular section and provide with clean section text.
We filter the URLs to remove the file formats other than webpage and pdf.
We use beautifulsoup in python to scrape the <p> paragraph text from above citation URLs, and pypdf2 to scrape the text from pdf.
Some files are too large(100s of pdf pages), hence we put a limit of 5 seconds while scraping each URL. We also handle common errors occurring during scraping, like non-existing pages.
Finally we tokenize the scraped text into individual sentences using universal sentence tokenizer in IndicNLP module

7 of 22

XWikiRef: Multilingual, multi-document, Multidomain Dataset

Domain

books

films

politicians

sportsman

writers

Languages

bn

hi

ml

mr

or

pa

ta

en

XWikiRef

~69K articles

~105K section specific summaries

8 of 22

Data Stats

Table showing number of wikipedia articles per domain per language in our dataset. An total of 68585 articles are present in our dataset.

8

	bn	hi	ml	mr	or	pa	ta	en	Total
Books	313	922	458	87	73	221	493	1467	4034
Film	1501	1025	2919	480	794	421	3733	1810	12683
Politicians	2006	3927	2513	988	1060	1123	4932	1628	18177
Sportsman	5470	6334	1783	2280	319	1975	2552	919	21632
Writers	1603	2024	2251	784	498	2245	1940	714	12059
Total	10893	14232	9924	4619	2744	5985	13650	6538	68585

9 of 22

XWikiGen Pipeline

2 Stage pipeline

Cross-lingual - Multi-document Summarization based approach

Approximately 90% of reference text in Indic Wikipedia is in English

Pipeline:

It is a cross-lingual multi-document summarization based approach where:

Input is

Set of citations
Section title in target language.

Extractive Summarization – Extract top K sentences.

HipoRank:

It is an unsupervised graph based extractive summarization technique.
It creates a directed hierarchical graph with sentence and section nodes and sentence-sentence and sentence-section edges with asymmetrically weighted edges.
Cosine similarity between node embeddings is used to compute edge weights. The importance score for a sentence node is computed based on a weighted sum of edges incident on the node.
We then sort these sentences in descending order based on the importance score and greedily select the top-K sentences as our extractive summary

Abstractive Summarization stage:

Finetuned (mBART/mT5) to generate section specific text because of their multilingual and cross lingual output capabilities

10 of 22

Methodology (Extractive Stage – Salience [1])

Concatenate Section title with reference text sentence.

Pass through Language Model and get the scores

Reverse sort the sentences based on scores (Cross Entropy loss)

Pick top K sentences as output of this stage.

We experimented with 2 different Extractive Summarization approaches: (1) Salience, (2) HipoRank.

Section title plays a critical role in identifying what information is available in the corresponding section text.

Therefore, we needed a way to integrate section title while performing the extractive summarization.

[1] QA-GNN: Question Answering using Language Models and Knowledge Graphs, Yasunaga et. al., NAACL 2021

11 of 22

Methodology (Extractive Stage –HipoRank [2])

It is an unsupervised graph based extractive summarization technique.

[2] Discourse-Aware Unsupervised Summarization for Long Scientific Documents, Dong et. al., EACL 2021

Directed hierarchical graph

Intra Sectional Links
Inter Sections Links

Asymmetric Edge Weighting

Over sentences
Over sections

Ranking Algorithm

Cosine Similarity

Importance Calculation

Weighted sum of inter-sectional and intra-sectional centrality scores

It is an unsupervised graph based extractive summarization technique.
The technique is divided into 4 parts:

Graph based ranking algorithm:

Calculates the cosine similarity between two nodes. Used to select most salient sentences from the input graph.

Hierarchical Document Graph Creation: Builds a hierarchical graph by creating intra and inter sectional links of sentences from sections.

Intra Sectional Links: Signifies the local importance of a sentence within it section. Each node here encodes the sentence information of that section. All the sentences are connected to other sentence within the same section.
Inter Sectional Links: Signifies the global importance of a sentence w.r.t. other sections in the document. Each node here encodes the entire sectional information

Asymmetric Edge Weighting by Boundary functions

To calculate the weight of an edge, similarity between a sentence-sentence pair sim(v_j ^I, v_i ^I) and a section-sentence pair sim(v^J , v^I_i ) is measured.
Asymmetric edge weighting over sentences:

It is is done by giving more weightage to sentences close to section boundary.

Asymmetric edge weighting over sections:

It is done in similar way by giving more weightage to sections that appear near document boundary.

Importance Calculation:

It is calculated as weighted sum of its inter-sectional and intra-sectional centrality scores.

Based upon these centrality scores, top-K sentences are being taken as output of this approach.

12 of 22

Underline pre-trained models for abstractive stage

We experiment with both mT5 and mBART in our abstractive summarization stage to generate the Wikipedia article text.

mBART

Multilingual variant of BART.
Autoregressive Seq2seq model
Seq2Seq denoising Auto-Encoder

Multilingual Variant of T5
Pre-trained on Common Crawl dataset.

mT5

mBART is a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora in many languages using the BART objective. The input texts are noised by masking phrases and permuting sentences, and a single Transformer model is learned to recover the texts. Different from other pre-training approaches for machine translation, mBART pre-trains a complete autoregressive Seq2Seq model. mBART is trained once for all languages, providing a set of parameters that can be fine-tuned for any of the language pairs in both supervised and unsupervised settings, without any task-specific or language-specific modifications or initialization schemes.

A Denoising Autoencoder is a modification on the autoencoder to prevent the network learning the identity function. Specifically, if the autoencoder is too big, then it can just learn the data, so the output equals the input, and does not perform any useful representation learning or dimensionality reduction. Denoising autoencoders solve this problem by corrupting the input data on purpose, adding noise or masking some of the input values.

The mT5 is a multilingual variant of Google's T5 model that was pre-trained over a dataset of more than 101 languages and contains between 300 million and 13 billion parameters. The architecture and training of the model followed for mT5 closely emulates that of T5.

Baseline Models:

mBART : 12 encoder + 12 decoder layers
mT5 : 12 encoder + 12 decoder layers

Fine-tuned these models on the output data taken from extractive stage.

13 of 22

Experimental Settings

Combined all languages across each domain

Multi-lingual setting

Combined all domains across each language

Multi-domain setting

Combined all languages and domain pairs

Multi-lingual - Multi-domain setting

14 of 22

Evaluation Metrics

ROUGE

chrF++

METEOR

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics and a software package used for evaluating automatic summarization and machine translation software in natural language processing. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation.
Note that ROUGE is case insensitive, meaning that upper case letters are treated the same way as lowercase letters.

ChrF and ChrF++ are two MT evaluation metrics. They both use the F-score statistic for character n-gram matches, and ChrF++ adds word n-grams as well which correlates more strongly with direct assessment.

METEOR, an automatic metric for machine translation evaluation that is based on a generalized concept of unigram matching between the machine-produced translation and human-produced reference translations. Unigrams can be matched based on their surface forms, stemmed forms, and meanings; furthermore, METEOR can be easily extended to include more advanced matching strategies. Once all generalized unigram matches between the two strings have been found, METEOR computes a score for this matching using a combination of unigram-precision, unigram-recall, and a measure of fragmentation that is designed to directly capture how well-ordered the matched words in the machine translation are in relation to the reference.

ROUGE is a standard metric for summarization-based tasks which considers recall as a parameter. Since it is a standard metric for this task, we had to use it otherwise reviewers would have asked why we didn’t use it.
METEOR on the other hand is considered as more correlated with human judgement as it considers F1 over word n-grams overlap between the generated summary and the reference summary. Although it is more suited for machine translation task but people use it for summarization as well.
chrF++ is one step ahead of METEOR as it considers character level n-grams along with the word level n-grams for calculating the scores.

15 of 22

Results

15

Overall results for all experiment settings

16 of 22

Results

16

Detailed results for multi-lingual - multi-domain setting (HipoRank + mBART)

17 of 22

Example Predictions

17

18 of 22

Example Predictions

18

19 of 22

Example Predictions

19

20 of 22

Contributions

These models significantly reduce the manual efforts required in writing a Wikipedia article and thus, help the community in efficiently enhancing the content in Wikipedia for LR Language.

The XWikiGen pipeline to generate Wikipedia article from citations.

The XWikiRef dataset for the task of cross lingual multi document summarization.

1

2

Codebase: https://github.com/DhavalTaunk08/XWikiGen

Corresponding author: Dhaval Taunk (dhaval.taunk@research.iiit.ac.in)

21 of 22

Thank you

22 of 22

22

Example Figure