1 of 53

Wikimedia’s new language metrics

as a resource for insights

about (minority) languages

Caroline Myrick

Wikimedia Foundation

User:CMyrick-WMF

2 of 53

About me

Wikimedia Foundation Research staff
Volunteer editor
Academic background

Sociolinguistics (MA, PhD)
Language variation in the Caribbean and U.S.

3 of 53

Overview

Language metrics (~10 min)

Context
Celtic language insights
Language metrics project

Live demo (~10 min)
Wrap up (~5 min)

What’s next
Q&A

4 of 53

Context

5 of 53

Currently, 334 languages have at least one HOSTED content project.

6 of 53

700+ additional languages have TEST projects in Wikimedia Incubator, Wikiversity Beta, or Multilingual Wikisource

7 of 53

Many Many Lots

projects + languages = of data

8 of 53

Let’s consider the six living Celtic languages…

Gàidhlig, Scottish Gaelic
Gaeilge, Irish
Gaelg, Manx
Cymraeg, Welsh
Kernowek, Cornish
Brezhoneg, Breton

“Map of Celtic Nations-flag shades.svg” CC BY-SA 3.0 by QuartierLatin1968

File derived from: “Celtic Nations.svg” (uploaded by OsgoodeLawyer)

9 of 53

There are 17 content project editions in Celtic languages

6 Wikipedia editions

6 Wiktionary editions

2 Wikisource editions

2 Wikiquote edition

1 Wikibooks edition

10 of 53

3 Wikivoyage test projects

3 Wikisource test projects

2 Wikinews test projects

2 Wikiquote test projects

1 Wikibooks test projects

There are 11 test projects* in Celtic languages

*test projects: pre-hosted projects located in Wikimedia Incubator, Wikiversity Beta, Multilingual Wikisource

11 of 53

There are 20,000+ Commons file captions written in Celtic languages

File: CC-BY-SA-4.0 | User:Llywelyn2000

12 of 53

The Celtic languages are written in the

Latin script.

Wikimedia projects are home to more than 35 different scripts.

13 of 53

Many of the Celtic languages are translatable using MinT.

Irish, Manx, Welsh, and Scottish Gaelic can be translated into and from

>250 languages

Breton can be translated to from 2 languages

Additionally, did you know that many of the Celtic languages are translatable using Wikimedia’s MinT?

What is MinT?

The MinT Service is designed to provide translations from multiple machine translation models.
MinT supports over 250 languages, with more than 70 languages not supported by other services (including 27 languages for which there is no Wikipedia yet).

Currently via MinT,

Irish, Manx, Scottish Gaelic, and Welsh can be translated into and from >200 languages
Breton can be translated into FROM French and English

MinT uses 14 Neural MT models to cover 250+ languages. We are looking for free and open-source MT models for more languages. If a language community wishes to use a new MT model, they can get in touch with us and we can check if we can host it as part of MinT.

————————

Currently, MinT uses the following models:

NLLB-200. The latest model from the No Language Left Behind project by a research team at Meta. This model supports translation across 200 languages, including many that are not supported by other vendors.
OpusMT. The OPUS (Open Parallel Corpus) project from the University of Helsinki compiles multilingual content with a free license to train the OpusMT translation models. Anyone can easily help improve the translation quality by participating in the different projects that contribute data to OPUS. For example, when using Content Translation to create translations of Wikipedia articles, the data on published translations will be incorporated as a new resource to improve the translation quality for the next version of the model. Another quick way to contribute is to provide sentence translations with Tatoeba.
IndicTrans2. The IndicTrans2 project provides translation models to support over 20 Indic languages. These models were developed by AI4Bharat@IIT Madras, a research group at the Indian Institute of Technology, Madras.
Softcatalà. Softcatalà is a non-profit organization with the goal to improve the use of Catalan in digital products. As part of the Softcatalà Translation project, translation models used in their translator service to translate 10 languages to and from Catalan have been released.
MADLAD-400. MADLAD-400 is a multilingual machine translation model by Google Research that supports 419 languages.

You can read more about the initial release of MinT and check some frequently asked questions in the summary page for the service.

14 of 53

Content Translation is available² for each Celtic language Wikipedia.

² For logged in users. In some languages it must be enabled as a beta feature, and in others it is a usual user preference enabled by default.

Speaking of translation, did you know the Content Translation tool is available for Wikipedias in all languages for logged-in users? Which means it’s available for each Celtic language Wikipedia.

As we’ve heard about in multiple presentations this week, this tool allows editors to create translations right next to the original article and automates the tedious steps: copying text across browser tabs, looking for corresponding links and categories, etc.

The tool also integrates tools like machine translation services. Those are not supported in all languages, but the tool can be extended to include more.

(Source: https://www.mediawiki.org/wiki/Content_translation)

——————

The Content Translation tool is still in development, but has been used already to create over a million articles resulting in reported improvements in translators' productivity.^[1]^[2]

Content Translation integrates tools such as dictionaries or machine translation (MT) services. Those are not supported in all languages, but the tool can be extended to include more.

15 of 53

Content Translation is available* for each Celtic language Wikipedia.

	languages translated into	languages translated from
Breton	15	11
Cornish	6	10
Irish	18	21
Manx	3	5
Scottish Gaelic	8	16
Welsh	29	34

16 of 53

Where can we find data about languages (or a specific language) within or across Wikimedia projects?

Where did I find these Celtic language stats?

17 of 53

Language metrics

project

18 of 53

Language-related data live in many places.

Special:SiteMatrix

(meta-wiki)

Incubator:Site creation log

(Incubator)

19 of 53

Requests for new languages

(meta-wiki)

Proposals for closing projects

(meta-wiki)

Language-related data live in many places.

20 of 53

Language-related data live in many places.

canonical-data/wikis.tsv (Wikimedia Gitlab)

21 of 53

Language-related data live in many places.

Analytics Data Lake

(Presto/Hive/Spark/data dumps)

MediaWiki database

(MySQL/MariaDb)

“Knowledge Gaps” datasets

(data dumps)

22 of 53

Wikimedia Statistics

(wikimedia.org)

Wiki Comparison (spreadsheet)

Incubator Dashboard (Toolforge)

Language-related data live in many places.

23 of 53

Language-related data live in many places.

language-data/..langdb.yaml

(Wikimedia Github)

Content Translation table

(Wikimedia Extension database)

MinT.yaml

(Phabricator)

24 of 53

Language-related data live in many places.

(And many more!)

25 of 53

TEST WIKI DATA

HOSTED WIKI DATA

BACK-END DATA

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

26 of 53

TEST WIKI DATA

HOSTED WIKI DATA

STATE OF LANGUAGES METRICS

LOTS OF DATA PROCESSING

BACK-END DATA

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

27 of 53

Language coverage across

hosted content projects?

STATE OF LANGUAGES METRICS

Scripts

and

directionality?

MiNT

availability?

Search support?

Language coverage in

Incubator?

Language coverage across multilingual projects?

Language coverage in Wikiversity Beta?

Test project graduation

rates?

Language coverage in Multilingual Wikisource?

Length of

time spent

Incubating?

Machine translation availability?

Content translation availability?

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

28 of 53

At the Wikimedia Foundation, these metrics will be able to help us…

Track language coverage and representation

Identify gaps in the “sum of all knowledge” and monitor gap closure

Identify knowledge gaps and monitor gap closure

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

29 of 53

At the Wikimedia Foundation, these metrics will be able to help…

30 of 53

Wikimedia volunteers could use these metrics to…

View language coverage trends

Identify gaps for individual volunteer work, or for campaigns and events

Track impact after interventions

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

31 of 53

Demo

https://public-paws.wmcloud.org/69910522/language_metrics/explorer/home.html

32 of 53

Gitlab repo

Explorer

33 of 53

What’s next?

34 of 53

There’s more work to be done

Develop additional metrics

Productionize datasets

Incorporate and/or connect to additional data

Standardize reporting

.“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

35 of 53

HOSTED WIKI DATA

TEST WIKI DATA

MORE INSIGHTS ABOUT

STATE OF LANGUAGES

ENGINEERING DATA

PRODUCT DATA

.“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

36 of 53

HOSTED WIKI DATA

TEST WIKI DATA

MORE INSIGHTS ABOUT

STATE OF LANGUAGES

GLOBAL LANGUAGE DATA

“Languages_world_map.svg” | GFDL CC-BY-SA-migrated-3.0: Julien1311

“Funnel (PSF).png” | Public Domain: Pearson Scott Foresman

ENGINEERING DATA

37 of 53

Q&A

38 of 53

Questions or comments about these metrics?

How could you use these (or other) metrics?

Questions about the coverage of Celtic languages which weren’t covered?

39 of 53

Learn more

Research:Incubator and language representation across Wikimedia projects

Leave feedback

Discussion page

gitlab.wikimedia.org/repos/research/incubator-data-exploration

Language Metrics Explorer

40 of 53

Thanks for your attention!

Get in touch with me:

Caroline Myrick

Email: cmyrick@wikimedia.org

Office hours: mediawiki.org/wiki/Wikimedia_Research/Office_hours

41 of 53

42 of 53

APPENDIX

43 of 53

Metrics Tables

44 of 53

45 of 53

46 of 53

47 of 53

48 of 53

Additional language metric visualizations

49 of 53

Of the 333 …

Most have 1 or 2 content projects.

Thirteen have all 8 content projects.

50 of 53

Most of those languages have 1 or 2 projects.

Thirteen have all 8 content projects.

51 of 53

Test wikis outnumber hosted wikis across project types

52 of 53

Celtic language	Wikibooks	Wikinews	Wikipedia	Wikiquote	Wikisource	Wikiversity	Wikivoyage	Wiktionary
Breton	–	–	Hosted	Hosted	Hosted	–	Test	Hosted
Cornish	–	–	Hosted	Test (closed)	Test	–	–	Hosted
Irish	Test (closed)	–	Hosted	Test (closed)	Test	–	Test	Hosted
Manx	–	–	Hosted	–	–	–	–	Hosted
Scottish Gaelic	–	Test	Hosted	–	Test	–	–	Hosted
Welsh	Hosted	Test	Hosted	Hosted	Hosted	–	Test	Hosted

1 of 53

2 of 53

3 of 53

4 of 53

5 of 53

6 of 53

7 of 53

8 of 53

9 of 53

10 of 53

11 of 53

12 of 53

13 of 53

14 of 53

15 of 53

16 of 53

17 of 53

18 of 53

19 of 53

20 of 53

21 of 53

22 of 53

23 of 53

24 of 53

25 of 53

26 of 53

27 of 53

28 of 53

29 of 53

30 of 53

31 of 53

32 of 53

33 of 53

34 of 53

35 of 53

36 of 53

37 of 53

38 of 53

39 of 53

40 of 53

41 of 53

42 of 53

43 of 53

44 of 53

45 of 53

46 of 53

47 of 53

48 of 53

49 of 53

50 of 53

51 of 53

52 of 53

53 of 53