1 of 7

Reference Risk �A Research-Based Approach to�Source Reliability in Wikipedia

Pablo Aragón, Research Scientist, WMF

2 of 7

“Verifiability means that people can check that facts or claims correspond to reliable sources.”
“Even if you are sure something is true, it must have been published in a reliable source before you can add it.”

https://en.wikipedia.org/wiki/Wikipedia:Verifiability

Verifiability (WP:V)

ㅤJost Amman (Public Domain)ㅤ

3 of 7

Perennial Sources List (RSP)

https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources/Perennial_sources

4 of 7

RSP is very effective in English Wikipedia

The number of articles with references to deprecated and blacklisted sources suddenly decreased after the introduction of RSP.�Baigutanova et al. (2023). Longitudinal Assessment of Reference Quality on Wikipedia. �ACM TheWebConf 2023. https://doi.org/10.1145/3543507.3583218

RSPs in other language editions are sparse or missing

Only 14 language editions maintain a RSP.
Except for English Wikipedia, RSPs are generally not comprehensive.�Baigutanova et al. (2023). A Comparative Study of Reference Reliability in Multiple Language Editions of Wikipedia. �ACM CIKM 2023. https://doi.org/10.1145/3583780.3615254

Our team, the Research team at the Wikimedia Foundation, has been studying verifiability through source reliability.

In 2023, a study of reference quality over time found that the perennial source list has been very effective in English Wikipedia:

- “the number of articles with references to deprecated and blacklisted sources suddenly decreased after the introduction of the perennial source list”.

A following study of source reliability across language editions of Wikipedia found that perennial sources lists in other languages are sparse or missing.

- “Only 14 language editions maintain a RSP”

- “Except for English, these other RSPs contain a very limited number of annotated web domains.”

A suggestion of this second study was to analyze the patterns of how editors from different language editions handle unreliable sources.

5 of 7

Reference Risk

The permanence/survival of a web domain: the average proportion of edits in which the domain remained in articles after its initial addition, for a given language edition.

The heuristic is based on features evaluated in machine learning experiments using the English Wikipedia RSP as the ground truth.

D’Ignazi et al. (2026). Language-Agnostic Modeling of Source Reliability on Wikipedia. ACM TWEB. https://doi.org/10.1145/3777444

6 of 7

Sources used by at least 500 French Wikipedia editors

High risk: top 10		Low risk: top 10
domain	survival_avg	domain	survival_avg
fr.wikipedia.org	0.16	inpn.mnhn.fr	0.96
en.wikipedia.org	0.33	rottentomatoes.com	0.95
de.wikipedia.org	0.34	metacritic.com	0.95
agoravox.fr	0.44	boxofficemojo.com	0.94
cassini.ehess.fr (down)	0.48	retronews.fr	0.94
footballdatabase.eu	0.55	cnc.fr	0.94
amazon.fr	0.58	jpbox-office.com	0.94
facebook.com	0.59	pitchfork.com	0.94
fr.wiktionary.org	0.60	deadline.com	0.93
conseil-constitutionnel.fr	0.61	ew.com	0.93

To showcase Reference Risk and since we are in Paris, I checked popular sources on French Wikipedia: sources added by at least 500 distinct editors.

On the left are the Top 10 highest-risk sources:

These include:

- language editions of Wikipedia, in most cases Wikipedia should not be used as a source for Wikipedia articles;

- controversial French news outlets like AgoraVox (according French Wikipedia it has been used to spread disinformation)

- databases that are no longer available

- platforms like Amazon or Facebook.

On the right are the Top 10 lowes-risk sources:

These include:

- the French National Inventory of Natural Heritage, providing data on biodiversity

- well-known catalogues, many about culture and movies like Rotten Tomatoes, Metacritic.

Actually, this is a general pattern: catalogues and databases tend to have very low risk because they provide factual information that is instrumental for building encyclopedic knowledge

—————————————

Higher risk: top 10 (as usual, Wikipedia itself, social media, and fringe media.)

- fr.wikipedia.org

- en.wikipedia.org

- de.wikipedia.org

- agoravox.fr: A French citizen journalism website where contributors publish opinion pieces and news articles with limited editorial oversight.

- According to frwiki: “Conspiracy Watch, StreetPress et d'autres critiques ont relevé que le site hébergeait ou avait hébergé certains articles et tribunes reflétant diverses théories du complot[1], le site est également connu pour diffuser de la désinformation pro-Kremlin.”

- cassini.ehess.fr *(down)-: Website by École des Hautes Études en Sciences Sociales with information on French communes and administrative history.

- footballdatabase.eu: Website with football statistics.

- amazon.fr

- facebook.com

- fr.wiktionary.org

- conseil-constitutionnel.fr: Conseil Constitutionnel (constitutional decisions, legal texts, and institutional information).

- I have no explanation for this, feel curious if attendees can provide some context.

Lower risk: top 10 (as usual, catalogues with factual data)

- inpn.mnhn.fr: French National Inventory of Natural Heritage with data on biodiversity, species, habitats, and protected areas.

- rottentomatoes.com: Film and television review aggregation website.

- metacritic.com: Review aggregation website (movies, TV, music, games, and books).

- boxofficemojo.com: Website that tracks and reports box office revenue and commercial performance for films.

- retronews.fr: Digital archive from the Bibliothèque nationale de France providing access to historical French newspapers and magazines.

- cnc.fr: Centre National du Cinéma (France's public agency for cinema, audiovisual media, and animation).

- jpbox-office.com: Website specializing in Japanese box office data and film industry statistics.

- pitchfork.com: Website covering music criticism, reviews, interviews, and industry news.

- deadline.com: An entertainment industry news website (film, television, streaming, and media business).

- ew.com: Entertainment Weekly (news, reviews, and features about movies, television, music, books, and pop culture).

7 of 7

Thank you!

Contact:

This talk featured research with Ai-Jou Chou, Aitolkyn Baigutanova, Andreas Kaltenbrunner, Changwook Jung, Diego Saez-Trumper, Jacopo D'Ignazi, Jaehyeon Myung, Kyriaki Kalimeri, Mariano Beiró, Meeyoung Cha, Michele Tizzani, Miriam Redi, and Yelena Mejova.

Scan the QR code to view the Reference Risk Model Card for technical details, data information, API access, and related documentation.