1 of 10

Reproducibility in Machine Learning Journals: a discussion

Mendoza Lab Group Meeting

March 3, 2026

Susana López Moreno

2 of 10

What is reproducibility?

In Mathematics,

Reproducibility means that given the same axioms and definitions, anyone can derive the same result.

When moving to computational experiments,

Provided the same method and initial conditions, anyone can get the same result.

(Results from “On the computation of tensor functions under tensor-tensor multiplications with linear maps”, J-H. Ju, S. López-Moreno)

3 of 10

What about Machine Learning?

Image from https://integranxt.com/blog/researchs-latest-challenge-reproducibility-of-results/

4 of 10

Why are we interested in this subject…?

(Because the lack of code from ML papers is delaying a lot of our projects…)

5 of 10

Reproducibility and verification�in journals

Conferences are preferred by ML researchers over journals (because of quick review times?).
Among all journals and proceedings, only Neurocomputing (NC) and Pattern Recognition (PR) require the code at the time of submission.
There were some Reproducibility challenges at ICLR 2018, ICLR 2019 and NeurIPS 2019, but it has been made into its own independent event since then.
No post-publication verification of code.

6 of 10

ACM artifact review and badging

Badges recommended by the ACM:

Available
Functional/reusable
Reproduced/replicated

7 of 10

The following figure appears in a ”Reviewer training” manual from ACM, but not on the main page

However, ACM-affiliated conferences (and ACM journals) are not implementing this system post-publication

8 of 10

What about other platforms?

Papers with Code (2018~2025)

Community-based post-publication verification platform + benchmarking of models and datasets.
After its dissapearance, ALL its data was lost. RIP.

Hugging Face

Private company attempting to follow PwC steps.
What happens if Hugging Face dissapears as well?

Paperswithoutcode.com

Shaming system.
Although interesting approach, not very active. The linked reddit community doesn’t even exist anymore and the public list has had the same 24 papers for the last 3 months.

9 of 10

Call to action: Post-publication verification should be done in-house by journals

Trusting outside platforms is risky.
Journals should bring the post-publication verification process in-house and give badges to papers that have been independently reproduced.
We propose that there is a:

Separate submission system for reproductions,
that papers are awarded public badges (at most one reproduced badge and one replicated badge) visible at the article website,
with the name of the team that reproduced it and the link to the repository with the code on the metadata of the article.