Dr Jonny Coates

BSky: @JACoates

​

​

jonny.coates@ripplingideas.org

​

​

Open Preprint Peer Review; the future of peer review

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Original content is licensed under a Creative Commons Attribution 4.0 International License.

Overview of today

• Quick history of peer review in the life sciences
• What’s wrong with peer review
• Preprints and preprint peer review
• The future of peer review
• Why does this matter?

​

twitter.com/ASAPbio_ | mas.to/@ASAPbio

What do you perceive as the biggest issues in academia & publishing?

?

@ASAPbio_ | #ASAPbio | @JACoates

When do you think peer review started?

?

@ASAPbio_ | #ASAPbio | @JACoates

When did peer review start? A brief history

“Peer review has been around for over 350 years and is therefore a well-established gold standard”

5th century BC

Pre-history peer review

Historical peer review

Modern peer review

17th century

1970’s

Preprint peer review

Henry Oldenburg

William Whewell

Medical Essays and Observations

Written by teams of eminent scholars, these reports might, he argued, be “often more interesting than the memoirs themselves” and thus a great source of publicity for science

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002234

YouTube

2017

Peer review prehistory

“Ethics of the Physician”, by a Syrian author, states that:

“It is the duty of a visiting physician to make duplicate notes of the condition of the patient on each visit…The notes of the physician were examined by a local council of physicians, who would adjudicate as to whether the physician had performed according to the standards that then prevailed.”

5th century BC

NOT peer review, but the concept of peer review

~900 AD

How did modern(?) peer review start?

Henry Oldenburg

“Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters”

The sanction which the Society gives to the work now published…extends only to the novelty, ingenuity or importance …Responsibility concerning the truth of facts, the soundness of reasoning, in the accuracy of calculations is wholly disclaimed: and must rest alone…the authors

Oldenburg used his own personal judgement in the selection process without resorting to external opinion; he used no referee system.

1665

1831

William Whewell

Written by teams of eminent scholars, these reports might, he argued, be “often more interesting than the memoirs themselves” and thus a great source of publicity for science

1730’s

Medical Essays & Observations

NOT peer review as we would know it

post-WW2 & USA politics

Robert Bauman

John Conlan

William Proxmire

Peer review as we know it

Concerned with funding on the grounds that NSF was giving money to projects that were “frivolous and wasteful”.

All three thought that the NSF should be reined in and that there should be more Congressional oversight of the grant awarding process.

Modern peer review - a rocky start

Max Perutz

​

“As all papers sent to Nature are checked by members of the board, peer review is unnecessary”

“had not authorized you to show [our manuscript] to specialists before it is printed. I see no reason to address the—in any case erroneous—comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere”

‘Nature was a place for rapid publication’, and a week after the crucial insight, ‘the first drafts of our Nature paper got handed out’

Watson

Einstein

What is peer review? - The purpose

What do you think the purpose of peer review is?

Originally -> Promote scientific outputs and to aid with editorial decisions (ensure novelty)

​

Mutated -> “improving” a paper & measuring perceived impact and “good” science, driven by financial & self interest

​

Should be -> Are the conclusions supported by the data and citations within the paper?

Emojis by Mozilla (CC BY 4.0)

Journal 1

Journal 2

Journal 9

Private

Public

Peer Review

Submit

Manuscript

Peer reviewed paper

Revise

Journal 3

Closed peer review hides the extent of the problem

Emojis by Mozilla (CC BY 4.0)

Journal 1

Journal 2-8

Journal 9

Private

Public

Peer Review

Submit

Revise

Manuscript

Peer reviewed paper

Community feedback, ideas, discussion

Months to years

Revise

Retracted paper

Years

Does peer review work?

?

@ASAPbio_ | #ASAPbio | @JACoates

Science is dependent on a failed peer review system

• Modern peer review is a very recent experiment
• Peer review is full of bias (gender, race, prestige, Matthew effect)
• Reviewers consistently do not agree with each other¹ (more than would occur by chance^1a)
• Peer review does not detect fraud or protect the literature
• Peer review is poor at detecting gross defects²
• Peer review is expensive³ and slow⁴
• Peer review is over-relied upon and perceived as a “stamp of approval”
• Peer review is abused by some journals themselves; e.g. PNAS contributor track
• Process itself is subjected to fraud⁵
• Retractions are extremely slow⁶
• We only share positive peer reviews
• Peer review is mis-sold as a “quality control” step, leading to inappropriate use
• “little empirical evidence is available to support the use of editorial peer review…A large, well‐funded programme of research on the effects of editorial peer review should be urgently launched” - Cochrane, 2007⁷

https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 | https://f1000research.com/articles/6-1151 |

1. https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/peerreview-practices-of-psychological-journals-the-fate-of-published-articles-submitted-again/AFE650EB49A6B17992493DE5E49E4431 & https://pmc.ncbi.nlm.nih.gov/articles/PMC9594632/ 1a https://pubmed.ncbi.nlm.nih.gov/10960059/ | https://pubmed.ncbi.nlm.nih.gov/8182120/ |
2. https://pubmed.ncbi.nlm.nih.gov/9676667/ & https://www.ncbi.nlm.nih.gov/pmc/articles/PMC381220/
3. https://researchintegrityjournal.biomedcentral.com/articles/10.1186/s41073-021-00118-2
4. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/ & https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0166387
5. https://blogs.lse.ac.uk/impactofsocialsciences/2016/12/13/manipulating-the-peer-review-process-why-it-happens-and-how-it-might-be-prevented/
6. https://sitn.hms.harvard.edu/flash/2022/peer-review-in-science-the-pains-and-problems/
7. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.MR000016.pub3/full

Preprints make work available almost immediately

Emojis by Mozilla (CC BY 4.0)

Journal 1

Journal 2

Journal 3

Private

Public

Peer Review

Submit

Revise

Manuscript

Peer reviewed paper

Community feedback, ideas, discussion

Months to years

Preprint server

<48 hrs screening process

Revise

@ASAPbio_ | #ASAPbio | @IratxePuebla

Definition of “preprint”

Preprints are complete manuscripts, openly shared online prior to journal-organised peer review.

​

Preprints;

• are citable
• have a DOI
• are permanent
• can be peer reviewed
• are shared when the authors believe the work is ready

​

​

​

Preprints are NOT (necessarily);

• Preliminary or early work
• Low quality or incomplete
• Rushed work

@ASAPbio_ | #ASAPbio | @IratxePuebla

Preprints as a percentage of published literature is growing rapidly

https://www.biorxiv.org/content/10.1101/2024.04.19.590240v1.full | Data from Europe PMC

<5%

~11-13%

COVID-19

@ASAPbio_ | #ASAPbio | @JACoates

~70%

The majority of preprints are eventually published in a journal

https://direct.mit.edu/qss/article/1/2/618/96153/The-relationship-between-bioRxiv-preprints

@ASAPbio_ | #ASAPbio | @JACoates

If preprints are not published within say 2 years, should we be concerned?

?

@ASAPbio_ | #ASAPbio | @JACoates

Public feedback is an important remedy for misinformation

“[T]he reaction from the scientific community to the bioRxiv paper was swift. In a nutshell, commenters on bioRxiv and Twitter said, the author’s methods seemed rushed, and the findings were at most a coincidence. By Saturday morning, bioRxiv had placed a special warning on all papers about coronavirus. Later Saturday, the authors commented on their paper, saying they were withdrawing it. And on Sunday, a more formal retraction appeared.”

​

https://www.statnews.com/2020/02/03/retraction-faulty-coronavirus-paper-good-moment-for-science/

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Transparent feedback goes beyond just peer review

‘Markers’ of trust

Not traditionally considered peer review

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Researchers support the idea of publishing peer review

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0189311

asapbio.org/peer-review/summary

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Growing preprint review activity

Preprints reviewed per month on Sciety, excludes reviews by automated tools (ScreenIT) and those posted by journals after publication of the journal version

Source data at doi.org/10.5281/zenodo.7778275

@ASAPbio_ | #ASAPbio | @JACoates

Preprint review activity differs across fields

@ASAPbio_ | #ASAPbio | @JACoates

Preprint peer review platforms

Social media posts

Minimalist or freeform

Journal-like

Comments on preprint servers

Preprint feedback initiatives

Over 35 platforms reviewing preprints

@ASAPbio_ | #ASAPbio | @JACoates

Preprint peer review platforms

Highlighting

• Generally very open and accessible for ECRs
• More focussed on highlighting interesting preprints
• Does contain a level of peer review but definitely nothing approaching QC or traditional review
• Perhaps most related to “curation” type effort

@ASAPbio_ | #ASAPbio | @JACoates

Preprint peer review platforms

Community reviews

• Very open and accessible for ECRs
• Similar to traditional review, but transparent
• No real adoption by journals, limiting the impact
• Does not require authors to engage - reviews benefit the public by providing context to a preprint

@ASAPbio_ | #ASAPbio | @JACoates

Preprint peer review platforms

Portable reviews

• Reviews occur in a centralised service
• Preprint + Review package can be taken to partner journals
• Removes some of the redundancy and waste in the process of peer review
• More open to ECRs than traditional peer review

​

​

​

​

​

​

​

• Can choose to “publish” in PCI directly

@ASAPbio_ | #ASAPbio | @JACoates

Preprint peer review platforms

Journal switches

• Essentially still just traditional peer review
• Business model change rather than process
• Focus on value-added (i.e. the peer review & curation)
• Limited examples currently
• eLife model involves important additional aspects - editors summary, controlled vocabulary for reviews

@ASAPbio_ | #ASAPbio | @JACoates

Recognition for preprint peer review

29 doctoral schools have stated that preprints recommended by a PCI are considered the same value as articles of good quality published in journals

​

via Thomas Guillemaud & Denis Bourguet

@ASAPbio_ | #ASAPbio | @JACoates

Formal recognition for preprint peer review via policy change

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000273

twitter.com/ASAPbio_ | mas.to/@ASAPbio

An open & transparent system, directly supported by institutions & funders with preprints at the core, can achieve this

“all peer-reviewed scholarly publications authored or co-authored by individuals or institutions resulting from federally funded research are made freely available and publicly accessible by default in agency-designated repositories without any embargo or delay after publication”

“In its conclusions, the Council calls on the Commission and the member states to support policies towards a scholarly publishing model that is not-for-profit, open access and multi-format, with no costs for authors or readers.”

What do you think the future of peer review is?

?

@ASAPbio_ | #ASAPbio | @JACoates

What is the future of peer review?

Preprint review as a service & transportable

Focus back on the reviews (William Whewell)

PRC model

Focus on the individual, not proxies or poor group metrics

A comment on PRC - the P stands for Problem

Problems:

• Mostly just replicates the current system
• N.b we’re achieving PRC already even if not explicitly stated
• Fails to solve any of the major problems
• including assessment relying on brands/poor proxies
• Only minor improvements
• speeds things up a little bit but slows the whole advancement down compared to alternatives
• Poorly thought out and those pushing this are failing to consider the issues and reality
• Very open to publishers for hijacking and dominating as with APCs and OA
• Could result in a system even less equitable and more expensive
• Damages standalone preprinting
• Ignores evidence and “better” paths forward

Publish

Integrity

&

Trust

Scholar-led

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002234

twitter.com/ASAPbio_ | mas.to/@ASAPbio

• Non-profit to oversee bioRxiv & medRxiv

​

• Enables greater funding sources and sustainability

​

• Places scientists & the community at the forefront

​

• Will be doing more advocacy work - will be the leaders in the preprint space

twitter.com/ASAPbio_ | mas.to/@ASAPbio

This matters

twitter.com/ASAPbio_ | mas.to/@ASAPbio

… because it recognises ghost peer reviewers

https://elifesciences.org/articles/48425#s2

twitter.com/ASAPbio_ | mas.to/@ASAPbio

…because scientists do play the game and put career advancement first

Collated data on “Questionable Research Practices”

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005738 | https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0263023#sec008

2022 data on “Questionable Research Practices”

…because incentives promote unacceptable practices

twitter.com/ASAPbio_ | mas.to/@ASAPbio

• A number of researchers in the field knew that this was not trustworthy data
• Most-cited retracted paper ever

twitter.com/ASAPbio_ | mas.to/@ASAPbio

https://www.pewresearch.org/science/2023/11/14/americans-trust-in-scientists-positive-views-of-science-continue-to-decline/

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Opaque and bad science legitimises hijacking and injection of ideology

twitter.com/ASAPbio_ | mas.to/@ASAPbio

How can you get involved?

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Podcast

https://zenodo.org/records/14496847

@ASAPbio_ | #ASAPbio | @JACoates

Actively review preprints!

https://zenodo.org/records/12731704

@ASAPbio_ | #ASAPbio | @JACoates

Actively review preprints!

https://zenodo.org/records/10245909

@ASAPbio_ | #ASAPbio | @JACoates

Learn more with preprint-related programs

@ASAPbio_ | #ASAPbio | @JACoates

Why publish peer reviews?

• Encourages constructive reviewer behavior
• Transparency = Accountability
• Recognises peer review as scholarship
• Enables reviewer recognition
• Helps trainees & ECRs
• Marker of trust
• Makes journal decisions more transparent
• Reduces waste
• Enables the study of peer review

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002502

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Change is difficult

Survivorship bias is extremely pervasive in academia

99% who don’t make it

1% who “succeed”

“If it worked for me it must work for all!”

@ASAPbio_ | #ASAPbio | @JACoates

Publish

Integrity

&

Trust

Scholar-led

https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002234

twitter.com/ASAPbio_ | mas.to/@ASAPbio

Thank you!

More info

​

YouTube:

History of peer review

History of preprints

Robert Maxwell

Preprint reliability

​

​

These slides

@ASAPbio_ | @JACoates | jonny.coates@asapbio.org

BSky: @JACoates

@ASAPbio_ | mas.to/@ASAPbio

​

jonny.coates@ripplingideas.org

https://bit.ly/2025-peer-review-developments