A Partial History of Partial Randomisation
Tom Stafford, AFIRE programme lead
16 December 2024
t.stafford@sheffield.ac.uk
“Details aside, the basic principle is clear; instead of dodging the fact that chance plays a big part in awarding money, the system will sanctify chance as the determining factor. After a few years, let's look back and evaluate the science that came out of this system.”
Greenberg, D. S. (1998). Chance and grants. The Lancet, 351(9103), 686.
Greenberg, 1998
Avin, S. (2019). Mavericks and lotteries. Studies in History and Philosophy of Science Part A, 76, 13-23.
Brezis, E. S. (2007). Focal randomisation: An optimal mechanism for the evaluation of R&D projects. Science and Public Policy, 34(10), 691-698.
Referees cannot recognise the true value of an invention, λi, and they all give the value λ*.
Uvi = α Di + γ B i + βλ* (4)
Moreover, we assume that, since referees have difficulty in recognising the future value of inventions, they tend to underestimate it, and give to λ* a lower value than the average of λi.
Brezis, 2007
Fang, F. C., & Casadevall, A. (2016). Grant funding: Playing the odds. Science, 352(6282), 158-158.
“Selection of the best of the best resembles a lottery in its unpredictability, but one that lacks the benefit of being truly random, due to bias”.
Fang & Casadevall, 2016
Fang, F. C., & Casadevall, A. (2016). Grant funding: Playing the odds. Science, 352(6282), 158-158.
“Selection of the best of the best resembles a lottery in its unpredictability, but one that lacks the benefit of being truly random, due to bias”.
The Health Research Council of New Zealand is the first major government funding agency to use a lottery to allocate research funding for their Explorer Grant scheme. … This paper presents the results of a survey of Health Research Council applicants from 2013 to 2019.
Health Research Council of New Zealand 2013 -
Liu, M., Choy, V., Clarke, P., Barnett, A., Blakely, T., & Pomeroy, L. (2020). The acceptability of using a lottery to allocate research funding: a survey of applicants. Research integrity and peer review, 5, 1-7.
Woods, H. B., & Wilsdon, J. (2021). Why draw lots? Funder motivations for using partial randomisation to allocate research grants. Research on Research Institute. https://doi.org/10.6084/m9.figshare.17102495.v2.
Research on Research
Partial Randomisation Trials Catalogue
These slides:
These slides:
https://researchonresearch.org/project/a-f-i-r-e/
https://researchonresearch.org/project/a-f-i-r-e/
RoRI’s Accelerator for Funder Experimentation - AFIRE
Partial Randomisation at The British Academy
19 February 2025 00:00 PDT / 08:00 BST / 09:00 CEST
Ken Emond, The British Academy & Adrian Barnett
Capacity building
Forum
Experiments
Sprint on AI/ML in reviewer selection (Feb -May 2025)
Distributed Peer Review, Core Outcome Sets for PR.
END
(reserve slides follow)
Wouldn’t this be gambling with scientists’ careers? ….These criticisms are without merit. The current system already gambles with scientific careers, just in a haphazard way.
Azoulay, P. (2012). Turn the scientific method on ourselves. Nature, 484(7392), 31-32.
AFIRE
These slides:
https://researchonresearch.org/project/a-f-i-r-e/
https://researchonresearch.org/project/a-f-i-r-e/
Forum
Support
Experiments!
Experiment: Core Outcome Sets for Partial Randomisation
Sprint #1 : AI in reviewer selection
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
2024
2025
Sprint #2 : Designing RCTs
Experiment: Distributed Peer Review @ VolkswagenStiftung
Metascience2025
AFIRE Funders’ Forum (monthly)
AFIRE phase 2 experiments announced!
These slides:
Bendiscioli, Sandra; Firpo, Teo; Bravo-Biosca, Albert; Czibor, Eszter; Garfinkel, Michele; Stafford, Tom; et al.
(2022):
The experimental research funder’s handbook (Revised edition, June 2022, ISBN 978-1-7397102-0-0).
https://doi.org/10.6084/m9.figshare.19459328.v2
18 Core Partners
10 codesigned projects
Partial Randomisation at The British Academy
19 February 2025 00:00 PDT / 08:00 BST / 09:00 CEST
Ken Emond, The British Academy & Adrian Barnett
Do Grant Proposal Texts Matter for Funding Decisions? A Field Experiment at NWO
19 March 2025 07:00 PST / 15:00 GMT / 16:00 CET
Müge Şimşek, University of Amsterdam
An experiment with Distributed Peer Review
23 April 2025 00:00 PDT / 08:00 BST / 09:00 CEST
Hanna Denecke, The Volkswagen Foundation
Register for updates. Speaker suggestions welcome!
AFIRE
2024-25 Funder Forum series
AFIRE
2024-25 Funder Forum series
Villum Experiment – a grant scheme with double-blind evaluation
23 October 2024 00:00 PDT / 08:00 BST / 09:00 CEST
Anders Smith, Villum Foundation
Predicting Progress: A Pilot of Expected Utility Forecasting in Science Funding
20 November 2024 07:00 PST / 15:00 GMT / 16:00 CET
Alice Wu, Federation of American Scientists & Jordan Dworkin, Open Philanthropy
Enhancing EDI in application processes in the CES Transformation Fund, including evaluation of an anonymisation-deanonymisation review process
18 December 2024 07:00 PST / 15:00 GMT / 16:00 CET
Rachel Kendal, Principal Investigator CES Transformation Fund
AFIRE Experiments - phase 2
AFIRE exists to support funder use and generation of higher quality evidence, across all topics. Topics where funders can take advantage of active experiments, recent experience and strong interest from other funders interested in running new experiments are below. For more ideas, see The Research Funders Handbook!
Distributed Peer Review | applicants for a funding scheme become peer review college for that scheme |
Partial Randomisation | reducing deliberation time and bias in the grey zone |
Can AI help find reviewers? | Testing if language models can help match proposals to peer reviewers |
Anonymisation | How and where to use anonymisation in proposal evaluation |