1 of 13

AFIRE

Desk rejection shadow experiment

Introduction

​

Josie Coburn

15/7/2026

2 of 13

Summary

  • Your organisation is invited to take part in a multi-funder shadow experiment to test the feasibility and effectiveness of using desk rejection in funding application assessment processes
  • Desk rejection is the practice of using funder staff expertise and experience to identify and reject applications which are unlikely to be successful
  • Rationale: Against the backdrop of growing applications numbers, desk rejection would allow funders to reduce the volume of applications sent out for peer review
  • Anticipated results: Co-produced in partnership with RoRI and a global coalition of funders, the results could inform changes to funding delivery practice at a global scale
  • Seven funders are taking part (UKRI, BA, Wellcome, CIHR, CIFAR, NCRST, SNSF)
  • Results will be reported in aggregate form and individuals participating in the experiment will not be identified

3 of 13

4 of 13

5 of 13

Why use desk rejection?

​

Why use desk rejection? It offers a range of benefits:

  • Reducing the burden on the research community by cutting the volume of applications sent out for peer review
  • Reducing the number of reviewers that staff must recruit (and chase)
  • Reducing the number of reviews that staff need to check before sharing with applicants
  • Cost savings (in funding opportunities where peer reviewers are paid)

​

Why not just roll out desk rejection at scale? We need more information about:

  • The feasibility of using desk rejection (e.g. resourcing requirements)
  • The accuracy and error rate

6 of 13

7 of 13

8 of 13

Requirements

We are looking for:

  • Funding opportunities that expect to receive more than 10 applications
  • Each application received to be assigned to at least one funder staff member involved in delivering the funding opportunity or with relevant expertise or experience relating to the funding opportunity (and preferably two)
  • In each case, scoring will be performed by staff who are involved in the delivery of funding opportunities as part of their ordinary work (this could include, but is not limited to, grants delivery roles, research portfolio managers, and investment leads)
  • Participating funder staff members will be asked to review and score each application assigned to them
  • Staff should review ahead of accessing peer review scores and observe standard conflict of interest policies

9 of 13

Requirements

What scoring systems will funder staff use?

​

  • Funder staff will be asked to score (but not provide written comments for) applications using the following scoring system:

​

    • Assessment score: staff will score applications using the advertised assessment criteria and the funder’s own scoring system (e.g. 1 to 6 scoring)
    • Confidence score: staff will provide a confidence score for each assessment score (1 to 5, where 1 = Very unconfident and 5 = Very confident)

​

10 of 13

What other data will be collected?

Staff scorers will be asked to complete two short surveys, one before participating and one afterwards to provide important contextual information (e.g. length of service, type of role, expertise, experience of administering funding calls).

Before reviewing the applications, survey topics will include:

  • Expertise: Formal or informal training, if any relevant to the discipline(s) or topics covered by the call.
  • Experience: Number of years they have worked for this funder, number of previous rounds of this particular call, or closely related calls, they have worked on.

​

11 of 13

What other data will be collected?

After reviewing the applications, survey topics will include:

  • Length of time spent on reviewing – in total and per proposal
  • How easy or difficult funder staff found the reviewing and why
    • Were the assessment criteria clear enough?
    • What was their confidence score based on?
    • If any part of the process was difficult, what made it difficult?
  • Expertise and experience - Whether/how a staff members’ training, background or other factors affected their ability to score applications

Some staff may also volunteer to participate in an optional short online interview after taking part. Optional interview questions will focus on the task of desk rejection and desk rejection as a funder mechanism

12 of 13

Contact

If you have any questions or feedback about the desk rejection shadow experiment, please contact:

​

josie.coburn@ucl.ac.uk

​

13 of 13

JOIN THE CONVERSATION

researchonresearch.org

​

​

@RoRInstitute

Sign up to the RoRI newsletter for the latest research on research:

researchonresearch.org/about/contact