Summary: Professor Chris Chambers is a tenured Professor at Cardiff University,  UK, and the co-inventor of the Registered Reports formats. He is already doing some Registered Reports advocacy but is time-constrained due to his university teaching load. Buying him out of his teaching and administrative work at the University would free up his time, so he could readily do more advocacy and introduce the Registered Reports to more and higher prestige journals. Our analysis suggests that funding this advocacy effort will lead to a  cost-effectiveness of roughly $365-1460 per journal adopting the Registered Reports format, though this should not be taken literally. If a few hundred high-quality journals would sign up for the format, we believe it is not inconceivable that this could lead to non-linear increases in the adoption of the format and cause a paradigm shift in scientific culture, where the Registered Reports would be seen as the gold standard for hypothesis-driven science. We believe this is where most of the value of this project lies. We believe that this project has at the very least £112,000 ($150,000) room for more funding to free Chambers from teaching administrative duties for two years, but that the project could productively absorb even more, so that funding is covered for several years. We close with a section in which we summarize some risks, reservations, and drawbacks of this funding this grant.


Table of Contents

About Professor Chris Chambers

Cost-effectiveness: General considerations

$ per journal signed up for the Registered Reports format

Room for more funding

Risks, reservations and drawbacks

Our research process

References


About Professor Chris Chambers

We believe that Professor Chambers is an excellent science communicator who is very motivated to continue and intensify the advocacy for Registered Reports. We believe this mostly based on his previous track record of Registered Reports advocacy work – examples of which include:

Chris Chambers has a detailed grant proposal on a potential scale-up of his Registered Reports advocacy plans, which has remained unfunded so far. There he states that his “primary aim is to see RRs offered at all reputable peer-reviewed journals (across all science) that publish the outcomes of empirical hypothesis testing.”

Cost-effectiveness: General considerations

There are several reasons for why we believe that funding Chambers for Registered Reports advocacy is highly effective:

Due to Brexit, the Pound sterling value has declined significantly (the UK's trade-weighted currency index being quite low historically[13]). So for donors with diversified assets in particular it is relatively cheap to fund researchers in the UK.

$ per journal signed up for the Registered Reports format

Chambers keeps an online spreadsheet with all the journals that have adopted the format [14].

To date, 140 journals have adopted them so far and the fields covered are:  

Figure 1 taken from [15] “Planning ahead: Study preregistrations on the Open Science Framework (OSF) are doubling every year”; 140 journals have introduced Registered Reports

Figure 2: Approximate number of new Journals that have started offering Registered Reports within a given year based on Figure 1.

Figure 1 shows that in the last two years, 2017 and 2018, around 50 journals have introduced the new format per year. We believe most of this can be attributed to Chris Chambers’ efforts. Given that he has only had time to work on this part-time next to his teaching and neuroscience research (the latter of which takes up 50% of his time), we estimate that so far he has only spent 10%-25% of his time on Registered Reports advocacy. A teaching buyout freeing 50% of his time might thus enable him to do around 2-5 times as much work on Registered Reports advocacy. This means that he might be able to sign up around 50-200 additional journals per year for the £56,000 / $73,000 teaching buyout (approximately 50% of his salary). This suggests a rough cost-effectiveness of $365-1460 per journal adopting the Registered Reports format. Interestingly, 200 journals per year is roughly similar to what Chambers deems realistic in his grant proposal (“I would aim for 1000 adopting journals within 5 years”). This estimate should not be taken literally but only gives a general indication and might help compare it to other cost-effectiveness analyses. Many caveats apply to this model that might make donating more or less effective. For instance, on the one hand, there might be economies of scale, which are mentioned in Chambers grant proposal (“scaled delivery plan for the format, instead of contacting journals on an ad hoc basis to request that they consider RRs – or in some cases having them approach me.”), and indeed, the publisher Wiley has now a feature on introducing Registered Reports across diverse research communities[16]. Getting big academic publishers to introduce the format might be a high leverage opportunity: big publishers like Elsevier have a software for paper submission that is similar across different branches of science. If journal editors were given the option to just conveniently tick a box that would allow the Registered Reports format, then this might be a big win. However, there needs to be more awareness of the format for this to happen. Another way to rapidly get more journals to adopt this format might be by getting big research funders to accept the first stage of Registered Reports as grant proposal, so that if a study gets provisional acceptance, the study might be funded directly (read more in the section “Improving the Efficiency of Grant and Journal Peer Review: Registered Reports Funding” in our report on Registered Reports).

On the other hand, there might be diminishing returns on further investment in Registered Reports advocacy with the most eager journals having needed less convincing and now signing up more journals will get progressively harder.

Given that there were about 25,400 active scholarly peer-reviewed journals in early 2009[17], this might seem too slow progress and too much of a drop in the bucket. However, what matters is also the quality of scientific journals that sign up for the Registered Reports format. Chambers 5-year proposal aims to “to have RRs adopted by at least one top-ranking generalist journal (e.g. Science, Nature, PNAS) and to see RR funding models offered by several major funding agencies”. If a top-ranking generalist journal were to offer the Registered Reports format, then many other more specialist journals might follow suit.

So far, Registered Reports has been moderately successful at being implemented at high-quality journals. In political science, the American Journal Of Political Science and American Political Science Review, which are the two journals with the highest Eigenfactor, a rating of the total importance of a scientific journal, have adopted Registered Reports[18]. In neuroscience, NeuroImage the journal with the third highest Eigenfactor in neuroscience has adopted the format. There are several other journals that have adopted the Registered Reports format with good impact factors (another rating of the total importance of a scientific journal): for instance, Cochrane Reviews and Campbell Systematic Reviews. If a few hundred high-quality journals would sign up for the format, we believe it is not inconceivable that this could lead to non-linear increases in the adoption of the format. In other words, it might cause a paradigm shift in scientific culture, where the Registered Reports would be seen as the gold standard for hypothesis-driven science. We believe that this is perhaps where most of the value of this project lies.

Moreover, even though many journals offer the format, as of to date, a list shows that only 128 Registered Reports were published; the list might be non-exhaustive, and there are likely more Registered Reports about to be published and ‘in the pipeline’, but the number is probably not very far off. [19] Ultimately, it is the number of quality-adjusted Registered Reports published that matter. We believe that as more journals are signed up, there will likely be more papers published using the Registered Reports format because researchers would like to derisk their experimental process and get accepted to a journal independent of the results (see our analysis on Registered Reports). We believe that these papers will be trusted more by other researchers, funders, and so on and the quality will shine through, which might lead other academics to switch to the more reputable format. One might be able to spot the beginning of such a trend because scientists increasingly put tags such as ‘A preregistered study’ in the title of their publications in order to indicate that their study is to be taken more seriously [20].

Room for more funding

Generally, there are always diminishing returns to scale as funding increases, but at first there are increasing returns to scale and then constant returns to scale:

Conservative case: We believe that this project has at the very least £112,000 ($150,000) of room for more funding in the very near term. A conservative donor could fund a year of a teaching buyout and then reevaluate in the next year. This would cover the funds necessary to free Chambers from teaching and administrative duties at his university for two years, which take up 50% of his time. The other 50% are spent on a neuroscience research grant. £56,000 is the approximate cost of a teaching and administrative work buyout at Cardiff University, for 50% of a full Professor’s time. The average Cardiff University Professor makes around £73,500/yr (£65K - £95K) according to Glassdoor.com (also see Cardiff University payscales [21]). Note that a buyout from teaching also includes infrastructure from the university such as HR and accounting services, offices, library access, finding a replacement teacher etc.

Medium case: However, we believe that it is quite likely that the project could productively absorb £168,000 ($220,000) so that a teaching and administrative tasks buyout funding is covered for three years. This would give more security and enable Chambers to make longer term plans.

Optimistic case: We think that it is conceivable that there is room for more funding and increasing returns to scale even above this level. Chambers might be able to spend money productively by:

Hiring staff for several years could would cost a few hundred thousands of dollars. This is quite common when it comes to academic grants. For comparison, Chris Chambers has been awarded a European Research Council Consolidator Grant (for “excellent researchers with 7 to 12 years experience after PhD”) in 2015[23]. This was about €2 million ($2.3 million) over 5 years, which equates to €400,000 ($460,000) a year [24]. The next biggest grant size at the European Research council  is €2.5 million or €500,000 ($574,000) a year. The mean UK cancer research grants for principal investigators are around £500,000, with a median around half that [25].  Very similar numbers can be seen for UK infectious disease research [26]. The Open Philanthropy Project also gives grants for science funding at levels such as this[27].

All this suggests funding at a level of around £500,000 / €550,000/ $650,000 of funding would not overwhelm a typical academic such as Chambers. We think that this could be seen as a rough upper bound on the amount of funding that can be productively absorbed before steeply diminishing returns kick in.

We have a more detailed note on diminishing returns here for the interested reader. In brief, according to the law of logarithmic utility,  which has been applied to research funding[28], a simple rule of thumb is that a dollar is worth 1/x times as much if you are x times richer (and that doubling someone’s income is worth the same amount no matter where they start).[29] This would imply that the dollar donated after $50,000 have been donated already might have 10x as much impact as those donated after $500,000.

Note that the funds will be intentionally unrestricted so that Chambers can use the money in whatever way seems best to do advocacy for Registered Reports. We think that Chambers believes that the teaching buyout is first thing that will be spent money on, because his time seems the biggest bottleneck on this project. However, if the funds raised are lower than £56,000 it is likely that funds raised will not be spent on a teaching buyout, but rather on an assistant, travel etc.

Risks, reservations and drawbacks

Even though we do really recommend donating to this campaign we want to summarize the main risks, reservations and drawbacks of this grant.

Our research process

Our research on science funding, meta-research funding, Registered Reports advocacy by Chris Chambers consisted of desk research, academic literature review, interviews with experts (such as Ben Goldacre) and draws on experiences during a science Ph.D.

During our research, we have considered several other funding opportunities at a more shallow level, such as:

For this grant evaluation, we cold contacted Professor Chambers with whom we had no prior contact and so declare no competing interests.

References


[1] "The Seven Deadly Sins of Psychology - Princeton University Press." 25 Apr. 2017, https://press.princeton.edu/titles/10970.html. Accessed 29 Sep. 2018.

[2] "Trust in science would be improved by study pre-registration | Science ...." 5 Jun. 2013, https://www.theguardian.com/science/blog/2013/jun/05/trust-in-science-study-pre-registration. Accessed 28 Sep. 2018.

[3] "A recipe for rigor | Science." 20 Sep. 2018, http://science.sciencemag.org/content/361/6408/1192.summary. Accessed 27 Sep. 2018.

[4] https://www.theguardian.com/science/head-quarters/2017/aug/24/clinical-trials-revolution-could-change-the-future-of-medical-research 

[5] See Google Video results for “Chris Chambers Registered Reports”

[6] "Does research organization influence academic ... - Science Direct." https://www.sciencedirect.com/science/article/pii/S004873330400071X. Accessed 2 Oct. 2018.

[7] "factors associated with research productivity - Journal of Agricultural ...." http://www.jae-online.org/attachments/article/371/43-03-01.pdf. Accessed 2 Oct. 2018.

[8] "Faculty Time Allocations and Research Productivity: Gender, Race ...." https://muse.jhu.edu/article/30080/summary. Accessed 2 Oct. 2018.

[9] "List of UK universities by endowment - Wikipedia." https://en.wikipedia.org/wiki/List_of_UK_universities_by_endowment. Accessed 29 Sep. 2018.

[10] "Bias in Research Grant Evaluation Has Dire Consequences for - PLOS." 3 Jun. 2016, https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0155876. Accessed 2 Oct. 2018.

[11] "Interdisciplinary research has consistently lower funding ... - Nature." 29 Jun. 2016, https://www.nature.com/articles/nature18315. Accessed 2 Oct. 2018.

[12] "The FDP Faculty Burden Survey. - NCBI - NIH." https://www.ncbi.nlm.nih.gov/pubmed/20563268. Accessed 18 Sep. 2018.

[13] "Monthly average, Effective exchange rate index, Sterling (Jan 2005 ...." https://www.ons.gov.uk/economy/nationalaccounts/balanceofpayments/timeseries/bk67/mret. Accessed 23 Oct. 2018.

[14] https://docs.google.com/spreadsheets/d/1D4_k-8C_UENTRtbPzXfhjEyu3BfLxdOsn9j-otrO870/edit#gid=0 

[15] "A simple strategy to avoid bias—declaring in advance what ... - Science." 21 Sep. 2018, http://science.sciencemag.org/node/715589.full . Accessed 29 Sep. 2018.

[16] "Discover the Future of Research : 2018 : September : 14 | Wiley." 14 Sep. 2018, https://hub.wiley.com/community/exchanges/discover/blog/2018/09/14?referrer=exchanges?elqTrack=true. Accessed 28 Sep. 2018.

[17] "An overview of scientific and scholarly journal ... - STM Association." 13 Oct. 2009, https://www.stm-assoc.org/2009_10_13_MWC_STM_Report.pdf. Accessed 30 Sep. 2018.

[18] http://www.eigenfactor.org/projects/journalRank/rankings.php?search=UU&year=2015&searchby=isicat&orderby=Eigenfactor 

[19] "Zotero | Groups > OSF > Library > Registered Reports." https://www.zotero.org/groups/osf/items/collectionKey/KEJP68G9. Accessed 29 Sep. 2018.

[20] See https://scholar.google.co.uk/scholar?q=%22preregistered%22&hl=en&as_sdt=0,5 

[21] "Pay scales - Jobs - Cardiff University." https://www.cardiff.ac.uk/jobs/what-we-can-offer/financial-benefits/pay-scales. Accessed 23 Oct. 2018.

[22] "Meta-scientific studies of reproducibility at Cardiff ... - Find A PhD." https://www.findaphd.com/search/ProjectDetails.aspx?PJID=100639. Accessed 19 Oct. 2018.

[23] "Professor Chris Chambers - People - Cardiff University." https://www.cardiff.ac.uk/people/view/133632-chambers-chris. Accessed 20 Oct. 2018.

[24] "European Research Council." https://erc.europa.eu/. Accessed 19 Oct. 2018.

[25] "A systematic analysis of UK cancer research funding by ... - BMJ Open." https://bmjopen.bmj.com/content/8/4/e018625. Accessed 23 Oct. 2018.

[26] "Systematic analysis of funding awarded to institutions ... - SAGE Journals." http://journals.sagepub.com/doi/abs/10.1177/2054270415577056. Accessed 23 Oct. 2018.

[27] "Grants Database | Open Philanthropy Project." https://www.openphilanthropy.org/giving/grants. Accessed 23 Oct. 2018.

[28] "The law of logarithmic returns - Future of Humanity Institute." 20 Oct. 2014, https://www.fhi.ox.ac.uk/law-of-logarithmic-returns/. Accessed 24 Oct. 2018.

[29] "The value of money going to different groups - Centre for Effective ...." 2 May. 2017, https://www.centreforeffectivealtruism.org/blog/the-value-of-money-going-to-different-groups/. Accessed 24 Oct. 2018.

[30] "Is the staggeringly profitable business of scientific publishing bad for ...." 27 Jun. 2017, https://www.theguardian.com/science/2017/jun/27/profitable-business-scientific-publishing-bad-for-science. Accessed 28 Sep. 2018.

[31] "Overcoming Bias : Policy Tug-O-War." 23 May. 2007, http://www.overcomingbias.com/2007/05/policy_tugowar.html. Accessed 24 Oct. 2018.

[32] "On methods for correcting for the look-elsewhere effect in searches for ...." 11 Feb. 2016, https://arxiv.org/abs/1602.03765. Accessed 1 Oct. 2018.