Registered Reports at the APSR
This version: November 19 2024
A registered report is a form of research publication in which the theory, methods, and proposed analyses are reviewed before the research is conducted. If the design is considered promising enough, the journal agrees to publish the paper as long as the ultimate implementation hews closely enough to the design.
The key feature of a registered report, then, is that the decision to publish is essentially made before either the author or the reviewers have seen the results. The decision to publish a paper is based primarily on how much we might be able to learn from it, given its design.
While there are good reasons to consider findings in deciding what research to publish (Frankel and Kasy, 2022), the publication of empirical results based on findings can make it more difficult to interpret the body of published work, for two broad reasons.
First, when publication depends on findings, the body of published work becomes unrepresentative of the entire distribution of findings. We must therefore consider how journals selected on findings to infer the true distribution of findings from published papers.
Second, when publication depends on findings, authors may distort their research practices in unobserved ways to increase the probability of publication, thereby making each finding less reliable. This concern grows to the extent that reviewers and editors are thought to prefer certain results (e.g. significant coefficients, conclusions that fit with existing beliefs, conclusions that conflict with existing beliefs) and authors have the motive and opportunity to cater to those preferences.
The registered report publication format lessens these concerns, which may make the results of the underlying studies more informative.
In the simplest case, researchers submit a Stage 1 registered report when they could not have conducted the analysis yet, e.g. because the event being studied has not yet occurred or the experiment has not yet been conducted. This unobservability proves that the design was not chosen on the basis of the results.
In some cases, at Stage 1 it would have been technically possible for the researchers to conduct the analysis, but they are able to offer some reassurance that they have not conducted it. For example, if the data is controlled by a single government agency or company, that entity may be able to provide an affidavit verifying that they have not provided the dataset. In such cases, the (un)observability of the data should be explained at the time of Stage 1 submission.
For considerations about (un)observability of registered reports, see this report by PCI RR.
It can be easier to assess a registered report’s likely contribution if the submission is informed by pilot data or data from similar previous studies. Analysis of these data can help the reader assess a study’s power, validate its important measures, and (perhaps most importantly) assess the strength of the “first stage” in a study that relies on a treatment affecting a mediator.
A possible downside to the inclusion of pilot data is that, for a sufficiently large pilot, the reader may have a good idea of what the ultimate finding is likely to be. To the extent that this affects the way reviewers assess the submission, this could undermine the anti-publication bias objective of the registered report format.
If authors think this is possibly an issue, then they can report analysis of pilot data that helps assessment of the design within revealing the results, for example by simply omitting the results or demeaning the outcome by treatment group.
Above all, a registered report submission must make clear what we can expect to learn from the study. It should therefore do the following:
“Making clear what we can expect to learn” does not necessarily mean accurately predicting what we will find in the study. It may be useful to clarify how different theoretical perspectives suggest different possible findings, and what prior empirical research suggests about the relative importance of these perspectives. But authors should not feel required or even encouraged to state what they personally expect to happen. In cases where the empirical findings stand to adjudicate between rival theories or expectations, the author’s own expectations may not be relevant, and in fact taking a position at the design stage could introduce bias in the estimation and testing process.
To help clarify what will be learned, authors should communicate any procedures they will use for drawing conclusions from the results. This could include both conventional null hypothesis significance tests and equivalence tests (i.e. tests of the null hypothesis that the parameter is outside of a certain interval), though such tests are not always necessary or even helpful. (Also, designing these hypothesis tests does not require predicting the outcome.) Authors should clarify what theoretical or practical conclusions could be drawn from the range of possible empirical findings. Ideally, we learn something from the study regardless of the result.
We may decline to publish a registered report submission for various reasons, including doubts about research design that could be resolved by actually implementing the study. When a registered report submission is rejected, we will not ordinarily allow authors to submit the completed study as a new submission. This policy is intended to conserve editorial resources and ensure that authors use the registered report track only for projects that would be especially suitable for that review process.
These instructions are provided on the APSR website.
You may submit a Stage 1 Registered Report via Editorial Manager choosing the “Registered Report” designation.
Please add an extra note to the editors (e.g. in a very brief cover letter) if your registered report is time-sensitive, e.g. because the data will become publicly available within five months of initial submission.
Your Stage 1 Registered Report should not contain the results of your analysis, though it can contain results of analysis based on pilot data or simulated data, which would be replaced with the results of analysis based on the actual data if/when that becomes available.
Note that, if your Stage 1 submission is accepted, you will be required to register your final approved design with the OSF or another recognized repository before conducting the research.
If the analysis proposed in the paper is based on data that already exists, be sure to explain in the Stage 1 submission any reassurance that can be offered that the analysis has not already been conducted.
Reviewers of Stage 1 registered reports will be instructed to assess
These instructions are provided on the APSR website and sent to authors when an In-Principle Acceptance is offered at Stage 1.
Now that you have received an In-Principle Acceptance (IPA), you may proceed with the research you proposed and prepare a manuscript for Stage 2 submission.
Before conducting the research, you are required to register your approved design on the Open Science Framework (https://osf.io/) or other recognized repository. OSF provides a special template for registered reports: https://osf.io/rr/ Your registration must agree with your (in-principle) accepted Stage 1 manuscript in all key respects, though it may add precision to any features left ambiguous. If the data is potentially observable at the time of pre-registration, include any available evidence that you have not yet conducted the research.
Your Stage 2 submission:
Editors and reviewers will assess a Stage 2 submission on the basis of its adherence to the design described in Stage 1 and the PAP, not on the basis of the results or the quality of the design approved at Stage 1.
Specifically, reviewers will be told to assess
Editorial decisions will not be based on the perceived importance, novelty, or decisiveness of the results.
These instructions are provided with reviewer invitations at Stage 1.
Please note that this is not a standard submission. You are being invited to review a Stage 1 Registered Report.
A registered report is a form of research publication in which the methods and proposed analyses are reviewed before the research is conducted. If the design is considered promising enough, the journal agrees to publish the paper as long as the ultimate implementation hews closely enough to the design.
The key feature of a registered report, then, is that the decision to publish is essentially made before either the author or the reviewers have seen the results. The decision to publish a paper is based primarily on how much we might be able to learn from it, given its design.
For more on registered reports at the APSR, see https://docs.google.com/document/d/1PSwOLWYOF78PNEw515kBcamIyopWQzBHrO6D3TZ2qD8/pub
For more on registered reports in general, see https://www.cambridge.org/core/open-research/registered-reports
https://www.cambridge.org/core/services/peer-review/how-to-peer-review-registered-reports
Because the Stage 1 submission you are being asked to review does not contain the results of the analysis, your review should focus on what you think we can learn from this design if it were to be implemented. Specifically, we ask that you focus on
-- The importance of the research question(s)
-- The logic, rationale, and plausibility of any proposed hypotheses.
-- The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where appropriate).
-- Whether the clarity and degree of methodological detail is sufficient to exactly replicate the proposed experimental procedures and analysis pipeline.
-- Whether the authors have specified analysis and testing procedures that allow for informative tests of any stated hypotheses and more broadly provide informative results.
These instructions are provided with reviewer invitations at Stage 2.
You are being invited to review a Stage 2 registered report.
A registered report is a form of research publication in which the methods and proposed analyses are reviewed before the research is conducted. If the design is considered promising enough, the journal agrees to publish the paper as long as the ultimate implementation hews closely enough to the design.
The key feature of a registered report, then, is that the decision to publish is made before the results exist. The decision to publish a paper is based almost entirely on how much we might be able to learn from it, given its design.
This manuscript received an In-Principle Acceptance at Stage 1, meaning that we judged the design to be promising and agreed to publish the resulting paper as long as the analysis followed the plan stated at Stage 1.
You are being asked to review whether this submission adhered closely to the design described in Stage 1. You are asked not to assess the paper on the basis of the results of the analysis.
Specifically, we ask that you assess
-- Whether the data are able to test the authors’ proposed hypotheses by satisfying the approved outcome-neutral conditions
-- Whether any changes to the sections of the paper included in the “In-principle accepted” Stage 1 submission (other than the addition of results and interpretation) are properly highlighted and well justified
-- Whether the authors adhered precisely to the registered experimental procedures, with any deviations well justified
-- Whether any unregistered post hoc analyses added by the authors are justified, methodologically sound, and informative
-- Whether the authors’ conclusions are justified given the data
Editorial decisions will not be based on the perceived importance, novelty, or decisiveness of the results.
The Center for Open Science provides many resources for authors of registered reports. We relied on their list of instructions to authors.
CUP also provides useful guidance on registered reports:
https://www.cambridge.org/core/open-research/registered-reports
https://www.cambridge.org/core/services/peer-review/how-to-peer-review-registered-reports
The Journal of Politics provides extensive guidelines including a FAQ. We have fewer explicit requirements than they do, but their requirements may provide very useful guidelines to authors, especially if authors are considering submitting to both journals.
This document is released under a CC BY license and may be used and repurposed by others with attribution.