A guide to being �a journal editor �at Collabra: Psychology

Created by Simine Vazire in November 2022

Last updated 13 February 2023

Disclaimer

This guide was created by Simine Vazire, Editor in Chief of Collabra: Psychology. Its aim is to share advice with Collabra editors so that we can bring our informal practices out in the open, and increase consistency across editors. Of course, there will always be some differences among editors, and not all editors will agree with or follow everything in this guide. Our editors are volunteers, and they are encouraged to use their judgment on a case-by-case basis. However, by sharing this guide and encouraging discussion of its content among the Collabra editing team, we hope to come to more thoughtful and consistent practices across the journal.

�By sharing the guide publicly, we are also hoping to increase the transparency of our informal practices, to help potential authors, reviewers, and future editors know what to expect. However, the content of this guide is not binding and individual editors may deviate it from it - only the policies on the Collabra website are formal and official. We also hope this guide may inspire other journals/editors to reflect on their practices, and share their own ideas and approaches.

​

If you have any feedback about the guide, please get in touch with Simine Vazire (first name at gmail dot com).

OUTLINE

1. General principles
2. Common reasons for rejection
3. Step-by-step guide
1. New manuscript
1. Conflicts of interest
2. Desk rejection
3. Finding reviewers
4. Inviting reviewers
2. Making a decision
3. Handling a revision
4. Special cases
• Registered Reports
• Streamlined review

General Principles

General principles

1. Own your decisions. You are the decider, not the reviewers. You do not have to go with what the reviewers think, even if they agree with each other, and you should be prepared to go against the reviewers if your evaluation differs, after considering their points.

Justify your decisions (even desk rejections), and make it clear what the major factors were that led to your decision. Imagine that your decisions will be made public (we give authors the right to share their decisions and reviews however they want), and make sure you would stand by what you wrote.

General principles

2. Reason from principles, values, and evidence, not from norms or habits. It can be tempting to apply a widely-used norm or rule without really thinking about it. Or, we might feel like we have to be tolerant of a research practice because it’s so widespread. But our job is not to reinforce existing norms and practices - we should be thoughtful and think critically about how we evaluate research. I think of editing as applied critical thinking - all the tools and skills we teach in introductory research methods are relevant here. Just because “everybody’s doing it” doesn’t mean it’s a good practice or that we have to accept it. Conversely, we can apply standards that aren’t widely shared across the field if we can back them up with good reasoning and/or evidence.

General principles

3. Admit when you were wrong. Editors make mistakes, and part of owning our decisions is owning our mistakes. It’s fine to reverse a decision if you’re genuinely convinced that you were in the wrong. If you’re tempted to reverse a decision because the authors are angry and/or powerful, come talk to a senior editor or the editor in chief, and we’ll help you.

General principles

4. Resist status bias. As editors, we are very susceptible to authors’ status and reputation. It’s almost impossible not to be influenced by what we already know about an author or their institution, country, etc. There might be some valid information in that background knowledge, but there is also a LOT of bias. I try to read every paper without looking at the authors’ names and affiliations first, and only look at their names after I’ve made my decision. (This is hard to do as the online system doesn’t mask their identities from us as editors.) Even if you can’t shield yourself from the authors’ identities, try to reflect on whether you are being too charitable or uncharitable because of who the authors are or where they’re based.

General principles

5. Recognize your power - don’t act like a co-author. As editors, we often underestimate our status and power. We are, however, in a powerful position as editors, and we should recognize that power and be careful with it. Keep this in mind when you make decisions, and avoid getting so involved that you take on the role of a co-author. When you request revisions, make it clear which revisions are required and which are just suggestions that authors can take or leave, and keep your word.

Editors can sometimes think they are doing authors, especially early career authors, a favor when they give extensive feedback and take a big role in reshaping the paper. However, this heavy-handed approach can easily go too far - we are not co-authors, we don’t need to like everything about the paper, and we should not make authors write the paper that we think they should have written. We are here to evaluate the paper that the authors chose to write, and that they think is good enough to publish. Of course, we are supposed to use our power to identify “dealbreakers” - things the authors need to change to meet our standards if they want to publish in Collabra. But we should be careful not to overstep - don’t ask for things that are idiosyncratic preferences (or if you do, make it clear that authors are free to ignore them). Also, don’t ask authors to cite your work (I think this can be acceptable sometimes for reviewers, but almost never for editors).

General principles

6. Do the right thing. Each of your decisions shapes the incentive structure for researchers, and contributes to the journal’s reputation. Our decision should be guided by what’s best for science, and our journal’s stated policies and values. It can be difficult to reject a submission because of shoddy research practices, or to accept a submission that seems boring or unimportant, but those decisions protect the long-run track record of the journal, and create incentives that make science better. ��One trick I use to help guide my editorial decision-making is to imagine that on every paper, there is one author who was pushing their co-authors to do the right thing (use a better measure, recruit a more diverse sample, make more calibrated inferences, etc.), and my job is to vindicate that author. This sometimes means writing a quite critical rejection letter (in which case I imagine the idealistic author telling their co-authors “I told you we should’ve done xxx!”), and sometimes this means accepting a paper that other journals or editors might have rejected (“I told you it was worth it to do xxx!”).

Common reasons for rejection

What is ‘quality’?

• Remember that at Collabra, we emphasize transparency, ethics, rigor, and calibration.
• I find it very useful to think back to the core topics taught in intro to research methods classes:
• construct validity (did the authors successfully measure and manipulate the constructs they are studying?),
• internal validity (are causal inferences valid?),
• external validity (are generalizations warranted?),
• statistical conclusion validity (are the statistical inferences sound? are statistical errors minimized?).
• This tool (seaboat.io), developed by Sarah Schiavone, can help with identifying threats to validity
• See next slides for more concrete tips.

Some common reasons for rejection

Research design does not match aims. Examples include:

• aims are to study change over time but study is cross-sectional (see also next slide),
• aims are to study cause-and-effect relationships but design lacks adequate attention to confounds/alternative causal mechanisms (common in observational studies, but can also be a concern for experiments),
• aims are to understand a clinical phenomenon but population sampled is non-clinical,
• aims are to understand behavior but behavior is not measured,
• aims are to understand real world phenomenon but study lacks realism

Of course these can sometimes be ok, but very often authors do not calibrate their claims to match the limitations of the design.

Some common reasons for rejection

Research design does not match aims: Causal claims

• Most study designs are not well-suited to support strong causal claims. This can include:
• Cross-sectional (one time point), non-experimental studies
• Longitudinal (multiple time points), non-experimental studies - most of the time many plausible alternative causal explanations still can’t be ruled out (e.g., third variables). Includes mediation claims, cross-lagged panel models, and more.
• Experimental studies used to test mediation. Most of the time there is no attention to very plausible alternative causal models (e.g., other mediators correlated with M, reverse causality between M and Y, etc.). Comparing the indirect effects in different models does not tell you which model is more likely to be correct.
• If strong causal claims are made without appropriate attention to alternatives, and without strong conceptual/theoretical justification, consider rejecting the manuscript, or at a minimum require authors to calibrate their claims and make it clear their empirical results alone do not justify their causal inferences, and to explicitly state the additional assumptions needed to warrant their causal claims..

Some common reasons for rejection

Methods are poor. Examples include:

• measures have validity problems,
• manipulation has confounds or lacks appropriate control group,
• sample size is too small,
• sample does not match population of interest,
• used only one stimulus, or a narrow range of stimuli,
• inappropriate statistical tests,
• interpreting null results as evidence of absence without appropriate tests,
• interpreting significant result as support for hypothesis when the pattern of results doesn’t fit with hypothesis,
• ignoring inconsistent or inconvenient results

Again, if conclusions are appropriately calibrated, this can be ok, but they often are not.

Some common reasons for rejection

Signs of increased risk of false positives. Examples include:

• weak p-values (e.g., p > .02) interpreted as clear evidence of an effect
• a series of weak p-values across multiple studies (internal meta-analysis does not help here! See Vosgerau et al., 2019)
• obvious decision points with potential for flexibility (e.g., dropping participants, items, measures, etc.),
• no pre-registration or the pre-registration did not constrain decisions or was not followed,
• complicated effects (e.g., interactions) especially with small samples

Any of the these could be ok individually, but in combination they are reasons to worry about false positives.

Some common reasons for rejection

Writing quality. Generally, I err on the side of desk rejecting papers that are hard to understand, and certainly if this is because the ideas/thinking are unclear. However, I’ll make exceptions when the writing is poor if the core structure is there and it's clear what the author is trying to say, but usually only if there’s something exceptional about the paper. See this slide for handling these exceptions. Don’t do this often - you will get burnt out!

Exaggeration or hype. This includes all of the ways in which claims may be miscalibrated to the flaws of the study or the weakness of the evidence, as described above, but can also come in other forms (e.g., outsized claims of implications for applied settings, minimizing serious flaws or sources of uncertainty, etc.). While the individual claims can be adjusted in a revision, if the hype is extreme enough this could be a sign that the research is likely to be biased, or that there was an insufficient commitment to scientific integrity.

Some common reasons for rejection

Many errors. If I notice sloppiness or errors in the manuscript (e.g., some numbers are wrong, labels on graphs are switched, discussion misstates results, etc.), I begin to worry that the authors may not have put enough care into how they conducted the research, analyzed the data, etc. Ultimately, we need some trust in the authors’ process, and if there are signs that they were not careful, there may be too little trust left for it to make sense to consider the manuscript for publication.

Results are implausible. This is rarely grounds for rejection on its own, but in combination with other issues (e.g., no pre-registration, many errors, signs of flexibility, etc.), it can be a factor in the decision to reject.

Inappropriate reasons for rejection

Remember, at Collabra we do not reject for lack of novelty, lack of potential impact, having null results, not being interesting enough, or being too incremental. (If you think an exception is warranted, check with the Senior Editor or Editor in Chief.)

Step-by-step guide

Overview

Typical steps for handling a manuscript:

1. Check whether you have a conflict of interest. If not,

2. Read the manuscript and either desk reject (any editor can desk reject, not just senior editors) or

3. Invite 2-3 reviewers. As reviewers turn you down or don’t reply, invite more until you have 2-3 who have agreed to review.

4. Occasionally check in on the progress, nudge reviewers (using your personal email) as needed, until you receive at least 2 reviews (or, in rare circumstances, make a decision with only 1 review).

5. When enough reviews are in, read the manuscript carefully and thoroughly before reading the reviews. Start drafting your decision letter. Then read the reviews and finalize your decision letter.

New manuscript: Conflicts Of Interest (COIs)

• Check if you have a conflict of interest:
• Are you collaborating (on papers, grants, etc.) with any of the authors now?
• Have you collaborated or published with any of the authors in the last four years?
• Are you at the same institution as any of the authors?
• Were you ever a mentor or mentee of any of the authors?
• Do you have a close personal relationship with any of the authors (e.g., good friends, a romantic/sexual relationship, family members)
• If their paper is published with your name listed as the handling editor, would a reasonable person think you had a conflict of interest?
• Are any of the authors senior editors at Collabra?
• If yes, tell the Editor in Chief of Senior Editor that you can’t handle the manuscript because of a COI
• If no, proceed!

New manuscript: Desk rejection

• It’s often better for everyone (authors, reviewers, etc.) to desk reject manuscripts that have a low chance of acceptance. Use desk rejection as often as needed.
• Assume the authors wrote the best manuscript they could, and want to be evaluated on what they submitted. It can be tempting to “work with the authors” to have them do a major revision, but that’s rarely a good idea. The authors like their paper, and they think it’s publishable. If Collabra doesn’t like it or would need it to be heavily revised, they’d probably rather find another journal that likes it just the way it is (or close).
• If you decide to desk reject the manuscript, write a letter that gives the major reasons, in language specific to that manuscript (do not only use a form letter). (Collabra editors can access example letters - see the Slack channel “Editing tips” for info on how to access them.)

New manuscript: Desk rejection

Most common reasons for desk rejecting a manuscript:

• See “Common reasons for rejection” section above. If fixing these problems requires a major revision, it’s often better to desk reject the manuscript. Exceptions can be made when there is something uniquely valuable about the research, but even then it may be better to give a “desk reject & resubmit” and ask authors to fix the major problems before sending the manuscript to reviewers. This is also helpful because sometimes authors aren’t willing to make those changes, and in that case it’s best not to waste everyone’s time.
• Violates journal policies (e.g., regarding transparency or ethics). Usually our managing editor tries to catch this before it gets to you, but we could have missed something, or the authors could have misrepresented things (e.g., said the data are available but they’re not).

​

New manuscript: Finding reviewers

• Try to find reviewers on your own first (without looking at authors’ suggestions). Use your knowledge of the literature to identify people who do high quality research on similar topics or with similar methods.
• If you’re having a hard time thinking of people, search the literature the same way you would to find high quality papers on a topic (e.g., if you can identify at least one potential reviewer, search their collaborators’ papers, papers that cite and are cited in their papers, etc.)
• You can also look at the references in the manuscript for ideas, but don’t limit yourself to those names - look for researchers not cited, too.
• Check the authors’ cover letter (if there is one) to see if authors recommend any reviewers. If so, choose AT MOST one reviewer from this list (fine to choose none).
• If the authors list any opposed reviewers, honor that request unless you see any red flags (e.g., long list, lists all intellectual adversaries). It can be a good idea to solicit a review from someone in the same “camp” as the opposed reviewers, if you believe they are qualified.
• Before inviting reviewers, do a quick check that they don’t have an obvious COI. You don’t need to do a thorough check, but take a second to think about if you know of any COIs. Don’t invite reviewers you know to be recent co-authors or collaborators with any of the authors, or at the same institution as any of the authors.

New manuscript: Inviting reviewers

• Invite 3 or so reviewers. If you identify more potential reviewers that you might need to invite later, you can leave yourself a note in the “Notes” section on the left side of the page in Scholastica.
• If fewer than 2 reviewers accept the invitation after 1 week, invite more reviewers until you have at least 2. In some circumstances, you may want to get more than 2 or 3, e.g., if you want to make sure multiple perspectives are represented, if you need reviewers with special expertise, or if you believe the manuscript needs extra scrutiny. In some cases, it may be reasonable to proceed with only 1 reviewer, if your expertise is a close match to the manuscript and the reviewer does a thorough job.
• Make a decision when you have enough reviews (might be 1, 2, or more). Consider moving ahead with a decision even if there are reviewers who agreed to review who haven’t submitted their decision yet (especially if their review is overdue). Send these reviewers a message saying you will make a decision on X date (e.g., in 5 days), and that you would be grateful for their review if they can get it in by then, but if not, you will proceed without their review.

Making a decision

• Read the paper very carefully before reading the reviews. Take notes on the strengths and problems in the paper. Make sure to read the tables and figures carefully. If there is a preregistration, have it open as you read the paper and compare the paper to the preregistration. Check the links to open data, code, materials, etc., to make sure they work and look vaguely like what they’re supposed to be. I don’t typically read supplemental materials or open the data or code, unless something in the paper makes think I should, but if you are up for it, I think it’s a great idea to check those, too.
• Before turning to the reviews, you should have a good idea of the decision you would make at this point, and a list of the points you would raise in your decision letter. Sometimes I start drafting my decision letter now.
• Next, read the reviews. Consider whether the reviewers’ evaluations change your decision - you do not have to agree with the reviewers, even if they are unanimous, but you should be able to explain why you have a different overall assessment than the reviewers. If the reviewers raise any concerns that go against Collabra’s values or policies (e.g., criticizing a paper because the findings are unsurprising, saying that the paper doesn’t make a substantial contribution, etc.), ignore those points in your evaluation (and if you are giving a revise & resubmit, make it clear in your letter that the authors do not need to address those points).
• Consider each of the reviewers’ points raised and whether to incorporate them into your decision letter. You don’t have to repeat every valid point, only incorporate the ones you especially want to emphasize, or ones you have something to add to.

Making a decision

• You have several options: reject, invite a revision (“revise & resubmit”), or accept.
• It’s rare to accept a manuscript on first submission, but not unheard of, and I think we should be more open to it than we usually are. You can also give a “conditional acceptance” which asks authors to make very small & straightforward revisions (see “very minor revisions” slide).
• If major revisions would be needed, consider whether it might be better to reject - see “revise & resubmit” slides for more tips.
• If you’re making a decision on a revised submission, try to push yourself to make an accept (or conditional accept) or reject decision. Avoid the trap of never-ending “revise & resubmit” cycles.
• When you’re ready to accept a manuscript (usually a revision), you don’t need to write a unique decision letter, just use the “accept” template and edit the text to fit your style.

Making a decision: Reject

In your letter, describe the most important reasons for your decision to reject the manuscript. If the reviewers also raised any of these points, repeat the point in your own words and refer to their comments. You do not need to list every problem with the paper, or refer to all of the reviewers’ points, but list all of the major problems that led to your decision. In some cases, I also list some more minor problems (especially if they are clear errors that the authors will want to fix before submitting elsewhere), but I explicitly state that these smaller issues were not the basis for my decision. Sometimes, I also give feedback that is more subjective, or are more like suggestions than errors, but it’s really important to be clear that these were not the basis for the decision (because often authors will justifiably feel that they have good counterarguments to these types of points).

Making a decision: Revise & resubmit

Think of a revise & resubmit as a contract - spell out all of the changes authors will have to make, or issues they will have to resolve, in order to have their paper accepted. Then, you can also list changes that are not required but are just suggestions. However, if the paper is exceptionally strong on an important dimension (or strong in a rare way that our field needs to see more of), I may be willing to invite a revision even if I’m not convinced it’ll be accepted. Most of the time, though, I only give a revise & resubmit if I can see a clear path to acceptance, and can articulate what would need to happen in the revision. If that’s not the case, consider rejecting the paper instead.

Your decision letter should list all of the points you think are important, and either state that all reviewers’ points are relevant, or identify which ones do or do not need to be addressed. If you think any of the reviewers’ points should be ignored, you need to say so. Be tactful (the reviewer will see your letter), but be clear.

For all revise & resubmit decisions, ask authors to include a cover letter detailing whether and how they have addressed each point raised by you and the reviewers (this should already be in the template).

Revise & resubmit - special cases

New conclusion. One tricky issue I face in revise & resubmit decisions is when I would need authors to change their conclusions before I would accept the paper. This happens most often when the evidence is mixed or even supports the null hypothesis, but authors conclude that the evidence supports their research hypothesis. In these cases, I make it clear how major the change would need to be, including changes to the overall conclusion and the abstract (and sometimes title), and I explicitly say that I understand if authors are not prepared to make this change and choose not to submit a revision. (If the results are inconclusive and collecting more data is feasible, you can offer this and ask that the authors submit their revision as a Stage 1 Registered Report, with the existing data as preliminary results, but the overall conclusion will be determined by the results of the proposed new study(ies).)

New data. Another special case is when you are asking for or encouraging new data collection. Again, in these cases, you should seriously consider whether a rejection might not be more appropriate (remember, evaluate the paper that the authors submitted, don’t try to shape it into the paper you wish they’d written). However, there are cases where it makes sense to invite a revision that would entail new data collection. In these cases, consider asking the authors to submit their revision as a Stage 1 Registered Report, where the existing data and results are presented as preliminary studies and the main conclusion will be based on the new data, collected as part of the Registered Report. At a minimum, indicate that new data collection is expected to be preregistered (but we are open to exceptions).

Revise & resubmit - special cases

Writing quality. As mentioned above, I generally, err on the side of (desk) rejecting papers that are hard to understand, and certainly if this is because the ideas/thinking are unclear. However if the writing is poor but the core structure is there and it's clear what the author is trying to say, I’ll consider working with the author to heavily revise the writing, but usually only if there’s something exceptional about the paper (e.g., a really valuable dataset, a special population or topic that’s important to study). Don’t do this often - you will get burnt out!

In rare cases where I decide it’s worth helping fix the writing, I try to make very clear and concrete requests, including specific corrections/changes, an alternative structure, etc. Rarely, I’ll track my edits in a Word doc and send it to the authors (or I might do this only for the first few pages, if I think that’s enough to give the authors a clear idea of what the revision should look like).

Don’t make assumptions about the authors (their fluency in English, career stage, etc.), just give feedback and make it clear what revisions are required. However, if you know that at least one of the authors is well-established and can write well in English, then in my opinion, it’s not worth putting this effort in. If they didn’t take the time to make it readable, it’s not fair to expect editors to do it for them. I don’t usually like treating papers differently based on who the authors are, but this is an exception I make.

Making a decision: Very minor revisions

It’s rare that a manuscript will be accepted on its first submission, but not unheard of. In most cases, there will be at least some points you would like the author to consider, or some small changes you would like them to make.

Similarly, you may want to ask for small changes after the first round of revisions.

In those situations, you can give a “conditional acceptance”, but you must select “revise & resubmit” as the decision in the online system, and explain to authors that the decision is recorded as a “revise & resubmit” in the system but is actually a conditional acceptance, and they will need to upload a revision and cover letter.

Handling a revision

If you feel that you can evaluate a revision without sending it back out to the reviewers, do so. I would consider sending a revision back out to reviewers if 1) you lack the expertise to evaluate it yourself, 2) if a reviewer has clearly indicated that they want to see the revision (this is very rare), or 3) you think the paper is likely to have a big impact (e.g., it’s controversial, makes bold claims, or will garner a lot of attention for some other reason) and needs extra careful vetting. However, in 90% of cases, I don’t send revisions back out for review.

Don’t send revisions out to new reviewers except under rare circumstances, e.g., 1) if the content of the manuscript has changed so much that you feel you need someone with different expertise to evaluate it, or 2) if there is an aspect of the manuscript (e.g., a method or argument) that the first round of reviewers did not evaluate, but now needs to be examined.

Try to avoid multiple rounds of “revise and resubmit” decisions. If you can’t give an acceptance or a conditional acceptance after one round of revisions, this might be an indication that you should reject the paper.

Special Cases

Streamlined Review

Authors can submit their manuscript for streamlined review if it was reviewed and rejected at (or they withdrew it from) another journal.

• The manuscript must have been sent out for review at the previous journal, and the authors must include the previous editors’ decision letter & reviews with their submission.
• The authors must include a letter describing their response to each of the points raised by the previous editor and reviewers, including any changes they made to the manuscript (like a cover letter for a revision).
• The Collabra editor can decide whether or not to send it out for review - all options are on the table. You can:
• Desk reject the manuscript
• Invite a revision, or give a conditional acceptance without sending it out for review
• Accept the manuscript without sending it out for review
• Send it out for review, then reject, invite a revision, give a conditional acceptance, or accept it.
• If the manuscript is accepted & published at Collabra, only the Collabra peer review history will be published, not the previous journal’s peer review history.

Registered Reports

If you are assigned a Stage 1 Registered Report to handle, use following resources, and ask the senior editor or editor in chief at any time if you have questions:

Collabra’s Registered Reports guidelines.

Some tips, especially on pages 34-35 ("Box 2. Top tips for reviewers"):�https://osf.io/preprints/metaarxiv/43298/

​

"Resources for editors" tab here: https://cos.io/rr/

When you accept a Stage 1 RR (“in principle acceptance”), select the “Revise & Resubmit” decision from the drop-down menu. Make it clear in your letter that this is an in principle acceptance, but in the Scholastica system it needs to be a revise & resubmit so that the authors can submit the Stage 2 manuscript as revision.