EMBRACE THE PEER REVIEW: GUIDELINES AND STRATEGIES FOR WRITING A HIGH QUALITY PEER REVIEW AND GETTING AHEAD OF CRITIQUES AS AN AUTHOR

MICHAEL FITZGERALD

THE WORST THING YOU CAN DO TO AN ACADEMIC IS IGNORE THEM

GOALS OF THE WORKSHOP

• Learn how to get involved in peer reviewing
• Identify qualities of an effective peer review
• Dos and Don’ts in providing feedback
• Using two theoretical frameworks, describe the core principles used to evaluate manuscripts

​

SETTING EXPECTATIONS FOR THE WORKSHOP

• We’ll be learning how to evaluate manuscripts, so we’ll also be talking about academic writing, building arguments, and the purpose of specific sections of manuscript (hopefully this is redundant)
• We’ll also learn how to provide or frame effective feedback
• Today could feel very similar to an academic writing workshop – we’ll use many of the same principles
• Spend time at the theoretical and conceptual level and move into the concrete (e.g., how to write an effective review)
• Concrete strategies will be provided as will examples (both good and bad)
• Secondary gain will be to improve your own academic writing
• Can be difficult to see the forest through the trees: Easier to recognize these errors in others’ writing rather than our own

SETTING EXPECTATIONS FOR THE WORKSHOP

• The motivation for the workshop are fourfold:
• Receiving poor-quality peer reviews of my own work
• Reading the reviews provided to authors on manuscripts which I have also co-reviewed
• Reflecting on my own training (or lack thereof) in learning how to do a peer review
• Safeguarding HDFS and science more generally
• We should be frustrated by poor peer reviews and grateful for high quality peer reviews
• You will see my frustration with poor peer reviews – I will get animated about this!
• You will see my gratitude for high quality reviews
• There are numerous examples and we’ll get through as many as we can but we will unlikely get through all of them

INTERACTING WITH THE WORKSHOP

• Ask questions throughout the workshop – workshop material is already available but questions and conversations about the material are not!
• Feel free to bring in your own examples (the good, the bad, and the ugly)

​

MY PEER REVIEW EXPERIENCE

• Reviewed for 34 journals
• Review an average of 50 articles per year (55 in 2025 – so far)
• Editorial Board Member
• Couple and Relationship Therapy
• Psychological Trauma: Theory, Research, Practice and Policy
• Traumatology
• American Journal of Orthopsychiatry
• Journal of Child and Adolescent Trauma
• Journal of Marital and Family Therapy
• Published 55 articles (and had many, many more rejected)

​

OBJECTIVES OF THE PEER REVIEW

• Evaluate the scientific merit of a manuscript prepared for publication in an academic journal
• Aspects of scientific evaluation: scientific advancement, theoretical and conceptual merits, the validity and reliability of the findings, and the overall quality of the manuscript
• Not about demonstrating your own knowledge and showing off to authors
• Serves as a gatekeeping mechanism in the dissemination of scientific knowledge
• Filter out research that is not rigorous or does not address a meaningful scientific problem

​

GETTING INVOLVED IN PEER REVIEWS

• Once you begin submitting articles to journals, you may be contacted by the journal to peer review (even as a graduate student)
• My first peer review occurred when I was in graduate school after submitting an article to a journal
• Registering in an academic journal
• Find a journal in your discipline and simply create an account
• I’ve registered for a journal to submit my own manuscript and within 24 hours, they invited me to peer review (bold strategy)
• Journals may reach out to you despite never having published in that journal
• Once you have an established knowledge base / expertise / reputation
• Rarer for graduate students
• Started happening during my 3^rd year as an assistant professor

​

GETTING INVOLVED IN PEER REVIEWS

• Sign up / reach out to journal editors
• Many journals have programs to get graduate students and emerging professionals involved (e.g., APA)
• Cold call journal editors to express interest
• Co-Review with Advisor/Mentor/Faculty
• Ask to peer review with a faculty member / mentor

PEER REVIEW PROCESS

1. A manuscript is submitted to a journal where it is appraised to be worthy of a peer review or not (e.g., desk rejection).
• Desk reject can be for manuscript quality or incongruence between journal scope and manuscript content
2. 2-4 peer reviewers are selected by the editor – some journals request that that authors recommend possible reviewers to make this easier – most I’ve seen on one paper is 5
• Reviewers are generally anonymous – sometimes their names are published AFTER acceptance (e.g., Frontiers)
3. Reviewers typically have 30-45 days to read article and submit (you can commonly get an extension if you need it)
4. Submit recommendation to editor and you are generally notified when a decision has been made

​

WHY BECOME A PEER REVIEWER

• In many journals, you can see the comments other reviewers and you can compare how you are thinking vs how others in the field thinking – a wonderful learning process
• Doesn’t mean your comments were “wrong” but invites a different perspective and a learning opportunity
• Stay on top of current and cutting-edge knowledge
• Learn what to do and what not to do in publishing research – insider information
• Can enhance your critical thinking skills that can then be applied to your own manuscript development
• Build relationships and professional connections!
• An editor I worked under wrote me a recommendation letter for an internal fellowship

WHY BECOME A PEER REVIEWER

• A way to get your name “out there” and demonstrate your experience and competence
• Join Editorial Boards
• Helps with the “national reputation”
• Service to the profession (often a part of your time allocation or is otherwise expected in TT positions)
• CV Line

PEER REVIEWING IS A VOLUNTEER POSITION

• Peer reviewing is an unpaid / volunteer position
• I believe it should be required for all researchers
• Not peer reviewing due to the position being unpaid is not an acceptable
• Others peer-review your articles, so you should repay the favor
• If you do not conduct research, then I don’t see peer reviewing as a requirement
• Don’t like peer reviewing being free / believe it is exploitative? More than fine, but….
• I don’t disagree with peer reviewing being exploitative
• By not reviewing articles, you are punishing the wrong entity (the researcher) when the entity that you have an issue with is completely different (e.g., journal, publisher)
• Peer reviewing can be viewed as your own professional development and investment into your own knowledge + skills
• If you don’t learn anything from peer reviewing, then that’s on you – there are opportunities!
• See previous slides for all the good things that come from peer reviewing

​

CHOOSING WHETHER OR NOT TO REVIEW A SPECIFIC ARTICLE

• Is the article within your circle of competence?
• You don’t have to be an expert in every area, but informed enough to evaluate the manuscript
• Example: I don’t really do qual research, but I’ve reviewed qual studies before because I’m a content expert on the topic and have a working knowledge of qual research methods + analysis
• I’ve also (rarely) reviewed articles outside my area of competence due to editorial board membership– this can be broad my horizons and allow the authors to defend their research to more general audiences outside their domains of expertise
• Do you have time to dedicate the 2-3 hours it takes to write an effective peer review?
• Can you complete the review within the allotted time-frame
• 30-45 days is quite standard
• Some journals are 21 days
• Others are 7 days (I never accept these – they are generally from open access journals)
• Do you know the journal? Is it where you have some content or methodological expertise?

CHOOSING WHETHER OR NOT TO REVIEW A SPECIFIC ARTICLE

• If you don’t have the time to review an article (or simply don’t want to) you can reject the request
• You won’t get blacklisted if you reject an invitation
• For some journals, they actually keep inviting you despite several declines (looking at you Current Psychology)
• Reviewing should not be a central feature of your day to day but is an adjunctive contribution you make to the profession

​

THEORETICAL UNDERPINNINGS

FRAMEWORKS FOR EVALUATING PEER REVIEWED MANUSCRIPTS

• Cohestion and Integration: Focuses on the connection and cohesion between the different sections of a manuscript
• Focuses on the relationship between the sections and evaluating the extent to which they are connected and consistent
• Macro focused
• Validity: Focuses on the four types of validity including construct, internal, external, and statistical conclusion (Shaddish et al., 2002)
• Focuses more on the content within the different sections
• Micro focused

BRIEF INTRODUCTION / REVIEW OF MANUSCRIPT SECTIONS

• Cohesion and Integration framework focuses on consistency across different manuscript sections -> what is the purpose of each section
• Sections:
• General Introduction: Introduction to the problem and variables, and “big picture” of the study (i.e., importance of the problem beyond the study)
• Answers the “so what” question
• Literature Review: More nuanced discussion of existing literature (e.g., what is known and what is unknown), theoretical and conceptual underpinnings, and hypotheses / study objectives
• Methods: How did you recruit your participants (i.e., sampling), what were the study protocols, and what are the specific ways you collected your data (i.e., measures)
• Statistical Analysis: Outlining the proposed analytic plan including specific data-related considerations, assumption testing, preliminary analyses and primary analyses
• Discussion Section: Integration findings with existing research and theory, limitations and future directions, implications and conclusion

COHESION AND INTEGRATION FRAMEWORK

• There should be a cohesive flow that connects the different parts of a manuscript
• Each section of a manuscript has a unique purpose, but all the sections should be connected
• The problem/gap, theory, methods, measurement and analyses should be connected to and support each other
• Results and discussion are also included, but their quality is largely dependent on everything that comes before

​

Gap

Theory

Literature

Sampling

Measurement

Analyses

A VALIDITY PERSPECTIVE TO EVALUATING MANUSCRIPTS

• Shadish, Cook, & Campbell (2002) propose four types of validity
• Internal: The extent to which there is a causal (or correlational) relationship between the independent and dependent variable
• External: The extent to which the findings of the study generalize across person, place, time, and setting
• Construct: The extent to which you are measuring (i.e., operationalization) what you are intending to measure (e.g., conceptual definition, focal concepts)
• Statistical Conclusion: Employing appropriate data management procedures and applying the most appropriate statistical method to evaluate the research question / hypotheses

A VALIDITY PERSPECTIVE TO EVALUATING MANUSCRIPTS

• Each form of validity has specific threats (non-inclusive list below) that temper the inference
• Internal: maturation, history, testing
• External: generalization across person, setting, time, outcome
• Statistical conclusion: low power, violated assumptions
• Construct: Inadequate conceptualization, incongruence between conceptualization and operationalization
• I’m thinking about each of these threats to validity when reviewing the manuscript and making comments that connect to these threats
• Because I am focused on a validity perspective, this workshop will have a focus on research methods

COMBINING THE FRAMEWORKS

• Integration and cohesion framework provides a “bigger picture” connection between the different sections and how they work together
• Validity perspective will be more “nuanced” and seen more within each section of the manuscript and far less seen between the different sections
• Validity is not ”confined” to a specific section and should be seen in EACH of the section of the manuscript
• If construct validity is being addressed as a gap then it should be seen in the literature review, methods, analyses (potentially), and discussion

CHARACTERISTICS OF A HIGH QUALITY PEER REVIEW AND STRATEGIES FOR PROVIDING FEEDBACK

QUALITIES OF A STRONG PEER REVIEW

• Provides clear, concrete, and specific comments that are aimed to help the authors strengthen their manuscript (even if you are recommending rejection)
• Highlight both strengths and weaknesses
• You provide a rationale and justification for the comments
• We expect authors to defend their decisions in the research, peer reviewers should justify (very briefly) their comments
• Not always needed (e.g., address typographical errors)
• Comments and feedback are focused on the scientific merit of the study and the ability consume such information

STRUCTURE AND FORMAT

• Introduction:
• It was a pleasure to review the manuscript entitled “XXXX” Below I offer my thoughts, comments, and reflections in hopes that it will help the authors strengthen their manuscript.
• Feedback
• My peer reviews are typically 1-2 pages long, single spaced; My reviews average about 600 words and the average peer review averages around 400 or less
• Bulleted list is more than sufficient
• You may or may not provide your disposition (e.g., accept/reject) to the authors (I rarely do this but I’ve seen it done quite often)

DO

• Highlight strengths / what you appreciate of the manuscript (I certainly need to do this more)
• Provide suggestions or ideas to the authors on how to address your concerns
• Don’t be tied to the author using your specific suggestions – it’s their research
• Suggestions can help illustrate your point more clearly
• When making a comment, link it to a specific place in the document (e.g., page 3)
• When making a comment, tell the author why you are making the comments (when necessary)
• Be open to your comments being “wrong” – can occur during a revision and 2^nd round of peer reviews
• The authors may educate you – don’t get defensive and consider their perspective

​

DO: PROVIDE SUGGESTIONS

• Positive comments that I have received from reviewers
• “The abstract would benefit of a first sentence where authors present the issue and/or literature on the theme. The aim and the structure of model are not clear. For example, authors should specify which kind of relationship they referred to (e.g. romantic one). The same is for abuse and depression. ”
• Identifies the problem and provides a general way of addressing the problem
• “It would be helpful to learn more details about the recruitment process and the study sample, especially given the subject matter. The Discussion also comments on the mean age of the sample being older than traditional college students. Is this related to recruitment, the setting, or any other factors that should be considered earlier or in the Discussion?”
• An area of concern connected to the science (i.e., external validity)

DO: BE SPECIFIC AND CONCRETE AND LINK TO AREA IN DOCUMENT

• “The introductory paragraph on mindfulness [pg 2, line 52 – pg 3, line 26] would benefit from strengthening the theoretical argument for mindfulness as key component of mental health. I would recommend adding an explanation of…”
• A more specific recommendation is provided

​

DO: TELL THE AUTHOR WHY YOU ARE MAKING COMMENTS

• Notice the difference between two different ways of providing the same critique
• “The authors should use the Common Fate Model instead of the Actor Partner Interdependence Model”

VS

• “There is a theoretical disconnect between the discussion of family level processes (Page 7) and the implementation of the APIM. The APIM does not assess family level processes, it only assesses how individual influence once another. Essentially, each dyad members assessment of family processes (e.g., conflict) still reflects an individual’s perception, thus remaining an individual level process. The common fate model is more appropriate to assess family level processes because it estimates common variances from multiple individuals reports of family level processes (thus measures those processes at the level in which they occur).”

DO: TELL THE AUTHOR WHY YOU ARE MAKING COMMENTS

• Notice the difference between two different ways of providing the same critique
• “The authors should use the Common Fate Model instead of the Actor Partner Interdependence Model”

VS

• “There is a theoretical disconnect between the discussion of family level processes (Page 7) and the implementation of the APIM. The APIM does not assess family level processes, it only assesses how individual influence once another. Essentially, each dyad members assessment of family processes (e.g., conflict) still reflects an individual’s perception, thus remaining an individual level process. The common fate model is more appropriate to assess family level processes because it estimates common variances from multiple individuals reports of family level processes (thus measures those processes at the level in which they occur).”

Feels very definitive as if the reviewer cannot be wrong

Reviewer recognizes the issue could be a framing issues in the introduction OR and analytic issue. Also provides a very brief explanation of the differences if the authors are not informed

DON’TS

• Add in personal preferences (unless you name it as a preference), you are evaluating the SCIENCE
• You do not have to agree with the author’s decisions, but evaluate whether the author’s decisions are scientifically justifiable/defensible
• You comment could be a change in the paper OR a limitation
• Ask researchers to do something that is fundamentally not a part of the manuscript (e.g., different outcome variable)
• Correct every minor grammatical/typographical mistake (you can, but copy editor isn’t your role)
• Add in your published studies to cite unless your studies are uniquely suited to be cited
• This is NOT an opportunity to increase your h or i10 index
• You also could inadvertently let the authors know who you are (blind reviews are no longer blind)
• Have the authors try and guess what you are trying to say (i.e., mind-reading)
• Frame comments about the researcher’s character

​

DONTS: REQUIRE CERTAIN CITATIONS

• My team and I had submitted an article and it went though three rounds of peer review
• During each round of the review, the same reviewer asked us to cite very specific authors after every round of reviews
• Final round of reviews was simply to add their ”suggested” articles
• Most of the citations were not relevant to our research question / topic
• ESPECIALLY if they are your citations
• There will be seminal citations that should be included
• When talking about a theory, using the seminal citations is appropriate and if authors do not use them, it is fair to recommend their use
• Cite Bowlby 1969/1982 when using attachment theory

ADDING IN YOUR OWN CITATIONS

• “ACEs: The authors need to provide a rationale for using the sum score of the ACEs measure as there are substantial theoretical and empirical problems with a total score (see articles below)
• Fitzgerald, M., & Bishop, A. (2024). Challenging the use of the overall adverse childhood experiences (ACEs) score: Comparing total ACEs, maltreatment, and household dysfunction on mental health problems among White, African American, and Native American women under correctional control. American Journal of Orthopsychiatry.
• Fitzgerald, M., & Gallus, K. L. (2024). Conceptualizing and measuring childhood adversity: A comprehensive critique of the adverse childhood experiences measure and offering a new conceptualization of childhood adversity. American Journal of Orthopsychiatry.
• Kelly-Irving, M., & Delpierre, C. (2019). A critique of the adverse childhood experiences framework in epidemiology and public health: uses and misuses. Social Policy and Society, 18(3), 445-456.
• Negriff, S. (2020). ACEs are not equal: Examining the relative impact of household dysfunction versus childhood maltreatment on mental health in adolescence. Social science & medicine, 245, 112696.
• McLennan, J. D., MacMillan, H. L., & Afifi, T. O. (2020). Questioning the use of adverse childhood experiences (ACEs) questionnaires. Child Abuse & Neglect, 101, 104331.”
• I include two of my own studies in the feedback to the author
• One is a conceptual article and the other is an empirical follow-up
• I included them as they were published most recently and provided additional information that was not available in the other three studies
• I didn’t ONLY provide my studies

​

DONTS: MINDREADING

• “I’m not convinced that your statistical modeling is accurate.”
• Doesn’t tell you why or provide a defense to their position
• Presumes that the reviewer knows more than the author – power imbalance
• “The following sentence/paragraph is unclear”
• Guess what…so your feedback!
• This involves mind reading.
• How I respond:
• “We have clarified our argument by doing XXX”
• “We are unsure what specifically this reviewer was alluding to, but we have attempted to clarify our position by doing XXX”
• Don’t get overly snarky with reviewers

​

DON’T: GOING BEYOND THE ARTICLE

• “First, the paper does not discuss the considerable literature on child abuse and risk for substance use, risky sexual behavior, and dating violence, all of which are important considerations when considering young adult romantic relationships.”
• Comments from another reviewer on a manuscript that I also reviewed
• The problem: None of these were variables in the manuscript…If you want to investigate those things, you can go for it! But it is not the focus of the current paper

​

DON’T: CONDESCENSION

• Comments that I have received from peer reviewers
• “Quite frankly, the paper was a little boring.”
• Problem: Unhelpful, nonspecific, and unnecessary
• “Here is a mention of "females" as 'significantly underrepresented.' Female what, what species? Please add 'respondents' or 'participants' after the gender or 'female-only participant samples' etc”
• Problem: In the social sciences, we only study humans…this is self-evident.
• Comments I’ve seen
• “Although some of our remarks might appear to be excessively critical, like Cudeck (1989) we strongly believe ‘‘that it is good for one’s character, not bad for it, to acknowledge past errors and clearly be capable of learning’’ (p. 317)”
• Problem: Contempt

​

​

DON’T: CORRECT EVERY MINOR GRAMMATICAL/TYPOGRAPHICAL MISTAKE

• Your time is valuable – so focusing on the specific grammatical, typographical, or clerical errors isn’t an effective use of your time
• Can be incredibly time consuming and this is not an effective use of your psychological resources - apply those resources to the science!
• Copy editing is the responsibility of the authors and the journal
• Feedback can be simplified to one sentence related to need for clarification
• I provide one sentence and one example place in the document where the issue is present

DON’T: PROVIDE SHORT AND UNINFORMATIVE REVIEWS

• The entire peer review:
• “I would like to thank you for the opportunity to review this manuscript. The present study aimed at understanding the role of COVID-19-related workplace stress and institutional betrayal on mental health among nurses. I read the main text of the article but h did not find any innovation in the results. I believe that this paper is not a significant contribution that explains these findings more than previous research has already done. So, I do agree the importance of this topic, but I'm not sure how your added research will dispel many of the misunderstandings surrounding this difficult situation since the data was only described and you did not contribute significantly to better manage these problems.”
• “I would like to thank you for inviting me to review this manuscript. I read the article carefully but didn't find the novelty in the subject of the research and the obtained results. Besides, the biggest limitations of this study are the cross-sectional nature of the data collection and low statistical power, which severely affects the accuracy of the results and its generalizability.”
• Problems: Very little effort given to providing a thorough review and feedback is vague and largely non-specific
• Hint: If you have to qualify that you read the document, then you

REFLECTION: WHICH THINGS THAT I LEFT OUT DO PEER REVIEWERS DO THAT ARE MADDENING OR HELPFUL

Maddening

• What are

Helpful

APPLYING THE THEORETICAL FRAMEWORKS

NUANCED APPLICATION OF CONCEPTS AND EXAMPLE COMMENTS

GETTING MORE SPECIFIC

• We’ll now transition to focusing on what I look for in specific sections of manuscripts
• Example comments from my peer reviews will be provided
• Note: The examples are for illustrative purposes only – they are not indicators of requite knowledge to conduct a peer review
• Apply the aforementioned etiquette to the theoretical frameworks
• The two frameworks will be integrated together and presented together

EVALUATING A MANUSCRIPT: GENERAL INTRODUCTION

• What are the key variables in the study and how are they defined?
• Innovation of the study – is this study going to make a significant contribution to the field and spark new ideas and research / influence practice and policy?
• The study should clearly state how it is innovative, new, novel, and addresses a significant problem/gap – you shouldn’t have to look for it
• This can be a practical, applied, theoretical, methodological, or policy advancement (or multiple)
• Contributions / advancement is incremental
• Importance of the study – does the study state important gaps in the research and does it effectively address the gaps?
• A gap in the research is a requisite but insufficient motivation
• Who cares about the study of 71 degree classrooms vs 70 degree classrooms and the number of questions students ask? NO!

EVALUATING A MANUSCRIPT: GENERAL INTRODUCTION

• What is the “bigger picture” of how the study / how could it be informative to larger scientific progress (i.e., significance)
• Example: What are the problems with existing ways of conceptualizing the variable of interest and how will your reconceptualization address the existing issues?
• Example: Currently inappropriate or ineffective application? Are prevention and intervention efforts addressing the wrong mechanism? Is there a critical piece moving? Can the program be condensed?

​

EXAMPLE COMMENTS: PURPOSE OF THE MANUSCRIPT

• “The largest issue that I see is that I can’t quite decide whether this is a case study or a theoretical integration with a clinical vignette / demonstration. I think the paper would have more value for a theoretical integration piece where there is a brief summary of EFT and IPNB and then discussing how the concepts of IPNB work with EFT in a more developed way (e.g., bringing in the specific building blocks of EFT and discussing how IPNB can aid in the development of those blocks). Not that the authors have to do this, but I think there is somewhat of an identity issue with the paper. Once resolved, I think it could be an impactful work”
• “The importance of the gap is missing – why is a specific focus on justice involved individuals needed. Why can’t existing research on EMDR be generalized to this population? What is unique about this segment of folks?”
• “Why was social support from partners chosen? What is the "so what?" (i.e. relevance / importance) for the choice of partners?”

EXAMPLE COMMENTS: IMPORTANCE / INNOVATION

• “Likewise, what is the theoretical importance of dissimilarity between partners? What does this give us that the APIM or CFM don't give us? For example, the shared experience of betrayal, powerlessness and other traumagenic dynamics may create mutual understanding and bond between partners that cannot be captured by other analyses.”

​

EVALUATING A MANUSCRIPT: LITERATURE REVIEW

• Theoretical framework(s) and conceptual underpinnings
• Is the framework consistent with the proposed relationship among variables / purpose of the study?
• Is it the best possible way of understanding the variables?
• If multiple theories are used, are they integrated together or do they simply co-exist?
• Literature review
• Is the reviewed literature consistent with the research questions being proposed?
• Are the reviewed studies integrated together and is a conclusion reached or are you left to form your own conclusions?
• Does the lit review “prove the gap” and build an argument for the study?
• Does the progression of ideas in the general introduction parallel those in the literature review?

EVALUATING A MANUSCRIPT: LITERATURE REVIEW

• Literature review generally brings about many critiques from reviewers
• Building an argument for a study is difficult and complex!
• Common issues in the literature review
• Lack of cohesion and progression of ideas within and between paragraphs
• Inconsistent arguments or even contradictory evidence
• Failing to “prove the gap”
• Lack of a theoretical framework
• Lack of integration of multiple frameworks
• Insufficient description of gaps

EXAMPLE COMMENTS: GENERAL COMMENTS MADE IN A LITERATURE REVIEW

• These sentence are addressing different processes: “On the other hand, school connectedness is a potential protective factor against childhood maltreatment (Diggs et al., 2024). A survey indicated that school connectedness is especially beneficial for adolescents who have experienced early adversity (Markowitz, 2017).” – first is discussing maltreatment prevention and the second is a moderating effect following maltreatment.
• I would like to see a little bit more nuanced in the empirical findings related to the research questions. For example, add one study that describes the findings of how parental mental health influences parenting following the third sentence (paragraph 1 on page 7) would ease the transition between the “effect” of parenting idea and the “adaptation” idea in the following sentence. This could also be an opportunity to foreshadow specific variables of interest within the methods section (i.e. variables that the PMTO intervention has be effective at addressing).

​

EXAMPLE COMMENTS: LITERATURE REVIEW (CONSTRUCT VALIDITY)

• “Adverse childhood experiences appears to be used interchangeably with maltreatment which is not the case – they are conceptually distinct ideas.”
• The problem: Differentiating constructs within the nomological network pf adversity
• “It would be helpful to clearly conceptualize and define what metabolic syndrome is in the current study. From a medical standpoint, it’s generally defined as 3 or more of the 5 criteria exceeded (hence my questions above about how many young adults actually have metabolic syndrome), but it isn’t measured that way as scores can range from 0-5. To be clear, I think it’s a good decision to have scores ranging 0-5 in the study, but the lack of conceptualization adds to the mystery already surrounding metabolic syndrome throughout the entire manuscript. Based on the operationalization of metabolic syndrome, the authors are not assessing the syndrome itself, as indicated in the title, they are measuring the number of symptoms. Metabolic syndrome would be a dichotomous outcome.”
• The problem: Incongruence between conceptualization and operationalization
• Note: This comment was made by my doctoral student (Viktoria Papp)

​

EXAMPLE COMMENTS: LITERATURE REVIEW

• “Why is it problematic to assume that the effects of maltreatment influence distal outcomes through the same mechanism? This needs to be expanded on rather than stated. Relatedly, at the end of the “New Ways of Conceptualizing Child Maltreatment” section, it would be helpful to briefly mention the importance of using the dimensional model for physical health outcomes. The authors mentioned that threat vs. deprivation makes a difference in mental health outcomes (e.g., internalizing and externalizing problems), but since the study also includes a physical health outcome, I’d like to know how the authors think about the differential impacts of threat vs. deprivation in a physical health context.”
• The problem: inconsistency between conceptualization and operationalization
• “There are far too many theories being utilized – through the first page there are three different theories mentioned and it is unclear how they work together and from which the hypotheses are derived.”

EVALUATING A MANUSCRIPT: METHODS

• Everything within the methods should provide enough detail for an independent researcher to replicate the study
• Some of that detail can be in online supplementary material
• Who are the people that the study recruited and how?
• How did the researchers get access to the people and how connected to the population is the sample (i.e., sampling method)?
• Do you know who the larger population that the researchers are trying to make generalizations about?
• Specificity is key
• An online sample from social media is far too vague: which sites, any specific domains within those sites (e.g., specific groups)
• What are the inclusion or exclusive criteria and what are the implications for the study (i.e., external and statistical conclusion validity)

​

EVALUATING A MANUSCRIPT: METHODS

• What are the study’s procedures (i.e., step by step walk through of the research design)
• Are these procedures realistic and valid given the field of study?
• In what ways were the data collected and are those valid and reliable (i.e., theoretically derived) ways of collecting the data given the study’s objective(s)?
• What indicators of reliability and validity do the measures have (i.e., construct validity)
• What is the temporal sequence of assessment in longitudinal research? Would the construct of interest change in the specified time between assessment timepoints?
• Variables can be operationalized in numerous ways – does the current operationalization make sense?

​

EXAMPLE COMMENTS: THREATS TO INTERNAL VALIDITY

• “Did the authors consider regression artifacts as a threat to internal validity. One of the inclusion criteria was frequent corporal punishment which could lead to parents just getting better because they were high on frequent corporal punishment. Were any sensitivity analyses run?’
• “Why were those with psychiatric conditions removed from the sample? The outcome variable is psychological distress – essentially, the authors have cut out the top portion of the distribution (i.e. restriction of range) which may result in attenuated or even inverted associations (Berkson’s paradox).”
• Problem: inclusion criteria may be a threat to all forms of validity due to restriction of range of dependent variable (without justification for considering subclinical distress)

​

EXAMPLE COMMENTS: RECRUITMENT

• “The authors should speak to MTurk as a valid and reliable way to collect data and to provide context for external validity”
• The Problem: who does Mturk represent? Key to both internal and external validity are discussed here
• “What were the specific convenience sample methodologies used. For example, email lists for what? What type of online advertisements and where we they posted. This has significant implications for generalization (external validity). If it was in a facebook group for relationship problems, for example, that would create very different conclusions than a general facebook advertisement.”
• The Problem: Lack of specificity in recruitment which makes evaluating external validity difficult if not impossible

​

EXAMPLE COMMENTS: MEASUREMENT

• “What is the rationalization for dichotomization? The authors are equating someone who experienced one incident of sexual abuse with someone else who experienced more than 10 incidents of sexual abuse….. Thus, acute and chronic abuse are equated which seems highly problematic. The total score does not represent more threat/deprivation. It represents a greater number of types of deprivation (physical and sexual abuse; emotional and physical neglect). The authors eliminated the more exposure interpretation via dichotmization”
• “Why was the CTQ overall score used in contrast to investigating specific subscales. For example, emotional abuse has the strongest effects to depressive symptoms (Nanni et al., 2012; Nelson et al., 2017)”

EVALUATING A MANUSCRIPT: STATISTICAL ANALYSIS

• Goal is for each step in the statistical analysis be clearly outlined and follow a logical and progressive pattern so that it could be reproduced
• Could you reproduce the analytic plan given the data set?
• Statistical Analyses should describe the methods of how the research question will be answered and each step should be discussed
• Are assumptions discussed?
• How will accuracy of results (e.g., interrater reliability, multiple coders, sensitivity analyses, model fit in SEM)
• How will “significant” results be determined
• Are the analyses appropriate / able to answer the research question?
• How robust will the results be with the proposed analyses?

EXAMPLE COMMENTS: STATISTICAL ANALYSIS

• “The effect used in power analysis is not specific, the effect size of what? Is it a regression coefficient? Interaction?”
• Problem: There is a lack of precision in what is being calculated. Specify is needed so the appropriate parameter’s power can be estimated
• “I’m not an expect in qualitative analyses, so I will defer major analytic comments to those who are more knowledgeable than I. “
• It’s perfectly ok not to know anything about a major part of the study – skip right over it!

​

​

EXAMPLE COMMENTS: STATISTICAL ANALYSIS

• Inappropriate Modeling
• “The power analysis provided is not appropriate, largely because it ignores dyadic data. Instead, use one of several Shiney Apps or a monte carlo simulation. See also Ledermann et al., (2022) – but their recommendations are nothing more than estimates for a set of effect sizes and should not be used as a definitive rationale for sample size calculations.”
• “Throughout the introduction and literature review, the authors referred to infertility stress several times as a shared experience within couples, and also sometimes framed interpersonal emotion regulation in ways that could reflect dyad-level processes (e.g., “shared problem solving”). While I was reading these introductory sections, it seemed like the authors may be talking about these processes from a common fate perspective rather than an actor-partner interdependence perspective. What was the reason for using an APIM? I think there could be justifiable reasons for choosing the APIM, but I would like to see a clearer justification and would caution against using language that is more indicative of common fate processes"

​

EXAMPLE COMMENTS: STATISTICAL ANALYSIS

• Choosing most appropriate analysis
• “It will likely be worthwhile to mention (in 1 sentence) why the latent congruence model was not used as it would also examine similarity and differences in levels of maltreatment. What does this model give us that the LCM doesn’t?”
• “The authors note that there is variability in state to state policies related to immigration, wouldn’t the authors want to use a multilevel model to examine within and between state differences in policies? States policies related to immigrants are unlikely to be independent of one another due to being embedded within the United States. The between group would likely represent the average state policy and the within level would represent state to state differences that could be used to predict child mental health. The benefit is disaggregrating the policies to capture similarity and differences across states and this would isolate the effects of state whereas currently the states are presumed to be independent which I don’t think would be the case. Running at ICC would be a good first step (with the continuous score). This would still allow for an interaction with race to be tested.”
• Note the disconnect between introduction/theory and analyses

​

EXAMPLE COMMENTS: STATISTICAL ANALYSIS

• Inappropriate Analyses
• “Adding covariates is not overfitting – adding in unnecessary and theoretically unjustified covariates is over fitting. Age, for example, is a factor in birth outcomes where pregnancies among women who are in the later 30s are at higher risk so there may be infertility stress due to the female partner’s age and that needs to be accounted for. Currently, age is treated as if it doesn’t influence any of the variables in the model. ”
• The Problem: Covariates are needed to rule out alternative explanations (Internal Validity) and not overfitting (adding in unnecessary covariate that are not theoretically justified)
• “EFA and principle components analysis are two separate analyses. For the use of a PCA, which implies a formative latent construct (i.e., the indicators cause / give rise to the latent construct), there needs to be theory. Nearly all psychological constructs presume reflective latent constructs (i.e., observed indicators are caused by the latent factor). Moreover, the use of a principle components analysis and a CFA presume opposite causal directions of latent variable - observed variable associations.”
• The problem: CFA/EFA presumes a reflective latent variable while a PCA assumes an emergent latent construct

​

EXAMPLE COMMENTS: STATISTICAL ANALYSIS (STATISTICAL CONCLUSION VALIDITY)

• Missing analyses
• “Longitudinal measurement invariance needs to be established among the measures over time.”
• The problem: Excluded needed analyses
• “There is critical information missing in the statistical analysis including 1) factor loadings, 2) eigenvalues / scree plot, 3) model assumptions (e.g., linearity), and 4) percent of variance accounted for by the factors, 6) KMO test, and 7) how the authors went from 18 items to 8 items.
• “There is no statistical analysis section outlining the analytic method so the reader cannot evaluate statistical conclusion validity”
• The problem: Missing Information. You should be able to replicate the statistical analyses if given the same dataset + software
• Unclear Analyses
• “How did the author address the issue of local minima in LPA? What was the procedure for random starts? What statistics were used to differentiate the classes (AIC, BIC, BLRT) because of often conflicting statistics (e.g., higher entropy but sig BLRT test)?”
• The problem: Procedures in statistical analysis and how statistical decisions were made

​

EVALUATING A MANUSCRIPT: RESULTS

• The results section vary widely
• Qualitative research vs quantitative research or simple vs complex modeling procedures
• Feedback is commonly clerical; critiques of the results section will be straight forward – most problems should be apparent in the methods / statistical analysis section
• Common source of feedback is model-data fit in structural equation modeling
• Did the authors include the results of the covariates?
• Results of the study are presented in a complete (in accordance with the statistical analysis section) and clear manner
• Are results presented for each of the analytic steps proposed in the statistical analysis section?
• Are the results answering the research questions appropriately?
• Are the results interpreted correctly?

EXAMPLE COMMENTS: RESULTS

• “Some of the results speak to parenting more generally (e.g., basic needs, parental readiness), and seem to stray from parenting children with DBD – these results don’t align with the research questions so it feels like a distraction”
• Results of the study stayed significantly from the purpose of the study / research question
• “Some more detail in the qualitative findings would be helpful; some examples include: ‘Parents also noted how helpful it was to build a sense of community and belonging with the other participants in the group.’ Maybe add details of how interventionists specifically facilitated this community-building during the sessions. ‘Parents also described needing more intentional conversations on the intergenerational transmission of unhealthy parenting practices.’ This is important; add some examples of specific unhealthy parenting practices.”

​

EVALUATING A MANUSCRIPT: DISCUSSION

• Common outline of a discussion section
• Do the authors briefly summarize the purpose of the study and major findings
• Contributions to new knowledge and integration of all findings with existing theory and research
• Do the authors clearly outline the primary contributions AND contextualize them within the larger knowledge base (theory and empirical findings) / not restating the results
• Does discussion return to the theory offered in the introduction / literature review?
• Unsupported findings:
• Are explanations offered for unsupported hypotheses
• Limitations and future directions
• Limitations should be thoughtful rather than “boilerplate” (i.e., convenience sample); why are these actually limitations?
• Could the limitations mention be address within the context of the study – if so, why weren’t they?
• Future directions are offered to address the limitations

EXAMPLE COMMENTS: DISCUSSION

• Not sure if this would work statistically (I think it would, but I would have to dig deeper into the literature), but wouldn’t the authors need to constrain the path coefficients in the model to make the following conclusion: “CM remains the most favorable configuration for couple outcomes, including sexuality.” The coefficient may be larger but is it statistically larger?
• Larger vs statistically different are not interchangeable
• “There needs to be much more theoretical discussion for the unexpected results and should have it’s own dedicated paragraph on why these unexpected results may have occurred.”
• Problem: Selection effects when only discussing significant findings – its ok to have unsupported hypotheses, but tell us why you think that is?
• Hint: Writing up explanations for non-significant hypotheses IS YOUR NEXT study

​

EXAMPLE COMMENTS: DISCUSSION

• “The discussion section could be expanded and integrate more literature – a lot of it felt redundant with the results section. Notably, the authors could remind readers of the specific, unique, and novel contributions the paper makes and discuss the implications of such findings.”
• Problem: lack of theoretical application to the results section

MY MOST COMMON CRITIQUE IN THE DISCUSSION SECTION

• “I could not comment on the discussion section as what is discussed may fundamentally change as a result of the addressing the aforementioned conceptual, methodological and statistical issues.”

LACK OF COHESION BETWEEN SECTIONS

COMMENTS RELATED TO ISSUES RELATED TO COHESION

LACK OF COHESION AND INTEGRATION BETWEEN THEORY AND SAMPLE

• “The introduction should focus more on the differences between groups (e.g., immigrant vs non-immigrant) rather than the immigrant paradox, which is a generational difference. The sample included a sample of non-immigrants, so the immigrant paradox discussion isn’t applicable to them. It’s worthy of discussion as it is relevant to the immigrant folks and could inform the covariates. As is, the introduction seems to be exclusively focused on immigrants when the analysis is comparing immigrants to non-immigrants.”
• “I also don’t understand why the items were developed in collaboration with those who did not directly experience the earthquake, but the sample included some who did directly experienced the earthquake.”

​

LACK OF COHESION AND INTEGRATION BETWEEN THEORY AND MODELING

• “Throughout the introduction and literature review, the authors referred to infertility stress several times as a shared experience within couples, and also sometimes framed interpersonal emotion regulation in ways that could reflect dyad-level processes (e.g., “shared problem solving”). While I was reading these introductory sections, it seemed like the authors may be talking about these processes from a common fate perspective rather than an actor-partner interdependence perspective. What was the reason for using an APIM? I think there could be justifiable reasons for choosing the APIM, but I would like to see a clearer justification and would caution against using language that is more indicative of common fate processes.”

LACK OF COHESION AND INTEGRATION BETWEEN THEORY AND RESULTS

• Much of the course of treatment section is describing the EMDR protocol with less attention focused on how EMDR can be applied to justice involved folks. Since this is the unique contribution, what is unique, novel, or specific to justice involved folks needs to be increased. I’m trained in EMDR and I read through this section of the paper I see the protocol clearly but I didn’t see anything that was new or novel. EMDR has been previously applied to forgiveness, suicidality, and has been applied in prison settings. As is, reading this article wouldn’t change much about my clinical practice of EMDR. Identification of a more specific gap and using the session information to more explicitly address that gap could make the manuscript helpful and impactful.

​

LACK OF COHESION AND INTEGRATION BETWEEN THEORY AND MODELING

• “I'm not quite sure what the rationale for the bi-factor model - the authors will need to provide a rationale for why a bifactor model was utilized over a simple 2 construct factor analysis and what the implications for doing so are.”

​

OVERALL LACK OF COHESION / PRESENTATION

• “In the third paragraph, the authors are revisiting the definition of survivor guilt which was already discussed in the opening paragraph. There is a lack of cohesive flow / progression of the ideas. I would recommend first starting with the conceptual definition of SG and then move towards the inadequacy of existing measures”

​

SUMMARY COMMENT

• “Overall, it was an absolute pleasure to read the article. Unlike a majority of studies, the authors did an exceptional job defending and supporting their various decisions (e.g., choice of recruitment from churches), they clearly integrated the introduction, method, and analysis together to create a cohesive study, and presented a good argument that was not overly technical and was highly accessible to various readers. I would suspect that the participants in their focus groups could consume this article with relative ease which is exceptionally commendable. I review about 50 articles a year and this is among the most thoughtful, cohesive, and well executed studies that I've read in the past 3-5 years. I hope to review more of the authors' manuscripts in the future.”

MOST COMMON ERRORS (IN SOME SEMBLANCE OF AN ORDER)

1. Only discuss gap but no justification for why the gap is important
2. Measurement problems
3. Statistical modeling Issues
4. Lack of cohesion between sections
5. Lack of sufficient information or information is disorganized and unclear (especially in the methods)
6. Going beyond the findings in the discussion section / misinterpreting the results

MAKING A DECISION

DECISION

• Decision will vary from journal to journal but generally include the following:
• Accept
• Minor Revisions
• Major Revision
• Reject (with and without transfer)
• You will provide comments to the authors as well as your recommendation (not necessarily directly to authors)
• Should be a holistic evaluation of the manuscript (strengths and weaknesses)
• You also have the opportunity to provide the editor confidential comments not seen by the authors
• I don’t use this a whole lot, but commonly do when 1) authors are unresponsive to reviewer’s concerns, and 2) When I’m torn between revise / reject

DECISION

Reject

• Contribution to science is unclear or the research question is not of significant enough important
• Methodologically flawed
• Consistent problems across multiple manuscript sections
• Contribution could be significant and important, but the execution of the study is fatally flawed
• There are “red flags” in the manuscript
• Lack of transparency
• The manuscript would need to be overhauled and many section would need significant medication or starting over

Accept

• The study is scientifically sound and makes a significant contribution
• No further revisions are necessary and the manuscript is able to be published as is / in current condition

DECISION

Minor Revisions

• The manuscript is largely ready for publication and any errors are clerical
• There are no theoretical, methodological, or statistical flaws that would preclude publication
• It is anticipated that the manuscript will be published

Major Revision

• There is high potential for publication but there are also significant problems that can be addressed
• There are no fatal flaws in the methods / statistical analyses
• If you blow measurement and sampling, the study cannot be saved
• There are some substantial issues with theory, literature review, and/or discussion; methods and analyses could use additional clarification or supplementary analyses

​

WHEN IN DOUBT, I REJECT

AS AN AUTHOR

• A secondary aim of this workshop is to help you strengthen your own research skills
• If we are evaluating other’s research then we can apply those same ideas to our own research
• If we have to evaluate statistical conclusion validity in others study then we should be doing this in our own research
• Use this framework to in your own work
• Review your existing work or that manuscript that you need to resubmit to a journal but you’re avoiding working on it
• Get ahead of the critiques proffered by peer reviewers
• As a peer reviewer, nothing frustrates me more than seeing a problem, writing out a critique and the critique is addressed in the next paragraph

​

CONCLUDING REMARKS

• My philosophy isn’t the only way to conceptualize the peer review process
• Take the good and leave the bad
• A high-quality peer review integrates a tactful way to provide feedback with an understanding of various aspects of research (i.e., validity)
• The best way to write high quality reviews is to review! Get involved!
• Applied learning is the best way to get something “in your bones”

ADDITIONAL RESOURCES

• If you want to review but are unable to get experience from advisors, editors ect , email me, tell me you were in this workshop, let me know what your research interests / knowledge base is, and I will keep you in mind for when I get invited to review
• I will co-review with you and we’ll meet on Zoom to discuss the article and I will submit an integrated review to the journal
• If you registered for the TCRM in 2025 (TCRM Paper Session 1), I served as a discussant and provided feedback for three papers
• These papers are available are on the TCRM website and provide additional examples of how I conduct a peer review
• https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html

​