How to Review a Research Paper
Dmitry Berenson
Reviewing papers is an essential part of the scientific process and is one of the main responsibilities of a scientist. However, reviewing is more than just a duty, it also benefits the reviewer! Though we may prefer to concentrate on our own work, the in-depth understanding of a paper that you gain during the review may get you to think in new directions and may end up benefiting your own work. Reviewing also allows you to get a snapshot of what is currently going on in your research community; i.e. what methods are growing in popularity? What work are people citing? etc. While not every paper (or even the majority of papers) you review will give you these benefits, reading a great paper once in a while makes working through the mediocre ones worth it. Finally, reviewing helps you develop your own paper-writing and research skills by seeing examples of quality work (which you can imitate) and examples where papers fail (so you can avoid the same pitfalls). Overall there is good reason to actually be excited about reviewing!
This document describes the basic principles of reviewing as well as some ways to make your reviews helpful to the paper’s author(s). This guide is intended mainly for “standard” papers which are proposing some new method to solve a problem. The guide is not intended for the more exotic types of papers such as literature reviews or position papers. Before starting to write your review, you must make sure to have a good understanding of the paper. See the how to read a paper guide and read the paper at level 3 (described in the guide) for reviews.
Your review is being judged
All of the peer-reviewed conferences and journals I know are either single-blind (the reviewer sees the name of the author but not vice versa) or double-blind (both the reviewer and the authors are anonymous). In both cases, the author never sees the name of the reviewer. So you may think that, since the authors don’t know who you are, there is no real consequence for doing a lax review. However, this is not the case.
The person who is judging your review is not the author but the editor who assigned the paper to you. Often the editors are some of the top people in your field so it is very important that you make a good impression on them. If your review isn’t up to par, it will cause big headaches for the editor; they will either ask you for changes or drop your review and ask someone else. In either case, you not only look incompetent but also cause others to have to do more work. You don’t want this to happen! The investment of time required to do a good review is worth it even if you only consider this factor.
What matters?
The review request you get may come with instructions to rate the paper on a variety of factors. While they are sometimes informative, don’t spend too much time on these ratings, as they are largely ignored. Three things really matter for a review: 1) Comments to the editor, 2) Your comments to the authors, 3) Your overall rating of the paper.
Comments to the editor
This should be brief! Keep in mind that the editor is probably overseeing a lot of papers, so they may not have time to read very long comments. Mention the top reasons for your decision in one or two sentences and make clear how strongly you feel about your decision. I always like to include one of the following: “strong accept”, “weak accept”, “weak reject”, or ”strong reject” in the comments. You may be tempted to say “borderline” in some cases. Don’t do it; editors hate ambiguity! The editor is usually asking you to make an accept/reject judgment about this paper, so tell them which it is.
Comments to the authors
This is where you describe in detail the strengths and weaknesses of the paper. For conference papers, this should be at least 4 paragraphs long. For journal papers, the review should be well over a page.
Style:
The main point of these comments is to help the author(s) improve their work for future revisions and/or future publications. Thus the goal is to provide constructive criticism. If you really like some (or all) the aspects of the paper, make it clear to the authors what you like and why. You can also suggest future work or applications for the method that the authors haven’t discussed yet. If you really dislike the paper, make clear exactly what aspects of the paper you don’t like and why. It is very important to explain why using specific details, as this is essential for improving the work.
Sometimes you may read a paper and find so many glaring problems you become very frustrated. Don’t let your frustration show in your review; it is not helpful to anyone. The authors will be unlikely to accept your criticism if you are yelling at them (they will just get defensive) and you will look mean and petty in the eyes of the editor. No matter how bad the paper is, stick to clear scientific language and treat the work seriously. By no means should you hold back on criticism, but make sure it is done in a way that doesn’t insult the authors and provides a path for improvement. For example, consider these two statements:
1) “This paper is a complete mess. The writing is unintelligible, the problem addressed is irrelevant, and the method presented is fundamentally unsound.”
2) “This paper requires significant modification and improvement. The writing was difficult to follow and would benefit from thorough proof-reading. It was unclear why the authors chose to pursue the particular problem described in the paper and an explanation of why this problem is relevant to the field would be helpful. The method presented to solve the problem should be studied in more detail, as it seems it would produce the incorrect result when....”
These two statements contain exactly the same criticisms. The benefit of statement 2 is that the authors are far more likely to listen to what you are saying because you are giving them advice on how to improve the work, not berating them.
In cases where you strongly dislike the paper it’s also important to balance out the review. Think of at least one positive thing to say about the paper to encourage the authors. This also makes you look more objective in the eyes of the editor.
You should also be careful not to say your accept/reject decision in the comments (though you can make it quite obvious). This is because the editor may choose to go against your recommendation (remember that there are usually multiple reviewers for a given paper), and clearly stating your decision makes it more messy for the editor to do this.
Content:
The first paragraph of your comments to the authors should be a brief description (2-3 sentences) of the method presented in the paper. This paragraph is necessary mainly to show that you understood what the paper was about so that the subsequent comments you make appear informed. After that, the review is more free-form. In general, it will consist of comments about the follow issues:
Clarity: One of the most important aspects of a paper is clarity. If the paper is not clear, the reader can not evaluate what it is claiming. When you address clarity issues, be as specific as possible. Point out specific sections that were particularly unclear and suggest ways to improve them. A common suggestion is to add a figure to clarify some complicated concept.
Background understanding: The authors need to clearly establish that they understand previous work in the area. This is done mostly in the related work section of the paper. If they have very few citations, it’s usually a sign that they are not very familiar with the field. The pitfall here is that, without understanding previous work, the authors could be re-inventing a previously-published method, or ignoring a key issue discussed in the literature. When commenting on understanding of background, suggest previous work that the authors may have overlooked and describe how it is relevant. You can also suggest survey papers and books on the topic to help the authors.
Claims vs. Results: The claims made in the paper must be interesting and relevant to the field and they must be backed up by results and/or proofs. Grand claims are very exciting but are very rarely justified, while boring or irrelevant claims raise the question of why the work was done in the first place. Regardless of their grandness, unsupported claims are probably my #1 reason for rejecting papers as a reviewer. It is crucial that every single claim made be backed up by the methods and results sections. If the claims are somewhat unclear, you can suggest that the authors explicitly list the contributions of their paper in the introduction. If the claims are extremely unclear, this can be grounds for rejection.
Novelty: The work presented in the paper has to be significantly new. This issue comes up frequently when the authors are building on previous work (often published by the same authors). While building on previous work is expected (and in many cases desirable), the paper must propose a new method. Taking an existing method and using it as a black box to solve a particular problem is not novel. Tweaking an existing method and generating slightly better results is also not particularly novel. Of course these concepts are necessarily vague (e.g. where exactly is the line between a tweak and a fundamental change?), so it’s up to you to use your knowledge of the standards of the field to assess novelty.
Significance: Significance issues are raised when there seem to be only small improvements over the state-of-the-art. This is perhaps the hardest aspect of the paper to judge because it requires knowing the field quite well. Even if the method is sound, producing only a marginal improvement over the state-of-the-art is questionable. Significance is especially important if the presented method is much more complex than the state-of-the-art. Basically, a significant increase in complexity needs to be justified by a significant increase in performance.
Writing: Some people like to point out specific spelling and grammar mistakes in the paper, but I would advise against this. This is mainly because the paper will probably change in subsequent revisions (likely introducing new spelling/grammar mistakes), so your suggestions may be irrelevant. However, you should definitely say something if the writing is poor and needs significant revision. If the writing is too hard to follow, this can be grounds for rejection.
Improvements: It is very helpful to authors when reviewers suggest improvements to their method. While it’s possible the method could be improved by shedding assumptions, some assumptions are so fundamental to the method that they cannot be removed unless you change everything. In your suggestions, focus on what the next step would be if you were to continue making the fundamental assumptions that this method makes.
Finally, your overall rating must be justified by the content in the comments you give to the authors. When the editor reads these comments, it should be obvious to them why you chose to give the paper the overall rating you did.
Overall rating of the paper
This is perhaps the most important part of the review. It is where you step back from all the comments you have written and judge the paper as a whole. It is important to do this fairly by thinking of the standards of the conference/journal. Some conferences/journals have much higher standards for novelty and significance than others. Usually journals have higher standards for novelty than conferences and require much more extensive results. You can find out these standards by looking at previously-published papers in the conference/journal you’re reviewing for and by asking your colleagues or advisor. One question you can ask yourself to help determine the rating is “If I had never seen this work before and came across this paper in the conference proceedings/journal issue, would I be surprised seeing it next to the other papers I’ve read from that venue?” If you’d be shocked to see a paper of this quality alongside others from this conference/journal, it’s a good sign that the decision should be a reject.
Reviewing Papers from a Competing Research Camp
It can be very difficult to review a paper from a direction of research which is fundamentally at odds with your direction of research. For example, you and the authors are trying to solve similar types of problems but are making fundamentally different assumptions about what’s important, problem specifics, etc. In such cases, you must suspend your disbelief.
The poet Samuel Taylor Coleridge coined the term “willing suspension of disbelief” to justify using fantastical elements in a modern narrative. For instance, we may think that time travel is implausible, but should we automatically discard all literature that contains time travel? Surely there are other parts of a book containing time travel that could be worthwhile.
Likewise, when reviewing a paper, you may believe that the assumptions the authors are making are implausible (or, more likely, in conflict with your cherished assumptions) but if these assumptions are indeed common to a serious direction of research in this area you must judge the paper relative to the standards of its research direction, not your own. Of course assumptions that are clearly and demonstrably incorrect are grounds for rejection, but that is quite rare in my experience. What’s more common is that the authors see the problem domain in a fundamentally different way than you do, which is OK, as long as the paper meets the standards described above.