Research Paper Review Guidelines

Dmitry Berenson

Reviewing papers is an essential part of the scientific process and is one of the main responsibilities of a scientist. However, reviewing is more than just a duty, it also benefits the reviewer! Though we may prefer to concentrate on our own work, the in-depth understanding of a paper that you gain during the review may get you to think in new directions and may end up benefiting your own work. Reviewing also allows you to get a snapshot of what is currently going on in your research community; i.e. what methods are growing in popularity? What work are people citing? etc. While not every paper (or even the majority of papers) you review will give you these benefits, reading a great paper once in awhile makes working through the mediocre ones worth it. Finally, reviewing helps you develop your own paper-writing and research skills by seeing examples of quality work (which you can imitate) and examples where papers fail (so you can avoid the same pitfalls). Overall there is good reason to actually be excited about reviewing!

This document describes the basic guidelines for completing your review. Before starting to write your review, you must make sure to have a good understanding of the paper. See the how to read a paper guide and read the paper at level 3 (described in the guide) for reviews.

To submit the file

Go to Piazza and click “New Post” on the left.
Select Post Type: “Note”
Select Post To: “Individual Student(s)/Instructor(s)”
Type “Instructors” in the field below “Post To”
Select the “reviews” folder
Attach your PDF file to the message.
Click “Post My Note”
You can update your post until the time of the deadline, don’t make multiple posts!

Format

You review should be 11pt, single-spaced, with 1-inch margins. The required length is approximately 1.5 pages.

Reviewing the paper

The first paragraph of your comments should be a description (about half a page) of the method presented in the paper. This paragraph is necessary mainly to show that you understood what the paper was about so that the subsequent comments you make appear informed. After that, you should have a separate section devoted to each of the topics below:

Background understanding: The authors will have citations to previous work in the area. In what ways does this work improve on existing methods? In what ways is the proposed method different from other literature? How do the authors justify the need for their work?

Claims and Results: The claims made in the paper must be interesting and relevant to the field and they must be backed up by results and/or proofs. Grand claims are very exciting but are very rarely justified, while boring or irrelevant claims raise the question of why the work was done in the first place. Regardless of their grandness, unsupported claims are probably my #1 reason for rejecting papers as a reviewer. It is crucial that every single claim made be backed up by the methods and results sections. What claims do the authors make about their method. Describe how the results either do or don’t justify each claim in the paper.

Significance: Significance issues are raised when there seem to be only small improvements over the state-of-the-art. This is perhaps the hardest aspect of the paper to judge because it requires knowing the field quite well. Even if the method is sound, producing only a marginal improvement of the state-of-the-art is questionable. Significance is especially important if the presented method is much more complex than the state-of-the-art. Basically, a significant increase in complexity needs to be justified by a significant increase in performance. Are the methods in the paper making a significant contribution over what is described in related work? Justify your answer with information from the paper.

Improvements: It is very helpful to authors when reviewers suggest improvements to their method. While it’s possible the method could be improved by shedding assumptions, some assumptions are so fundamental to the method that they cannot be removed unless you change everything. Suggest some future work that would improve the method. In your suggestions, focus on what the next step would be if you were to continue making the fundamental assumptions that this method makes. Don’t make throw-away suggestions about implementation like “try it on a real robot” or “make it faster by running on a different computer”.