SPOT Group Rebuttal Guide

Many conferences (including CHI and UIST) have a "rebuttal" step where you can respond to reviewers' feedback using some limited number of characters. Writing good rebuttals can be challenging so this is my (Steve's) guide for the process^[1] I take when writing rebuttals. I also took some inspiration from other resources.

Step 0: Avoid the urge to complain

The review process is often frustrating. It's OK to vent with co-authors and friends. However, unless there is something very egregious about your reviews, you should not complain with the program chairs or (worse) by sharing via social media.

Step 1: Shared Google Doc with Reviews

Other guides say to sleep on this but I prefer to do some "routine" things immediately after getting reviews (which also in turn help me deal with frustration). First, create a Google Doc for the rebuttal and share it with collaborators. Copy all the reviews at the bottom. I like to use a fixed-width font for this (and I honestly have no idea why…maybe just habit):

Rebuttal: (empty)

===================================

Reviews:

R1: (Score: 2)

The authors did a good job of framing their paper. The system seems to be effective, though I worry about its scalability. I had some concerns with the methodology of the user study.

R2: (Score: 4)

This paper brings a fresh perspective to the topic. I think the design is very interesting. But I think it's missing some related work that the authors need to consider. Further integration with recent literature would strengthen the narrative.

R3/2AC: (Score: 3)

This is a relevant contribution, presenting its aims and motivations clearly. The approach is well-executed, showing strong potential for impact. However, I am not clear on the scalability and certain sections might benefit from a more rigorous examination. Clarifying these points would add to the work’s credibility and impact.

1AC: (Score: 3)

The paper presents a fresh perspective (noted by R2) with a well-defined problem framing (noted by R1 and R3). Reviewers also appreciate the innovative design of the system and its potential for impact in the field. Given the positive reception of the design and overall framing, I believe this paper has merit. However, before it can be accepted, the authors should address the reviewers' concerns.

Step 2: Color-Code Reviews

Next, do something to help make it easier to differentiate reviews. I like to make every reviewer's feedback a different color:

Rebuttal: (empty)

===================================

Reviews:

R1: (Score: 2)

The authors did a good job of framing their paper. The system seems to be effective, though I worry about its scalability. I had some concerns with the methodology of the user study.

R2: (Score: 4)

R3/2AC: (Score: 3)

1AC: (Score: 3)

Step 3: Copy/Paste and Extract Reviewers' Feedback

Divide all reviewers' feedback into "positive" (+) and "negative" (-) feedback. You should paste in the exact text from the reviews and then add in their reviewer numbers:

Rebuttal: (empty)

===================================

POSITIVE:

R1: did a good job of framing their paper.

R1: system seems to be effective
R2: fresh perspective
R2: design is very interesting
R3: relevant contribution
R3: presenting its aims and motivations clearly
R3: approach is well-executed, showing strong potential for impact
1AC: fresh perspective (noted by R2)
1AC: well-defined problem framing (noted by R1 and R3)
1AC: innovative design of the system and its potential for impact in the field

NEGATIVE:

R1: worry about its scalability
R1: concerns with the methodology of the user study
R2: missing some related work
R2: integration with recent literature would strengthen the narrative
R3: not clear on the scalability and certain sections might benefit from a more rigorous examination

===================================

Reviews: (...excluding for brevity but keep this in the document...)

Step 4: Categorize Negative Feedback

Next, try to place all the negative feedback into categories. No need to do this for the positive feedback. You might also have sub-categories for some categories.

Rebuttal: (empty)

===================================

CATEGORIES:

SCALABILITY CONCERNS

R1: worry about its scalability
R3: not clear on the scalability and certain sections might benefit from a more rigorous examination

MISSING RELATED WORK

R2: missing some related work
R2: integration with recent literature would strengthen the narrative

USER STUDY

R1: concerns with the methodology of the user study

===================================

POSITIVE: (...excluding for brevity but keep this in the document...)

NEGATIVE: (...excluding for brevity but keep this in the document...)

===================================

Reviews: (...excluding for brevity but keep this in the document...)

Step 5: Prioritize Categories

Next, you'll want to choose what you want to prioritize. There isn't an exact process for this but some things to consider:

Feedback from 1AC and 2AC are usually most important. They are the ones who ultimately decide whether the paper gets accepted.
Check if any reviewers identified any piece of feedback as a "key" issue to address.
See how negative the reviewers who bring up this point are. It might be more helpful to address negative feedback from a reviewer with a very low score than negative feedback from a reviewer with a very high score.

Step 5½: Sleep on it

Take a break from reviews (and calm down if necessary).

Step 6: Double check your categories vs. the original reviews

Go back and look at the original reviews, now that you have prioritized feedback categories. Do the categories you have capture everything? Are you prioritizing the right things? Revise as necessary.

Step 7: Draft a Rebuttal

Use the prioritized categories (in order) and do your best to draft a rebuttal with each of these categories as headings. Some things to consider:

Always thank the reviewers, no matter how much you disagree with their reviews.
I like to start by reminding reviewers what they liked about the paper

Keep in mind that 1ACs have ~10–12 papers to manage. They might forget what they liked about the paper if you don't remind them.
I like to use reviewers' own words in a short paragraph at the beginning.

If there's feedback that you disagree with, do your best to work with rather than against reviewers (unless their feedback is objectively wrong).

Try to understand and acknowledge reviewers' perspective and think of them as a collaborator who is helping you improve your paper
If the review makes a big deal of something that you feel is minor, do not suggest that they are wrong to care about it. Try to (i) understand their point of view and (ii) address it (or if you can't really address it, suggest why this work is helpful/impactful/etc. despite this flaw).

Be specific about how you plan to change the paper (or if this is a revise/resubmit, how you did change the paper). If there's a space limitation, you might also want to specify how you will make room for any potential additions
Sometimes it helps to have an "other specific points" section at the end to have one-sentence summaries

Thank you for your thoughtful reviews. We appreciate that reviewers felt our work offered a "fresh perspective" (R2, 1AC) and that the system "seems to be effective" (R1)

SCALABILITY CONCERNS (R1, R3):

We have tested our system with thousands of simultaneous users and found that it scales very well. Blah blah blah

MISSING RELATED WORK (R2):

Thank you for pointing out this related work. Unlike XYZ, our system does ABC. We will add a discussion to this to the paper.

OTHER SPECIFIC POINTS:

- R1: participants were not repeated in our studies

- R2: …

===================================

CATEGORIES: (...excluding for brevity but keep this in the document...)

===================================

POSITIVE: (...excluding for brevity but keep this in the document...)

NEGATIVE: (...excluding for brevity but keep this in the document...)

===================================

Reviews: (...excluding for brevity but keep this in the document...)

If this is a revise & resubmit process (rather than just a rebuttal), you might want to instead make a table with columns:

Feedback: What the feedback was from reviewers
Response: Your response to that feedback
Changes to paper: How you changed the paper in response

Step 8: Discuss with Collaborators and Revise

You might need to reduce wording to fit within a page limit, remove items that you can't effectively address, get collaborators' feedback on your review draft, etc.

Step 9: (If applicable) Revise the Paper

If you're doing a revise & resubmit, you'll also have a chance to update the paper. Once you've agreed on how to respond to reviewers' feedback and (on a high-level) how to revise the paper, it's time to actually make the revisions. Many places require you to submit both a revised version of the paper and a "marked up" version of the paper that highlights the changes made. If you're working in LaTeX, the easiest way to do this is by adding these commands to your document (assuming you've imported xcolor or some color package):

\newcommand{\add}[1]{\textcolor{blue}{#1}} % FOR MARKED UP VERSION

\newcommand{\del}[1]{\textcolor{blue}{\sout{#1}}} % FOR MARKED UP VERSION

%\newcommand{\add}[1]{#1} % FOR FINAL VERSION

%\newcommand{\del}[1]{} % FOR FINAL VERSION

Then, put all the text you've added inside of \add{...} and everything you're deleting in \del{...}. Compile the "marked up" version by commenting out the last two lines. Compile the "final" version by commenting out the first two lines.

Other Resources

The inspiration for this was April Wang's Peach Group's Rebuttal Slides

[1] OK—the idealized process