HOW TO BE A GOOD REVIEWER?
REVIEWER TUTORIAL FOR ECCV 2024
Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, Gül Varol
ECCV 2024 Program Chairs
[Slides adapted from ICCV 2021, CVPR 2021, CVPR 2022, CVPR 2023, CVPR 2024]
THANK YOU FOR SERVING AS REVIEWER!
We are all counting on you:
TUTORIAL GOAL: TO UNDERSTAND…
The nature of the review process
The role of reviewers in the review process
The expectation on reviewers
What to include/avoid when writing
reviews
1
2
3
4
THE DECISION PROCESS: OVERVIEW
Desk reject;
Recommend reviewers for papers
Confirm all decisions
PAPER SUBMISSION
PAPER DECISION
PROGRAM CHAIRS
AREA CHAIRS
REVIEWERS
AUTHORS
Authors submit
Assign papers to ACs
Provide thorough expert
reviews
Confirm decisions of other ACs within the triplet
Submit rebuttal
Decision to authors
Engage in rebuttal discussion;
Update reviews
Oversee reviewers;
Discuss within AC triplets;
Write consolidation report ("meta-review");
Recommend a decision
1
2
3
4
5
6
7
8
9
10
THE DECISION PROCESS: DETAILS
LEVEL OF DECISIONS
Your job as a reviewer is to provide well-reasoned recommendations to ACs to enable them to make final decisions on all papers
Major advances that will heavily impact the field; will be used by many people, create new capabilities. E.g.: ResNet (CVPR 2016 Best Paper), Mask R-CNN (ICCV 2017 Best Paper), NeRF (ECCV 2020 Best Paper Runner-Up), ...
Outstanding paper: top ~10% of all accepted papers
Unlikely to be significant, or papers with major flaws
Poster (default for all accepted papers): works that add bricks to the cathedral of knowledge; papers introducing useful tools; papers of interest to a subcommunity; creative ideas that are hard to judge but could be promising
AWARD
CANDIDATE
ORAL
POSTER
REJECT
WHAT PAPER SHOULD BE ACCEPTED?
Any paper that, in accordance with ECCV community standards,
Note: ECCV is very inclusive. Historically, rejection solely for out-of-scope is rather rare.
WHY NOT ACCEPT EVERYTHING?
Papers can have a negative impact:
REVIEW FORM OVERVIEW
Explain the key ideas, contributions, and their significance. This is your abstract of the paper. The summary helps the AC and the authors understand the rest of your review and be confident that you understand the paper. You must NOT use an LLM to write the summary.
What about the paper provided value? E.g., interesting ideas that are experimentally validated, an insightful organization of related work, new tools, impressive results, something else? Most importantly, what can someone interested in the topic learn from the paper? Short bullet lists are not enough.
What detracts from the contributions? Does the paper lack controlled experiments to validate the contributions? Are there misleading claims or technical errors? Is it possible to understand (and ideally reproduce) the method and experimental setups by reading the paper? Short bullet lists are not enough.
Carefully explain why the paper should be accepted or not. This section should make clear which of the strengths and weaknesses you consider most significant.
Minor suggestions, questions, corrections, etc. that can help the authors improve the paper. Not crucial for the overall recommendation.
SUMMARY
STRENGHTS
WEAKNESSES
RATING AND JUSTIFICATION
ADDITIONAL
COMMENTS
ECCV 2024 REVIEW FORM DETAILS
1. By taking this review assignment and checking on "I agree" below, I acknowledge that I have read and understood the reviewer guidelines. * (visible to meta-reviewers)
[x] I agree
2. Summary. In 5-7 sentences, describe the key ideas, experimental or theoretical results, and their significance. * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
In your own words, explain the key ideas, contributions, and their significance. This is your version of the abstract of the paper. The summary helps the AC understand the rest of your review and be confident that you understand the paper. In particular, try your best to answer the following questions: (1) What problem is addressed in the paper? (2) Is it a new problem? If so, why does it matter? If not, why does it still matter? (3) What is the key to the solution? What is the main contribution? (4) Did the experiments or the theoretical analysis sufficiently support the claims?
3. Strengths. Consider the significance of key ideas, experimental or theoretical validation, writing quality, data contribution. Explain clearly why these aspects of the paper are valuable. Short bullet lists do NOT suffice. * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
What are the key contributions and why do they matter? Please consider the following aspects: interesting ideas that are experimentally or theoretically validated, a clearly formalized scientific hypothesis, an insightful organization of related work, new tools, impressive results. Most importantly, what can someone interested in the topic learn from the paper, i.e., clearly articulated knowledge advancement? Some specific examples include, but are not limited to - Clear explanations and illustrations - Contributions clearly stated and validated - Innovative problem formulation or solution - New technical insights - Thorough experiments - Thorough theoretical validation (if applicable) - High practical impact - High impact on the research community
4. Weaknesses. Consider the significance of key ideas, experimental or theoretical validation, writing quality, data contribution. Clearly explain why these are weak aspects of the paper, e.g., why a specific prior work has already demonstrated the key contributions, or why the experiments are insufficient to validate the claims, etc. Short bullet lists do NOT suffice. Be specific! * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
What are the aspects of the paper that most need improvement? E.g., - Key ideas and techniques of the paper are difficult to understand - Typos and grammar problems make reading difficult - Contributions are not clearly and accurately stated - Potential importance of contributions is not convincingly described or demonstrated - Paper contains technical or experimental errors - Experiments or theoretical grounding are insufficient to validate the contributions
ECCV 2024 REVIEW FORM DETAILS
5. Paper rating (pre-rebuttal). * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
6. Recommendation confidence. * (visible to other reviewers, visible to meta-reviewers)
7. Justification of rating. What are the most important factors in your rating? * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
Do NOT fill with “See weaknesses above”. Carefully explain why the paper should be accepted or not. This section should make clear which of the strengths and weaknesses you consider most significant. Explain how you weigh the strengths and weaknesses you identified above. What are the most important concerns that need to be addressed in a potential rebuttal? If your rating is borderline, carefully explain what additional information you are looking for (either from the authors in the rebuttal or from other reviewers during the discussion) that will enable you to make a more concrete recommendation.
ECCV 2024 REVIEW FORM DETAILS
8. Are there any serious ethical/privacy/transparency/fairness concerns? If yes, please also discuss below in Question 9. * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
If you check this flag, please provide details in the following question. Feel free to alert your AC as soon as possible so that the paper can be handled appropriately.
9. Limitations and Societal Impact. Have the authors adequately addressed the limitations and potential negative societal impact of their work? Discuss any serious ethical/privacy/transparency/fairness concerns here. Also discuss if there are important limitations that are not apparent from the paper. (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
If limitations and social impact are not discussed appropriately, please include constructive suggestions for improvement. Are there any serious ethical, privacy, transparency, fairness concerns such as lack of appropriate licenses, inappropriately sourced data, etc.? If yes, explain.
10. Is the contribution of a new dataset a main claim for this paper? Have the authors indicated so in the submission form? * (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
If the paper is claiming a dataset release as one of its scientific contributions, it is expected that the dataset will be made publicly available no later than the camera ready deadline. The authors should have also appropriately flagged this in the submission form. You can check the submission form answers by clicking on the paper ID.
11. Additional comments to author(s). Include any comments that may be useful for revision but should not be considered in the paper decision. (visible to authors during feedback, visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
12. Confidential comments to AC, such as concerns about plagiarism, other ethical violations, or your ability to evaluate the paper (only visible to area chairs and program chairs). (visible to meta-reviewers)
13. If another person wrote or helped you with the review, please identify that person here (only visible to area chairs and program chairs). (visible to meta-reviewers)
If you ask someone to help you with your review, you must check the review for quality and assume full responsibility for it.
ECCV 2024 REVIEW FORM DETAILS
14. Final rating based on ALL the reviews, rebuttal, and discussion (post-rebuttal). * (visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
15. Final justification (post-rebuttal). * (visible to authors after notification, visible to other reviewers, visible to meta-reviewers)
What are the main reasons to accept or reject the paper? Any other considerations? Suggestions to improve in revision or resubmission can also be listed here.
THE NATURE OF THE REVIEW PROCESS
THE ROLE OF REVIEWERS IN THE PROCESS
Provide an independent, objective, critical, and comprehensive review
Key: What is the knowledge advancement in the paper?
Discuss with AC and reviewer buddies to (hopefully) reach consensus
Explain clearly the basis of your review and recommendation
It is OK if the reviewers disagree with one another even after discussions
AC will form recommendations weighing in reviews, rebuttals, and discussions
Make your final recommendations with solid justifications
Read the rebuttal and discussions. Do they change your position? Why?
This facilitates the ACs to make final recommendations for the paper
REVIEW
DISCUSS
RECOMMEND
THE EXPECTATIONS ON REVIEWERS
Be constructive to the authors
Be friendly to your buddy reviewers and ACs
Be on time, responsible, and responsive
YOUR ACs ARE THERE TO HELP YOU
GUIDELINES
Take the time to do a good review
Be impartial
GUIDELINES
Be specific and detailed
Be professional and courteous
GUIDELINES
Be aware that different kinds of papers require different levels of evaluation
Do your work on time!
ETHICS
Avoid conflicts of interest
Protect the authors' ideas
TAKE AWAY POINTS
Respect authors and protect their ideas.
Take the time to do a good review.
Clearly justify your ratings.
Do your work on time!
1
2
3
5
Be constructive.
4
WHAT SHOULD BE INCLUDED IN THE REVIEW?
What problem is addressed in the paper? Is it a new problem? If so, why does it matter? If not, why does it still matter? What is the key to the solution? What is the main contribution? Do the experiments sufficiently support the claims?
What are the key contributions and why do they matter?
What aspects of the paper most need improvement?
Are the assumptions and theories (mathematically) sound? Are the experiments scientifically sound and valid? Is the problem addressed trivial? Did the paper miss important prior work? Has it been done before? If yes, where?
A CONCISE SUMMARY OF THE PAPER
A CLEAR STATEMENT OF STRENGTHS AND WEAKNESSES
A COMPREHENSIVE CHECK OF POTENTIAL FUNDAMENTAL FLAWS IN THE PAPER
WHAT SHOULD BE AVOIDED IN THE REVIEW? COMMON MISTAKES
Arrogance, ignorance, and inaccuracy
Pure opinions
Novelty fallacy
Blank assertions
Intellectual laziness
Policy entrepreneurism
ARROGANCE, IGNORANCE, AND INACCURACY
ARROGANCE
IGNORANCE
INACCURACY
SAFE BEHAVIOUR
SCRIPT:
AUTHORS: We did A by means of B
REVIEWER: The only way to do A is through C (i.e., my way or highway) … …
ERROR: you should know or check
SCRIPT:
AUTHORS: All A are B
REVIEWER: I do not think all A are B
ERROR: you should know or check
SCRIPT:
AUTHORS: A is a ring, not a field
REVIEWER: All rings are field
ERROR: They are NOT ……
Do not provide an opinion on things you do not know about
PURE OPINIONS
SCRIPT:
REVIEWER: This is not good enough for ECCV 2024. Why?
SCRIPT:
REVIEWER: CNN is not that interesting Why?
SCRIPT:
REVIEWER: Adversarial losses guarantees distribution matches. No theoretic proof indeed!
These remarks are pure opinions and not grounded
SAFE BEHAVIOUR
Check if you grounded your statement with a “because … …”
ERROR
NOVELTY FALLACY
SCRIPT:
REVIEWER: This should not be accepted because it is not novel.
Why? By whom and where has this been published before?
Many important things are not that novel. Many novel things are not that important & most really silly things are novel
SAFE BEHAVIOUR
Focus on whether or not the paper presented well-grounded knowledge advancement
ERROR
SCRIPT:
REVIEWER: This should be accepted because it is novel
Why? Provide additional justification.
BLANK ASSERTIONS
SCRIPT: REVIEWER: This has been done before By whom? Where? Why?
Making ungrounded statements. Comment about authors instead of focusing on the paper content
SAFE BEHAVIOUR
Provide evidence to support your assertions, Confine the discussion on the technical content of the paper, not on the authors
ERROR
SCRIPT: REVIEWER: Intrinsic images are not longer important Really? To whom? Why?
SCRIPT: REVIEWER: Experiments on unpublished datasets are not scientific Really? Why?
SCRIPT: REVIEWER: Authors are ignorant/careless/incompetent… Be humble, nobody is perfect.
SCRIPT: REVIEWER: If the authors were smart enough, they would…. Be humble, nobody is perfect.
POLICY ENTREPRENEURISM
SCRIPT: REVIEWER: You must publish your dataset! No such policy!
You imposed your own policies which are 1) not part of the official review policy and 2) against scientific review principles.
SAFE BEHAVIOUR
Make sure you follow common principles in scientific review. Most importantly, focus on whether the paper produced significant knowledge advancement.
ERROR
SCRIPT: REVIEWER: You must beat SOTA! No such policy!
SCRIPT: REVIEWER: You must have a theorem! No such policy!
SCRIPT: REVIEWER: You must beat arXiv papers! No such policy!
INTELLECTUAL LAZINESS
SCRIPT: REVIEWER: Does not beat SOTA so it must be rejected!
Does the paper present sufficient knowledge advancement?
Overemphasize certain factors instead of giving a comprehensive assessment
SAFE BEHAVIOUR
Make sure you follow common principles in scientific review. Most importantly, focus on whether the paper produced significant knowledge advancement
ERROR
SCRIPT: REVIEWER: Beat SOTA so it must be accepted!
Does the paper present sufficient knowledge advancement?
SCRIPT: REVIEWER: Theorem V looks wrong
It is either wrong or correct. You cannot be unsure.
SCRIPT: REVIEWER: There is this error hence it should be rejected
Is the error making the main knowledge advancement invalid?
EXAMPLE OF REVIEWS
REVIEW QUALITY GOOD
Rating: 9: Top 15% of accepted papers, strong accept
Review: First of all this paper was a delight to read. The authors develop an (actually) novel scheme for representing spherical data from the ground up, and test it on three wildly different empirical tasks: Spherical MNIST, 3D-object recognition, and atomization energies from molecular geometries. They achieve near state-of-the-art performance against other special-purpose networks that aren't nearly as general as their new framework. The paper was also exceptionally clear and well written.
The only con (which is more a suggestion than anything)--it would be nice if the authors compared the training time/44 of parameters of their model versus the closest competitors for the latter two empirical examples. This can sometimes be an apples-to-oranges comparison, but it's nice to fully contextualize the comparative advantage of this new scheme over others. That is, does it perform as well and train just as fast? Does it need fewer parameters? etc.
I strongly endorse acceptance.
+ Clearly explains why the paper should be accepted
- Does not contain many details about the contribution or why it is novel, so relies on the AC trusting the reviewer's judgment on these points
Note: though the proposed method does not achieve the best results (according to the review), the paper is highly valued for proposing a more general framework.
+ Indicates that the reviewer tried to think of weaknesses but could not come up with anything that should negatively impact the paper rating
+ Constructive feedback for the authors
Though missing a summary of contribution, the review clearly explains why the paper should be accepted
REVIEW QUALITY Ok BUT NOT GREAT
RATING Rating: 8: Top 50% of accepted papers, clear accept
The paper proposes a framework for constructing spherical convolutional networks (ConvNets) based on a novel synthesis of several existing concepts. The goal is to detect patterns in spherical signals irrespective of how they are rotated on the sphere. The key is to make the convolutional architecture rotation equivariant.
PROS: + novel/original proposal justified both theoretically and empirically; + well written, easy to follow; + limited evaluation on a classification and regression task is suggestive of the proposed approach's potential; + efficient implementation
CONS: - related work, in particular the first paragraph, should compare and contrast with the closest extant work rather than merely list them;
- evaluation is limited; granted this is the nature of the target domain
Presentation:
* While the paper is generally written well, the paper appears to conflate the definition of the convolutional and correlation operators? This point should be clarified in a revised manuscript.
* In Section 5 (Experiments), there are several references to S‘2CNN. This naming of the proposed approach should be made clear earlier in the manuscript. As an aside, this appears a little confusing since convolution is performed first on S42 and then SO(3).
Evaluation:
* What are the timings of the forward/backward pass and space considerations for the Spherical ConvNets presented in the evaluation section? Please provide specific numbers for the various tasks presented.
* How many layers (parameters) are used in the baselines in Table 2? If indeed there are much less parameters used in the proposed approach, this would strengthen the argument for the approach.
Minor Points:
- some references are missing their source, e.g., Maslen 1998 and Kostolec, Rockmore, 2007, and Ravanbakhsh, et al. 2016.
- Figure 5, caption: "The red dot correspond to" --> "The red dot corresponds to"
+ Highlights key ideas and contributions.
- The summary should also include one sentence on experimental setup and one sentence on significance of the contribution
+ Itemizes strengths and weaknesses
- Does not provide enough detail. E.g., what is original about the paper? How is the evaluation limited?
+ Includes clarifications questions and constructive feedback for authors
+ Makes it clear that "Minor Points" are not an important factor in decision
Makes general factors in decision clear and provides detailed feedback to authors, but does not provide adequate explanation for strengths and weaknesses
REVIEW QUALITY BAD
Rating: Rating: 4: Ok but not good enough - rejection
Review:
1. The idea of multi-level binarization is not new. The author may have a check at Section "Multiple binarizations" in [a] and Section 3.1 in [b]. The author should also have a discussion on these works.
2. For the second contribution, the authors claim "Temperature Adjustment" significantly improves the convergence speed. This argument is not well supported by the experiments. | prefer to see two plots: one for Binarynet and one for the proposed method. In these plot, testing accuracy v.s. the number of epoch (or time) should be shown. The total number of epochs in Table 2 does not tell anything.
3. Confusing in Table 2. In ResBinNet, why 1-, 2- and 3- level have the same size? Should more bits required by using higher level?
4. While the performance of the 1-bit system is not good, we can get very good results with 2 bits [a, c]. So, please also include [c] in the experimental comparison.
5. The proposed method can be trained end-to-end. However, a comparison with [b], which is a post-processing method, is still needed (see Question 1).
6. Could the authors also validate their proposed method on ImageNet? It is better to include GoogleNet and ResNet as well.
7. Could the authors make tables and figures in the experiment section large? It is hard to read in current size.
Reference
[a] How to Train a Compact Binary Neural Network with High Accuracy. AAAI2017
[b] Network Sketching: Exploiting Binary Structure in Deep CNNs. CVPR 2017
[c] Trained Ternary Quantization. ICLR 2017
- Cites papers that make the idea “not new”, but it does not say how these methods relate
- Not clear. Because it is not tested by experiments, or that the convergence speed is not different?
- Points 3-6 may help authors improve the paper, but it is not clear if they are a significant factor in the rating to reject
Big problems:
- AC can't make good use of the review without reading the paper, due to lack of justification.
- No strengths listed, which may indicate that reviewer is just looking for reasons to reject.
- Author and AC don't know which of the listed points are important for reject rating.
The review lists only weaknesses and requests for clarification, omitting a summary and justification for decision. Thus, it is unclear to author or AC which of these points are the primary basis for the rating.
ADDITIONAL RESOURCES
https://sites.google.com/view/reviewing-the-review-process/
http://luthuli.cs.uiuc.edu/~daf/CVPR21Training.html
https://cmt3.research.microsoft.com/docs/help/reviewer/reviewing-guide.html