How to Review for ICER

2021 Program Co-Chairs

Amy J. Ko and Jan Vahrenhold

Overview

Thank you for reading these training slides! They will explain the ICER review process and prepare you to constructively and fairly evaluate submissions.

Reviewing this guide should take less than 1 hour.

At the end, we’ll include a self-administered quiz. Take it at the end to verify you understand the process and criteria, and if you get any wrong, re-review the slides.

Prior to reviewing

ACTION ITEM

Read the call for papers at icer2021.acm.org. This is the ground truth for scope and submission requirements. We expect you to account for these in your reviews.

ACTION ITEM

Read the author guidelines. We expect your reviews and meta-reviews to be consistent with these guidelines.

HotCRP

We’re conducting all reviewing on HotCRP this year. The next few slides will provide a basic tutorial to its features. We’ll include screenshots throughout this slide deck to help you navigate the site.

But first, make sure you can sign in, then bookmark it:

http://icer2021.hotcrp.com

If you have trouble signing in, or you need help with anything, contact Raymond Pettit <rp6zr@virginia.edu> and James Prather <james.prather@acu.edu>, our submission chairs, for help.

HotCRP Email Settings

You can get to this page by clicking “Profile”

​

​

​

​

Make sure you select the right boxes for your preferences.

ACTION ITEM: Make sure you can 1) log in to HotCRP, 2) that your name and other metadata is correct, 3) that emails from HotCRP aren’t marked as spam, and 4) email notifications are enabled.

Principles behind ICER reviewing

Goals

The ICER review process is designed to:

• Maximize the alignment between a paper and expertise required to review it,
• Minimize conflicts of interests and promoting trust in the process
• Maximize our community’s ability to make excellent, rigorous, trustworthy contributions to the science of computing education.

ICER is a broadly scoped conference

The call for papers and author guide should make this clear, but ICER is broadly scoped, considering all kinds of research papers on all kinds of topics related to computing education. It publishes learning that happens in any context, not just post-secondary and not just formal education. We strongly encourage research on primary education, secondary education, adult education (e.g., bootcamps, training). We strongly encourage research on informal settings, outside the context of formal learning settings.

Your reviews should not critique papers for being about a topic you personally perceive to be less important to computing education. If the work is sufficiently ready for publication and you believe it is of interest to some part of the computing education community, let it be published and the community will decide its importance over time.

Conflicts of interest

ICER takes conflicts of interest, both real and perceived, quite seriously.

The conference adheres to the ACM conflict of interest policy as well as the SIGCSE conflict of interest policy. These state that a paper submitted to the ICER conference is a conflict of interest for an individual if at least one of the following is true:

1. The individual is a co-author of the paper
2. A student of the individual is a co-author of the paper
3. The individual identifies the paper as a conflict of interest, i.e., that the individual does not believe that he or she can provide an impartial evaluation of the paper.

Managing conflicts of interest

No reviewer, meta-reviewer, or program chair with a conflict of interest in the paper will be included in any evaluation, discussion, or decision about the paper.

It is the responsibility of the reviewers, meta-reviewers, and PC chairs to declare their conflicts of interest throughout the process.

It is the responsibility of the PC chairs to ensure that no member of the Program or Review Committees is assigned a role in the review process for any paper for which they have a conflict of interest.

Review process roles

Program committee (PC) chairs

Each year there are two program committee co-chairs. The PC chairs are solicited by the ICER steering committee and appointed by the SIGCSE board to serve a two year term. One new appointment is made each year so that in any given year there is always a continuing program chair from the prior year and a new program chair. Appointment criteria include prior attendance and publication at ICER, past service on the ICER Program Committee, research excellence in Computing Education, collaborative and organizational skills to share oversight of the program selection process. The ICER Steering Committee solicits and selects candidates for future PC chairs.

Program committee (PC) members

PC members write reviews of submissions, evaluating them against the review criteria. The PC chairs invite and appoint the reviewers. The committee is sized so that each reviewer will serve for 5-6 paper submissions, or more depending on the size of the submissions pool. Each reviewer will serve a one year term, with no limits on reappointment. Appointment criteria include expertise in relevant areas of computing education research and past reviewing experience in computing education research venues. Together, all reviewers constitute the program committee (PC).

Senior Program Committee Members (SPC)

SPC members review the PC members reviews, ensuring that the review content is constructive and aligned with the review criteria, as well as summarizing reviews and making recommendations for a paper’s acceptance and rejection. They also provide feedback on reviews if necessary, asking reviewers to improve the quality of reviews. Finally, they participate in a synchronous SPC meeting to make final recommendations about each paper, and review authors’ minor revisions.

The PC chairs invite and appoint Senior PC members, with the approval of the steering committee . The committee is be sized so that each meta-reviewer will handle 8-10 papers. The PC chairs are responsible for inviting new members of the committee and retiring longer serving members, balancing the need for intellectual diversity and expertise necessary for the pool of submissions.

The review process

1. Authors submit abstracts

Authors will submit a title and abstract one week prior to assigning papers. We’ll ask you to bid on papers for which you have sufficient expertise—in both phenomena and methods—and then the PC chairs will assign papers based on your bids.

Authors are allowed to revise their title and abstract before the full paper submission deadline.

2. You bid

The purpose of bidding is not to express interest in papers you want to read. It’s to express your expertise and eligibility for fairly evaluating the work. These are subtly but importantly different purposes.

• Specify all of your conflicts of interest. Conflicts are any situation where you have any connection with a submission that is in tension with your role as an independent reviewer (you advised an author, you’ve collaborated with an author, you’re at the same institution, you’re close friends, etc.). After declaring conflicts, you will be excluded from all future evaluation, discussion, and decisions of that paper. Program chairs and submissions chairs will also specify conflicts of interest at this time.
• Bid on all of the papers you believe you have sufficient expertise to review. Sufficient expertise includes knowledge of research methods used and prior research on the phenomena. Practical knowledge of a topic is helpful, but insufficient.
• Don’t bid on papers about topics, techniques, or methods that you strongly oppose. That precludes authors from being fairly reviewed by authors without such bias.

HotCRP - How to bid

After paper registration deadline has closed, but before the full paper deadline, you can bid on papers you’d like to review. To do that, click “Review Preferences” on the homepage.

HotCRP - Review Preferences

On the review preferences page, you can indicate your level of interest in a particular paper so that the automated assignment algorithm can accurately place you into reviewing the right papers.

ACTION ITEM

Block 2 hours on your calendar the week after the abstract deadline to bid. We will send an email reminder with instructions.

3. Authors submit papers

Submissions are due one week after the abstracts are due.

As you read in the call for papers, submissions are supposed to be sufficiently anonymous that a reader cannot determine the identity or affiliation of the authors. The call for papers defines what constitutes sufficient anonymization. If you forget, refer to the call, and if the call is ambiguous, write the PC chairs for clarity at chairs@icer.acm.org.

4. Chairs desk reject

The PC chairs, with the help of the submissions chairs, will review each submission for papers that violate anonymization requirements, length restrictions, or plagiarism policies. Authors of desk rejected papers are notified immediately.

We may not catch every issue. If you see something during review that you believe should be desk rejected, contact the chairs before you write a review; the chairs will make the final judgement about whether something is a violation, and give you guidance on whether and if so how to write a review.

Conflicts of interest. PC chairs with conflicts are excluded from deciding on desk rejected papers, leaving the decision to the other program chair.

5. Chairs assign reviewers

Based on your bids and their judgement, the PC chairs will collaboratively assign at least 3 PC members and 1 SPC member for each submission. We will likely rely on HotCRP’s assignment algorithm, which depends on your bids being high quality.

Remember, for these assignments to be fair and good, your bids should only be based on your expertise and eligibility. Interest alone is not sufficient for bidding on a paper. The chairs will review the algorithm’s assignments to identify potential misalignments with expertise, with you are best positioned to assess your expertise.

Conflicts of interest. PC chairs with conflicts are excluded from assigning reviewers to any papers for which they have a conflict. Assignments in HotCRP can only be made by a PC chair without a conflict.

5. Chairs assign reviewers — a note on load

It’s critical to review on time.

• We’ve given you 4 weeks to review before discussion begins.
• We’ll likely give you 5 papers to review, but it could be more or less.

We recommend reviewing one or two papers per week to avoid fatigue.

Remember, your job isn’t to “find the best paper in your pile” and there is no per-reviewer quota; your job is to find all submissions worthy of archiving and sharing for the community to build upon. That could be none of your papers or even all of them.

HotCRP - Homepage After Reviews Assigned

After papers have been assigned to reviewers, you can see your assigned reviews on the homepage. Click each one to see the submission and review it.

HotCRP - Reviewing

The review criteria is the same as is listed below. Please make sure to read the training slides before conducting your reviews.

​

Note: the example above is “Ray’s awesome paper title” but all papers are anonymous for review. In this case, we made up a silly title for testing. Real submissions won’t have the author’s names in the submission title.

HotCRP - Offline Reviewing

There is an option to review offline. Click “Download form”, fill it out, and then upload it using the “Choose File” button. If you do offline review, DO NOT delete the word “ready”.

ACTION ITEM

Add reminders to your to do list, one for each paper. Spread your reviews out to be a happier, more constructive evaluator! Reviews are due Friday, April 30th, AoE.

6. Review

Assigned reviewers submit their anonymous reviews through HotCRP by the review deadline, evaluating each of their papers against the review criteria

PC members, SPC members, and PC chairs with conflicts cannot see any of the reviews of their papers during this process.

6. Review — Criteria

ICER currently evaluates papers against 7 criteria, as independently as possible. To be published at ICER, papers should be positively evaluated on all of these.

These have been carefully chosen to be inclusive to many phenomena, epistemologies, and contribution types. Let’s discuss each of the criteria in detail.

6. Review — Prior work

1. The submission is grounded in relevant prior work.

Papers should both cite relevant prior work and explicitly show how it relates to the paper’s questions. After reading the paper, you should feel more informed about prior literature and how that literature is related to the paper’s contributions. Such coverage of related work might come before a work’s contributions, or it might come after (e.g, connecting a new theory derived from observations to prior work).

Identify related work the authors might have missed and include it in your review. Missing a paper that is relevant, but would not dramatically change the paper, is not sufficient grounds for rejecting a paper. Such citations can be added upon reviewers’ request prior to publication. Focus on missing prior work that would significantly alter research questions, analysis, or interpretation of results.

6. Review — Prior work

Because prior work should be sufficiently but not completely covered:

• Don’t critique work for missing 1 or 2 peripherally related papers. Just note them, helping the authors to broaden their citations.
• Don’t critique authors for not citing your own work, unless it really is objectively highly relevant.
• Don’t critique work for where in a paper they address prior work. Sometimes a dedicated section is appropriate, sometimes it is not. Sometimes prior work is better addressed at the end of a paper, not at the beginning.
• Do critique work for simply listing papers without meaningfully addressing their relevance to the paper’s questions or innovations.

6. Review — Theory

2. The submission leverages available theory when appropriate.

Theory can be an informative guide in shaping a research question. If there’s theory that’s relevant to a submission, the submission should discuss it.

That said, not all types of research will have relevant theory to discuss, nor do all contribution types need theory to make significant advances. For example, a surprisingly robust but unexplained correlation might be an important discovery that later work could develop theory to explain.

6. Review — Theory

Because theory is just a guide:

• Don’t penalize work for making discoveries inconsistent with theory. The point of empirical work is to test and refine theories, not conform to them.
• Don’t penalize work for not building upon theory when there’s no sufficient theory available.
• Don’t penalize work for not using the same interpretation of a theory as you; many theories have multiple competing interpretations and multiple distinct facets that can be seen from multiple perspectives.
• Do be explicit about what theory should have been built upon if you critique a work for not building on theory.

6. Review — Methods

3. The submission describes its methods and/or innovations sufficiently for others to understand how data was obtained, analyzed, and interpreted, or how an innovation works.

You should be able to understand most of the key details about how the authors conducted their work or made their invention possible. This is key for replication and meta-analysis of studies that come from positivist or post-positivist epistemologies. For interpretivist works, it is also key for what Checkland and Howell called “recoverability” (See Tracy et al. 2010 for a detailed overview for evaluating qualitative work). Focus your critiques on omissions of research process or innovation details that would significantly alter your judgement of the paper’s validity.

6. Review — Methods

​

Because there are always more details a paper can describe about methods:

• Don’t penalize a paper for not describing every detail.
• Don’t ask authors to write substantially new method details unless you can identify content for them to cut, or there is space to add those details within the length restrictions.
• Do ask authors for minor revisions that would support replication or meta-analysis for positivist or post-positivist works, and recoverability for interpretivist works using qualitative methods.
• Don’t ask authors of theory and argument contributions for a traditional methods section; such contributions do not require them, as they are not empirical in nature.

6. Review — Soundness

4. The submission’s methods and/or innovations soundly address its research questions.

The paper should answer the questions it poses, and it should do so with rigor, broadly construed. This is the single most important difference between research papers and other kinds of knowledge sharing in computing education (e.g., experience reports), and the source of certainty researchers can offer.

Note that soundness is relative to claims. For example, if a paper claims to have provided evidence of causality, but its methods did not do that, that would be grounds for critique. But if a paper only claimed to have found a correlation, and that correlation is a notable discovery that future work could explain, critiquing it for not demonstrating causality would be inappropriate.

6. Review — Soundness

​

Because soundness is relative to claims and methods:

• Don’t apply criteria for quantitative methods to qualitative methods (e.g., critiquing a case study for a “small N” makes no sense; that is the point of a case study).
• Don’t critique a lack of a statistically significant difference if the study demonstrates sufficient power to detect a difference; a lack of difference can be discovery too.
• Don’t ask for the paper to do more than it claims if the demonstrated claims are sufficiently publishable (e.g., “I would publish this if it had also demonstrated knowledge transfer”).
• Don’t rely on inexpert, anecdotal judgements (e.g., “I don’t know much about this but I played with it once and it didn’t work”).
• Don’t assume that because a method hasn’t been used in computing education literature that it isn’t standard somewhere else. The field draws upon methods from many communities. Look for evidence that the method is used elsewhere.

6. Review — Advances knowledge

5. The submission advances knowledge of computing education that is of interest to the computing education community.

A paper can meet the previous criteria and still fail to advance what we know about the phenomena. It is up to the authors to convince you that the discoveries advance our knowledge in some way, whether it’s as incremental as confirming uncertain prior work, or adding a significant new idea.

Secondarily, there should be someone who might find the discovery interesting. It does not have to be interesting to you, and you do not have to be 100% confident that an audience exists. A possible audience is sufficient for publication, as the PC does not necessarily perfectly reflect the broader audience of readers.

6. Review — Advances knowledge

Because advances can come in many forms, there are many critiques that are inappropriate in isolation (if many of these apply, they may justify rejection):

• Don’t penalize a paper because a single paper was already published on the topic. Discoveries accumulate over many papers, not just one.
• Don’t penalize a paper for “only” being a replication. Replications are important.
• Don’t penalize a paper that contributes a really new idea for not yet having everything figured out about it. That can require multiple papers.
• Don’t penalize papers because you don’t think the results are generalizable enough or were only done at a specific university; generalizability takes time, and some types of qualitative work don’t intend generalizability.
• Don’t penalize a paper for investigating phenomena you personally don’t like (e.g., “I hate object-oriented languages, this work doesn’t matter”).

6. Review — Interpretation and implications

6. Discussion of results clearly summarizes the submission’s contributions beyond prior work and its implications for research and practice.

It is the authors’ responsibility to help interpret the significance of a paper’s discoveries. If it makes significant advances, but does not explain what those advances are and why they matter, the paper is not ready for publication.

That said, it’s okay if you disagree with the paper’s interpretations or implications. Readers will vary on what they think a discovery means or what impact it might have on the world. All that’s necessary is that the work presents some reasonably sound discussion of one possible set of interpretations.

6. Review — Interpretation and implications

​

Because there is no single “right” interpretation or discussion of implications:

• Don’t critique papers because you don’t think the idea would work in your institution.
• Don’t critique papers because you don’t agree with a paper’s speculations about impact. Everyone can have their own speculations.
• Do critique papers that make interpretations that are not grounded in evidence, or propose implications that are not grounded in evidence.

6. Review — Clarity and Conciseness

7. The submission is written clearly enough to publish.

Papers need to be clear and concise, both to be comprehensible to diverse audiences, but also to ensure the community is not overburdened by verboseness. We recognize that not all authors are fluent English writers; however, if the paper requires significant editing to be comprehensible to fluent English readers, or it is unnecessarily verbose, it is not yet ready for publication.

6. Review — Clarity and Conciseness

Because submissions should be clear enough:

• Don’t penalize a paper for having easily fixed spelling and grammar issues.
• Don’t penalize a sufficiently clear paper because it could be clearer. All writing can be clearer in some way!
• Don’t penalize a paper for not using all of the available word count. It is okay if a paper is short but significant! (e.g., many significant papers published in Science are <4,000 words).
• Don’t ask for more detail unless you are certain there is space, or if there is not space, you can provide concrete suggestions for what to cut.

6. Review — Plagiarism

If after reading a submission, you suspect that it has in some way plagiarized from some other source, do the following:

• Read the ACM guidelines on Plagiarism, Misrepresentation, and Falsification
• If you think there is a potential issue, write the PC chairs at chairs@icer.acm.org to escalate the potential violation, and share any information you have about the case.
• The chairs will investigate and decide as necessary prior to the acceptance notification deadline.

6. Review — Recommendation

Based on the criteria above, this paper should be published at ICER.

Based on all of the previous criteria, decide how strongly you believe the paper should be accepted or rejected, assuming authors make any modest, straightforward minor revisions you and other reviewers request before publication. Papers that meet all of the criteria should be strongly accepted (though this does not imply that they are perfect). Papers that fail to meet most of the criteria should be strongly rejected.

Each paper should be reviewed independently of others, as if it were a standalone journal submission. There are no conference presentation “slots”; there is no target acceptance rate. Neither should be a factor in reviewing individual submissions.

6. Review — Recommendation

Because each paper should be judged on its own:

• Don’t recommend accepting a paper because it was the best in your set. It is possible that none of your papers sufficiently meet the criteria.
• Don’t recommend rejecting a paper because it shouldn’t take up a “slot”. The PC chairs will devise a program for however many papers sufficiently meet the criteria, whether that is 5 or 50. There’s no need to preemptively design the program through your review; focus on the criteria.

6. Review — Award consideration

On the review form, PC members may signal to the SPC that they believe the submission should be considered for a best paper award. Selecting this option in the review form is visible to the other PC and SPC members as part of your review, but it is not disclosed to the authors.

Reviewers should recognize papers that best illustrate the highest standards of computing education research, taking into account the quality of its questions asked, methodology, analysis, writing, and contribution to the field. This includes papers that meet all of the review criteria in exemplary ways (e.g., research that was particularly well designed, executed, and communicated), or papers that meet specific review criteria in exemplary ways (e.g., discoveries are particularly significant or sound).

​

6. Review — A strategy

1. Remember to refer to this slide when reviewing. Bookmark it!
2. Before reading, remind yourself of the reviewing criteria by reviewing these slides.
3. Read the paper, and as you do, note positive and negative aspects for each of the 7 criteria.
4. Use your notes to outline a review organized by the 7 criteria, so authors can understand your judgments for each criteria.
5. Draft your review based on your outline.
6. Edit your review, making it as constructive and clear as possible. Even a very negative review should be respectful to the author(s), helping to educate them. Avoid comments about the author(s) themselves; focus on the document.
7. Based on your review, choose scores for each of the criteria.
8. Based on your review and scores, choose a recommendation score and decide whether to recommend the paper for consideration for a best paper award.

7. Discussion

After the reviewing period, the SPC member assigned asks the reviewers to read the other reviewers’ reviews and begin a discussion about any disagreements that arise. All reviewers should:

• Read all the reviews of all the papers (and re-read yours).
• Engage in a discussion about sources of disagreement.
• Remain open to other reviewers shifting your judgements.
• If your judgement does shift, update your review to reflect your new views. There is no need to indicate that you changed your review.

Conflicts of interest. PC members, SPC members, and PC chairs with conflicts cannot see any of the discussions of their papers during this process.

​

ACTION ITEM

Schedule time on your calendar to read the other reviews and participate in discussion, attempting to build consensus toward a recommendation. Discussion is April 30th-May 14th.

8. SPC Meta-Reviews

After the discussion phase, and after marking excellent reviews, SPC members use the reviews, the discussion, and their own evaluation of the work to write a meta-review and recommendation.

There are four possible meta-review recommendations: reject, discuss, conditional accept, and accept. Your recommendation should be entered in your meta-review.

Conflicts of interest. PC members, SPC members, and PC chairs with conflicts cannot see any of the recommendations or meta-reviews of their papers during this process.

HotCRP - Metareviewing

You will notice an “M” icon next a review if you are the metareviewer. The “R2” there means “round 2” which is the metareview.

HotCRP - Metareview: Seeing Reviews

As a metareviewer, you will be able to see all reviews for a given submission, including who submitted that review. Seeing reviewer identity is necessary to help metareviewers interpret judgements and arguments relative to their expertise.

HotCRP - Meta-reviewing Form

The meta-review form looks different than the normal review form with these fields (shown on the right). You will also be able to see all the reviews submitted by reviewers.

9. SPC Meta-Reviews: #reject

Reject. Ensure that the meta-review constructively summarizes the reviews and the rationale for rejection. The PC chairs will review all meta-reviews to ensure that reviews are constructive, and request SPC members to revise their meta-reviews as necessary. The PC chairs will make the final rejection decision based on the meta-review rationale.

10. SPC Meta-Reviews: #discuss

Discuss. Ensure that the meta-review summarizes the open questions that need to be resolved at the SPC meeting discussion, where the paper will either be recommended as reject, conditional accept, or accept.

Papers marked discussed will be scheduled for discussion at the SPC meeting.

11. SPC Meta-Reviews: #conditional

Conditional accept. Ensure that the meta-review explicitly and clearly states the conditions that must be met with minor revisions before the paper can be accepted.

It is okay if there are no conditions; all conditionally accepted papers will be accepted, assuming authors deanonymize and meet the final version deadline.

To accept with conditions, the conditions must be feasible to make within the 1-week revision period, so they must be minor.

​

11. SPC Meeting - Prior to the meeting

The PC chairs will host synchronous SPC meetings with all available SPC members to discuss and decide on all borderline papers.

Before this meeting, a second SPC member will be assigned to each paper tagged #discuss or #conditional, ensuring that there are at least two SPC members to facilitate discussion. Each SPC member assigned to a paper should come prepared to present the paper, its reviews, and the HotCRP discussion. Each SPC member’s job is to present their recommendation, and or if they requested discussion, present the uncertainty that prevents you from making one.

All SPC members who are available to attend a SPC meeting session should, at a minimum, skim each of the papers to be discussed and their reviews (excluding those for which they are conflicted), so they are familiar with the papers and their critiques prior to the discussions.

Conflicts of interest. SPC members conflicted on a paper will not be assigned as a 2nd reader.

12. SPC Meta-Reviews: Excellent reviews

Excellent reviews are:

• Constructive, explicitly identifying the merits of the work, as well as feasible ways of addressing any of its weaknesses.
• Insightful, demonstrating expertise on the topic and methods in a work.
• Organized, helping the authors clearly understand the reviewer’s opinions of strengths and weaknesses of the work.
• Impartial, demonstrating a commitment to the reviewing criteria of the conference, and not personal interests, speculation, or bias.

Reviewers will be recognized with an “Excellent Reviewer” distinction from the program chairs after the PC’s work is complete.

13. SPC Meeting - Deliberation

At the meeting, the goal is to collectively reach consensus, rather than relying on the PC chairs alone to make final decisions. Papers may move from #discuss to either #reject, #conditional, or #accept; if there are conditions, they must be approved by a majority of the non-conflicted SPC and PC chairs at the discussion.

After a decision is made in each case, one of the two SPC member will add a summary of the discussion at the end of their meta-review, explaining the rationale for the final decision, as well as any conditions for acceptance, and updating the recommendation tag in HotCRP.

Conflicts of interest. Any SPC member or PC chair conflicted on a paper will be excluded from a the paper’s discussion, returning after the discussion is over.

14. SPC Meeting - Awards

The meta-review form for each paper includes an option to officially nominate a paper to the Awards Committee for the best-paper award. PC members may flag papers for award consideration during review, but SPC members are ultimately responsible for nominating papers for the best paper award. See this slide for the desired characteristics of nominated papers.

• Each SPC member may nominate 0, 1, or 2 papers for the best paper award. Nominated papers may or may not have been flagged by one or more PC members.
• Nominations should be recorded in HotCRP and be accompanied by a paragraph outlining the rationale for nomination. NOTE: Whether a paper has been nominated and the accompanying rationale are not disclosed to the authors as part of the meta-review.
• SPC members are encouraged to review and finalize their nominations at the conclusion of the SPC meeting to allow for possible calibration.
• Once paper decisions have been sent, the submission chair will make PDFs and the corresponding rationales for all nominated papers available to the Awards Chair. Additionally, a list of all SPC members that have one or more conflicts of interest with any nominated paper will be disclosed to the Awards Chair, as those members are not eligible to serve on the Awards Committee.

​

15. SPC Meeting - PC chair review

Before announcing decisions, the non-conflicted PC chairs will review and optionally edit any meta-review for clarity and consistency with the review process and its criteria.

After this review, all recommendations will be converted to official accept or reject decisions in HotCRP.

ACTION ITEM

If you’re an SPC member, ensure the Meta-review deadline (May 14th, 2021) and SPC decision meetings are on your calendar.

16. Decisions announced

After the SPC meeting, the PC chairs will notify all authors of the decisions about their papers. Papers that are accepted will be encouraged to make recommended changes; papers that are conditionally accepted will be reminded of the revision evaluation deadline.

Authors will have one week to make the minor revisions.

Conflicts of interest. PC chairs cannot change the outcome of a decision after the SPC meeting.

17. Final versions

Authors of conditionally accepted papers submit their final versions for review by the assigned SPC member. SPC members and PC chairs will have one week to review and approve them. If the assigned SPC member reviews the minor revisions and makes a final recommendation to the PC chairs. The PC chairs then make final decisions on acceptance.

Authors will then have one week to submit to ACM TAPS for final publication.

After the review process

Session chairing

If you’re attending ICER, we expect PC and SPC members to help chair sessions.

Session chairing involves:

• Before the session, helping presenters who might be new to ICER
• Resolving A/V issues with speakers
• Introducing speakers before their talk
• Asking questions if the audience is silent.

The chairs will provide a session chairing guide, much like this reviewing guide.

Dates

Friday, March 19th, 2021, AoE: Titles, abstracts, and authors due�Friday, March 26th, 2021, AoE: Full paper deadline; PC bidding deadline�Friday, April 2th, 2021, AoE: Paper assignments complete�Friday, April 30th, 2021, AoE: Reviews due�Friday, May 7th, 20201, AoE: Discussions complete�Friday, May 14th, 2021, AoE: Meta reviews due�May 17-20, 2021 SPC meetings�Friday, May 21st, 2021: Preliminary decisions announced�Friday, June 4th, 2021: Final decisions announced�Friday, June 16th, 2021, AoE Final versions due

Self-administered quiz

Q: What’s out of scope at ICER?

A: Nothing, unless it really has nothing to do with people learning or teaching (broadly construed) about computing (broadly construed).

Q: What should bids be based on?

A: Only bid on a paper if

1) you have no conflict of interest,

2) you have sufficient expertise,

3) you have no personal bias against it’s topic, method, or area of work.

Bid more highly if you’re interested in the topic.

Q: What criteria should you evaluate papers against?

A: 1) grounding in prior work, 2) use of theory if applicable, 3) replicability/recoverability of methods, 4) soundness, 5) advances, 6) interpretation and implications, and 7) clarity/conciseness. Nothing else, including personal biases against particular topics.

Q: What is a sufficiently large sample for an empirical paper?

A: There isn’t one. Whether a work can justify its claims, regardless of whether a study is qualitative, quantitative, or both, is dependent upon what the researchers are trying to discover and how easily observable that thing is from the evidence gathered (e.g., the size of an effect). The sample sizes required are circumstances dependent.

Q: What were the action items in this slide deck?

A: 1) Read the call for papers, 2) read the author guidelines, 3) verify your HotCRP configuration, 4) reserve bidding time, 5) add reviewing to do’s, 6) reserve discussion time, 7) ensure the SPC meeting is on your calendar if on the SPC.

Thanks for reviewing this slide deck!

​

Let the chairs know if you have any questions about this training by adding comments to these slides or writing chairs@icer.acm.org.