Report from the Committee to Study the ISCA Review Process (R2)

Natalie Enright Jerger, David Kaeli, Christos Kozyrakis,

Gabe Loh, Tom Wenisch, David Wood

​

June 2017

Charge by SIGARCH and TCCA

• Phase 1
• Survey the community about issues with the ISCA review process
• Consider best practices used by other SIG/TCs
• Identify issues and opportunities to improve the ISCA review process
• Phase 2
• Make recommendations for changes and/or improvements

​

2

Goal of this Presentation

• Review results of community survey
• Understand any problems before we proceed with solutions
• Get feedback on further sources of info on review process
• That can help us identify or quantify issues

​

3

Survey Background

• Developed by R2 committee with help from polling/survey experts
• Feedback on structure, length, and maintaining objectivity
• Goals
• Identify what works well and what not in our current process
• Understand both authors’ and PC/ERC members’ perspective
• Anonymous survey
• 621 responses collected over 2 weeks
• Survey sent to SIGARCH/TCCA members and attendees of recent ISCAs

4

Survey Participants

5

Survey Participants

6

Main Takeaway Points

• Overall, our review process functions well
• Significant satisfaction with overall outcome and various other metrics
• But there are certain issues that we need to improve
• Before they escalate to major problems

7

Survey Says: ISCA is a Top Quality Venue

8

What is the Role of the ISCA PC?

9

What we Agree that Works Quite Well Already

• ISCA acceptance rates are about right (70% agree)
• Number of reviews per paper are about right (90% agree)
• Double-blind review process (74% agree)
• Conflict criteria are about right (85% agree)
• In person PC meetings are important (74% agree)
• PCs should include both junior & senior members (93% agree)
• Pre-PC meeting, online discussions are useful (87% agree)

10

What Needs to be Improved

• Reviews not always fair (8% strong, 25% somewhat, 20% unsure)
• Reviews not done by experts (4% strong, 21% somewhat, 15% unsure)
• Reviews not always helpful (5% strong, 17% somewhat, 15% unsure)
• PC members are overloaded (48% agree)
• Of the non-student authors of recent papers: 50% have not served on�PCs, 25% have served on ERC, 17% have not reviewed any papers
• Review turnaround time too long (15% strong, 29% somewhat, 25% unsure)

11

We are Split About

• Usefulness of rebuttals
• Mixed value for authors (50% find value, 35% do not, 15% unsure)
• How to review resubmitted papers
• From scratch (41% of authors, 44% of PC/ERC members)
• With continuity (30% of authors, 37% of PC/ERC members)
• Usefulness of revision-based review process
• Positive (39% of authors, 40% of PC/ERC members)
• Negative (27% of authors, 35% of PC/ERC members
• Maybe too early to make a call

12

We Agree on: Reviewer Accountability

13

We Agree on: Connecting ISCA to a Journal

14

Noteworthy Differences between Groups

In general, there is strong similarity in the survey replies of

• Authors | PC/ERC members
• Senior | junior | grad students

But there are a few differences worth noting

15

Noteworthy Differences between Groups

• PC/ERC members (vs. non-PC/ERC) are more likely to think that:
• Acceptance rates are too low (34% vs. 20%)
• Papers are reviewed by non-experts (28% vs 21%)
• Reviews are more helpful in improving my paper (65% vs 57%)
• There should continuity in reviewing rejected papers (36% vs 26%)

​

• Junior members (vs. senior) of the community are more likely to think that:
• Papers are not reviewed by experts (29% vs 24%)
• Reviews are not fair (37% vs 33%)
• Reviews are not helpful (25% vs 19%)

​

​

16

Next Steps for R2 Committee

Survey results

• We will work with SIGARCH/TCCA to release full results

Give us feedback

• What other sources of information should we examine

R2 committee

• Recommendations phase

17