SciPy 2017 Chair and Reviewer Guidelines

The Program Committee’s job is to select the talks that maximize the benefit to the entire conference audience, within the space and time constraints given.

SciPy 2017 will implement a double-open review: authors and reviewers are known to each other. Area chairs will make acceptance decisions, informed by reviewer comments and conference schedule limits. The intention is to offer transparency in the process, building trust in the community. (There is also research attesting to higher quality in  signed reviews, compared to anonymous reviews.)

But, research shows that peer review can suffer from implicit bias—just like the hiring processes—that disadvantages women, minorities and people from less-prestigious institutions. There is also research attesting to higher quality in signed reviews compared to anonymous reviews[1],[2],[3]. For this reason, we want to educate our reviewers about bias in the review process, and equip them with tools to interrupt this bias.

Bias in the review process

Human beings are consistently, routinely, and profoundly biased. We not only are profoundly biased, but we also almost never know we are being biased. (Interested?[4],[5])

First, by realizing and accepting that we all have bias, we can learn to watch for it in ourselves and help others who work with us to do the same. This process of building awareness is analogous to what happens when we step on the clutch in a manual-transmission automobile. The motor doesn’t stop running (bias doesn’t stop), but the car is no longer moving forward. When we are on the lookout for our biases, they are less likely to blindly dictate our decisions.[6]

Be on the lookout! As you review, ask yourself …

Things to think about as you prepare to review:

Tips for the lead chairs

Tips for the Reviewer 

Other interesting links on the topic about gender bias:

You should analyze the presentation in the aspects described below, thanks to Doug Hellmann for these tips:

  1.  The Abstract: Occasionally a title comes along that is so compelling that I have to remind myself to keep reading before voting +1 and moving on to the next talk. It isn’t enough for a proposal to cover an interesting topic. It has to indicate that the talk will be interesting, too. While I am reading, I look for several factors.
  1. Is the abstract clear?  The speaker should describe the topic they plan to talk about in terms I can understand, even if I don’t know anything about that subject area. A clearly written abstract, without a lot of domain-specific jargon, tells me the speaker will be able to communicate with the audience.
  2. Is the abstract complete? An incomplete proposal is the largest negative factor I consider. If a proposal is incomplete, I can’t really say what the speaker will talk about, or even if they know the subject matter for their talk. If a proposal does not have a detailed summary or outline, I as the submitter to provide more detail.
  3. Is the abstract compelling? Without regard to the actual subject, is the abstract written in a way to attract an audience? I look for an abstract that shows the speaker is excited about the topic, and that they will be conveying that same excitement to the audience.
  1.  The Topic, For some people, the subject matter of a talk is the most important, or only, aspect taken into consideration when voting. I have seen presentations on topics I thought would be boring, but which were delivered with such enthusiasm that I enjoyed them more than talks I thought would be interesting from the outset. In my mind, the topic is an important factor, but not necessarily the most important, to be considered.
  1.  How relevant, immediately useful and novel is the topic? I look first at whether the topic is relevant to the conference attendees. Attendees will have a range of experience levels, interests, and backgrounds. Although we want a broad set of topics, we do need to be careful to avoid talks that are too narrowly focused on a niche. I often recommend that new projects which show a lot of promise convert their talk proposal into a poster proposal. As a counterpoint to considering whether a topic is too niche, I also try to take into account whether the audience will take away something immediately useful.

This section prepared by the SciPy 2017 Diversity Committee and Program Chairs. Thanks to Philip B. Stark for useful links.

Material compiled from different sources, listed below:

  3. Everyday Bias: Identifying and Navigating Unconscious Judgments in Our Daily Lives. Howard J. Ross.

Conflicts of Interest

The definition of a Conflict of Interest in peer review is a circumstance that makes you "unable to make an impartial scientific judgment or evaluation.” (PNAS Conflict of Interest Policy).

Conflict of Interest (COI) is different from individual bias. It is concerned with independence from any interest that could impair your objective assessment, any concern other than the intellectual quality of the work.

As a reviewer, your present or previous association with certain authors poses textbook COI: recent collaborators in research funding or publications (in the past 4 years), family members, business partners, and thesis student/advisor or mentor. Other associations could be more tenuous: being employed by the same institution, for example. In this case, your obligation as reviewer is to disclose the potential COI to the area chair or Program Chair, and state whether you think you are able to make an impartial assessment of the work.

If you have a conflict of interest with a submission, you should decline to review it and disclose the specific reason to the Program Chairs.

As Area Chair of a conference track or mini symposium, you should avoid conflicts of interest in making acceptance decisions: you should recuse yourself from decisions on the work of your collaborators (or indeed, your own submissions), and let that decision be handled by your co-chair (or Program Chairs).

Declaring the potential of a conflict of interest is required under professional ethics. If in doubt: ask the Program Chairs.







[7] Threats to objectivity in peer review: the case of gender