WI Logo image.gif

Judging Bias Report

This report was compiled by 2014-2015 APDA Women’s Initiative members Juliana Vigorito, Sarah Margulies, and Jerusalem Demsas. Input was solicited through an anonymous Google Form to inform the contents. Specific anecdotes are included below and are unedited unless otherwise noted. If you have questions or concerns about this report, please email APDAWomenInitiative@gmail.com.

The anonymous Google Form remains open, and you can submit to it here.

Introduction

APDA is a league that relies majorly on its judges to facilitate fair, enjoyable, and competitive debate. When judging quality is low, it hurts not only the individuals who get poor feedback, but also the league as a whole, since debaters who feel dissatisfied are less likely to continue competing. Improving the quality of judging on our league is a multifaceted effort, with one key area being the elimination of biased judging. Women’s Initiative defines ‘biased judging’ as feedback, speaker points, ranks, or a win/loss that was affected by the debater’s race, ethnicity, gender, social class, appearance, age, accent, school affiliation, or any other trait not directly linked to debate performance. In order to elucidate instances of judging bias and educate debaters on how to combat it, Women’s Initiative collected anecdotes through an anonymous form and shared the contents during a meeting at the Brown University tournament in October 2014. This document is a follow-up to that event, and serves to inform those who were unable to attend.

We urge readers to keep in mind the fact that there is a distinction between biased judging and bad judging. There is overlap, however, and the problems may exacerbate each other; for instance, a bad judge may pick up a female/female team even though they clearly lost the round while also giving a male debater better ranks than he deserved. Women’s Initiative supports the end of bad judging as well, but herein primarily focuses on biased judging. You can calibrate and train a bad judge, but may need to engage more fundamentally to change a biased judge. We seek to offer recommendations on how to do the latter.

People Who Don’t See A Problem

There are certainly many debaters who never experience judging bias, or who do not feel that it impacts their debate experience. The input from our form suggests that this is a relatively small group, though there is selection bias when we ask specifically for instances that do demonstrate biased judging. Below are a few examples of these outliers.

“I think it's also important to hear from women who don't feel they've experienced bias, so I'm filling this form out just to say that I have never felt my gender influenced a decision in a round.”

“I felt that in my time on the circuit I have not encountered judging bias.”

People Who Leave

One important aspect of judging bias is its ability to actually make people quit APDA. Some debaters stick around in spite of biased judging, but others choose to leave. This is not by any fault of their own, but rather a rational decision in the face of an extracurricular activity that belittles them. As long as there is judging bias on APDA, our league will suffer the loss of individuals who are the victims of it. Below are a few examples of these instances.

“I was never around long enough to know people's names specifically, but I have had judges that explicitly told me that they don't think women can be as persuasive as men, and that they think women are always either too bitchy or too passive, and yet these men still judge tournaments every weekend.”

“As someone who debates [sic] across numerous different formats in high school, I was immediately struck with a sense that APDA had a whole different universe of bias. Nearly every round I entered, I felt like my arguments were devalued because they came from a woman instead of a man. People were too willing to call me bitchy or dismiss my arguments for being "too aggressive" when my style would likely be viewed as average if I were a man. I also have the sense that APDA weights reputation - and school reputation - much more than any other circuit. My sense was that if I were to make the exact same argument - word for word - as a debate [sic] from [redacted], the assumption would be against me and in favor of the [redacted] debater. In this sense, I felt like I had my school affiliation and my gender stacked against me in every round. Ultimately, I left APDA and stopped debating because I didn't think it was worth sticking around in a forum [sic] of debate where my input was so devalued.”

When instances of judging bias like these occur, a debater may or may not raise the concern with a friend or teammate. The responsibility each of us can take on is to be a supportive friend or teammate when such stories are shared. Try to ensure that you communicate to younger debaters on your team a willingness to believe them when they have a bad experience, and follow up accordingly. Even just habitually asking about other peoples’ rounds can help get to this end. When you notice a pattern of biased judging emerging, consider sitting down with the affected person to talk at length about the problem. What we do not recommend is placing blame on the debater or asking them to change, whether by adjusting their speaker position or attempting to lower the pitch of their voice or dress differently. If several complaints mount against a particular judge, or at a particular tournament, it is wise to seek out an EOF, a member of Women’s Initiative, or an APDA Board member to discuss the issue.

Reputation-influenced Judging

Even the best debaters on APDA have some stories of dropping rounds to teams that were simply better known on the circuit. This problem of reputation-based (“reppy”) judging furthers inequity at tournaments and contributes to less informationally accurate outcomes. A few examples are included below.

“I feel that many times the judging result is based more on rep than on actual quality, and I'm frustrated that judges often don't pay enough attention during the round to note which arguments are carried through and which are not. Rep is a systemic issue and probably something not going to be corrected immediately, but it is something we should look at now.”

I'm a decent varsity -- I'm basically always on [sic] the bubble and never have a losing record, but will probably also never win a tournament. Because I really want to break, and because top varsity often judge the bubble, this creates a massive incentive for me, as a judge, to vote for debaters who are perceived as good, even if 1. they're not actually that good or 2. they're generally very good but are losing to a team with no rep. I consider myself a pretty ethical person, but I've found myself countlessly searching for ANY reason to vote for a team and justify an RFD just so I'll get picked up by that team at their tournament (or at the very least, not auto-dropped). I think this is actually even more pernicious than racial/gender bias, mainly because many of the problems we're talking about regarding inclusion also are related to this (e.g. black female novices drop a round they should have won to white male varsity, conclude APDA is rigged, and quit).

Combatting reppy judging is one of the hardest areas of bias to alleviate, since it doesn’t quite constitute an equity violation and isn’t the specific responsibility of any committee. Still, it hurts our league and is worth drawing attention to. When hosting a tournament, you should calibrate judges and rank their ballots “blindly” - that way, the judges who are the most competitive debaters, and perhaps more likely to have the problem described in the above anecdote, do not automatically become the top ranked judges. When choosing a video to use for calibration, try to make sure it does not include any debaters who will be attending your tournament, and ideally none who are still actively debating. A list of gender-balanced rounds that Women’s Initiative recommends for calibration can be found here - you should also consider contacting Quinn Maingi, the current Video Task Force head, for assistance with calibration. It is helpful to have a judging clinic with as many of your judges as possible prior to the start of the tournament. You can emphasize the importance of judging on the flow alone, and avoiding preoccupation with the reputation of the debaters. If you feel strongly that reppy judging might pose a problem for your tournament, you can reduce it in part by anonymizing school names (often accomplished by designating a numeric code in place of each school name).

A particular problem with reppy judging is the notion that one partner on a team “carries” the other. This idea is pervasive even beyond pro-am teams, and is an unfair pre-judgment for the “carried” partner. When judging a team where one debater has more rep than the other, it is doubly important to question that assumption. Adjudicate slowly and carefully in these cases, since oftentimes your assumption will be confounded by a particularly impressive performance by the “carried.” Relatedly, the knowledge of someone’s year in school or debate should not factor into adjudication - while our speaker scale is ill-defined, many novices regularly perform better than a 24. Try to avoid entering a round with assumptions about how the debaters will perform, particularly at novice and pro-am tournaments, and refrain from asking debaters their year in school.

Just as important as trying to avoid reppy judging is avoiding chastising judges when you are the debater in question. Part of the reason that judges can become reppy is because of debaters who respond aggressively to a loss, speaker points, or feedback they disagree with. Judges who feel afraid of having to justify their decisions are less empowered to adjudicate fairly when faced with overly confident or well-known debaters. This denies newer teams their due as well, since their talent and performance becomes secondary to the perceived demands of their opponent. Treating your judges with respect makes judging a more enjoyable thing to do, and when people like judging, the adjudication on our circuit gets better for everyone. Whether or not you’ve ever experienced any kind of judging bias, it is your responsibility as a member of APDA to respect the people who have volunteered their time to judge.

Sexism in Judging

One of the most common types of responses we received was commentary on sexism in judging. Several examples are included below.

“In speaking with a judge outside of a round at [redacted], I was told he had not turned in his ballot yet, but if I were to give him my number he would give us the round. I did not give him my number; we lost. I reported this to tournament staff, and I was told he was asked not to come back and judge initially. However, I can confirm he judged several rounds afterwards-- his name was on postings for inrounds and outrounds.”

“I was once told by a fellow judge in sems at [redacted] (he was chair, I was on the panel) that it was unbelievable that I could vote as I did because of my gender (female). He ranted on this point for at least a few minutes, making me and the other panelist uncomfortable. His final comment to me was that, "you should be ashamed of how you voted". He was a dino, and I brought this up with tournament staff (other panelist was there to express his discomfort as well), but [redacted] did nothing to my knowledge.”

“I am a male debater who regularly partnered with a female debater over the course of a season; in a round at one particular tournament run by a team with a history (or at least a rumored history) of serious issues relating to gender on their team in the past (actively pressuring female debaters to leave the team/not run for e-board positions), a judge from that team judged my partner and I running a case that my partner was intimately familiar with and I had never run before. I PMed the case, she MGed. I gave a decent PMC, and we faced a very strong opposition. Her MG, with knowledge of the case, a large density of argumentation and excellent rhetorical flow and style. We had a solid opp block, and I gave an objectively bad PMR; not a "well, I'm being modest, it was actually ok" PMR, but one which was stuttering, halting, didn't emphasize correct points, barely glanced over our winning arguments, and didn't properly respond to good points in the opp block. We ended up picking up, primarily based on arguments that came directly out of her MG that I barely mentioned in PMR- yet I got the 1 with a 26.5, and she got the 4 with a 25.5. I have no doubt that if our genders had been swapped, I would have deservedly been the 4, potentially with an even lower score. It was insidious gender bias, and although I won one of the few varsity speaker awards I earned in my career, it tainted its symbolic value and made it feel distinctly unearned.”

“As a novice, I was told by a very experienced dino that I received a 25 because I was too "shrill" and "shrieky." He proceeded to tell me that this was a problem for a lot of women on the circuit and advised me to get voice lessons to improve the problem.”

“I had a judge at [redacted] tell me he docked me half a speaker point because the way I spoke "wasn't ladylike". I also had a judge tell me that he didn't weigh my arguments regarding how a policy would impact women because it was "anecdotal evidence" (because I was a woman talking about women).”

“I've had a judge at a tournament ask me if the topic was too complicated for me to understand (only female in the room) before even POCs had started. I asked a judge what topic they wanted to hear about and they said "nothing girly"”  

“Many judges will not justify a high score for a debater unless it is something "special" or "above and beyond." This is fine by itself, because a 26.75 or 27 should be given only to special, above and beyond speeches. However, since we don't have any particular definition for those concepts, I worry that stylistic issues may functionally cap certain debaters, especially women, at a 26.5. For example, in my experience, lots of judges will bump a 26.5 up if they feel it was a perfect speech that was also funny (and would have remained a 26.5 without the humor). It is well documented that women have a harder time being comfortable cracking jokes and being perceived as funny, by both men and women. Justification is probably one of the most subjective and ill-defined areas of judging, and unfortunately also has an enormous impact on SOTY, which may help account for the gender discrepancies we see at the very top of the SOTY board every year.”

Reducing sexism in judging is an enduring challenge for APDA. One tool that can help reduce judging bias across the board, but particularly with regard to the types of feedback described above, is the judge feedback form. Tournaments can distribute forms to all teams before each round, the same way they distribute ballots to judges. If a team returns a form to the tab room that indicates sexist or otherwise severely biased judging, the tournament should react, ideally by pulling that judge from the pool. Because many tournaments run short on judges, this is not always a practicable solution. The obvious alternative is dropping the offending judge’s ranking, but that is an imperfect response - the judge may be “punished” by watching less competitive rounds, but still has the chance to offend more debaters (and likely novices specifically).

Confronting the judge about the issue is also imperfect since it might lead him or her to retaliate against the complainant debater, but such is the way of a student-run league - when handling equity complaints, equity officers face the same concern. Use your judgment, and consult an EOF or your tab observer if you feel it necessary - sometimes pulling a truly terrible judge is worth holding up the tournament for an hour to find another, sometimes it’s not. Importantly, try to include the complainant debater in the decision making process, and do not lie to him or her about the resolution, as described in one of the above anecdotes. Should an equity officer bring an incident to the attention of the tournament staff, you should take the complaint seriously and work with the equity officer and complainant to find a mutually agreeable resolution.  

Racism in Judging

Like sexism, racism was a frequent complaint in the comments we received. Several examples are included below.

“On every occasion accept [sic] for a couple, my partner (Caucasian) has won the first place speaker in the room. This wouldn't be problematic if he was clearly and significantly a better speaker than me, but we both agree that's not the case. In the rounds he was getting first, I was getting 3rd or 4th and was told that the other team was getting "the benefit of the doubt" in order for them to still have a chance at breaking. Coincidentally, in all the cases it was the white male that was given the second. If their partner was a female or a minority, they were usually placed in 3rd or 4th. This is a consistent theme amongst my entire team, because we have a rare dynamic of mixed (minority, woman/white male) teams. There have been rounds where the male would get 1st and the female would get 4 in a round they won. This problem is relatively easy to avoid when you've created a name in the league over a period of time, but if you are a new face the bias is blatent [sic] and obvious. This could deter minority students in the future from joining APDA, because of the time it would take for them to build the reputation to dissuade the unintentional bias. There have been times where my arguments have been completely ignored, I've been given scores with no warrant as to why I did not get higher, and have went 5-0 never recieving [sic] a first place in one of the round, and did not place at all in speaker awards, while my partner (white male) placed second.”

“During the third round at [redacted] last year, I ran a case about a hypothetical Chinese-American student choosing between EDing to Harvard or Williams (the idea of the case is about the pressures of Chinese culture with respect to the college process). On the ballot, the only comment the judge wrote for me was "Why did you write a biography?" (I am Chinese-American.) Maybe he only meant it about the ridiculously detailed answers I had during case construct, but it felt like he was just assuming that because I, as an Asian, was presenting this case, it must've been based on my life (which it wasn't).”

“I had written up a case pertaining to my ethnicity/culture that I personally love. I first ran this case with a partner of the same ethnicity as myself, in front of a judge who happened to be of the same ethnicity. We dropped, and apparently the judge made a comment to tab to the effect of "Of course, two debaters of x ethnicity would run a case about x in front of me. What else would you expect?" We dropped, on the RFD that the case placed an unfair burden on the opp team. Mind you, the opp team had loved the case and neither the LO or the MO made a single comment about how we used spec knowledge or were being abusive. In fact, they had taken classes pertaining to the subject of the debate and had openly expressed their admiration for the case during the round. In retrospect, I do think that part of the reason why the judge decided this way is because they themselves were afraid of giving off the appearance that they were being biased in our favor. It’s funny because no matter who the judge was, we would have run the case anyway—we were novices and I’m pretty sure that’s the last one we had.”

The same recommendations that exist for sexist judging certainly apply to racist judging. The last anecdote above indicates a particular problem seen for cases about lived experiences, whether they involve race or another facet of identity. Judges sometimes feel they are being pandered to, and are punitive to the gov team in response. At past Women’s Initiative events, some debaters who were WI members or women without committee affiliation complained that they felt feminism cases were habitually run in front of them, despite being judges who wanted to see a broad range of cases. This phenomenon is unique, because it is not considered strange to ask any given judge what sort of case they want to see - debaters are encouraged to ask their female or minority judge for case preferences the same way they would any other. For their part, judges who belong to minority groups ought not penalize debaters for running cases about that group in front of them.

Final Remarks

This report represents commentary shared over only a short period of time, and unfortunately advertised to only a relatively small group of debaters. Creating a more robust history and assessment of bias on APDA is a longer term project that future committee members will hopefully take up. Whether one or one hundred debaters experience bias, it is a problem worth addressing to build a more fair and inclusive league. The major tools APDA has to combat judging bias are calibration and our own willingness to volunteer as judges. As a debater reading this, we hope you will consider judging at future tournaments besides your own - hopefully this document has convinced you that judging bias is a pernicious problem worth working against. Being diligent about calibration of your judges and offering judge training to teammates is also a service to the league that will increase the draw and reputation of your school’s tournament. When qualified debaters do not break at your tournament, you might also consider asking them to judge outrounds - some tournaments this season have used this strategy to add gender diversity to their judging pools, to positive effect. We hope that the APDA Board and committees will lead the way on this effort by instituting judging requirements for their members in addition to hybrid requirements. Finally, Women’s Initiative hopes that judging will become a respected and honored tradition on our circuit, and looks forward to your help making this shift.

As always, feedback is welcomed at APDAWomenInitiative@gmail.com and through this anonymous form. You may also continue to submit anecdotes about judging bias here.