Can Voters Detect Malicious Manipulation of Ballot Marking Devices?

Matthew Bernhard, Allison McDonald, Henry Meng, Jensen Hwa,

Nakul Bajaj, Kevin Chang, and J. Alex Halderman

What’s a ballot marking device (BMD)?

2

Where are BMDs used?

3

Source: Verified Voting

BMD Security Assumptions

​

4

Select

Print

Review

Scan

Audit

BMD Security Assumptions

Attacking the Scanner

5

BMD Security Assumptions

Attacking the Scanner

6

Post-election audit will catch malicious scanner!

BMD Security Assumptions

Attacking the BMD

7

BMD Security Assumptions

Attacking the BMD

8

Post-election audit will confirm wrong result!

BMD Security Assumptions

Attacking the BMD

9

Voter inspection will catch malicious BMD/printer!

BMD Security Assumptions

Attacking the BMD

10

Voter inspection will catch malicious BMD/printer?

BMDs in Context

​

​

Stark speculated that voters are likely not to check their ballots on their own [1], and prior work about other types of voting equipment supports this [2,3].

Prior work on users’ response to warnings in other settings like phishing [4] and certificate warnings [5] suggest that if voters do not check their BMD ballots, well-designed warnings may help.

​

11

[1] Stark, P. B. "There is no reliable way to detect hacked ballot-marking devices, 2019." (1908).

[2] Acemyan, Claudia Ziegler, Philip Kortum, and David Payne. "Do voters really fail to detect changes to their ballots? An investigation of ballot type on voter error detection." In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 57, no. 1, pp. 1405-1409. Sage CA: Los Angeles, CA: SAGE Publications, 2013.

[3] Selker, Ted, Elizabeth Rosenzweig, and Anna Pandolfo. "A methodology for testing voting systems." Journal of usability studies 2, no. 1 (2006): 7-21.

[4] Egelman, Serge, Lorrie Faith Cranor, and Jason Hong. "You've been warned: an empirical study of the effectiveness of web browser phishing warnings." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1065-1074. 2008.

[5] Akhawe, Devdatta, and Adrienne Porter Felt. "Alice in warningland: A large-scale field study of browser security warning effectiveness." In Presented as part of the 22nd {USENIX} Security Symposium ({USENIX} Security 13), pp. 257-272. 2013.

Research Questions

Other research questions

• Does ballot style matter?
• Does candidate position on the ballot matter?
• Does type of manipulation matter?
• Are voters sensitive to the content of the warnings?

13

Study Design

Worked with AADL to setup mock polling place environment

14

Study Design

Built custom BMD out of older voting machines

15

Study Design

Used truncated version of Ann Arbor 2018 midterm ballot

​

16

Study Design

On every ballot, one of the participants’ choices was printed incorrectly in a randomly selected race

​

​

17

E.g., Participant vote for Candidate Alice is printed as vote for Candidate Bob

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

​

​

​

18

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

• Two variations on ballot style

​

​

19

Ballot Style

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

• Two variations on ballot style
• Deselecting a candidate rather than choosing a different candidate

​

​

20

Deselection

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

• Two variations on ballot style
• Deselecting a candidate rather than choosing a different candidate
• Three sets of poll worker instructions

​

​

21

Poll worker Instruction

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

• Two variations on ballot style
• Deselecting a candidate rather than choosing a different candidate
• Three sets of poll worker instructions
• Two sets of instructions while asking the participants to vote a provided, random list of candidates

​

22

Slate + Instruction

Study Design

Ran battery of 9 experiments to evaluate what impacts detection:

• Two variations on ballot style
• Deselecting a candidate rather than choosing a different candidate
• Three sets of poll worker instructions
• Two sets of instructions while asking the participants to vote a provided, random list of candidates
• Usage of a sign

​

23

Signage

Participant Demographics

Recruited 241 participants from in and around Ann Arbor

24

Study Design

Observed whether participants reviewed their ballots and reported discrepancies

25

Quantitative Results:

Non-intervention experiments

​

26

Ballot Style

Deselection

Quantitative Results:

Ineffective intervention experiments

​

27

1

Poll worker

Instruction

Signage

Quantitative Results:

Effective Script experiments

​

28

3

2

1

Poll worker

Instruction

Quantitative Results:

Effective Slate experiments

​

29

Slate +

Instruction

Other Findings

​

• Voters will find and report errors if they review the ballots
• p < 0.01 in two-sample permutation test
• Echoed in recent work by Kortum et al. [1]
• Participants finding errors was correlated with the position of the error
• i.e. the higher on the ballot an altered race was, the more likely a participant was to find it
• Pearson’s of -0.64
• Voters may be sensitive to particular language
• Informing voters of the correct behavior appeared to be important, echoing work in other settings like [2]

​

​

​

30

[1] Kortum, Philip, Michael D. Byrne, and Julie Whitmore. "Voter Verification of BMD Ballots Is a Two-Part Question: Can They? Mostly, They Can. Do They? Mostly, They Don't." arXiv preprint arXiv:2003.04997 (2020).

[2] Akhawe, Devdatta, and Adrienne Porter Felt. "Alice in warningland: A large-scale field study of browser security warning effectiveness." In Presented as part of the 22nd USENIX Security Symposium (USENIX Security 13), pp. 257-272. 2013.

Research Questions

• Can voters detect errors introduced to the paper ballot? Yes
• Are there interventions that can improve the rate at which people detect? Yes
• Does ballot style matter? No
• Does candidate position matter? Yes
• Does type of manipulation matter? No
• Are voters sensitive to the content of the warnings? Probably

​

​

31

Limitations

• It is hard to simulate a realistic election environment
• Participants still took it seriously, with some complaining about our randomized slate

32

Limitations

• It is hard to simulate a realistic election environment
• Participants still took it seriously, with some complaining about our randomized slate

​

“I noticed that there was a [Republican] selected and I'd almost never vote a republican”

33

Limitations

• It is hard to simulate a realistic election environment
• Participants still took it seriously, with some complaining about our randomized slate

​

​

​

• Our sample is not generalizable
• However, forthcoming work with a more diverse sample corroborates our findings [1]
• Findings need further replication

34

[1] Kortum, Philip, Michael D. Byrne, and Julie Whitmore. "Voter Verification of BMD Ballots Is a Two-Part Question: Can They? Mostly, They Can. Do They? Mostly, They Don't." arXiv preprint arXiv:2003.04997 (2020).

Takeaways

• Voters can review their printed BMD ballots, but mostly don’t.
• Voters’ decision to review can be impacted by in-precinct intervention
• The jury is still out on BMD security, but issues with BMD verification likely apply to other election domains too

35

Can Voters Detect Malicious Manipulation of Ballot Marking Devices?

Matt Bernhard

matber@umich.edu

@umbernhard

BMDs in Context

Voter-verified paper is a human-in-the-loop setting, where users need to understand and act on risk, i.e. that the BMD may print the wrong thing

Cranor lays out five means of communicating risk in a system [1], however only warnings apply in an election context.

Because voters are novices, warnings need to be contextual and provide an obvious recourse [2].

37

[1] Cranor, Lorrie F. "A framework for reasoning about the human in the loop." (2008).

[2] Bravo-Lillo, Cristian, Lorrie Faith Cranor, Julie Downs, and Saranga Komanduri. "Bridging the gap in computer security warnings: A mental model approach." IEEE Security & Privacy 9, no. 2 (2010): 18-26.

Experiments Summaries

38