Critical Review of "Unclear Ballot"

Edited by Ray Lutz, CitizensOversight.org   2021-07-22; Updated 2022-02-16

https://mbernhard.com/papers/unclearballot.pdf 

The "unclear ballot" paper^[1] broaches important questions regarding the validity of ballot images that may be used in ballot image audits. We believe this paper considers a very specific vulnerability which does not exist in many cases, and it improperly minimizes the value of ballot image audits.

Considers a premeditated hack to scanner with a modifiable driver

The paper considers that a sophisticated fraudster with sufficient access to central scanning operations would manipulate a windows driver to modify images as they are created and sent to the host computer. As a result, the cast vote records that are created based on those images would be consistent with the images, but not with the original paper. Even if this hack were possible, it would take considerable planning and the number of ballots modified would have to be determined in advance, without knowledge of the ultimate outcome, and could not be adjusted or not implemented at all, if the outcome went their way.

Not compared with "opportunistic" cheating scenarios

The paper does not compare the effort needed to modify images with other hacks, such as directly modifying the totals in the database, which can be considered opportunistic because they can be implemented only if needed, and then only to the extent required, such as to "find" a given number of votes in the result. Any reading of the work results in a clear understanding that the design and implementation of the driver wrapper was not a trivial amount of work, and in fact, it was far more difficult than directly modifying the totals in the database.

The UnclearBallot hack assumes that a 1) a driver exists, 2) that the fraudster has access to it, 3) that the fraudster understands image processing 4) and has programming skills, and 5) is able to decode the styles on any given election and 6) cause bubbles to be altered in the images without detection. If this were the case, then that same fraudster would have access to the database and could just modify the results directly, without all the work to understand the styles, perform oval substitution in the images, and do it in a way that was undetectable.

Did not consider modifications to an embedded voter-facing scanner

They mention only in passing that it could be done to "other layers in the stack," but there was no admission in the paper that modifying firmware in a voter-facing scanner would be substantially more difficult. Yet they do admit the approach they used was "relatively easy".

A desktop scanner like the one used in UnclearBallot typically leverages computing power in the host computer to perform conversion of a raw bitmap into an image file, such as PDF, TIF, PNG, JPEG, etc. The horsepower in the scanner itself is minimized, in some cases, providing the scanner data in a set of raw image data as horizontal bands across the page. This is in fact how the TIFF format was originally designed so that the bands could be generated from a simplified scanner mechanism. Indeed, a functional extreme of this simplification can be seen in Group 3 facsimile machines, where the data is streamed out on the telephone channel one scan line at a time, compressed using 1-D Huffman compression, and then received, decompressed, and immediately printed as a line of dots. In such a scanner and printer, the page image is not buffered more than the single scan line.

This is in contrast with typical voter-facing voting machines, such as ES&S DS200 which completely buffers two images, one for each side, then analyzes the image to determine front/back and inverting any sheets that were inserted bottom-first, then saving many ballot images to flash memory. The opportunity to hack an integrated DS200 is far harder than hacking one that is split in function between the scanner device and the host computer.

The UnclearBallot driver wraps the stock scanner driver and alters images from the scanner before they reach the election management application. We chose this approach for simplicity, as the Windows driver stack is relatively easy to work with, but the attack could also be implemented at other layers of the computing stack. For instance, it could be even harder to detect if implemented as a malicious change to the scanner’s embedded firmware. Alternatively, it could be engineered as a modification to the tabulation software itself.

The phrase "easy to work with" also means "easier than modifying embedded firmware." Yes, it would be "harder to detect if implemented as a malicious change to the scanner's embedded firmware," and it would also be far more difficult to implement. Just saying it would be hard to detect does nothing to explain the relative difficulty of the attack. Modifying firmware in a precinct-based scanner is significantly harder than modifying a driver, and the work required even for the driver scenario is already substantially greater than other attack vectors, and is not opportunistic in nature. This requires a premeditated hack, and the size of the modification to the election must be decided in advance of scanning the ballots, and thus before any district-wide subtotals are obtained.

Also, they refer to a huge image processing library, OpenCV, but then claim that they can do it in 398 lines of C++ and Python code. OpenCV is about 200MB in size, and other libraries would undoubtedly also be required. Shoe-horning the 200MD OpenCV library into an embedded scanner would be a nonstarter.

Assumed the fraudster had access

In their attack analysis, they spend no time explaining how an external fraudster might be able to alter the driver when it is in the protective domain of an air-gapped election office, nor how to defend against the attack. This paragraph is not a responsible way to represent the various methods of possible attack and provide a sense for the relative likelihood of the various attacks. As a result, readers may incorrectly assume that a ballot image hack is just as likely as much easier methods.

Would substantially slow down the scanner

Even though they did not include all the processing in their model, they also said their manipulation code took 279ms per ballot (over 1/4 second).  This is a substantial fraction of ballot interpretation processing time, and these times would be summed. So a significant decrease in throughput would probably be detected. An ES&S DS-850 central operations scanner operates at 200ms per sheet. Adding 279ms seconds would slow it down by nearly 140%. If they included all the processing necessary, then it would likely slow down the scanner to more than one or two seconds per ballot, or indeed quite a bit more. This aspect of their analysis was completely disingenuous.

In their analysis, they decided to avoid doing template matching, and instead ran their experiments on individual contests, cropped out of the entire ballot:

Because the runtime of the template matching step of our algorithm is highly dependent on customization for the particular races on a ballot, we opted to skip it for this experiment. Rather than marking full ballots, we marked cropped races from each ballot style and then ran them through our program.^[2]

Indeed, we admit that template matching may not be necessary, but it is necessary to at least identify the style of ballot and where the ovals are located so they can be altered. This was not included in their design.

Styles Issue not Fully Addressed

Another factor that is largely misunderstood by the general public is the large number of ballot styles that exist, meaning that contests may move in location on the ballot from one style to the next. In some districts, there are more than 10,000 different styles, but typically there are at least a few dozen to 100s of styles, and they change every election. So the hackers would 1. Need the style data and 2. update the hack for every election. Doing this in embedded firmware in an air-gapped system is infeasible.

With that said, we must also admit that in some cases, consequential contests such as federal races are generally at the top of the ballot and may be in the same location among all the styles of one election. But each election would still need to be separately configured, and each jurisdiction would need to be individually penetrated.

Considered right-justified ovals only

The paper also focused on right-justified ovals that are used on some Dominion ballots. However, in our experience, the vast majority of ovals are left-justified and are relatively harder to move without accidentally incorporating text nearby and making it obvious. They had to fine-tune their hack for the specific situation, which is likely not going to be feasible unless it is an insider attack, and then we can't imagine the hacker doing all this work versus just swapping out ballots.

No recommendations for securing ballot images, endorses E2E hypothetical systems

The paper does not, unfortunately, actually provide specific proposals for securing ballot images, but speaks to the idea that E2E systems are a solution, but these have not even been demonstrated, and are a big leap from the installed base, and actually have their own problems.

We submit that a more productive approach would be to provide methods to secure ballot images and defeat the attack vectors they explored. They did not do this, instead opting for the "we don't like it at all" approach while also saying that some new system that does not exist is the solution, while not analyzing it at all for any security issues to make a fair comparison.

Ballot Image audits provide benefits even if ballot images are unverified

The paper does provide at least some measure of balance by saying that ballot image audits have some usefulness. For example this passage:

Uses for image audits.

So long as image audits are not the sole mechanism for verifying election results, they do provide substantial benefits to election officials. Using an image audit vastly simplifies some functions of election administration, like ballot adjudication in cases where marks cannot be interpreted by scanners or are otherwise ambiguous. Image audits can be used to efficiently identify and document election discrepancies, as has occurred in Maryland where nearly 2,000 ballots were discovered missing from the audit trail in 2016 [28]. Image audits also identified a flaw in the ES&S DS850 high speed scanner, where it was causing some ballots to stick together and feed two at a time [29].

It must be said that these benefits exist even if no other method is used to verify election results. So the first phrase "So long as image audits are not the sole mechanism for verifying election results," which implies dependency on some other method for these benefits, is technically incorrect. The benefits exist regardless of whether the argument can be correctly made that the election results are confirmed.

The two discrepancies mentioned were discoverable even if ballot image audits were the sole mechanism for verifying election results. But in reality, election officials never rely on just one method to verify the results as there are many checks and balances in their process. The question is also not just what officials can use, but what the public can. Ballot images and cast vote records should be made public so that anyone can independently audit those images and compare them with the cast vote record results.^[3] RLAs cannot be performed by the public.

The benefits mentioned are generally NOT available in other auditing methods, such as RLAs and FHCAs (Full Hand Count Audits), because those procedures do not compare at the ballot level for all ballots and all contests.

Did not admit that RLAs fail and FHCAs have a human-error rate

The author asserts that RLA audits will work, but we know they will fail (become impractical) when the margins are tight:

Image audits are an augmentation and a convenience for election administration, however, and should not be viewed as a security tool. Only physical examination of paper ballots, as in a risk-limiting audit, can provide a necessary level of mitigation to manipulated election results.

Instead, we assert that ballot image audits can and should be viewed as a security tool, because they are effective against opportunistic cheating, can cover all contests, and are not limited by margin of victory.

An RLA becomes impractical when the margins are tight, say less than about 1%^[4]. At some point, essentially all ballots must be included in the sample. If these are randomly drawn, then the cost and error induced is higher than performing a FHCA from the start. Ballot images can be confirmed by creating validation scanned images of a small subset of the paper ballots and comparing those results with the original ballot image audit, and are a good partner process with any RLA or FHCA process.

RLAs and Full Hand Count Audits have a substantial human-error rate

Officials generally turn to a FHCA, which has a lower overall error rate than an RLA, because they are easier and simpler to implement. But even then, university studies have shown an error rate of 0.48% to over 2% for manually tallying ballots^[5]. RLAs (sampling audits) can do no better than a FHCA. RLAs have more error mechanisms, due to the fact that auditors must choose random samples, then draw those samples, review them with the same likely error rate as FHCAs, and then enter the data into at least a spreadsheet and perform calculations to determine the sampling error rate (which does not include the base FHCA error rate). Also, it is improper to make corrections to sampled batches "as you go" with RLAs whereas with FHCAs, that is allowed. We find this last point is a constant issue because elections workers and the general public think it is okay fix problems with the official results "as you go" but are actually not allowed in any form of a sampling audit^[6]. All those steps do not exist in FHCAs. So if you have a very close election, it would be a poor technical decision to use an RLA, and instead a FHCA is generally used from the start.

Ballot Image Audits can improve RLA and FHCA accuracy

At that point, we need a method that has precision down to the ballot to be able to check the results of the FHCA. The FHCA can be compared with the original results on a batch basis, but generally not on a ballot basis because the typical procedures for a FHCA do not track every ballot, and if they do, then this further complicates the process, inducing even more error and increasing the time required.

A ballot image audit can compare with the official results on a ballot-by-ballot basis, and on a batch basis to check the results of the FHCA. This can further reduce the error rate in those situations when the margin is extremely close.

IN CONCLUSION

The "Unclear Ballot" paper did not compare the relative difficulty of this demonstrated hack with other methods of hacking the election. They assumed they had full access to the computers and drivers, and they did not describe that such a hack would be obvious due to the time delay induced and they minimized the usefulness of ballot image audits even if the images were not otherwise verified. They assumed full access to ballot styles information (which is available by request but then the attacker is identified), and they implemented a partial hack, without doing template matching or other methods to identify the location of the contests and options. And finally, they assume that RLAs are a solution when it is known that RLAs actually will not work in critical situations, and they finally declare that E2E systems will fix everything, without admitting that those have not been even demonstrated and that every system has its own limitations. (Because of this bias, we might guess that perhaps the writers were intent on promoting E2E systems or perhaps gaining research funding for this deeply technical area of investigation.)

We believe that ballot image audits are an essential component for public verification of elections because:

1. They can audit all contests on all ballots
2. Do not fail when margins are tight.
3. Find problems that RLA and other methods cannot.
4. Can be deployed by the public independently of officials.
5. Do not rely on verification when problems are actually found.
6. Work well with FHCAs to help to reduce the error rate.

Please take this paper with a very large grain of salt.