2020 Election Ballot Image Audit of

Bartow County GA

Raymond Lutz
Executive Director and Lead Developer,
Citizens' Oversight Projects

V2.2 2022-10-04  (M1983)

Citizens' Oversight Projects

is a 501(c)(3) nonprofit and

nonpartisan organization

http://citizensoversight.org 

Executive Summary

This document summarizes results of an audit of the November 2020 election in Bartow County, Georgia by reviewing the ballot images and other data. This audit was performed using the "AuditEngine" platform being developed by Citizens' Oversight Projects, also known as "Citizens Oversight" or "COPS".

This audit was conducted to demonstrate the capabilities of AuditEngine and the approach of ballot image audits in general, and to provide information about the reliability of the 2020 General Election.

The primary audience for this report are election officials in Bartow County and the state of Georgia, but we anticipate the general public will also be interested in these readily accessible results.

AuditEngine is an election auditing platform which performs "Ballot Image Auditing". Modern voting machine ballot scanners capture relatively high-resolution digital images of each ballot in polling places or central count operations. AuditEngine processes these ballot images to create an independent tabulation, and then it compares its evaluation of each ballot with the official cast vote record (CVR), which provides ballot-by-ballot detail of the official evaluation by the voting system.

AuditEngine can provide detailed reports which describe discrepancies between the official records and our independent tabulation. Comparing results from two systems like this can expose errors in each system which would be very hard to find otherwise. While election systems are usually accurate, various factors can introduce problems by mistake or on purpose^[1]. Software updates, changes in the election definition, or malicious activity may change the outcome.

Most voters have doubts. Only 13% of Republicans and 4% of Democrats in 2018 were "very confident that election systems are secure from hacking and other technological threats."^[2] The 2020 Election was more secure than recent elections because of the use of paper ballots in more districts but improvement is still possible.

AuditEngine’s analysis of the 2020 General Election in Bartow County, GA found:

• Among the ballots processed in the audit, there was no evidence of significant inconsistencies that would cast any contest into doubt.
• 50,678 ballots were cast in the official election results. The ballot image audit processes ballots based on the sheets involved. In this election, all ballots had only one sheet, so we expected to review 50,678 images. The inventory of images was an exact match.
  
• AuditEngine was able to process all the Bartow ballot images and found no images that were corrupted due to scanner maladjustment or lack of maintenance.
  
• AuditEngine found very few voter intent differences. None of the disagreements cast doubt onto the outcome of any contest.
  
• Bartow County performed extensive adjudication of all overvotes, and of all undervotes in the Presidential contest. The adjudication of these issues was recorded in the election data. They also adjudicated all write-in votes, which require review to determine whether they are for qualified write-in candidates or for other, unqualified persons (or entities–the Mickey Mouse write-ins).
  
• Since in 2020 Bartow County used  the newest Dominion software suite available at the time (Democracy Suite 5.5), the cast vote record includes the additional clarification of voter intent when the ballot has been adjudicated. Adjudication is done by election staff using the Dominion software suite using human-eye review of the ballot images. Consequently, most voter intent disagreements between AuditEngine and the voting system’s initial interpretation had already been correctly adjudicated by Bartow elections staff.
  
• AuditEngine enhanced its operation to accommodate the Dominion CVR unexpectedly classifying all adjudicated overvotes as undervotes. Also, AuditEngine found an incorrectly adjudicated ballot, which is a fascinating case (see "Inappropriate Adjudication" under "Red Flags" later in this report).
  
• We found no significant issues that could not be explained.
  
• After refinement of our BMD parsing algorithm, AuditEngine parsed 100% of the 39,967 BMD ballots^[3] (except for one) so the result can be compared with the official cast vote record, and no variations were discovered. The very clean and complete data set provided by Bartow officials also facilitated refining AuditEngine to accommodate prospective audits in other Dominion jurisdictions.
  
• Audit Engine was able to "read" the human readable text and generate an independent cast vote record of each ballot, then compare it with the official result. AuditEngine found zero discrepancies among the nearly 40,000 BMD ballots cast.
  
• This is an extremely important feature when considering the recently exposed "flaws" in the Georgia voting system, as reported by J. Alex Halderman, a computer science professor at the University of Michigan. As an expert for plaintiffs in an election security lawsuit, Halderman gained access to Georgia voting equipment for 12 weeks and produced a 25,000-word secret report.

Halderman found that malicious software could be installed on voting touchscreens so that votes are changed in QR codes printed on paper ballots, which are then scanned to record votes, according to court documents. QR codes aren’t readable by the human eye, and voters have no way to know whether they match the printed text of their choices^[4].

• AuditEngine’s initial automated vote extraction had different results than the Dominion voting system, in some instances, and did quite well in guessing the final adjudicated resolution for overvotes and for a few instances of undervotes eventually resolved to votes.
  
• When the voting system and AuditEngine disagree on voter intent, the correct interpretation becomes clear by looking at the disputed ballot image. By “correct interpretation” we mean the human eye determination, which is the deciding interpretation under Georgia voter intent law.^[5]
  
• The first pie graph below shows the total ballot sheets in the election, the number of images analyzed by AuditEngine.

• 1,044,051 votes were on 50,681 ballot sheets (including blank votes and unprocessed sheets).
  
• The Fully Agreed sheets (47,621, or 94.0% of all ballots cast) were completely agreed between AuditEngine and the voting system and had no variations, such as write-ins, overvotes, or gray-flags.
  
• Partially Agreed Sheets: 3,056 sheets (6.0% of all ballots cast) had 54,133 contests (5.2%) that were non-variant and agreed, while 8,910 contests (0.85% of all contests) on those same sheets were classified as "variant contests" and were "pulled" from the partially agreed records and individually classified in separate records for each contest, for further reporting categorization.
  
• Total of Nonvariant Contests: Thus, a total of 1,035,141 votes (aka, ballot-contests, 99.1%) on these ballots were interpreted the same and non-variant in every respect by AuditEngine and the voting system, and there was no additional scrutiny required due to write-ins, overvotes, or disagreements.
  
• Contest Variants: The "Contest Variants" (8,910 votes, 0.85%) were further categorized by AuditEngine. These are the individual contests ("votes") which had either write-ins, overvotes, gray-flags, or were "disagreed". A contest is "disagreed" when AuditEngine and the voting system did not interpret the vote exactly the same. Please note this is across all contests. There were 27 contests in the election, and these variants are spread over all contests.
  
• Normal Disagreed: Of those, 328 were classified as "disagreed", while the rest were write-ins, overvotes or gray only. These will require additional scrutiny in close elections.
  
• Closest Contests: Contests were individually considered. The 2 most discrepant contests had disagreements between 0.32% to 0.13% of the margin of victory:
  

• US Senate (Loeffler) - Special

• Margin of victory: 5,236 votes (10.55%)
• 17 votes "Disagreed" (0.32% of margin)
• 160 contest variants (3.06% of margin)
  

• US Senate (Perdue)

• Margin of victory: 10,303 votes (20.57%)
• 13 votes "Disagreed" (0.13% of margin)
• 83 contest variants (0.81% of margin)

The Presidential Contest was of particular interest in this election:

• County Margin of Victory: 25,582 (50.69%)
• Statewide Margin of Victory: 11,779 votes (about 0.23%)
• 10 votes disagreed (0.04% of county, 0.084% of statewide margin)
• 160 contest variants (0.63% of county, 1.35% of statewide margin)

Across all contests:

• Most Variant: 9.09% was the highest level of variant votes in any contest, as % of the margin of victory in the contest "Adairsville City Council Post 4 - Special" had a margin of 5.29% (99 votes) and there were 9 contest variants but no disagreements.
  
• AuditEngine's Correct Evaluations when Voting System required adjudication:
  252 votes were correctly interpreted by AuditEngine while the Dominion voting system initially misinterpreted those votes ('x_y_x' group), and these were corrected by Bartow County staff during adjudication to match the AuditEngine evaluation.
  
• 2 votes were very difficult to interpret and were not interpreted correctly by AuditEngine nor by the voting system but were corrected or confirmed by Bartow County staff during adjudication.

The most common reasons for discrepancy were:

• where the voter circled or checkmarked the oval, but did not darken the middle of the oval, and the election system did not look outside the oval.
• where the voter hesitated and slightly marked one oval and then definitely marked the other one
• where the voter scratched out one oval with a very large mark while marking the desired option with a correct but smaller mark.

The second pie chart shows the major categories of votes with write-ins or overvotes, with the "other" category including the disagreed votes where there were no write-ins or overvotes.

About Bartow County

Bartow County, and all counties in Georgia, use the most recent release of the Dominion Voting System (Dominion). We were able to compare the results between AuditEngine and the voting system down to each contest on each individual ballot, because those systems can provide the ballot-level "Cast Vote Record" (CVR) file, which is the digital record of voter intent for every contest on that ballot. In addition, this county used the Dominion adjudication system, and any ballots that were adjudicated and changed had both the 'original' and 'modified' CVR records.

This table provides the overall profile for a ballot image audit of this election:

We appreciate the high level of cooperation by Bartow County and their thorough processing of this election and adjudicating all write-ins, all overvotes and many undervotes.

Although there were a couple of minor details to discuss with Bartow County, we found no inconsistency in the outcome of the election based on our independent ballot image review and comparison with the official cast vote record.

TABLE OF CONTENTS

Executive Summary        1

1. Background        9

2. Details of this audit        9

3. Setup and Mapping Comments        10

4. Discrepancy Report        10

4.1 Discrepancy Report -- High Level Reconciliation        11

4.2 Discrepancy Report -- AuditEngine Flags of Ambiguous Votes        14

4.3 Discrepancy Report -- Contest Variant Breakdown        16

4.4 Discrepancy Report -- Disagreements        19

4.5 Discrepancy Report -- Contest Discrepancy Report        21

4.6 Discrepancy Report -- Precinct Report        22

4.7 Discrepancy Report -- Presidential Contest        22

4.7.1 Normal Disagreements not including overvotes and write-ins.        24

4.8 BMD Disagreements:        29

5 Discussion of Interesting Issues in Bartow County        32

5.1 General evaluation        39

5.2 Undervotes were adjudicated in presidential contest        41

6. Overall Summary and Findings        42

7. Conclusion        44

Primary Author: Raymond Lutz        44

How to Comment        45

APPENDIX 1 -- Links to detailed reports        46

Appendix 2 -- Overvotes marked as Undervotes Issue        46

Appendix 3: Other Audit Details        49

Ballot Image Archive (BIA) Report        49

EIF -- Election Information File        51

Styles Report        51

1. Background

To reduce the size and complexity of audit reports, background information has been moved to a companion document:

"Auditing Elections Using Ballot Images and AuditEngine -- General Background"
https://docs.google.com/document/d/18A1K8mXXHnhisLqBQigx0ibboz39FAh9hOSykcR-jT4/edit?usp=sharing 

Please fully read and study this document before attempting to digest the rest of this report, particularly with respect to the terminology defined.

A note on writing style

Throughout this document, we will use "programmer" style quotes, which always frame the terms and do not include punctuation. Also, as a matter of style, numbers are always shown in numerical form, commas will always be included in conjunctive lists, and all quotes are straight.

2. Details of this audit

1. With the general description completed, the results of the audit in Bartow County, GA will be described. Regarding some of the issues described in the general description above, we can refine this description as follows:

1. Bartow County was able to provide 100% of the ballot images, including 50,678 ballots which were predominantly BMD (Ballot Marking Device) ballots (39,967), and 10,711 hand-marked ballots.
   
2. Due to the recent changes in Georgia law, ballot images are recognized as public records that are not exempted from release to the public^[7]. Bartow County provided their ballot images and CVR records at no charge.
   

1. Files: The files were very clean and in perfect organization. However, we would appreciate it if, in the future, Bartow creates separate JSON file chunks for the CvrExport.json file instead of one large file. Our instructions already have the settings desired:

1. In the CVR Export screen, you should choose <<ALL>> for all filters, PLEASE:
2. CHECK "create a separate file per batch"
3. CHECK "Published batches only"
4. UNCHECK "Use tabular format"
   

3. Setup and Mapping Comments

3. For this audit, we initially used computer-assisted manual mapping using our TargetMapper application. Later, we were provided with the Ballot Style Masters and we implemented a faster method by parsing those PDF files. The combination of Ballot Style Masters and human input results in mapping being completed far faster and more accurately. The time required was reduced substantially as it is now proportional to the number of contests rather than the (number of styles) x (number of contests). There were 41 styles in this election with 16 different patterns of contests and 28 different contests. Georgia does not rotate ballot options and this can reduce the number of styles slightly.

4. Discrepancy Report

4. Vote Evaluation Method: Georgia is a voter-intent state. This means that the intent of the voter is to determine the vote rather than based on how the machine would read it. For example, if a voter fills in an oval, then crosses it out, and writes "No" next to it, and fills in another oval, then the second oval would be interpreted as the intent of the voter^[8].
   
5. Comparison. After the vote is extracted and evaluated by AuditEngine, it is compared with the cast vote record. This is the most sophisticated stage in the process, and our reporting methodology is more precise and detailed than other ballot image auditing solutions.
   
6. This process was performed for Bartow County because we were provided with the complete detailed CVR.
   
7. The detailed discrepancy report as prepared for this election by AuditEngine is extensive and provides images of the ballots of concern. It is not intended nor recommended that this report is printed out on paper. Instead, it is best to review it in a browser so the hot links will operate and so that specific patterns can be searched for. Here, we will summarize the important points from this report.
   
   If there is any discrepancy between this narrative report and the machine produced report linked below, the machine produced report may have been slightly updated and should be considered the official audit result. Here is the link to the report.

https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Discrepancy_Report.html 

8. In Bartow County, we encountered zero hand-marked paper ballot images that could not be aligned and the vote extracted by AuditEngine. This is exceptionally good and a first for AuditEngine. We believe this was the case because these early voted ballots were processed by central count scanners which do not have the same quality issues as some of the in-precinct (and therefore inexpensive) ballot scanners.
   
9. There were initially a number (fewer than a dozen) BMD ballots that had slightly corrupted images which resulted in one contest selection not being fully recognized, and "gray flagged" so they could be reviewed if desired. AuditEngine has been enhanced in a general way so that these ballots were completely processed (except for one ballot that had curved lines), and these changes will be applicable to future audit projects.
   

4.1 Discrepancy Report -- High Level Reconciliation

10. Contest Variant Definition: A contest variant is a contest on one ballot where AuditEngine disagreed with the voting system evaluation of that contest, or where there were write-ins, overvotes, or "gray" marks. Undervotes are not considered a variant unless they are considered "disagreed" or are flagged as "gray".
    
11. Agreed Undervotes: If undervotes are disagreed, then we do treat it as a contest variant. We do not routinely classify all agreed undervotes as contest variants. If this were done, it would result in a vast number of contest variants, one for every contest that voters skipped. Yet, this can be an important consideration on hand-marked ballots, particularly in "critical contests" where voters might circle the ovals, circle names, or for other reasons. The total number of undervotes in any specific contest is provided in the contest summary in the contest detail report.  A planned enhancement for AuditEngine is to treat agreed undervotes as contest variants in  "critical contests", such as the Presidential Contest, because it is rare that a voter would skip this contest compared with down-ballot contests. Critical Contests can be any contest designated by those running the reports.

When adjudicated by the staff in Bartow County, all adjudicated and confirmed overvotes were converted to undervotes in the Presidential Contest, inflating the number of undervotes.

12. High-Level Reconciliation by Sheets: 
    The following pie chart shows the High Level Reconciliation by Sheets.

2. 9. No Images: 0 sheets (0%).
4. 10. Fully Agreed sheets: 47,621 sheets (94.0%): The AuditEngine evaluation for all contests on these ballot sheets agreed with the CVR from the voting system. Any ballots with write-ins, overvotes, or gray marks would mean the ballot sheet would be categorized as Partially Agreed, and the contests that deviated would be logically "pulled" from those sheet records and included in Contest Variants.

2. 11. Partially Agreed sheets: 3,056 sheets (6.0%): At least one contest was found that disagreed or had write-ins, overvotes or were gray-flagged on these sheets. Each record in the Partially Agreed set is for one entire sheet but with at least one variant contest removed from that record, and a separate record is created for each variant contest in the group "Contest Variants". There were 54,133 contests on these sheets that agreed with no variations.
   
3. 12. Sheets with Contest Variants: 3,056 sheets (6.0%) had at least one or more contest variants.
   
   
4. 13. High-Level Reconciliation by Contests: When we view the same data by ballot-contests, "Fully Agreed Sheets" and "Partially Agreed Sheets" categories are the number of contests remaining in those sheet records after Contest Variants have been removed. Please note that these numbers are for the contests on those sheets that were agreed and had no variations.

6. 14. Agreed and Non-Variant Ballot Contests: There were 1,035,141 non-variant ballot-contests, including 981,008 contests on sheets that were "Fully Agreed" and 54,133 contests on sheets that were "Partially Agreed".

15. Total Agreed Ballot-Contests: AuditEngine processed 1,035,141 ballot-contests and 99.1% of these contests on 50,677 sheets had no variations and were fully agreed from both the Fully Agreed and Partially Agreed groups.

7. 16. Contest Variants: 8,910 ballot-contests on 3,056 sheets were classified as Contest Variants (0.85%). Contest Variants have either write-ins, overvotes, gray-flags, or are considered "disagreed".
   
8. 17. Initial Consistency Screen: The set of Contest Variants provides an initial consistency screen. If we had processed the entire election and if the tightest margin of victory was greater than twice the number of Contest Variants, then the outcome could be deemed as consistent, because even if all Contest Variants are fully reviewed and altered in favor of a losing candidate, it can not alter the outcome. This is a very conservative threshold because all Contest Variants are among all contests.
   
9. 18. Presidential Contest: In GA, the statewide margin of victory in the Presidential contest was only 0.23% and 11,779 votes, which is less than 8,910 x 2 = 17,820. Therefore, it is necessary to break down the "Contest Variants", and also look at each contest in detail.
11. 19. Other County Ballots: All ballots were checked for the proper county by looking for the words "BARTOW COUNTY" at the top of the ballot sheet by OCR to determine if any ballots had been mixed in from other counties. All ballots were appropriate for Bartow County. AuditEngine detected 0 ballots from other jurisdictions.
    
    20. Unprocessed: 1 BMD ballot was not successfully processed due to curved lines of text.
12. This ballot can be reviewed at this link in the report:
    
    https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Discrepancy_Report.html#unprocessed-ballots-report 

14. 4.2 Discrepancy Report -- AuditEngine Flags of Ambiguous Votes

22. 22. If we had no cast vote records (CVRs or "CVR Files"), then we could not compare the AuditEngine tabulation on a ballot-by-ballot basis, but we can compare with aggregated totals. AuditEngine also provides "gray flags" when it uses heuristics or is unsure of ambiguous marks. To refine the results, we can take a look at the write-ins, overvotes, and contests flagged as "gray" based on the evaluation of AuditEngine alone. These are shown in the following pie chart, without reference to the CVRs.
    
    Of course here, we have the detailed CVR and we will use that instead, but this is shown here for demonstration of how the result could be used to identify ballots that need further review. These can then be reviewed using the AuditEngine Adjudicator App.
    
23. In the following sections, these categories will be discussed.

24. 23. Total Flagged Contests: There were a total of 8,913 ballot-contests flagged for additional scrutiny, which is spread across all contests. It is not less than half the statewide margin of victory of 11,779 votes in the presidential contest, so conservatively, it would need additional scrutiny particularly because of the fact that there are 159 counties in GA, and so we need only 74 changes on the average in each county to get the 11,779 votes.
25. 
    24. CVR was Available: Because the CVR is available, we need not depend only on the flags by AuditEngine. Thus, these are discussed here only to explain the capability available when there is no CVR. The category "ae_uncategorized" are the ballots that were discovered as variants by the CVR comparison. Thus, we would not have discovered the ae_uncategorized group without the CVR comparison.
    
26. 25. Across All Contests: We must be cognizant that these figures are across all contests, and there were 27 contests in the election. This would further dilute the effect of the contest variants. Also, each variant should be considered based on whether it would decrease or increase the margin of victory.
27. 26. AuditEngine flagging detailed breakdown:

Non-Gray Write-ins:         2,451 contests         (27.5%)
which are flagged because they involve write-ins. These are generally reviewed by the election department in detail, and so we are unlikely to find that these were misinterpreted by the time the canvass is completed.

BMD Write-ins:                5,940 contests        (66.6%)
Gray Write-ins:                 51 contests                 (0.57%)

Non-Gray Overvotes:         104 contests         (1.2%)

Gray Overvotes:                 5 contests                (0.06%)

Other Gray:                         146 contests         (2.4%)
Uncategorized:                216 contests                (2.4%)

These Uncategorized variant contests were not flagged by AuditEngine but were identified due to comparing AuditEngine with the CVR. Thus, these will be found in the "Normal Disagreed" group.

29. 4.3 Discrepancy Report -- Contest Variant Breakdown

27. When the CVR is available, the Contest Variants can be further categorized and reviewed. This is the most powerful way to analyze an election, but since these are across all contests, the review will more appropriately inform considerations about the general quality of the canvass rather than reflecting on exact outcomes.
    
28. Contest Variants: 8,873 contest variants out of 1,044,051 total ballot-contests (0.85% of all contests)  were either disagreed, write-ins, overvotes, or gray-flags pulled from the "Partially Agreed" sheets. Because the Dominion Voting System includes a "modified" record if the ballot was adjudicated and some change was made, this allows our analysis to be even more detailed. But we must emphasize that if a contest is not changed in the 'modified' record, then we are not sure if these were inspected and confirmed or not.

Of the Contest Variants, there were:

29. Write-Ins:

The most common form of variant are write-ins. All write-ins total 8,439 variant contests, 95.1% of all contest variants.

We can break these down based on whether the CVR has a modified record, which means that it may have been reviewed and adjudicated. But the modified record may have been created due to review of a different contest on the ballot. If so, then the entire 'modified' record is created, and if write-in is unchanged, then it probably was not actually reviewed and confirmed. Unfortunately, in the CVR, there is no indication available for "reviewed and confirmed". Therefore, although these are called adjudicated (meaning having the modified record) they may not have been reviewed.

1. Unadjudicated Write-ins: 5,603 Contests on 2,022 sheets (63.1% of contest variants) were write-ins without adjudication records. Write-ins on BMD ballots are keyed-in and do not need to be individually reviewed and decoded from voter's writing, and there is no doubt on a BMD ballot if the contest has a write-in.
   
2. Adjudicated Write-ins: 2,836 Contests on 854 sheets (31.9% of contest variants) included adjudication records, and were not necessarily changed.

"Officially Qualified Write-in Candidates": Write-ins have another wrinkle because although the write-in may be properly marked and a name written-in, it is usually not valid unless the name refers to an officially qualified write-in candidate. The voting system may indicate a write-in candidate was correctly indicated, but later, the list is reduced to only the qualified write-in candidates. However, this is a matter of state and local election statutes that may vary from place to place, and procedures used by election officials may vary as well.

Generally, at  this stage, the write-ins are not often reduced, based on whether they are on the qualified write-ins list.

Sometimes, the write-ins are for one of the official candidates. In those cases, when reviewed, the vote is awarded to the official candidate. So for example, in this election, if the voter marked the oval for the write-in line and wrote "Trump", then that vote would be awarded to that candidate, even though the candidate is an "official" and not a "write-in" candidate.

For example, this ballot is the very first one in the ballot image set (00010_00000_000871) and it has an agreed write-in. Since "ANYONE ELSE" is not a valid write-in candidate, and although there is no question the voter entered this as a write-in vote, this should optimally be regarded as an undervote.  However, at this stage it is in the set 'bmd_wi_wi'.

At this juncture, we do not attempt to re-adjudicate all these rulings on the write-ins by the voting system adjudication process. We call these "agreed write-ins." That is, we agreed that these are write-ins, and we generally accept these as adjudicated by the voting system. Yet, we did carefully examine all of these in critical contests, such as the Presidential contest.

Sometimes, the write-ins are for one of the official candidates. In those cases, when reviewed, the vote is awarded to the official candidate. So for example, in this election, if the voter marked the oval for the write-in line and wrote "Trump", then that vote would be awarded to that candidate, even though the candidate is an official and not a write-in candidate.

30. Overvotes:
    All overvotes total 106 contest variants, 1.2% of all contest variants. Again, we can break these down based on whether they have an adjudicated record.
    

1. Adjudicated Overvotes: 106 contest variants on 85 sheets (1.2% of contest variants). These appeared on ballots that had modified records, and were not necessarily changed.
   
2. Unadjudicated Overvotes: There were 0 overvotes that did not have a modified record.

Bartow County employed a useful method of marking overvotes as undervotes when they have been checked and adjudicated as overvotes. This has the beneficial side effect of marking these as "done" so that in the end, there are no unadjudicated overvotes. However the downside of this method is that overvotes are under-reported in final automatic reports by the voting system, while undervotes are over-reported.

We investigated this issue with Bartow County election officials and determined that it is a matter of procedure that they elected to use and not a function of the adjudication system. The details of this issue are provided in Appendix 2.

31. Other Contest Variants - "Normal Discrepancies":
    Contest Variants other than Write-ins and Overvotes are "Normal Discrepancies". There were a total of 328 contest variants in this category, 3.7% of all contest variants. Included in this category are contests with true disagreements, but also contests where initially the voting system and AuditEngine disagreed, but after adjudication, the evaluation was in agreement. By "Normal" we only mean that they do not include write-ins or overvotes as their primary categorization by AuditEngine.

4.4 Discrepancy Report -- Disagreements

32. "Normal Disagreed" Contest Variants included 323 contests. By "Normal" we only mean that they do not include write-ins or overvotes as their primary categorization by AuditEngine.

33. Originally disagreed vote adjudicated to confirm the original AuditEngine interpretation (x_y_x):
    252 contests (82.1% of Contest Variants in the Normal Disagreed category). These were definitely reviewed by Bartow County adjudication staff, and their adjudication agreed with the original AuditEngine evaluation, but disagreed with the initial voting system evaluation. This shows that AuditEngine generally does a better job than Dominion in evaluating these contests.
    
34. Originally disagreed vote; CVR vote confirmed by adjudication (x_y_y). 
    2 contests (0.66% of Contest Variants in the Normal Disagreed category). These are typically very difficult ballots to interpret and call for human-eye review. Sometimes or very often, the adjudication is incorrect because no adjudication occurred, even though the adjudication record exists. These were difficult to interpret ballots that required human-eye review.
    
35. Agreed Undervotes, confirmed as votes by adjudication (uv_uv_x). 
    36 Contests (11.9% of Contest Variants in the Normal Disagreed category). These were all definitely adjudicated and the change may be due to other voter marks to explain voter intent. Since undervotes are largely not adjudicated as there are so many, these are likely reviewed in close or critical contests.
    
36. All other normal disagreed categories. 
    19 Contests (6.3% of Contest Variants in the Normal Disagreed category). These are best reviewed in contest-specific review.

4.5 Discrepancy Report -- Contest Discrepancy Report

37. The most effective report is the Contest Discrepancy Report because the disagreements can be related to the margin of victory in a specific contest.
    
    This portion of the report is at this URL: https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Discrepancy_Report.html#contest-discrepancy-report 

Here is a clip of the top portion of this report:

38. For any particular contest, we can focus on the "Disagreed% of Margin" or the "Variant% of Margin". The margin of victory in votes for the contest is between the last-winning candidate and the first-losing candidate. This is not the "pairwise" margin^[9], but the actual margin including all other candidates. For ease of reading, the closest 5 contests are highlighted in terms of the Disagreed% of Margin and Variant% of Margin, and also contests with margins of victory below 10% are highlighted. These contests are also detailed and can be accessed by the contest name link. (Other contests can be added to the report as needed.)
    
39. There were no contests that had no listed options. If such contests exist, then they will appear to be extreme variants and would need to be set aside.
    

    ---

40.  The closest two contests were as follows:

4.6 Discrepancy Report -- Precinct Report

41. The Precinct Report provides a breakdown of the ballots in each precinct. These values are ballot counts, and are not specific to any particular contest. This report can sometimes highlight issues that may be specific to any particular precinct, but in our opinion is not as valuable as the Contest Discrepancy Report. Nonetheless, we include it because some states have requirements for this report. This report highlights the highest 5 Disagreed% of Margin precincts.
    
42. Overall, there were 323 sheets with contests categorized as Normal Disagreed cases.

4.7 Discrepancy Report -- Presidential Contest

43. To give the reader an understanding of the detail to which AuditEngine provides an ability to analyze the results, we will focus on the contest 'President of the United States', since it is the most consequential contest and was quite close state-wide with an official statewide margin of 11,779 votes. However in Bartow county, the margin was 25,582 votes with a margin of 50.69%, with Joe Biden receiving 12,092 votes (24.0%) and Donald Trump receiving 37,674 votes (74.7%). Thus, although this was a landslide victory in this county, statewide this contest was still quite close, and any deviation of at least 11,780 votes might flip the election.

The analysis by AuditEngine shows that even if we consider all contest variants in this contest (160), they only account for less than 1.4% of the total margin needed to flip the election with these results alone.

As you will see, the election staff at Bartow County did a very good job of adjudicating this contest, and there is really no chance the results could be altered based on the ballots included in our audit. We found zero cases which were improperly evaluated in this contest.

The official results of the Presidential Contest had 0 overvotes and 209 undervotes. When confirmed in adjudication, overvotes are classified by Bartow County staff as undervotes. Thus, undervotes are over-reported and overvotes are under-reported.

Undervotes on nonBMD ballots in critical contests may be of interest for additional review. These are not included in contest variants in this version of the Discrepancy Report but will be added as an option in the future as Contest Variants for "critical contests". "Critical contests" is a setting in the Job file and any contests can be considered critical. Also, the adjudicator will have access to undervotes as contest variants for all contests.

44. General approach for this document: Here, we will take a sample of a few of the cases in each category. The full Discrepancy Report for the contest "President of the United States' can be reviewed for more details. It provides ballot images for (up to) the first 10 cases in each category. Although they are categorized as variants, we do not routinely review agreed write-ins and agreed overvotes or agreed votes flagged as "gray", unless the contest is extremely close, because even if all the variants are ruled for the losing candidate, there is no way to overturn the election.
    
    The cases we did review showed that the election office adjudicated the results well and there is no chance that the outcome could have been different, given that the ballot images are an accurate representation of eligible votes cast.

The notation is covered in the background document, but will be repeated here to refresh the memory:

[bmd_]AE_CVR_ADJ

We will see bmd_ if the contest variant applies specifically to BMD ballots, and left off if it is a nonBMD (hand marked) ballot.

AE is the AuditEngine evaluation, and will be either 'x', 'wi', 'ov' or 'uv' meaning a vote, a write-in, and overvote, or an undervote, respectively.

CVR is from the cast-vote record, and is the voting system evaluation. It has the same list of abbreviations except it might also have 'y' to mean a vote that differs from the vote 'x'.

ADJ is the adjudication, and if it differs from the CVR value, then we are sure that the contest was reviewed by election staff and changed, otherwise, we are not sure if it was reviewed.

So for example, if the designation were wi_uv_x then it means that AuditEngine evaluated it as a write-in, the voting system evaluated it as an undervote, and after adjudication, it was evaluated as a vote for a listed candidate.

4.7.1 Normal Disagreements not including overvotes and write-ins.

45. Originally disagreed vote adjudicated to confirm the original AuditEngine interpretation. x_y_x. (8 Cases)

These are perhaps some of the most interesting, because they point out that most of the time, AuditEngine will correctly interpret voter intent without additional adjudication. In these cases, the AuditEngine interpretation disagreed with the voting system, but then after adjudication, the vote was judged to be the same as the original AuditEngine interpretation. Here are a few cases.

46. Originally disagreed vote; CVR vote confirmed by adjudication x_y_y (1 Case)

At first glance, this case is where AuditEngine failed to provide the proper result, but as this was actually a write-in, it will still require human-eye review. This was gray-flagged. AuditEngine will more effectively evaluate this case in a future enhancement because the write-in area metric will also be used to trigger a write-in classification.

47. Agreed Undervotes, confirmed as votes by adjudication uv_uv_x: (1 case) This case is definitely adjudicated because the record is changed for this contest.
    

4.8 BMD Disagreements:

A key finding in this election was that there was no discrepancy between the votes cast based on the human-readable text and the official cast-vote record.

The QR Code on Dominion BMD ballot summary sheets is not compatible with QR Code readers available in cell phones as they are binary encoded rather than producing readable text. It is therefore not feasible for voters to check that the QR Code properly encodes the selections that are printed in the summary in any convenient fashion in the voting booth.

There is a fear that the human readable text might say one thing and the votes encoded in the QR Code may say something else^[10]. Since AuditEngine reads the human-readable text and does not use the QR Code to determine the votes on the ballot, it will detect this theoretical vulnerability.

It is important to note that BMD ballot summaries such as this cannot be audited regarding original voter intent. Hand-marked paper ballots have an important advantage in that the voter marks the ballot with their own hand, and can watch the oval being filled, and notice which candidate or ballot option they are selecting. The voter, when they read the ballot, can see all the options in that contest. Thus, auditors know by viewing a hand-marked paper ballot later that all the options were presented to the voter and the voter verified their vote. In contrast, ballot summary sheets from Dominion BMD devices cannot guarantee that the voter verified their vote and it is not possible to tell if the voter was shown all the available options in the voter's private voting session.

University studies have found that most voters do not check what is written in the ballot summary list, and when errors are inserted, they detect them only 7% of the time.^[11]

AuditEngine read nearly 100% of the ballot summaries (about 40,000 in this election with only one ballot that would not read successfully) and confirmed that these summaries matched the cast vote record. Although this is working well, fine tuning this to work was not a simple matter. The format and size of the summary can be improved to help both AuditEngine and human readers, particularly those with poor eyesight. The space between lines and the space between columns should be increased to aid in reading. Also, it would help if the contest titles were reduced in length and complexity.

We are also concerned that blind voters, if they attempt to read the text with text-to-speech readers, will  have trouble with the three-column layout. Text-to-speech readers would likely read the text line by line, and not understand that it is arranged into three columns.

We note that also, it may not be possible to read the BMD summary for non-English language readers. Most hand-marked paper ballots can be provided in alternate languages, usually in a bilingual format so the ballot can also be read by auditors. Election officials are attracted to BMD because they can provide the various languages in the touch-screen session of the BMD, and thereby simplify the logistics of providing alternate languages. But if the text on the printed BMD summary is not voter-verifiable by that same voter (due to this language limitation), then it does not accomplish the goal of providing voter verifiability.^[12]

For example, the following image made it a little bit harder to convert the text, but AuditEngine can actually convert this without any error. The vertical line image defect is likely in the image only and not on the paper printout, but we are not sure. If it was only in the image, then it would not impact voter legibility of the summary sheet.

5 Discussion of Interesting Issues in Bartow County

We will now go over the eight contests that were classified as "red flags" by our system.

48. Inappropriate Adjudication.

This first case is an interesting anomaly because the adjudication was clearly incorrect.

On the next page, the left image is a clip of that section of the ballot image of just this contest. It is clearly voted only for Raphael Warnock.

The Dominion voting system provides the voting system evaluation prior to adjudication in the ballot image file on the third page (the first page is the front, the second page is the back of the ballot, and the third page is the AuditMark, a term trademarked by Dominion.)

The AuditMark provides, in rendered text, the voting system interpretation of the vote. The top part of this page has the pre-adjudication machine evaluation by the voting system.

This initial machine evaluation is shown on the following page in the image on the right. It showed that the vote for Warnock was correctly recognized. The original voting system results were dated 10/28/2020 at 5:56 PM.

We found only one such mistake in the entire election, and this was not consequential in this contest, because both Warnock and Loeffler went on to the runoff election held on January 5, 2021. However, it does point out the effectiveness of AuditEngine in detecting this issue, so that the procedures by election office personnel can be improved.

We also checked every variant ballot for the presidential contest because of the intense interest in this consequential contest, and we found no inappropriate adjudications, and in our general review of the election, we did not find any other similar cases.

49. Undervote that should have been a vote.

In this contest, the voter made a mark slightly outside the oval. Had this been adjudicated, it would have been awarded to Daniel Blackman. No adjudication record exists in the CVR for this ballot. AuditEngine correctly evaluated this vote.

This variant is NOT significant in terms of the outcome of the contest.

Bartow County did a very good job of adjudicating ballots in this election. However, in this case, it was not adjudicated.

The correct assessment of voter intent by AuditEngine in ballots that have been seen as undervoted by the voting system is a much more common occurrence in other audits performed by AuditEngine. The adaptive thresholding technology, where AuditEngine tends to correctly estimate voter intent without human-eye adjudication, is useful in helping adjudicators locate ballots that require adjudication. Undervotes are sometimes missed by AuditEngine, but we missed very few in this election.

The next clip is from the AuditEngine discrepancy report for this ballot and contest, and the following clip is from the Dominion voting system’s AuditMark page of the CVR record for this ballot.

50. Write-in that was marked but treated as undervote.

The write-in bubble was marked by the voter but it was not regarded as a mark by the voting system. If it were correctly detected, it would have been ruled as an undervote anyway, as the write-in name was not qualified. This may have been adjudicated by Bartow County, but the adjudication process does not provide an indication if the ballot is unchanged in the process^[13]. The original evaluation was "undervote," and there was no adjudication record. This case is inconsequential in terms of the outcome of this contest.

51. Hesitation mark that was marked as Marginal by AuditEngine

AuditEngine ruled the undervote as a vote due to a hesitation mark, but it did mark it as "Marginal". There was only one candidate in the "Sole County Commissioner" so this would not have affected the outcome. This is a case where AuditEngine did rule incorrectly, however it would have been further reviewed in a close election.

5.1 General evaluation

Compared with other counties we have processed with AuditEngine, Bartow County had the least number of concerns. We noticed that all the overvotes and write-ins were adjudicated. And generally, undervotes in consequential contests were fully adjudicated as well.

In 252 contests on 78 ballots, AuditEngine correctly evaluated voter intent without requiring additional adjudication, while the voting system incorrectly evaluated these contests prior to human-eye adjudication. All of these cases were then adjudicated by Bartow County election officials and so ultimately, both groups agreed in these cases. When AuditEngine incorrectly interpreted voter intent without adjudication, AuditEngine itself flagged those cases as "gray", which means that the marks were ambiguous or heuristics were used, and thus adjudication would be called for. There were 146 contests on 103 sheets in this category. In those cases, the marks by the voters were very problematic and difficult to ascertain.

For example, the following ballot accounted for a large number of those difficulties, because the voter used a dark X to indicate not voted for and a dark small oval to indicate a vote. Thus, initially all contests would appear to be overvotes. The interpretation of this ballot would be based on “voter intent”. If the election were close and there was no adjudication provided in the CVR, these could be resolved by human-eye evaluation using our Adjudicator App.

Although AuditEngine can do very well in estimating voter intent in normal cases, extreme cases such as this one are difficult to determine even for the human eye.

5.2 Undervotes were adjudicated in presidential contest

We noticed in our review that votes that were originally classified as undervotes were generally also adjudicated by Bartow County election staff to make sure they were tabulated correctly.

For example, in this case, the voting system originally classified this as an undervote but after adjudication, correctly classified this as a vote for Trump.

In this case, AuditEngine correctly interpreted voter intent without adjudication.

The table below shows the results from AuditEngine under "Audit" and then CVR_orig and CVR_modi are the original and modified (adjudicated) CVR records.

6. Overall Summary and Findings

Overall, AuditEngine did not find any indications that the outcome should be cast into doubt. However, we did learn a great deal from this effort which led us to implement significant enhancements to the software to improve our ability to diagnose the issues. Our findings are summarized below:

1. The overall election processing by Bartow County was excellent with very few real issues.

The quality of the data was excellent, with no repeated ballot images, no missing ballots images, and no ballots excluded from CVRs. All variant ballots were explainable except for one.

2. AuditEngine parsed human readable text on BMD ballots and found zero discrepancies.

AuditEngine does not rely on the QR/barcode on BMD ballots, but instead parses the human readable text of the ballot summary which describes the selections of the voter. We do this because the human readable text is voter verifiable while the QR Code is not, even though it is much more expensive in terms of CPU processing requirements. AuditEngine then compares with the cast vote record on a ballot-by-ballot basis. We found zero discrepancies between the cast-vote records and the voter verifiable text summaries. However, we had to significantly improve our parsing capability and we eventually had no failures in fully parsing the BMD summaries except for one with curving lines.

We believe this is an important feature of AuditEngine because it checks on the possibility that the text on the ballot says one thing while the QR Code encodes something else.

3. Bartow uses an unexpected method for marking overvotes as undervotes after adjudication, requiring an adjustment to how AuditEngine interprets adjudication records.

The procedure used in Bartow and some other jurisdictions allows the tracking of which of the overvotes have been reviewed. When adjudicating an overvote, if it is confirmed as an overvote, Bartow then logs it as an undervote in the cast vote record.

If any contest is changed in the adjudication process for a given ballot, an additional adjudication record is added to the cast-vote record for that ballot. The adjudication record contains all contests on the ballot and provides the adjudicated result.

However, if a member of the adjudication staff reviewed a ballot and decided that the interpretation by the voting system was correct, then no adjudication record is created, even though it was actually reviewed and confirmed.

Therefore, it is unfortunately not possible to tell by looking at the cast vote record if a contest was reviewed and judged to be correctly interpreted.

This is the case except for overvotes. We have discussed this with election staff in Bartow County and confirmed that this is the procedure implemented by the election staff and not a procedure that is mandated by the voting system.

Given the methodology used by Bartow, if just the final adjudicated records are reviewed at the end of the election, no ballots have any overvotes. This may be a convenience by the adjudication staff to keep track of whether overvoted contests had in fact been adjudicated, and it does seem like a good idea even though this was unexpected. We had to modify our evaluation of overvotes to take into account the fact that if they were ultimately marked as undervotes, that simply meant that the adjudication staff had ruled that it was a legitimate overvote.

4. AuditEngine found far fewer disagreements in this election due to extensive adjudication by Bartow County.

The counties in Florida included in our case study (in the two counties that provided the ballot-level CVR so AuditEngine could compare ballot-by-ballot with the CVR) provided evidence that voter intent was frequently misinterpreted by the voting system, at the rate of about 0.23% (23 marks out of 1000 marks). In Bartow County, we would normally have detected about 115 disagreements if this rate was in effect here. In this case, we had only 8.

5. One "faulty adjudication" case was found where the adjudication was incorrect

The one disagreement we found quite interesting is the mistaken adjudication of one contest on one ballot. We detected only one case of this type, and therefore, it does not appear to be widespread. However, we must say that we did not review all the adjudications to check that they were all done correctly. If overvotes or write-ins are resolved to a vote by the adjudication process by Bartow workers, then we assume this was done correctly at this stage.

In this case, however, an agreed vote (both AuditEngine and the voting system both evaluated it to be a vote) was changed to an overvote (and thus to an undervote, by the way the adjudication record is marked).

What this points out is the need for possible improvement in double checking adjudication decisions. Also, it points out that human-eye adjudication is not without error, or even possible malfeasance. We doubt that it was true in this case, but there is a remote possibility in similar cases that election workers might insert an  "error" on purpose to test whether it would be possible to actively alter the results without detection.

We did review all the variants in the presidential contest and quite a number throughout the election and we found no other concerns regarding the adjudication in Bartow County.

6. AuditEngine sets the "Gray" flag when heuristics are used for overvotes.

When AuditEngine uses heuristics to make a guess as to the ultimate evaluation of an overvote, then the contest is flagged as "gray" so that they can ultimately be reviewed. The heuristics are designed to detect hesitation marks (very light) and also very dark marks (likely cross-outs) and also are sensitive to whether the ovals are circled vs. darkened. These heuristics have been manually fine-tuned to get excellent results in most cases.

7. Conclusion

This audit is one of a set that shows the value of performing ballot image audits to check on the tabulation of elections from modern voting systems that utilize ballot images. We must caution the reader that finding consistency between the ballot images and the official reported results is not sufficient to fully audit an election, as there are still concerns regarding voter eligibility, chain of custody, whether the ballot images are a faithful representation of the ballots, and other factors.

In this election, as all ballots were also audited by human-eye evaluation in a full hand count of the Presidential contest, it is unlikely that the ballot images are not a faithful representation.

However, if a ballot image audit does find any issues, these can be easily checked as legitimate issues in the portion of the process in which AuditEngine is checking for consistency. Finding no issues proves consistency between the two data sets: the ballot images and the cast vote records.

We appreciate the cooperation of Bartow County and spotless data exports which we had no trouble processing.

We hope that election officials and the public see the value of such a review of ballot images to increase voter confidence in election results.

For further information, please visit https://auditengine.org. We appreciate funding by the public for these independent audits.

Primary Author: Raymond Lutz

Raymond Lutz is the founder and executive director of Citizens' Oversight Projects, a 501(c)3 nonpartisan nonprofit organization that has been involved in providing oversight to elections for over 15 years. Lutz has a Masters degree in electronics and software engineering, with experience in the document management and printer/scanner/fax/copier industry, and medical device industry. He is the lead developer of AuditEngine.

How to Comment

Please send questions and comments about this report to support@citizensoversight.org

APPENDIX 1 -- Links to detailed reports

The following page provides links to the automated reports.

https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Final_Report.html 

The Discrepancy Report:
https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Discrepancy_Report.html

This Narrative Report:
https://docs.google.com/document/d/1oNyx0v0pNVBtkhF41IB0RWjR9n1sNlfmTYCT93tHXCA/edit?usp=sharing 

Appendix 2 -- Overvotes marked as Undervotes Issue

When overvotes are adjudicated in the Dominion system in this county, the adjudication record records the contest on that ballot as an undervote (blank contest), instead of leaving it as an overvote. The net result is the same, because both overvotes and undervotes (blank contest) contribute no votes for that contest in the tabulation. But marking the contest as an undervote (blank contest) can be considered incorrect in terms of any statistics that might be gathered from the final records.

Our guess is that this is a way to keep track that all overvoted contests were adjudicated, because it is not possible to note in the Cast Vote Record (CVR) that the contest was adjudicated (reviewed) and no changes were made.

Our question to Bartow County is whether this is a matter of procedure or if it is a requirement of the adjudication system being used.

By way of example, we will consider a contest on a ballot categorized initially as an 'ov_ov_uv' type, that is AuditEngine interpreted the contest as an overvote, the voting system initially also interpreted it as an overvote, and then after adjudication, it was classified as an undervote.

If we search for that type in the Discrepancy Report, we are provided with this example:

ballot_id:        00530_00023_000033
contest:        US Senate (Loeffler) - Special

The image attached to the right is that specific contest. It is clearly marked as an overvote. In this case, the voter voted for every Republican candidate, voting for six candidates instead of just one.

When we look at the Dominion system "AuditMark", the third page of the ballot image, the voting system correctly detected the marks shown, and classified it as an "OVER-VOTE" and each of the ballot options (candidates) selected are shown as "(NOT COUNTED)"

AuditEngine also detected these same marks.

When we look at the cast vote record as modified by adjudication, it is reported in Dominion’s AuditMark as shown in the snippet below:

Is it necessary to remove the marks in the adjudication?

When we look at the cast-vote record, it states that it is now an undervote, and a blank contest, and not an overvote. The cast-vote record has two parts, one called "original" and the other called "modified," which is after the adjudication.

      "ImageMask": "C:\\NAS\\Bartow Nov 2020 General\\Results\\Tabulator00530\\Batch023\\Images\\00530_00023_000033*.*",

      "SessionType": "ScannedVote",
     "VotingSessionIdentifier": "",
     "UniqueVotingIdentifier": "",
     "Original": {                     <-- THIS IS THE ORIGINAL SNAPSHOT
       "PrecinctPortionId": 12,
       "BallotTypeId": 6,
       "IsCurrent": false,
       "Cards": [
         { "Id": 27,
           "PaperIndex": 0,
           "Contests": [
             ...
             { "Id": 3,                        <--- only showing this contest here.
               "ManifestationId": 257,
               "Undervotes": 0,                <--- Not undervoted.
               "Overvotes": 1,                 <--- Overvoted.  This is correct.
               "OutstackConditionIds": [5],    <--- 5 = "Overvote"
               "Marks": [
                 { "CandidateId": 9,   "ManifestationId": 1193,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 97,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5] <-- 5 means "Overvote"
                 },
                 { "CandidateId": 11,  "ManifestationId": 1195,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 99,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5]
                 },
                 { "CandidateId": 13,  "ManifestationId": 1197,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 97,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5]
                 },
                 { "CandidateId": 16,  "ManifestationId": 1200,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 91,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5]
                 },
                 { "CandidateId": 19,  "ManifestationId": 1203,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 96,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5]

                  },
                 {"CandidateId": 24,  "ManifestationId": 1208,
                   "PartyId": 0,       "Rank": 1,
                   "MarkDensity": 98,  "IsAmbiguous": false,
                   "IsVote": false,    "OutstackConditionIds": [5]
                 }
               ]
             },
             ...

      "Modified": {                     <--- THIS IS THE MODIFIED (ADJUDICATED) SNAPSHOT
        "PrecinctPortionId": 12,
       "BallotTypeId": 6,
       "IsCurrent": true,
       "Cards": [
             ...
             { "Id": 3,                <--- (Only showing this contest here)
               "ManifestationId": 257,
               "Undervotes": 1,           <--- changed to undervotes
               "Overvotes": 0,         <--- changed to no overvotes.
               "OutstackConditionIds": [4, 6],        <---  4="Undervote", 6="Blank Contest"
               "Marks": []             <--- No marks at all
              },
           ...
           ],

We confirmed that Bartow County used this method as a matter of procedure, and it is not a requirement of the Dominion voting system.

Appendix 3: Other Audit Details

Ballot Image Archive (BIA) Report

1. The BIA BIF report is the metadata and other statistics that can be extracted by just reviewing the ballot image archives. This report for Fulton County can be found at this URL: https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/biabif_report.html 
   
2. The number of zip archives: 1.
   
3. The following is the file size distribution chart:

1. There were two groups of ballot images.
2. Sheet1 group is the hand-marked paper ballots with average size of 229KB
3. The BMD ballots had an average size of 85KB.
   

4. There were 50,678 raw image files and 50,678 unique ballot images, with 0 repeated images.
   
5. We must note that we did limited searching for rescans but did not find any. We must be aware that if batches of ballots are scanned twice and included in the totals, this may only be in the CVR and ballot images and it is not in the tabulation results of the voting system. We encountered this issue in the 2020 General Election in Volusia County, FL^[14], where the CVR and ballot images differed from the aggregated totals. That defect in the ES&S equipment we have not seen yet in the Dominion systems.
   
6. The number of precincts, parties, and groups is NOT something that can be discerned from the directory structure in the case of Dominion, so these are enumerated as just 1 each.
   
7. There were 313 batches.
   
8. There were 39,995 BMD ballots detected by inspecting the file size.
   
9. All ballots had only one sheet, and there were no unusual extremes.

EIF -- Election Information File

10. The "Parsed EIF Report" is at this link: https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/EIF_Parse_Report.html 
    
11. The Election Information File (EIF) describes all the contests and options, including the official names and names used on BMD ballots. The EIF was generated by first parsing the Cast Vote Record (CVR) files, and then editing manually. Dominion provides excellent metadata in their CVR but does not provide the exact strings used in BMD ballots, and as those are read using OCR (Optical Character Recognition), we need the exact strings of the contest names and options (candidate names or Yes/No), to ensure accurate BMD summary card reading.

Combined Metadata Report (BIF report)

12. The metadata from the BLT BIF is combined with similar metadata from the BIA and CVR metadata to create the Combined Metadata report.
    
13. The combined metadata report (BIF Report) can be found at this URL: https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/bif_report.html 
    
14. The official number of ballots cast was 50,678. There were 50,678 unique ballot images, with 0 marked as "skipped repeats" by AuditEngine.
    
15. There were 50,678 records in the CVR files.
    
16. There were 0 ballots that were missing from the ballot image archives.
    
17. There were 40 precincts identified in the CVR, and 41 different ballot styles defined in the CVR.
    

Styles Report

18. The Styles Report is at this URL: https://us-east-1-audit-engine-jobs.s3.amazonaws.com/US/GA/US_GA_Bartow_20201103/reports/Final_Report.html