Tag Archives: Pilot Election Review Committee

Pilots of voting systems not yet an adequate test of verification

The following is a letter to Colorado SOS staffer Dwight Shellman and members of Colorado’s Pilot Election Review Committee:

I'm including a message I sent to Dwight long ago.   Please include this in published public comment to the PERC committee on the SOS website. ( http://www.sos.state.co.us/pubs/elections/VotingSystems/reviewCommittee.html ).

Please send me the plans/schedule for the mock RLA and the test data dumps you got from the vendors.  And please send the spreadsheet that contains the schedule of election activities. I am shocked that it was not published on the SOS website, as far as I can tell. I have certificates of appointment for a statewide ballot issue committee to watch in any county.

While watching last week I also discovered by chance that the SOS issued apparently secret provisional watcher rules to some counties and asked them to pilot these. Fortunately the four pages of rules did not catch me by great surprise as they were the product of a committee I served on.  Why wouldn't the public be made aware of that watcher pilot?  Perhaps not your department.

As you know I am  attempting to evaluate the tabulation accuracy of the four systems as well as their facility to support transparency in the service of election verification- not only in the future via a RLA (Risk Limiting Audit) but the facilitation of deeper dives into verification of election results for accuracy. At least one such deeper dive is definitely appropriate to do prior to this UVS (Uniform Voting System-s) decision.  I am attempting to do it and am facing ridiculous costs on the order of $20,000 - a sum accumulated from estimates of some of the counties for access to ballot scans. This has forced me to constrain my requests with the hope of getting useful access to evidence from these elections in a much smaller scope.  My efforts will be something closer to a pilot of the functional part of an "evidence based election" - something that election quality advocates like myself and many election officials seek.

I have no doubt that the counties have been corresponding about how to answer my requests. the CORA requests are rendered challenging by the limitations in CORA law and the opportunities of new technology at hand. Perhaps Adams and Denver have been responding without much coordination. Most other counties seem to think it will take 5 seconds to find identifying marks on one side of a ballot scan shown on a computer. I think that would be excruciatingly slow and disastrously expensive.

If the SOS office has exposure to these conversations by email I would appreciate seeing the content of the discussions.  I know you were cc:d one one county response to my request.  It is important for the process of public access to election records to be a part of the legislative agenda and rules agenda for the future. Huge opportunities can be missed by failing to address the benefits of some or all of these new voting systems. One such benefit would be for campaigns to locate potentially mis-adjudicated ballots and to systematically include any such discovery in the process of auditing the election so that higher accuracy could be obtained prior to certification - particularly with a narrow margin contest.

For my part I have posted (at  considerable expense of my time) all of the  threads of my conversations with the counties regarding the CORA requests for ballot designs as well as scans of tabulated ballots, cast vote records, and new scans of duplicated ballots. These conversations represent a large range of facilitation of public access to verify elections. No doubt we will not achieve a "uniform election system" by enforcing the use of one or two or whatever prescribed voting systems. The UVS is a well distributed, well worn myth and the counties seem to know it.  I hope the SOS does. I hope he becomes well aware of this transparency project and the benefits that it might produce.

The record of these CORA conversations under my commentary is located here: http://electionquality.com/2015/11/colorado-pilots-of-new-voting-systems-allow-technical-evaluation-through-access-to-ballots/

Please be aware that the running of a normal low volume election with new voting systems does not necessarily produce the evidence of how these systems will perform in exceptional conditions or in case of narrow margins.  We must simulate these conditions or dig deeper into the evidence to see how they are performing in detail.  One such exceptional condition is the inevitable presence of people like me who will attempt to verify the accuracy of an election.  I have not noticed much simulation of that condition in the counties where I have been watching. In most cases I am the only  credentialed watcher who has appeared.

And of course thanks for making this a public process- something that some other states do not feel necessary. Simply the opportunity  to be involved is a very meaningful enhancement that I have every reason to appreciate and compliment. And thanks to the tireless efforts of many who are engaged in trying to get Colorado to a better election.

Harvie Branscomb

-------- Forwarded Message --------
 Mon, 21 Sep 2015 14:42:40 -0600
 Here I reprise my email of July 29 that refers again to the SOS preparation for RLA.

I understand that the past two PERC meetings were cancelled and that includes the meeting that would have included Philip Stark.  On 7/29 I asked again if I should ask you once more for the communications regarding RLA with the pilot counties. I do  not even yet know which 4 counties will conduct the RLA.

I am heavily involved with the Election Verification Network in a discussion of RLA and have been for years.  Philip Stark and Ron Rivest and many election officials and academics are part of that conversation.

It is clear that Colorado's actions in November will be of great importance to the audit community nationwide and I plan to learn as much as I can about what Colorado is doing as well as to learn as much from the pilots.

I plan to CORA the PDFs of the ballot styles from the 8 counties involved in the audit so that by the time I get ballot images I may be in a position to independently tabulate them
-------- Forwarded Message --------
are we ready for risk limiting audits?
Wed, 29 Jul 2015 16:18:38 -0600
Harvie Branscomb
Dwight Shellman

Here is an excerpt of something I am sharing with election verification experts:

Fortunately as Phil Stark pointed out ballot level risk limiting audits do not require aggregating multiple ballots and only actually require capture and comparison of patterns off of single ballots at a time. However this is also not trivial and even newest systems are not necessarily providing data that has good human factors for this purpose.

It appears that machines are better than humans at counting up the number of pieces of paper in front of them. It seems unavoidable that machines fail to capture idiosyncratic voter intent when humans succeed. Therefore an ideal combination is found in the sort and stack method that can be combined with machine counting of pages. For example during a single contest recount it would be ideal to have human judges separate the ballots by voter choice and batch tabulate them such that all ballots in a batch (or on a particular scanner or memory card) are hand sorted to be a single voter choice. The scanner can then provide a machine count of sorted pages and a sub-tally of vote counts to confirm that it recognized all as voted for the specific choice. This will easily expose discrepancies that can be resolved by human observation and improve recount accuracy.

I am currently analyzing the first 8,000 of 130,000 ballot images and cast vote records from the Dominion system used in Denver CO (the much advertised and visited May municipal pilot election). The principal reason for my analysis is to learn what we need to know to guide the evaluation of such a pilot system from the point of view of auditing (and transparency)… auditing the audit as it were.

The format for the Cast Vote Records that were provided 1) as a separate text file aggregated by batch as well as 2) the “AuditMark” voter selection digital printout that is embedded with the 200dpi 1bit per pixel ballot image are not suitable for convenient machine tabulation and they aren’t particularly desirable for hand comparison either. Human usability concerns (as well as machine formatting compatibility concerns) abound in the design of forms and scanned images and reports that are needed for auditing. No doubt this issue is not limited to Dominion.

Our CO pilot review program seems insufficiently concerned about the ways that the risk limiting audit will take place. Perhaps more attention will be paid within the next few weeks. We have until November when five such pilots will take place as part of the 2015 election. At present it is uncertain what the instructions for a risk limiting audit will be and it is even more uncertain how we will know if we have done a good pilot audit.

Dwight I am deeply concerned that our pilot evaluations will fail to measure and document the crucial back office aspects of the systems even as they are in use in real elections. The evaluations of these systems in real elections crucially must look beyond the motions of successfully performing an unremarkable election.  The RLA is a big part of my concern and that is why I asked you to forward the communication to the vendors about preparations for the pilots. Should I ask for that again?

I am hoping that I will get enough information from Denver's completed pilot in time to be able to make a sample partial evaluation of that system as an example for what the PERC or SOS could and should do as a minimum to evaluate each system. I am hoping I can do that before the PERC decisions on how and what to evaluate are made.  No doubt what I am doing by myself is not enough either.  I am already learning that the RLA may be painful with some systems. And the counties performing the pilots may have the tendency to act to defend the system rather than subject it to a critical evaluation in public. That should be a major concern of PERC and SOS and all Coloradans.

I don't think my bi-weekly writings to the PERC have been productive but I do think that you can benefit from the information I am collecting.  By random chance after examining only a few ballot images in Denver I already found an incorrectly captured voter mark in the Dominion May pilot- but the CO evaluation must be in a position to find all of them for each piloted system. And system evaluators must be in a position to understand and evaluate the remedies offered by  the vendors.   A technique such as I have briefly described above for sorting before scanning if used for a top of ticket contest would get part of the way there and without spending much extra time.

Likewise the documentary products of each system (logs, settings, etc.) must be studied in detail and each document's ability to convey exception cases and discrepancies evaluated.  Unfortunately I have not yet received any logs or reports from the Denver system, and the ballot images are coming slowly due to an understandable learning curve on privacy redaction. The timing of the PERC and any other SOS evaluation must allow for weeks of evaluation of reports and cast vote records and audits after the documents are produced. It will not be enough to say no one complained.

And on that matter, it is also important that well trained watchers/observers be present at the 10 pilots at important moments throughout -  of course including the ballot design, LAT and ballot prep and early tabulation and post election activities. And plans must be made for a super-LAT at each pilot that plans for potentially problematic ballots to be voted and results analyzed.  Remember what the clerks did to ballots for the UVS demo last year- crumpled them, etc.  Whatever the out of state certification is doing and whatever SOS is doing in house is probably insufficient to provide confidence about exceptional test cases and the ability of systems to detect, log and remedy these cases.

Finally you keep mentioning the hardware test and acceptance tests.  These are important and have been ignored in the past.  I think work should be done on what the hardware tests (every election) and acceptance tests (by county upon purchase) should be- and these tests should be performed with careful evaluation during the pilot or within its scope. Seening the lights come on and the system say "ready" doesn't provide enough information.

Please send this to the PERC if you feel it is constructive to do so.

PS I just received the following link from Ron Rivest- another very well respected statistician working on elections - with a simple one page description of a RLA. Unfortunately it is still only the mathematicians description and not one for election officials. Someone should turn this (or similar mathematical instructions) into a clear and specific multi-county process (all 5 RLA pilots coordinated plus any other counties separately doing RLA?) for the state-wide issue on the ballot and within county audits for the remaining contests.

http://people.csail.mit.edu/rivest/pubs.html#Riv15y  In this list of publications, the RLA description is number 311 or the second from the top at the moment.

Harvie Branscomb

Colorado Pilots of new voting systems allow technical evaluation through access to ballots

Colorado is tabulating the final numbers in the 2015 Coordinated Election.  Results on the Secretary of State (SOS) site are found here: Colorado 2015 results by county.

Eight counties are currently running their election with new hardware and voting system software from 4 companies.  On Dec. 15 the SOS Pilot Election Review Committee (PERC) will make a recommendation to the SOS about what system or systems should be made available for purchase by Colorado counties. Reportedly 24 counties are interested in purchasing replacement vote tabulation equipment (and software) in 2016 prior to primary elections. PERC is a public body and its meetings are recorded and testimony to it is posted here: CO SOS PERC website.

Reportedly on Dec. 31 2015 the Colorado Secretary of State Wayne Williams will decide what system or systems will be certified for purchase in Colorado.  It is my intention to conduct an in depth evaluation of the functionality of these four systems  using scans of individual ballots and the associated interpretations of those votes.

Each system creates an anonymous digital record of the marks on each ballot after a each is opened, scanned and then processed to detect the marks. Then exceptional cases are redirected to human adjudication as part of the paper ballot tabulation process The result is presumably an accurate representation of the anonymous voter intent. By checking potentially problematic looking marks on paper or on scans of paper in  comparison to the output recorded by the machine for that ballot I can locate ballots that might have been misinterpreted. And I can sum all these interpretations (called cast vote records) to check to see the published results match.

What I am describing is an evaluation of the tabulation portion of a paper ballot election. For practical reasons I will use scans of ballots rather than the actual paper as input.  Granted this is not the most perfect approach as the scans could fail to represent something marked on the ballot. Election officials already check some of the ballots to see if a hand count of these will balance against a machine count.  This is called the “post election audit.”  In 2017 Colorado will enjoy perhaps the first statewide Risk Limiting Audit  that will  caused judges to methodically hand check random paper ballots against the “cast vote records” and then sum all “cast vote records” to see if the election outcome would change if a hand count of all ballots were conducted.This is approach is part of an “evidence based election.”

A way to go beyond the maximally efficient risk limiting audit is to check more than a minimum number of ballots to see if they match the cast vote records – the records the ballot scanners produce for each ballot.  We could even retabulate the entire election from published scans if they are of high enough quality and can be trusted.  I have issued Open Records (CORA) requests to the eight counties to get access to these cast vote records and the scans of the voted ballots (after they are deemed to have had any identifying marks removed.) It is simply impractical to get physical access to count that many paper ballots, although I may attempt to check a random set of the scans against the paper if and when I have time. That would require a visit to each county.

What you will find on following pages is the record of the process of my attempt to get access to these important public records for the purpose of a one time massive evaluation of four new voting systems based on collected evidence at election scale. This is an unusual opportunity that is made possible by  far better technology in these new systems and Colorado’s transparency laws in the Colorado Open Records Act. What you may find is that there is  substantial lack of uniformity in the way the counties treat my requests and their custodianship of the public records.

Another thing that is really quite disturbing is that  our election officials are now clearly acknowledging the presence of identifying marks on the ballots although in reality there are probably very few. In Colorado the constitution requires – in Article VII Section 8- that “all elections by the people shall be by ballot, and in case paper ballots are required to be used, no ballots shall be marked in any way whereby the ballot can be identified as the ballot of the person casting it.”

According to our state constitution, it is illegal to make such a mark- whether the voter or an official does it. The state law ironically now authorizes counties to charge me as a member of the public to pay substantial and often insurmountable sums  to in effect hide   from myself the evidence of a constitutional violation. It would make so much more sense if the county  governments would prevent the identifying marks from being made or redact them at the moment of recognition. Then officials could reveal to the public through fulfilled records requests that the unconstitutional cases where identifying marks are present  have been remedied. One way many states handle this is to reject ballots that have been made identifiable. Thus they do not become part of a fulfilled request for all tabulated ballots.

What I hope to see is more  attention paid to transparency  so that Colorado can enjoy with the help of the new generation of voting systems the most verifiable and most verified elections in the United States.  The new technology is capable of it- and all that is needed now is an actively interested public and  election records custodians who are appreciative of the need for efficient and low cost public access. You will see in the records linked below that some of our officials are taking that approach.

Harvie Branscomb 11/9/2015

Links to county open records requests (CORA) conversations:

Adams quoted price near zero for about 73,103 ballot scans  = $0/ballot

read the Adams: 2015 CORA Conversation

Denver has not quoted a price yet for 80,019  ballot scans
read the Denver: 2015 CORA Conversation

Douglas quoted $4000 for 88,248 ballot scans  = 4.5 cents/ballot
read the Douglas: 2015 CORA Conversation

Garfield quoted  $990 for 11,172 ballot scans  = 8.8 cents/ballot

read the Garfield: 2015 CORA Conversation

Gilpin quoted only the cost of media for 1,519 ballot scans475273
read the Gilpin: 2015 CORA Conversation

Jefferson quoted $12500  for 185,467 ballot scans  = 6.7 cents/ballot
read the Jefferson: 2015 80019CORA Conversation73103+80019+

Mesa quoted $1500 for  29,154 ballot scans =  5.1 cents / ballot
read the Mesa: 2015 CORA Conversation

Teller quoted $550 for 6,591 ballot scans = 8.3 cents / ballot
read the Teller: 2015 CORA Conversation

Total number of ballot scans and cast vote records requested 475273 (plus some over under and late counted votes )… total  estimated number of ballots 395254 cost: $19540

The average cost for accessing an electronic copy of a voted anonymous ballot in CO is currently about 5 cents per ballot which means the cost to access ballots for a presidential election is about $100,000. Can you afford it?  I can’t.