Following up on the latest monumental failure by Diebold/Premier — whose voting system was recently found by a citizens’ transparency project to have lost hundreds of ballots in a single precinct in Humboldt County, CA — Wired’s Kim Zetter covered the issue earlier this week.
Her article touched on a number of technical issues involved in the failure of the Diebold GEMS central tabulator which, though we hadn’t noted them in our own coverage, are nonetheless important, even if somewhat down in the weeds, and perhaps most of note to technical election integrity geeks only.
As quoted below in some detail from Zetter’s report, she advances the story by noting:
- The problem, which Diebold admits has been in its system for years, likely wouldn’t have been noticed at all were it not for the unique, experimental, citizen-led “Humboldt Transparency Project.” Even CA’s required 1% random post-election “audit” wouldn’t have uncovered the problem, since absentee ballots are not included in that audit.
- The chances of the problem occurring, which resulted in ballots being deleted without notice by the Diebold GEMS tabulator, are rather high, even if chances of discovery of the error by election officials are very low.
- Diebold’s explanation for the failure may or may not be legitimate.
- The GEMS internal “audit logs” (the GEMS system was used in 34 states around the nation last November) aren’t actually audit logs at all, and may not actually note the complete set of events which occurred on the voting system, even if anyone is allowed to review those logs in the first place.
Let’s take a look at the details…
Thanks to the unique oversight program allowed by Humboldt County’s election chief Carolyn Crnich, the deck of some 200 absentee ballots scanned, but then deleted from the totals without notice by the Diebold/Premier GEMS tabulator, was discovered and the previously-certified results had to be corrected and re-certified.
The special project used open-source software, written locally, and off-the-shelf scanners to scan the same paper ballots that were scanned and tabulated by the Diebold op-scan system. However, had it not been for that parallel counting project, using alternate hardware and software, the previously-certified results — which did not include the ballots deleted by Diebold — would have stood, and the lost ballots would likely never have been found, as Zetter notes:
The audit requires every county to hand-count ballots in 1 percent of randomly chosen precincts in order to compare the totals against digital tallies. But the audit involves only ballots cast physically at a precinct, not mail-in ballots, which are the ballots that the Premier/Diebold system dropped in Humboldt. Even if the Humboldt ballots had been precinct-cast ballots, Crnich would not have known they’d been dropped from the system if they weren’t cast in a precinct that was included in the 1 percent of precincts that were hand counted.
But while the chances of the problem that occurred in Humboldt are actually very high, the chances of discovering the failure are low, unless the ballots are rescanned or recounted by an alternate system that actually works, like the one created by citizens, for free, on open-source software in Humboldt…
The system never provides any indication to election officials when it’s deleting a batch of ballots in this manner.
That last graf, noting that no indication of the error is given to officials, is both troubling and reminiscent of Diebold’s other recent admission that its GEMS central tabulator — on both the paper ballot systems and touch-screen systems — routinely drops thousands of ballots when they are uploaded to the central GEMS server, but without giving any indication of that failure to the system administrator, who is left believing that all votes were properly recorded.
Ohio’s Secretary of State Jennifer Brunner is currently suing Diebold/Premier over that particular failure, as she and I discussed in an exclusive interview late in the Summer, though no other state — including California — is currently suing Diebold/Premier to our knowledge. That, even though the same GEMS system is currently used in some 34 states around the nation.
With all of this, however, it’s not even certain that the reason Diebold offers for the deletion of those ballots was actually the reason for the deletion, as noted by Zetter:
And, as if all of that is not bad enough, as they say on Late-Night TV: But wait! There’s more! And, ultimately, it could be the worse flaw of all!
As it turns out, the GEMS system’s internal audit logs, used (when election officials and/or courts will allow it) to look back and determine what operations were actually carried out by the system and when, doesn’t appear to be a real “audit log” at all!
Rather, as Parke Bostrom, one of the Humboldt citizens who worked on the project described it, the audit log is actually a “‘re-imagining’ of what GEMS would like the audit log to be, based on whatever information GEMS happens to remember at the end of the vote counting process.”
The system creates a “deck 0” for each ballot type that is scanned. This means, the system should have three “deck 0” entries in the log — one for vote-by-mail ballots, one for provisional ballots, and one for regular ballots cast at the precinct. Crnich found that the log did show a “deck 0” for provisional ballots and precinct-cast ballots but none for vote-by-mail ballots, even though the machine had printed a receipt at the time that an election worker had scanned the ballots into the machine. In fact, the regular audit log provides no record of any files that were deleted, including deck 132, which was deleted when Crnich intentionally aborted it. She said she had to go back to a backup of the log, created before the election, to find any indication that “deck 0” had ever been created.
Parke Bostrom, one of the Transparency Project volunteers, wrote in a blog post about the issue, “This means the audit log is not truly a ‘log’ in the classical computer program sense, but is rather a ‘re-imagining’ of what GEMS would like the audit log to be, based on whatever information GEMS happens to remember at the end of the vote counting process.”
Zetter didn’t offer the link to Bostrom’s blog post, but it’s right here. He notes, in addition to the points mentioned by Zetter above, what appears to be a contradiction to her reported information on the other two “deck 0” batches — for provisional ballots and for mail-ballot-precincts (those precincts that are too small and remote to have actual physical precincts, so they are all done as vote-by-mail).
While Zetter notes that the other two “deck 0” batches did show up as having been scanned in the “audit log,” in fact Bostrom notes that they “do not appear in the audit log, but apparently their votes are included in the final report.”
Not sure whether it’s Bostrom or Zetter who has that point correct, and I’ve been on the road and out-of-pocket for weeks, so don’t have the time for the moment to find out. But, suffice it to say, if we’re talking about “audit logs” that are made at the end of the process, instead of concurrently, as events occur, then they are worthless for purposes of auditing, no matter what.
Bostrom adds one other point of note in his blog item which illustrates just how difficult it is to audit the job that Diebold’s GEMS system — again, it’s used in 34 states! — has done in scanning ballots, even if (and that’s a big “if”!) officials and/or courts allow regular citizens to examine the system at all:
As we noted in our Monday coverageof this story, this entire sorry affair reveals yet again the enormous scam behind the entire federal e-voting legislation contained in the Help America Vote Act (HAVA) of 2002, allocating some $3.9 billion for states to hand over to private voting machine corporations like Diebold for their proprietary trade secret-protected voting systems which don’t work.
In the meantime, on the other hand, a handful of citizens in Humboldt County, using fully open source code, developed at no cost to the county, with off-the-shelf commercial scanners, have created a system in a number of months that did a better job of counting ballots than Diebold, even with all of their millions/billions of tax-payer dollars, hundreds of programmers, secret hardware and software, and enormously expensive long-term maintenance contracts.
HAVA also created the U.S. Election Assistance Commission (EAC) which was supposed to have served as a “clearinghouse” for voting system problems such as the one that hit Humboldt. The flaw in Diebold’s system has been known since 2004, but the EAC failed to notify anybody about it, and has failed to set up any useful clearinghouse system at all, in the 6 years since HAVA was signed into law.
Given all of those failures, every jurisdiction in the country which insists on using optical-scanners should immediately look at moving over to the software and hardware developed by the citizens in Humboldt. Diebold/Premier should be dumped, sued for fraud, malfeasance, and gross negligence and be put out of business — at least the elections business — once and for all.
As to what should happen to the EAC, we’ll leave that for now, for another day…
Brad,
If you need follow up, since I live in the heart of Humboldt Co. (Eureka) and Mitch Trachenberg, who did the actual programming for the Transparency Project, given as you say, you’re on the road and living out of pocket, drop me a line if you need more clarification on who’s really right, Bostrum or Zetter and I’ll try and get an objective assessment.
Not sure whether it’s Bostrom or Zetter who has that point correct, and I’ve been on the road and out-of-pocket for weeks, so don’t have the time for the moment to find out. But, suffice to say, if we’re talking about “audit logs” that are made at the end of the process, instead of concurrently, as events occur, then they are worthless for purposes of auditing, no matter what.
This year I helped our local election officials with the absentee ballots. Like regular ballots each ballot (absentee or on day of election) was given a number on a small slip of paper. When a voter votes they submit their slip of paper to the attendant at the voting machine before they run their ballot through the op-scan machine. With absentee ballots we were assigned in pairs of election afficials to assign the ballots numbers, give the slips of paper to the op-scan machine attendant and then run the ballots through the machine. At the end of the day we checked that all the slips of paper had been accounted for (we found some in the trash from when the lines were longer). We needed to have the same amount of slips turned in as ballots cast, or a record of why they were not matching. I believe all our slips/numbers were accounted for. We also ran our absentee ballots during the time that all the rest of the ballots were being cast (except the last 100, we had almost 2000 absentees in our district). This reduces the chance that absentee ballots could be treated differently by a software issue (software written that after a certain time of day votes tallied would be put into a “deck 0”).
This system seems to reduce the chance that there might be a different count of ballots versus votes, and reduces the chance of a “deck 0”. This system does not account for all malfeasence in the tabulating software, but it helps keep more people honest.
Wow, excellent work in Humboldt County and an excellent post detailing the situation with Premier/Diebold.
I have a question. In an election at any given precinct, is there a count made of the number of ballots received (independent of the machine count)?
For example, in Precinct A, 837 people came to vote, and 3 ballots were voided and replaced, and 1 person left without voting, and 85 absentee ballots were received, so there should be 836 ballots plus 3 voided ballots plus 85 absentee ballots, or 924 total ballots, of which 921 were scanned for counting.
Then, say you looked at the total from the Premier/Diebold system and checked that 921 ballots were scanned and, for a particular race, say there were 567 votes for the Democrat, 352 votes for the Republican, and 2 undervotes, totaling 921.
Wouldn’t you at least be able to tell if a “deck” of votes had been dropped? (Granted you would not be able to tell if any vote was counted as intended.)
Lora asked:
While all jurisdictions do things a bit differently, in general, there should be end of the night reconciliation between the number of voters as recorded in the poll books, and the number of ballots cast on the machines (at least in the case of precincts with precinct-based op-scan systems).
Often, however, I’ve found that when the two numbers don’t match up, poll workers just figured they did something wrong, or someone forgot to sign the pollbooks or some such. So sometimes it’s reconciled, sometimes it’s not. Usually seems to depend on if there are any close races out of that particular jurisdiction.
The “decks” we’re talking about in this story, however, were absentee ballots scanned at central headquarters prior to Election Day, as opposed to ballots cast at the precinct.
So what machine were they using to scan AB’s, was it the PCS high speed scanners?
The best resource regarding the audit logs will be the California Secretary of State’s office, which is now conducting an investigation. I’m hoping they’ll issue results soon. I think and hope that at some point they’ll come to Humboldt and go through the actual equipment used.
One other point: the very REASON that our system can give better results (bugs, potential fraud aside) is that it is open. This leads to two results, both critical:
First, errors in our system WILL be caught, because anyone in the world who is interested in it can see it and correct it. A worldwide community of interested programmers and citizens can do a better job finding issues than ANY set of paid programmers and consultants. It’s not even close.
And second, our system opens results at every level. For example, the ballot images can be projected before an audience as they are counted by my program or another, and any misses can be flagged. (That is, misses that are not hidden by secret code.) And, because Ballot Browser doesn’t just report the votes, but also reports the average intensities of the vote ovals, it becomes easy to distinguish between real overvotes (two dark ovals) and false overvotes (one dark oval, one darkish oval). That is, it says yes or no to a vote, but it also shows you WHY it said what it said.
Over the last twenty years, much open source software has caught on, especially outside of the United States. Because open source doesn’t have a budget to compare with that of, say, Microsoft, many non-computer people don’t realize that it is simply better. But it is, folks, it is.
Brad, thanks for the clarification.
Mitch, this is impressive and fascinating. I know very little about computer stuff, so I don’t follow everything, but thanks to your and Brad’s detailed explanations I get quite a bit out of it.
It’s wonderfully great to get confirmation of what we’ve been saying all these years — that citizen-based, fully transparent, and fully auditable paper-based, human-counted/monitored systems are hands-down superior by far, than billion-dollar, closed, private, error-prone, easily hackable, corporate-run elections systems.
Now, if the national media could be bothered to interest themselves (or be allowed to interest themselves) the word could really get out!
Sorry for the multiple comments on multiple posts. Does the system delete all the ballots for deck 0 or just the votes? Do the precincts balance but there’s missing votes?
[ed note: Don’t be sorry. Don’t do it. –99]