‘We are being eaten alive’: Behind the scenes as Johnson County scrambles to avoid yet another election disaster
Johnson County has tried to put a brave face on efforts to fix reporting software that failed in the August primary. County commissioners have reaffirmed their confidence in Ronnie Metsker, who runs the elections. Metsker, in turn, praises Election Systems and Software, the company that sold him on the brand new voting machines.
After all, they “owned” the problem. They announced they found, and are fixing, the offending code. And the accuracy was never in doubt. Nothing to see here, folks.
But a check of the county’s emails obtained through an open records request shows a reality quite different from the confidence Metsker and some elected officials projected to the public. In the days following the county’s reassurances that they were on top of things, they were in fact scrambling to make equipment work and get the vote count verified.
That time was filled with frayed nerves and sleepless nights as election officials also struggled to prepare for a possible recount — not to mention public relations damage control.
And the future is anything but certain, as ES&S pushes hard to get its code rewrites certified by the federal Election Assistance Commission in time for the November general election. The company hopes to speed things up, but the government does its own testing, which could take a month or more.
• • •
Election night was supposed to have been a triumph for Johnson County’s new $10.6 million voting machine system. The equipment was a top-of-the-line newest configuration that would finally give that paper trail skeptical voters have been wanting. Even better, it would have a reporting system that aggregates results and displays them on the screens of election employees, so they can know vote totals and precinct counts that are constantly being updated.
All that computing power was supposed to keep the county from another embarrassing reporting delay like it had in 2016, when it took all night to get totals to the public. So officials were stunned to find themselves waking the morning of August 8 to the same questions about why it took all night to get results out.
The company quickly stepped in to take the blame. The software system was, for some reason, slow to upload the totals. It had nothing to do with the accuracy of the county. ES&S was working on it.
Meantime, Metsker and county commissioners reaffirmed their support of each other and the company. Metsker called ES&S a “fine company” that has “demonstrated resolve they will find the problem and fix the problem and given us surety it will never occur again. This will be a one-time event.”
“I’ve tried to exhibit what I would call ‘common grace,’ where we cut each other some slack,” Metsker said at the time. “We need to adjust and allow some corrections to be made.”
Then things went quiet for three weeks, as attention turned to the outcome of the exceptionally close Republican primary for Kansas governor.
Things were far from peaceful within the election office, though. While software engineers were working on the reporting code, the county was having trouble with another piece of ES&S equipment. Just a couple of days before the canvass results were to be announced, a DS850 high-speed scanner and tabulator had likely broken a belt and was unusable, leaving the county with only one, according to county emails. The scanner in question was being used as the county reconciled poll ballots over the weekend. But it would be a key part of the recount. Janette Scobey, warehouse and voting systems technology manager, asked for a quick repair and a couple more of the machines for the impending recount ahead.
That was at 10:30 Saturday, Aug. 11. By 5:47 p.m. that same day, Metsker had lost his cool. He shot off a note to Christopher Kurland, ES&S account manager:
We cannot have any more failures. This is not acceptable … Seriously I have been extremely patient…[ellipses his] but that long-suffering patience is running out, because ES&S products are not performing as promised.
We are no longer a local story that should have never happened, now we are a national focus. We are being eaten alive in the media.
It was a very different Ronnie Metsker from the one who, just two days earlier talked about “common grace.” Now Metsker was worried about finishing the canvass in the less than 48 hours he had left. He wrote:
I assume we will somehow be ready for canvass. I am not sure. What I have seen today while our team has struggled to function on software that will not operate, gives me grave concern.
Metsker also alerted the company of an upcoming meeting between the county administration, Commission Chairman Ed Eilert, Commissioner Jim Allen, and “the top three attorneys from our county government.” He continued:
This is unprecedented. They are gathering to discuss the last minute ‘bad media coverage damage control.’ ES&S is the central focus of the situation. That group will ask me for an update. I hope I will be able to give assurance we are ready. I frankly do not know what outcome to expect if we have problems prepping for the canvass or performing or reporting out the results of the canvass. The public microscope is out. They are already relentless.
As it turned out, the canvass results were delayed a day and only the preliminary actions were taken to meet legal requirements on Monday.
The office’s emails, sprinkled with references to tiredness, turned positively giddy, though, when the news broke that Gov. Jeff Colyer conceded, eliminating the possibility of a recount. In an Aug. 15 note to ES&S vice president of governmental relations Kathy Rogers, Metsker comments on having “successfully survived the canvass” with workarounds the company suggested:
And the GOP Governor’s race primary was conceded last night and we are no longer facing a recount. Yay.
Rogers’ reply:
Our team is 100 percent committed to supporting Johnson County in all public relations efforts including taking full ownership of the issues and communicating corrective actions and plans.
The lack of a recount should be a relief, she wrote.
We all did a happy dance for you guys when we heard the news!
Meanwhile, corporate officials are working against the clock to get corrected software into the system in time to be tested for the general election. They were to submit their applications for a new certificate on Aug. 29, with the expectation that the federal government’s independent testing would begin the next day. Just in case certification doesn’t happen in time, ES&S also is looking for workarounds on the software as it is.
• • •
What was the issue in August? It came down to what would seem to be a rookie mistake in coding and a lack of the kind of testing needed to expose the problem.
The reporting software was new, with features that put incoming totals on the screens of workers in the election office. The software was designed to do frequent updates.
But they were too frequent — many times per second, in fact — and those updates were eating up all of the processing cycles, said Gary Weber, vice president of software development for ES&S.
Weber compared it to an employee giving updates to a boss on job progress. Instead of a reasonable number of updates, the software was flapping them out with the enthusiasm of an overexcited hummingbird.
To make things worse, the ballot was filled with precinct committee positions adding to the number of votes to be counted. The 2,000 contests and 3,000 candidates were well within the limits that the system could handle, as was the volume of the election turnout, Weber said. The problem was that no one caught it in a test before the election.
“In November we won’t have this problem because of the optimization of the code that Gary and his team have now done. We should have caught it in testing. There should have been a test scenario that would have caught this particular issue. However we did not. But the votes themselves were always secure,” Rogers said.
Douglas Jones, associate professor of computer science at the University of Iowa, seconds that.
“This is the kind of error you see early in product development. It’s the kind of error that ought to be caught long before the software is in the hands of users. While I am willing to believe the explanation from ES&S, it’s evidence of very sloppy testing — twice over,” Jones said in an email after reading the county’s explanation. (Jones is a former member of the technical guidelines development committee of the Election Assistance Commission.)
“Releasing software to a county without running it through a realistic test is irresponsible,” Jones continued. “ES&S ought to have test data for a completely fictional election on the scale of a large-sized county that they routinely use to test every single election management product they produce. Such testing ought to reveal the kind of problems Johnson County, KS had.”
The county also should do its own thorough tests, he said.
The company is correct to hope for a fast track on certification and also look for workarounds, he said. But Jones warned that expedited certification has its dangers. Jones remembers a case 20 years ago in which a now-defunct voting system had a bug that allowed voters to see how the person ahead of them in line voted. Those were billed as only minor changes at the time.
• • •
Despite all this, county officials have maintained their belief in ES&S and remain hopeful that all will work out for the best in November. Metsker demurred when asked how much staff time the problems have cost, saying that the election office would normally spend a lot of post-election time analyzing how things went — although all-nighters generally aren’t a normal part of that. Hence no calculation has been made about cost to the county.
Commissioners have generally backed Metsker up, but are somewhat more circumspect about how things will end up when it’s time to pay.
After the problems were discovered, commissioners warned that payment would depend on the equipment working, but so far neither the commission nor Metsker have given any metrics for how that will be decided, or a timetable on when the decision will be made. The county is due to take delivery of 1,000 more machines this month to put online for the November election.
“I don’t see that this is a problem, because ES&S is going to deliver. So I’m not harboring that problem,” Metsker said about the payment. “Why would I talk about something that isn’t here?”
The commission and Metsker have given the company points for accepting responsibility. And ES&S officials do their mea culpas at every opportunity. But in the end, the county is heading toward an expected high turnout in November with little but the company’s fervent promises.
Rogers asserts that the new version of reporting equipment the county used will become the standard after its code is fixed.
“I’m sure everybody is going to upgrade to it [the software in question] before it’s said and done because what this county got was the latest and greatest including the latest and greatest security features,” Rogers says.
Yes, the testing and software fell short. But Rogers says the public should remember that the problem could have been worse. Johnson County’s problems were about slow reporting — not accuracy of the vote.
“We have never tried not to own it,” she says. “This belongs 100 percent to us. But we make great products and this county bought a great product and they are going to have excellent elections for years to come on this technology. And this is going to be a blip in time that we’ll all remember painfully. But they will run great elections on this technology.”
♦
