Iowa caucuses and the Shadow app: A lesson in critical infrastructure
On Monday 3 February 2020, Iowans caucused at almost 1700 precincts across the state to select their preferences among presidential candidates. Three days after the Iowa caucuses, the final tallies for the Democratic presidential candidates remained incomplete. The popular press blames a technical glitch in the Shadow app, but accountability resides with party leaders and the Shadow app executives who allowed technology that had not been previously fielded nor fully tested to be deployed as critical infrastructure.
The incident in Iowa is a lesson for owners and operators of any kind of critical infrastructure. CIOs, Chief Engineers, and business leaders should follow these recommendations to ensure the integrity and reliability of critical infrastructure:
- Audit critical infrastructure systems annually and situationally based on risk assessments
- Observe government, industry, international and other applicable standards and certification procedures when replacing or making changes to critical infrastructure technology
- Conduct risk assessments periodically and situationally
- Monitor and continuously audit the performance of critical infrastructure
- Stress test, retest, and retest again your procedures in the event of an incident
- Document the architecture of critical infrastucture including risks and interdependencies
- Prepare for social storms with crisis communications planning
For an explanation of how these steps are applied to electoral system critical infrastructure, see the recommendations for election officials at the end of this note.
Iowa is different
Unlike primary states, the reporting and tabulating of results of the Iowa caucuses are managed by political parties rather than government election management professionals. To make that clear, the announcement below from Iowa’s Secretary of State was posted on Twitter.
Election officials in primary states should not assume that, because Iowa is a caucus state, they can brush the lessons there aside.
Hopefully, the Iowa caucuses debacle is a one-off. But let’s not assume there won’t be more incidents.
Certainly, the candidates are not assuming. Skepticism, if not outright distrust, of official tabulation and reporting has led some campaigns to develop their own “ghost tracking,” that is, parallel tabulation and reporting systems.
According to Ray Laracuenta, CEO of Oblivion.io which produces the WeAct app for grassroots organizing, “When the IDP [Iowa Democratic Party] started reporting results, the Sanders campaign identified discrepancies between the tabulation efforts from their ghost app and the IDP results tabulated with the Shadow app. This delayed the release of the results.” To validate its tally, the IDP put off publicly reporting results while it completed a manual tally across all 1700 precincts.
Cybersecurity is not just about stopping hackers
If the incident weren’t so visible and consequential, IDP officials might be forgiven the failure of their new app. After all, these types of things happen in the business world all the time. But elections have consequences as President Obama once said, and so do cyber-incidents associated with elections.
A common misconception by non-IT people and even by many IT people is that cybersecurity is about stopping unauthorized access. But as George Westerman of MIT and Richard Hunter of Gartner explain in their book The Real Business of IT, good security starts on a foundation of availability. Without availability, nothing else matters. Next comes access – access to let the right people in and to keep the wrong people out. Then comes accuracy — can the data and reporting be trusted. Finally, there is agility – does the system respond well to changes and work in all kinds of scenarios.
Looking at the Shadow app website, it does not claim to have a tabulation and reporting tool, which means the IDP app likely was a custom solution. In critical infrastructure systems like voting tabulation and reporting, custom software should be tested and certified for availability, accessibility, accuracy, and agility beyond the expected peak load, such as tests that simulate all 1700 precincts reporting at once. The U.S. Election Assurance Commission promulgates standards for election management systems and voting machines. While the state of Iowa requires that electoral system technology be tested and certified by federally approved labs, the law may not apply to party-run caucus technology, and there is no evidence that the IDP or Shadow applied the federal certification standards.
Furthermore, procedures for response to a failure should be tested and retested. Failures may be simple such as precincts in rural areas with poor cell coverage not being able to get through – keep in mind half the town might be at the same place caucusing and all trying to make calls through the same cell tower. It could be a just a minor glitch, as appears to be the case with the Shadow app. Or it could be a massive failure due to tampering or a hack of electoral systems.
It is what it is: there won’t be a perfect electoral system
When considering solutions to improve trust in the electoral system, there are four critical elements that limit options:
There is not one U.S. electoral system: There are thousands. Each state and the District of Columbia have established their own systems, and they vary significantly. Furthermore, the management of voter registration and elections is devolved to over 3000 local jurisdictions, so that even within a state there can be many variations in electoral infrastructure.
The digital transformation of elections will continue. Like businesses, electoral systems have undergone a digital transformation. By the 1920s, paper-punch lever machines, which had seen their first use in New York in the 1890s, were common, and these led in the 1960s to punch card machines that could be computer tabulated. The “hanging chad” controversy in the 2000 U.S. presidential election led to the rapid adoption of electronic voting machines, some of which had no paper trail, adding uncertainty since votes could not be independently tabulated by hand for recounts or audits. Today, most states, including New Hampshire have electronic machines that use OCR to scan a paper ballot, thus providing the rapid tabulation and reporting of a purely digital system plus the ability to do a hand recount or audit with the paper ballots.
From paper, to mechanical lever, to computer punch card, and then to digital hybrid systems, the evolution of election technology will continue. Secure internet voting cannot be too far in the future.
There will always be aging electoral infrastructure. How many of us are carrying the same mobile phone we had 10 years ago? If it’s a smartphone we’ve likely gone through two or three newer models by now; our laptops have been changed out at least once. Yet, New Hampshire primary voters this year will use 30-year-old machines that no longer have manufacturer support, and the service contractor scavenges parts from old machines to maintain those still operating.
There’s nothing inherently wrong with old technology though, if it works, can be supported, and is secure. In 40 New Hampshire towns, the voting technology consists of a wooden ballot box dating to 1892. It’s a reliable and secure voting machine in which a voter inserts the ballot and turns a crank to deposit it into the securely locked box. The voter gets a satisfying ring of a bell when the ballot has been fully inserted and drops into the box. Local officials hand count the ballots and report the results to the state. The next time you’re in New Hampshire, make a detour to Durham, NH, and stop by the Durham Historic Museum to see this very clever and reliable machine.
Retrofitting electoral infrastructure is a very expensive process and is not undertaken lightly. And it’s not just the expense that should cause pause in major changes. Any change introduces new risks that must be assessed and managed. On the other hand, old infrastructure can be hard and expensive to maintain, plus create risks as it ages: remember, hanging chads.
Cybersecurity issues evolve. Voting machines and the election management systems that connect them for tabulation and reporting normally are not connected to the internet. However, machines may be temporarily connected for maintenance and firmware and software updates, and researchers have discovered a few connected machines. Still, a massive cyber-attack on the machines themselves is unlikely. Voting registration systems and voter databases though are often connected to office networks and thus could be and have been accessed by hackers (see Prediction 2020: Be prepared for U.S. election hacks).
However, hacking is not just done over the internet and there are machine vulnerabilities that must be effectively managed. For instance, with respect to the OCR machines used in New Hampshire and many other states, VerifiedVoting.org reports security concerns with:
- Exposed ports, memory card access, doors, and seams
- Locks that are pickable
- Ballot box access
- Sensitive memory cards that could be corrupted
- Challenges with reading red and blue inks that could be used by voters
These security risks are manageable, but the risks are not static. Changes in personnel, infrastructure, procedures and laws all can affect the cybersecurity posture of electoral systems.
The social amplification of risk generates social storms. As it became apparent that there was a problem in tabulating the results of the Iowa Democratic Party caucuses, a twitter storm erupted. Social media amplifies any election incident well beyond its real-world impact and jurisdictional borders. Election and political party officials must consider not just risks but the social amplification of risks in their incident planning.
Trust in the electoral system has been shaken in recent years, and the Iowa caucuses and Shadow app debacle indubitably leads some voters to think that it could be 2016 all over again. Election officials who shake this off as a one-off due to Iowa’s atypical caucus system may well be unprepared for other types of failures in upcoming primaries and the general election. They should take the following actions:
1 – Audit voting and election management systems. These audits should be done annually and situationally based on risk assessments with quarterly updates on deficiencies and remediation to the most senior electoral officials.
2 — Require federal standards for testing and certification when procuring and making changes to electoral technology. Ensure that election management systems and voting machines are certified to the latest versions of federal standards approved by the U.S. Election Assurance Commission. When standards change, assess whether recertification is required.
3 – Conduct risk assessments periodically and situationally. Changes in election law, technology, threat assessments, and other environmental factors, such as an incident in another state, can change the risks posed to electoral systems; periodic risk assessments, at least annually, should be conducted. For an event that happens in another local, a situational risk assessment should be conducted. Situational risk assessments also should precede any change, including major personnel and organizational changes, the acquisition, deployment and acquisition of new technology, changes in procedures, regulations and laws, and when there are other changes that impact the people, technology and processes associated with the electoral system.
4 – Conduct risk limiting audits immediately upon tabulating election results and ideally before reporting them. Hand-counting a statistically meaningful sample of results can usually be quickly completed. However, if margins are tight and the sample size needs to be larger, the hand count could impact the timeliness of reporting. An alternative to hand counting would be to run a large sample of paper OCR ballots through exquisitely secured voting machines to validate the counts.
5 – Stress test, retest, and retest again your procedures in the event of an incident. Even an incident that does not materially impact the election results can undermine public confidence in the electoral system. Stress test your procedures; that is, test them with several failure scenarios under peak load. Some of these tests can be what-if heuristic table-top exercises, but some should be physical walk-throughs of plans and procedures with the officials who will need to respond in real life. Precinct volunteers should be trained as well.
6 – Document the electoral system architecture including risks and interdependencies. Taking an enterprise architecture approach to the electoral system supports better decision making on changes and investments. Risks are more easily assessed as well. This may look like too big a task for smaller jurisdictions, but state officials could assist with a reference architecture and other resources.
7 – Prepare for social storms with crisis communications planning. A crisis communications plan must be a part of incident planning and should be tested alongside incident planning. These plans should include consideration of campaign ghost tracking systems and the proliferation of potentially conflicting reports from campaigns. Ensuring that campaigns are aware of incident and communications plans and have designated and trained people as points of contact is critical.