Democratic Iowa Caucus Shadow App - A Cautionary Tail for Digital Commerce Software Development

What was wrong with the Iowa Caucus voting app?

In case you have been hiding under a rock during the Iowa Caucuses, here’s a quick summary.

• The Iowa Democrats decided to provide an to caucus leaders for recording the results of their local caucus.
• The app and reporting system behind it were unmitigated disasters.
  • Caucus leaders didn’t understand the app.
  • Many users had difficulty logging in.
  • It is possible that not all data was recorded correctly.
  • The reporting system failed due to bad code.
  • Cyber-security specialists found massive security holes.
• The backup system (phone reporting) also failed.

In the end, party officials counted votes using paper backups. Many are calling for a repeat of the caucuses. Most concerning of all: critical voting data has been siphoned off to unknown entities.

Frustratingly, Shadow issued a glib “sorry for the delays, we’ve learned from our mistakes” statement. One wonders if they truly understood the weight of their responsibility.

What does the Shadow App have to do with Digital Commerce Software Development?

I’m always deeply concerned by failures in our democracy, especially with regards to elections. However, I do welcome the attention this brings to the importance of good software development. This goes for political and digital commerce alike.

The failure of the Iowa Caucuses goes far beyond the company Shadow; almost always the case in a failure of this import.

Basic Architecture of a Voting / Ecommerce Ecosystem

Application Server

Users download the application to their phones by visiting this server. Once downloaded, the application connects to the database over the internet. Users receive updates to the application from the server as well.

One of the key issues reported with the Shadow Caucus App was the lack of its availability in the Apple App store. Loading an app from a private system goes beyond the expertise of inexperienced users (though most teens would have no problem).

The front end

The app, developed by Shadow, Inc, provided the user interface to a voting recording and reporting system hosted online.

The admin panel

Election monitors must be able to update settings, see which caucus sites have reported, deal with exceptions and perform other administrative tasks.

Transactions

Ecommerce applications process purchases in a workflow. This checkout process verifies things like availability of the product, the identification of the buyer, and availability of funds before completing the transaction.

Similarly, a voting application must verify the submitter’s identity, check that data submitted is not duplicate, and so forth. If anything, the level of verification in a voting system should be higher than ecommerce.

When errors are created in ecommerce, merchants can issue refunds, take returns or otherwise ensure fairness to their customers. Election officials can recount votes but the voter has no way to verify their vote is correctly represented in the results.

Where it all went wrong

Recording vs Reporting

Shadow had separate logic for recording and for reporting data. On the recording side, the app user may have thought data was being recorded when it was not. Even worse, the vote tallies were incorrect.

According to Troy Price, the issue was on the reporting side. Thus, the data was in the database, but, the reporting mechanism was not spitting out complete results. A code fix unblocked the log jam.

As part of our investigation, we determined with certainty that the underlying data collected via the app was sound. While the app was recording data accurately, it was reporting out only partial data.

Official Statement
Troy Price, Iowa Democratic Party Chair

Recent reports, however, challenge this statement and indicate errors went far deeper.

When it all went wrong

Inadequate time

Nine women can’t make a baby in one month.

Fred Brooks
The Mythical Man month

Managers and business people involved in software projects often fall prey to the infinite divisibility fallacy. They believe that doubling the number of developers can cut development time in half.

With most software development projects, you can often speed up delivery – at exponentially increasing cost – to a certain extent. However, at some point, steps must be completed in order. For example, I cannot test a feature until the programmer is done writing it.

In the case of the Iowa Caucus system, the Iowa Democratic Party decided to create the application only a couple of months before the caucuses. In the best of cases, Shadow had a month to write the app and a month to test and make corrections.

With good planning and sufficient resources, this timeline would be uncomfortably tight.

Inadequate budget

$63,000 may seem like a lot of money for an application. However, given the critical nature and complexity of a voting system, I would estimate you need at least twice that budget. You need very senior developers working on it. Apparently that was not the case.

Honestly, the biggest thing is … the app was clearly done by someone following a tutorial. It’s similar to projects I do with my mentees who are learning how to code…. …the code looks like someone Googled things like ‘how to add authentication to React Native App’ and followed the instructions….

Kasra Rahjerdi
Vice.com article

Shadow’s use of inexperienced developers is unconscionable. Any programmer or development company with a sense of ethics would have refused to roll out the project.

Additionally, Shadow cut costs by using free distribution software designed for testing, not production. The level of irresponsibility and poor decision making is epic, possibly to the point of negligence.

…the app was distributed using the TestFairy platform’s free tier and not its enterprise one. That means Shadow didn’t even pony up for the TestFairy plan that comes with single sign-on authentication, unlimited data retention, and end-to-end encryption.

The Verge

Specifications

News reports indicate that the rules for the caucus had changed dramatically from past caucuses. This was the first live run through of this new process. Thus, it would have taken quite a bit of time to translate the political rules into technical logic. Thorough discovery was needed to make sure the system met requirements.

The multiple iterations of discovery would have taken a few weeks. Many implementation details specifically pertained to the new system making the timeline unrealistic in the best of circumstances.

This same type of time crunch can affect digital commerce projects. Clients who want their projects to succeed have to make sure the development team has sufficient time and enough rounds of communication to define a very precise result. This takes time and planning.

Those who do not have to produce the final product often see timelines as ‘directional’ or ‘padded’ and can drag their feet in making decisions. Politicians are not well known for either precision or wanting to be nailed down.

Testing

One thing is certain, however: It appears that there wasn’t enough testing of the app.

Silicon Angle

The system required logical testing of the back end. It is unclear if this took place. This system also involved applications running on mobile devices. At LEAST two weeks of field testing should have been done. Apparently little, if any, field testing took place.

In the same fashion, merchants must consider the complexities of their unique shopping experience. While loosing one sale has fewer repercussions than losing one vote, making sure web sites and applications are properly tested on mobile devices is critical.

In ecommerce, we test in both a sandbox environment – where fake credit cards are used – and the production environment – where we process an actual transaction. Shadow should have adopted this practice.

At Razoyo, when we know a merchant will experience a spike in usage due to a special event like Black Friday, we load test the servers. This ensures we have time to adjust servers and code to guarantee success.

In load testing in ecommerce, we work with the merchant to produce an educated guess of the amount of traffic. However, the Iowa Democrats knew well in advance how many locations would be involved. Not load-testing adequately was as much a failure of political leadership as it was a technical one.

Training

Numerous reports indicated that users were simply confused by the app and had trouble logging in. Due to the sketchy distribution system users also had difficulty downloading it.

It appears that inadequate resources were dedicated to confirm every single user not only successfully downloaded the app but also had a chance to practice. To make things worse, Shadow pushed out a software update just days before the caucuses. Who verified that users had the new version? Given the sketchy distribution process it was far from guaranteed.

Users spend most of their time on other sites. This means that users prefer your site to work the same way as all the other sites they already know.

Jakob’s Law, Laws of UX

Good user experience design should follow the Jakob’s law. When they do, most users intuitively figure out how to use new apps. However, applications like Shadows are unfamiliar to users. The consequences of uninformed usage are extremely high.

There was no excuse for the lack of user training. On this point alone, failure was assured.

Using the right company for the right job

The team at Shadow that wrote the app and back end systems was clearly unqualified to do the work. Additionally, there was no backup - no qualified developer reviewed the code for efficiency and security as evidenced by the results.

So, why Shadow?

Often, we are asked to work on projects that are outside our domain. When we feel our involvement puts a client or project at risk, we respectfully bow out. If it is something that stretches us but still makes sense, we disclose all the risks and let the client decide.

The folks at Shadow didn’t do this. The Chicago Tribune described the opacity of the developer selection process.

• The Democratic National Committee recommended Shadow in a closed-door process.
• Obvious conflicts of interest were rampant.
  • Tara McGowan of Acronym (Shadow’s only investor) is married to Michael Halle, a senior strategist for Pete Buttigieg’s campaign.
  • Tara McGowan’s brother also works for Buttigieg.
  • Buttigieg’s campaign spent almost as much with Shadow as did the Iowa Democratic Committee.
  • Troy Price, head of the Iowa Democrats, is also socially connected to Michael Halle.
  • Top Executives at Shadow all worked on Hillary Clinton’s 2016 campaign.

Not to be glib, but using a DNC-recommended development company funded and run by politicians is a sign of either hubris or corruption. Remember, the DNC and the Clintons – who are closely tied to the companies involved – both have extremely poor track records when it comes to cyber-security.

TL;DR - Lessons for Digital Merchants

• Insist your team and your developers team understand both the business goals and nuances of your environment.
• Identify and prioritize risks and list what is unacceptable.
• Test expected scenarios and ones that will break features.
• User test or at least A/B test your project.
• Select your implementation partner carefully and understand their strengths and weaknesses.