The success of many government agencies now hinges on their ability to successfully execute large custom software projects. And yet as a rule, the principals of those agencies lack the ability to ensure that those projects will succeed, or even to oversee them meaningfully. As a result, they have lost the ability to ensure that their agency can achieve its mission.
The Problem
Agencies are pretty specialized—in federal government, their key needs are unique, and in state government they’re one of just 50 (or as many as 56, depending on how you count) agencies with those needs. The truly generic needs can be addressed via commercial off-the-shelf software (COTS), but the mission-unique stuff must be met by what I call load-bearing software, which is inherently custom.
Load-bearing software became a thing in federal government midway through the last century. The ur-example of this is the IRS’s Individual Master File, their core computing system, written in COBOL and IBM System/360 assembly, which debuted in 1960. Load-bearing software became an increasingly common need for government agencies in the 1980s and 1990s, with that software almost entirely internal-facing. That changed in the 2000s and 2010s, as the public gradually came to expect internet-intermediated interactions with agencies, especially for application processes. In 2020, Covid forced agencies at all levels of governments to move service delivery online, and three years later there is no sign of that shift receding.
If a state unemployment agency’s UI system doesn’t work, in what sense are they a UI agency? If a state’s EBT system goes down, in what sense do they provide SNAP benefits? If the IRS’s Individual Master File crashes, in what sense are they a taxation agency?
Load-bearing software must work for agencies to achieve their missions. And yet, under the standard outsourcing paradigm, agencies outsource every aspect of the construction, maintenance, enhancement, support, and hosting of this software. In doing so, they outsource their mission. This is a terrifically dangerous practice.
When an agency principal lacks the knowledge or even interest to understand and control these software projects, they are handing their control of the agency to a consulting firm’s project manager. No leader wants to do that.
The Solution
In short, agency leaders need to give a damn about technology procurement, budgeting, oversight, and implementation. Load-bearing software is not a detail—it’s the whole ballgame.
There are four things that principals need to learn if they’re to control their agency’s ability to achieve its mission:
- How modern software is made
- What’s possible, at what level of effort
- How much software costs
- How to oversee software development
Let’s review each of these.
How modern software is made
To know how software gets built today, there are six core concepts that agency leaders need to grasp:
- User-centered design
- Agile software development
- Product ownership
- DevOps
- Building with loosely coupled parts
- Modular contracting
There’s a short overview of each of these in GSA’s “State Software Budgeting Handbook,” which I co-wrote in 2019, so I won’t re-explain them here. It’s not enough for agency principals to read a paragraph about each of these, though. Without about an hour of training in each of these subjects, agency leaders can have a good base of knowledge to how projects are being executed—or should be executed—by vendors and agency staff.
What’s possible, at what level of effort
Many of the work overseen by agency leaders draws from fields that they’re already equipped to understand the complexity of. If an agency needs to hire 500 new employees, a leader knows intuitively that this possible, and that will take longer than two months but less than two years. If it needs to buy new office equipment for 100 people, that’s very achievable, and will cost more than $100,000, but less than $1,000,000. If the agency needs to move into an entirely new building in two months on a budget of $5,000, that is not possible. And so on.
There is absolutely nothing that has prepared an agency principal for understanding the cost of software. There are a pair of xkcd comics that address this concept:
The best corrective for this is to observe actual Agile software development teams actually developing software, by joining a series of sprint review sessions for multiple projects. That makes it possible to see what e.g. six people are capable of accomplishing within two weeks of work.
How much software costs
Grasping the cost of software is difficult in the space of government software because of the absurd levels of pricing distortion brought about by decades of procurement practices unsuited to the problem. At a state or federal level, $100 million is a normal price for the development of a load-bearing software system, and that’s a price tag that’s not meaningfully decomposed to any part of that system.
Again, observing actual Agile projects will do a lot of good here. By understanding the level of effort, and connecting that to the billing rate of a vendor, it becomes possible to see that the work produced by our six-person team in two weeks cost $60,000 in billed time. Observing this for a while, it soon becomes evident what software should actually cost.
How to oversee software development
I’ve already written a guide to overseeing major software projects, albeit intended for legislatures, but the 14 listed plays largely apply to an agency principal, either for them to apply themselves (especially “Demos, not memos“) or to ensure that their staff are applying.
But there are a few leader-specific admonitions that I want to include here.
- No stoplight charts. Agency principals receive regular updates on major software projects that say everything is going great, with green lights all the way….right up until the update that says everything is red and the project has failed. What happened? In short, strategic misrepresentation. Nobody wanted to provide troubling news to their boss, so as the state of the project got handed up the chain, the view got rosier and rosier, until the principal was told, with every update, that things were going great. The solution to this is to eschew reports in favor of live demos of the actual work being done. If leadership requires reports, make them narrative, authored by the agency’s product owner for the software project.
- Require that live software be deployed to production regularly. Project will spin their wheels in isolation. This allows for bad decisions to be hidden, for a lack of progress to obfuscated. Leaders should insist that software improvements be incrementally delivered to end-users. Continuous delivery paired with continuous user research makes it very difficult to waste much money on a software project.
- Require weekly ship reports. At the end of each week (or each sprint), a project’s leadership should write a “ship report,” which briefly describes what was shipped since the last ship report, along with what’s coming up, what blockers are preventing the project from progressing, and how much of the budget has been spent on the project to date. For any project that requires particularly close monitoring, it helps to require the inclusion of a list of all user stories that were completed within the period in question—this makes it crystal clear what the project team has accomplished.
This is all meant to to avoid precisely this scenario:
“Deloitte presented much too rosy of a picture to us,” [Governor Gina Raimondo] said. “I sat in meetings with Deloitte and questioned them and they gave us dashboards that showed us everything was green and ready to go, and the fact of the matter was it wasn’t.”
“Raimondo Faults Vendor Deloitte For Delivering ‘Defective’ UHIP System,” The Public’s Radio, Feb. 15, 2017
It can be tremendously difficult for the principal of an agency to oversee a load-bearing software project, in no small part because leaders tend to get in the way, providing bad ideas, issuing self-important new project requirements, and measuring the wrong things. Learning how modern software is made will ensure that principals set their expectations properly and can engage in an appropriate fashion. Learning what’s possible, at what level of effort, will allow principals to understand and make demands that are reasonable. Learning how much software costs will allow principals to have reasonable expectations of costs, both in preparation for projects and when executing them. And learning how to oversee software development will draw together the prior three skills so that principals can effectively manage the projects on which the viability of their agency’s mission depends.
This approach will allow an agency principal to take back control of their agency from vendors, and to stop outsourcing their agency’s mission.