Beyond the stoplight chart: How to perform Agile oversight at scale.

Four years ago I co-wrote the federal government’s guide to budgeting and oversight of major software projects. In the intervening years, the advice contained in that document has been put into practice in a number of state legislature and governors’ offices, to the point where a new problem has appeared: How do you oversee software projects at scale?

So you’ve left behind the fiction of stoplight charts. You’re getting demos instead of memos. Teams are ostensibly delivering software incrementally, based on constant user research. You have two, five, ten, thirty projects working in this way. But, oh no…you have thirty projects working in this way. Now your calendar is just demos of functioning software, all day, week after week. This is a great problem to have, but it’s a problem just the same.

I have worked on solving this problem, and I’ve applied parts of a solution, but my solution is by no means a tested theory of change. But I think it’s important to work out loud, so what follows is the four-stage process that I recommend to anybody facing the problem of scaling up oversight of major software projects. (I wrote a bit about this last May, though targeted at agency heads, in “How an agency principal should oversee a major custom software project.” That’s meant for the actual head of an agency; this guide, on the other hand, is meant for their deputies, advisors, etc., for legislative staff in similar positions, and for state IT agencies and procurement shops.)

Here are the four steps: 1. Get access to the backlog. 2. Require sprintly ship notices sent to leadership. 3. Engage with the ship notices. 4. Provide direct support. Let’s take each of these in turn.

1. Get Access to the Backlog

Any Agile team practicing user-centered software development is going to maintain a backlog of user stories, in GitHub Issues, Jira, or a similar tool. (And if they are not, that is the biggest, reddest of red flags.) Tell the team that they need to provide you with access to that backlog, so that you can monitor their work. This is the single source of truth for any software project—without some serious deception, looking at this will reveal exactly how much work they’re getting done and how they are doing it. If the team is reluctant or claims technological obstacle, tell them that you’ll give them 30 days until you need access; they will use that time frantically getting their act together, like somebody tidying up their home before the house-cleaner arrives.

When you have access to the backlog, periodically review it to get a sense of what they’re doing, why they’re doing, how it’s going, and the extent to which that’s rooted in user needs uncovered through one-on-one user research. Look at the software they’re developing to see those user stories realized in staging and production. It is important to fight the the urge to extract metrics from the backlog. You might think “I can tally the story points completed per sprint to measure productivity,” or “I can measure the time allocation per activity.” Down this path lies madness. A scrum team is a small, self-organizing, cross-functional group that incrementally builds software to address user needs. The one external metric that they should be held to is “are they meeting user needs?” Turning their activities into metrics in the name of transparency will wreck that work product.

2. Require Sprintly Ship Notices Sent to Leadership

A reasonable request to make of a team is that they send ship notices at the end of each sprint—a summary of what they’ve been up to for the past two weeks. A ship notice should also include what’s planned for the next sprint, a list of blockers that they need help clearing, and a one-sentence budget update that indicates how much has been spent against what total dollar value with a forecast end date based on that burn rate. To report what they’ve been doing for the past two weeks, have them simply paste in a list of all user stories that they completed, with each one linked to that user story in the backlog, for traceability. These only take a few minutes to prepare, since it’s mostly copying and pasting from their backlog, and a cadence of sending one per sprint is sufficient for leadership. This email should be CCed to the most important people who need to receive these updates. A major initiative for an agency should be CCed to the agency principal.

3. Engage with the Ship Notices

It is not enough to simply receive the ship notices. For teams, that will eventually feel like they’re CCing /dev/null. It is important to actually read every ship notice, understand every ship notice and, when appropriate, engage. When you see that a particularly valuable or interesting user story has shipped, you should try out the new feature on the site and respond with praise for the team. If you see that there’s a blocker that you could or should address for them, do so. If you see that a team’s velocity has slowed down over the past few sprints, you should check up to see if they need support (not to complain, but to help). Remember that your emailed responses will be shared with the team—seeing that leadership is engaged, interested, and supportive can make a big difference to their morale. The frequency with which you engage with ship notices should be proportional to the level of oversight required for the project.

4. Provide Direct Support

Sometimes projects will need help. “Help” does not mean haranguing them, insisting that they work faster, or micromanaging them. Many projects will be performing Agile, but not actually practicing Agile. Sprintly ships will reveal this to be the case, but it won’t teach them how to work correctly. Demanding that they “be Agile” is not going to do it.

The solution is to teach them how to work, and that’s best done in the form of an internal digital service team. That can be a very small team, as few as four people: a user researcher, a software developer, an Agile coach, and an Agile project manager. It’s possible that one person could check more than one of those boxes. If your agency outsources a lot of software development, then a contracting officer should be in the mix, too. That team needs to have experience with consulting, because their task is ticklish.

If the software development team in question is comprised of agency employees, the digital service team can work directly with them to help them improve their processes. If the the software development is outsourced, the digital service team can help agency employees learn how to let the vendor team do their best work, or demonstrate that they need to replace that under-performing vendor with a better one (and then help them to do so).

It is not enough to tell them “do better,” because they may not know what “better” looks like, or how to get there within the realities of your agency. You need to give them the capacity to work better.

---

The goal here is to ensure that each project has spun up the flywheel of using constant user research to inform the development of in-production software that’s providing benefits to users.

This is all about scaling up “demos, not memos,” but it’s no replacement for actual demos. It’s important to attend sprint reviews as often as plausible, especially for high-risk and high-value projects, to ensure that value is being delivered to end users.

At the risk of straying into management consulting, I must caution that there is such a thing as too much oversight. Is the team consistently delivering high-value improvements to end users who are measurably, objectively happy? Then leave them alone! If they are falling short of that goal, you will determine that through getting access to their backlog, reading their sprintly ship reports, and engaging with those ship reports, and the way to address the problem is by providing them with direct support from an experienced, cross-functional digital services team. Just because you are qualified to observe that there is a problem does not mean that you are qualified to fix the problem. Only an experienced Agile coach can do that.

Again, this four-part approach is by no means a tested theory of change, but instead a series of things that I and/or my colleagues have tried at least once and had success with. I’m working every day on testing out these and other approaches to Agile oversight at scale, as I know many other people are doing as well. We’re on a path to collectively arrive at a theory of change over the next couple of years, and exercising this four-step process will help us to accomplish that.