The US Air Force Learned To Code - And Saved the Government Millions

Speed. Innovation. Move fast and break things. Ask for forgiveness, not for permission.

Not exactly the the MO for the military chain of command. But with software and hardware development moving much faster than the command and control decision-making process, failure to change meant competitive disadvantage - so necessity beat tradition. JL


Mark Wallace reports in Fast Company:

Best practices for modern software development would be impossible at DoD. Specs take years to write and  more years to deliver before code can be tested in the field—often making systems obsolete by the time they’re delivered. “The DoD violates every rule in modern product development. ”Technical people need to be in the room making technical decisions in places of authority.“Just because you followed the software requirements spec doesn’t mean you met the capability need.”

In October 2016, Eric Schmidt found himself in the Arab Gulf nation of Qatar. Then executive chairman of Alphabet, Inc., Google’s parent company, Schmidt was touring military installations in his role as chairman of the Defense Innovation Board (DIB), whose mandate was to provide the U.S. secretary of defense with “independent advice and recommendations on innovative means to address future challenges.”

At the Air Operations Center (AOC) at Qatar’s Al Udeid Air Base, Schmidt saw up close the way that today’s U.S. Air Force meets its enormous challenges: with magnets and colored plastic cards. The Air Force was then engaged in an offensive against the so-called Islamic State forces in Mosul, Iraq. But when Schmidt asked an AOC commander what his biggest concern was, he got a surprising answer. As Schmidt told me, “He said, ‘Well, frankly, it’s something different: I don’t want them to erase my whiteboard.'”
The AOC at Al Udeid is where AFCENT, the U.S. Air Forces Central Command, oversees air force operations for over 20 countries from Egypt to Kazakhstan. An enormous amount of data is pushed through the center every day, through a system of 43 software applications that help with everything from assessing targets to planning attacks, getting information to pilots, monitoring operations, analyzing damage done, and more.

Eric Schmidt (center) and other members of the Defense Innovation Board. [Photo: Cmdr. Patrick Evans, courtesy Office of the Secretary of Defense]

Not all the functions of the AOC are carried out on a computer, however. The whiteboard, as Schmidt and other DIB members soon discovered, was where daily planning took place for AFCENT’s aerial refueling operations. “We got the missions for the day, figured out what targets needed to be hit, and how much fuel was needed, who needed the fuel, and when they needed it,” explains U.S. Colonel Mike Drowley, AFCENT chief of staff. “It was an eight- or nine-hour process [for three or more people] to try and figure all the ins and outs. It was like a Tetris game of blocks and pucks.”That’s a literal description. “They would actually use physical distance on the whiteboard to determine whether they could meet their mission,” Schmidt said. To do so, a person called “the Gonker” entered data into an Excel spreadsheet known as “the Gonkulator,” while “the Planner” arranged magnetic pucks and plastic laminated cards on the whiteboard to indicate how long planes could stay in the air.

The whiteboard on which tanker refueling operations were planned. [Photo: courtesy of U.S. Air Force]

“They had a VBScript that they ran when the spreadsheet was correct,” said Milo Medin, a Google vice president who is also on the innovation board. “That would generate an input that had to be manually typed in.” The person doing that typing was called “the Thumper,” and the destination for that data was the Master Air Attack Planning Toolkit, which helped generate an attack plan known as an Air Tasking Order at AFCENT headquarters at Shaw Air Base in South Carolina.For Schmidt, the obvious question was whether the base had access to more modern software. “And the answer came back, ‘Yes, but it doesn’t work,'” he said.

“The comment around the group was, this just isn’t that hard a problem to solve,” said board member and Caltech professor Richard Murray, who was in Qatar with Schmidt and Medin. “It was sort of hard to comprehend why the original software couldn’t solve it. You could imagine it had been written a long time ago, it just couldn’t handle this number of tankers, whatever, fine. But can’t you just go fix this? And the answer was basically, no.”
“That can’t be the right answer,” Murray said.
But under federal regulations, it is. Standard DoD procedure requires systems like the AOC software to be competitively bid, and for the winning contractor to design, build, certify, and test the entire system before delivering it to users—and then to go through the entire process again each time any appreciable amount of code needed to be changed. That’s in part why the Air Operations Center software had been in use more or less unchanged since the 1990s.
But the board’s visit set off a singular chain of events, and less than six months later, tanker planning at the Qatar AOC was being done via state-of-the-art software that saved hours every day and millions of dollars every week. And much of the rest of the AOC software was also being overhauled—by a fast-moving cadre of Air Force engineers working hand-in-hand with Pivotal Software to step in where a $745 million contract with a big defense contractor had produced nothing over the previous five years.

The tanker refueling planning app that replaced the AOC’s whiteboard. [Photo: courtesy of U.S. Air Force]

The Air Force team writing this wartime code draws its inspiration from pop culture and the need to move fast. But its broader mission is more serious: to do away with the red tape that keeps the armed forces from moving past things like whiteboards and plastic cards, even for critical missions. The tools it has worked on were developed and deployed in a fraction of the time and at a fraction of the cost of most contracts that take on similar work. But can it get the Air Force, and the Department of Defense more broadly, to undertake the badly needed update to the way they conceive of, acquire, and deploy the software—and, just maybe, the hardware—that are the tools the armed forces rely on in war?

Reporting for coworking duty

The latest in U.S. Air Force technological advances can be found in an unlikely place: the WeWork coworking space on Portland Street in downtown Boston. There, a couple of dozen Air Force enlisted men and women report for duty every day. None are in dress uniform, or even flight suits; most are in jeans and sneakers. A handful wear T-shirts emblazoned with their outfit’s logo: the silhouette of a familiar-looking spaceship—the ship that made the Kessel Run in under 12 parsecs.

It may seem strange to find a bunch of Air Force personnel showing up in downtown Boston in Millennium Falcon shirts to write code. But strange—or at least, different—is exactly what the Air Force is after. The enlisted personnel in Boston are dressed less like pilots than like software developers, because that’s exactly what they are. Their small, open-plan office at WeWork holds ranks of keyboards and monitors—as well as another couple of dozen software engineers on loan from the nearby offices of Pivotal Software and other civilian companies.
On the day I visited it recently, it was impossible to tell enlisted from Pivots and consultants. The office was a hum of conversation and activity, and a passel of visiting dignitaries—including Schmidt and other DIB members—were moving around the narrow space getting a tour. This was the unofficial opening day of the Kessel Run Experimentation Lab, as the Air Force’s WeWork office has been dubbed, and the place looked more like a growing startup than a military outfit. While Kessel Run is an official Air Force program with an entertaining name, it’s also a highly unusual arrangement, one in which enlisted men and women are crafting the components of one of the Air Force’s most important weapons systems—a job normally done by big defense contractors, and overseen by beleaguered acquisitions officers in the Pentagon.

The Kessel Run office in Boston. [Photo: U.S. Air Force photo by Todd Maki]

But getting this far has not been easy. Kessel Run was born from crisis and serendipity, and midwifed by a Secretary of Defense who knew enough to put the pieces into place and then get out of the way. It’s taken the Air Force two years, sustained advocacy on Capitol Hill, and the efforts of everyone from the Secretary of the Air Force to Eric Schmidt to Pivotal Software to enlisted personnel from Qatar to Silicon Valley. And the project nearly died several times along the way.

The Defense Innovation Board (launched by Ashton Carter, then Secretary of Defense, in early 2016) was taken aback to learn that the AOC software in Qatar was more than two decades old, and that regulations made it nearly impossible to update. But at some point during the whiteboard presentation, Schmidt recalls, he leaned over and said to another person on the AOC tour, “Couldn’t you do something about this?”
That person was Raj Shah, and it just so happened that Shah had on his staff someone who had been waiting for the chance to tackle just such a project for some time.

Building stupid cards

Shah at the time was managing partner of the Defense Innovation Unit-Experimental, or DIUx, another office stood up by Ash Carter. He is also a former fighter pilot who has flown combat missions out of Iraq. “Having refueled four times a day when I was deployed, I was very familiar with the output” of the tanker planning process, he said. “I’d been in situations where the tanker didn’t show for a time, and as a receiver getting gas, if they’re not there you get very angry.”

For a force that flies something like 25,000 sorties a year, coordinating aerial refueling is no trivial task. Whiteboard planning is imperfect. So the Air Force keeps tankers loaded with fuel and ready to scramble at a moment’s notice, to cover gaps like the ones that irked Shah.

An Air Force KC-135 Stratotanker out of Al Udeid refuels a Navy F/A-18C Strike Fighter. [Photo: U.S. Air National Guard photo by Master Sgt. Andrew J. Moseley]

Shah knew the other side of the process as well, from his days as a young airman in Iraq. “When I was first deployed, in 2006, I would get a printout via email [from the AOC] of where everybody’s supposed to go,” he told me. “Then my job is to turn it into mission materials that pilots fly, because I’m the junior guy in the squadron.” That process involved spending six hours a night copying and pasting text to produce cards that pilots would consult to execute their missions. “After about two weeks there, spending six hours a night building stupid cards, I was like, there’s gotta be a better way. I was literally taking stuff off a sheet, highlighting it, and translating it. So I built an Excel macro that did it all.”In fact, many of the AOC applications ran at a similar level of sophistication. “It was endemic of the entire [AOC] that all the new tools were based on Microsoft Excel and PowerPoint, because those were the only approved solutions,” Shah said.
Even things like choosing and analyzing targets and passing targeting information to pilots in planes was done with the same tools, often requiring airmen to read data off one screen on one system and then type it into another. “They were cut and pasting, sometimes writing things down, and clearly affording themselves plenty of opportunities to make a mistake,” Lieutenant General Jeffrey Harrigian, commander of U.S. AFCENT, told me. “It was on the shoulders of our great folks that we very, very rarely had any bad data passed to the folks in the jets. But clearly there’s got to be a better way to fuse this information where you minimize the possibility of air-gapping information from one operation to another.”

Ashton Carter, then secretary of defense, speaks with reporters at DIUx headquarters, as Raj Shah looks on. [Photo: Senior Master Sgt. Adrian Cadiz]

If the AOC software hasn’t seen an appreciable upgrade in more than 20 years, it’s not for lack of trying. Lockheed Martin was initially awarded a $589 million contract in 2006 to “standardize, modernize, sustain, and transform” more than 20 AOCs around the world. Initial work to hash out an upgrade of the 1990s-era “AOC 10.1” to a networked and secure “AOC 10.2” began in 2007. But in 2013, when the Air Force solicited bids for the bulk of the work, Lockheed declined to submit one. The deal went instead to another big defense contractor, Northrop Grumman. But by the fall of 2016, when Shah and the DIB visited Qatar, Northrop was running years behind schedule and hundreds of millions of dollars over budget, leaving the future of one of the Air Force’s most important software systems uncertain, at best.The whiteboard problem resonated with Shah, who was five months into his job. DIUx locates startups and other innovative private-sector companies to help produce fast solutions to some of the challenges faced by the DoD. The scope of the tanker planning problem was just the kind of challenge Shah was looking for. And he knew that a fast-talking lieutenant colonel named Enrique Oti had been champing at the bit to tackle such a problem.

“I called Enrique literally that night,” Shah recalls.

Buying risk

A career Air Force officer, Oti was appointed head of Air Force programs at DIUx when the outfit launched in August 2015. He had already begun to sketch out concepts for an “air operation center of the future” with the help of a consultancy called BMNT, a of private-sector counterpart to DIUx led by retired colonel Pete Newell, who had headed the Army’s Rapid Equipping Force.
Shah had a more modest proposal. Instead of tackling the whole AOC, Shah suggested, “Why don’t you guys just build this tanker whiteboard? We’ll put a million bucks of DIUx money into it, but just go do this. Start tomorrow.”
Oti did. “I literally sent out a bunch of emails to all the squadron commanders I knew out there, saying, ‘Hey, does anybody have any airmen that can write code?'” he said. Within a week, Oti had dozens of responses (including some from airmen who claimed to be able to code but couldn’t). With no background in software development himself, Oti was winging it. “I figured, I’ll grab six guys and I’ll see what they can do,” he said.
Oti ran the plan by AFCENT commander Harrigian, who gave it his cautious blessing. “I had to accept the fact that we were going to try something a bit different while we were fighting a war. I had to be willing to buy some risk,” he told me.
To get Pivotal on the job fast, Oti leveraged a few contracting instruments that are normally eschewed by risk-averse Pentagon bureaucrats. With Pivotal on board, four of the Air Force coders were paired with four Pivots, as dictated by the company’s adherence to a software development methodology known as Extreme Programming.

The DIB had visited Qatar in October. By Christmas, Oti and Shah had coders and product managers back at the AOC to talk to prospective users about what they needed. By April 2017 the tanker planning tool—named JIGSAW—was in use in Qatar. “Four months from start to in production, in use in combat operations,” Oti says. The average DoD software project takes almost three years, according to a recent study.
JIGSAW cost a grand total of $2.2 million, according to Shah. The new software is not only saving time—planning now takes a matter of two to three hours, rather than all day—but is also more reliable: AFCENT now scrambles two to three fewer tankers each day—at a cost of about $250,000 each. That means the project paid for itself in under a week.
And the whiteboard on which airmen formerly did tanker planning? It’s being installed in a museum at Wright-Patterson Air Force Base outside Dayton, Ohio.

Soul searching in Silicon Valley

With JIGSAW under their belt, the team set their sights on more AOC software. But first they needed to sell the idea to the Air Force’s Life Cycle Management Center (LCMC), which is typically in change of such projects.
There was resistance, at first, to some of the ideas being presented. And other rough spots as well: Oti initially clashed with the program manager for the AOC, Colonel Jeremiah Sanders, one person told me.
“It was bumpy for a while,” says Hunter Price, director of Air Force programs at the Defense Digital Service, which helped Oti and DIUx in their campaign to win over Air Force leadership. “The program office went through a conversion process. They knew they were out in the wilderness, they knew the AOC wasn’t working well. There was some soul searching.”

That soul searching was brought on, in part, by the stalled AOC contract with Northrop Grumman, according to Sanders. “Through crisis came great opportunity,” Sanders told me. The success of the tanker planning tool “allowed us to try something really revolutionarily different.” Sanders describes the kind of iteration inherent to modern software development as a fundamental change in the DoD’s approach to software acquisition, one that “expansively transforms how we approach requirements.”
“I didn’t have a good appreciation, nor did my team when we started, of just how significant that transformation would become,” he said.

Kessel Run’s Adam Furtado welcomes new engineers to the team. [Photo: U.S. Air Force photo by Todd Maki]

Part of that transformation involves a close partnership with Pivotal, who not only helps write code for the AOC but also trains the Air Force engineers who are the backbone of the program. “We don’t want to be anybody’s outsource app development organization forever,” says Keith Salisbury, the vice president responsible for Pivotal’s work with the federal government. “We want our clients to understand what we do and how we do it, and most importantly, why we do it. We don’t want to teach them to fish. We want to teach them to be fishing guides.”After JIGSAW, the team got to work on a pair of applications that would help streamline the problems Harrigian described in assembling targeting information, deciding how and when to attack those targets, and getting that information to pilots. One of those two applications—named RAVEN and CHAINSAW—was in operation by the end of 2017, the other soon thereafter.
With offices in Boston, Chicago, and Silicon Valley, there are now about 100 people on the Kessel Run team. By this spring, Kessel Run had delivered five AOC applications and had eight more in development. The Boston operation is slated to grow to around 300 people this summer and take over an entire floor of the WeWork space. “By the end of this year I hope we’re going to be up to 15 to 18 [applications],” Oti says. It may not be the operations center of the future, but, he says, “We’re out to tackle the whole AOC.”
The name Kessel Run has now become part of a culture that the team’s leaders knew would have to differ markedly from how the military normally does business. “Culture is the hardest thing,” says Air Force Captain Bryon Kroger, Kessel Run’s COO. “One thing that’s really hard is psychological safety. Something we talk a lot about is, How do you create an environment where truly people feel safe to fail? Making that happen is hard. Adam [Furtado, Kessel Run’s chief of product] and I are both military, Colonel Oti is military, and we have these natural tendencies around command-and-control style organization, and we need the exact opposite of that.”

In addition to the name, the team chose a hashtag for its project that is a nod to both agile software development, and the fact that Kessel Run’s culture is more startup than strictly military: #agileAF
“The AF stands for Air Force,” Furtado assured me.

Unraveling the red tape

By the time the tanker planning tool went into operation, Northrop Grumman was running three years behind schedule on its overhaul of AOC software, and the budget for the project had nearly doubled, from $375 million to $745 million. In a stunning instance of bad timing, Northrop came back to the Air Force for yet more money for the AOC upgrade around the same time JIGSAW went into service. In mid-April—just as the AOC airmen were saying goodbye to their whiteboard—the Air Force issued a “stop work” order to Northrop, halting further work on the contract. By June the contract was cancelled. Northrop declined several requests for comment for this article.
“I think the Air Force and the Hill both realized that AOC 10.2 was not on a path to deliver anything, which made it pretty easy to let go,” says the DDS’s Price.
The Northrop contract envisioned building in security and networking capabilities without necessarily rebuilding the 43 apps that run on top of it, Oti says. The work of the Kessel Run team has begun to encompass that lower-level functionality, as they begin to build out an event-driven architecture to let development teams access a classified database and securely distribute information to thousands of troops.
Kessel Run is budgeted at only $68 million for fiscal 2018, according to Sanders. (Raytheon, another big defense contractor, won a $375 million contract last April to maintain AOC software over the long term, and is also working alongside Kessel Run engineers.) More money will likely be leveraged on Kessel Run’s behalf this year in the form of contracting instruments from DIUx and other sources.

When the project started, Sanders and Oti had their engineers look over the code that Northrop had delivered, but in the end they found they weren’t able to incorporate any of it into the new system. “Just because you followed the software requirements spec doesn’t mean you met the capability need,” Kroger said.
“There’s a lot of fault to go around on all of this, and we’re not sitting here trying to say the defense industrial base can’t write good software,” Oti said. “The requirements process that we are forcing on them, they’re forced to build to what we asked them for, even if they know it’s not the right thing.”
The notable thing about the decision to start working on low-level code—and about all of the team’s decisions—is that it was made on the fly, based on real-time conversations about users’ needs. That’s nothing more than best practices for modern software development, but at the DoD, such agility would normally be impossible. Specifications commonly take years to write and then more years to deliver on before code can even be tested in the field—often making systems obsolete by the time they’re delivered. “The DoD violates pretty much every rule in modern product development,” Schmidt told U.S. Congress recently.
The Air Force coders in Boston aim to change that. “Kessel Run is not just about building applications,” Oti adds. “We within the DoD need to relook at how we do requirements. We have to be able to take these technical requirements decisions and move them down to that product level, versus at the acquisitions level.”
Like software engineers, acquisition executives need the “psychological safety” that would support a break from the risk-averse DoD culture that has them tangled in red tape. “There’s plenty of additional things that can be done, even without new authorities being granted,” Schmidt said. “But there’s got to be cultural safety associated with doing that.”

[Photo: Senior Master Sgt. Adrian Cadiz]

The tip of the spear

Former Defense Secretary Ashton Carter told me Kessel Run is “a perfect example of what I wanted from the Defense Innovation Board.”

“It’s interesting for the history books that our contribution was observation,” says Eric Schmidt. “And then we left, and somebody else did the work.”
Kessel Run is providing a model for change, but it remains to be seen how broadly that model can be adopted. The Life Cycle Management Center has started trying to shift a number of other projects toward “a model of what we would call common-sense software development,” the DDS’s Price says, and other “centers of innovation” have sprung up to generate ideas. But big projects in which requirements decisions are made at the “product level” remain hard to find.
The most effective way to broaden change is to “set the right example, and don’t try to do everybody’s work for them,” according to former Carter. “This isn’t the kind of thing that it’s useful to make people do.”
But leading by example may not be enough. “We believe that technical people need to be in the room making technical decisions in places of authority,” says DDS director Chris Lynch. “Anything less than that will end up placing us back into the results we have right now.”
“Hopefully, we’re just the tip of the spear,” says Sanders, “and the Department of Defense relooks at how we do requirements, budgeting, and the acquisition process for both software and hardware systems.”
Kessel Run grew out of a moment of crisis and serendipity—and of war. Of the ability to move fast and rapidly iterate, AFCENT chief of staff Drowley said, “We’ve been granted those authorities because we’re a combatant command right now that’s in wartime operations.”

But it shouldn’t be necessary for us to rethink the Pentagon’s business practices in the midst of a war, just so we can get our fighting forces a UI that works—never mind a state-of-the-art fighter jet. I asked members of the Defense Innovation Board whether they thought it would be possible to create a military machine that could maintain an innovative edge in times of peace as well as war, and got a curious response.
“It hasn’t been peacetime in a long time,” I was told.
But as one board member said on learning that the AOC software couldn’t just be fixed: that can’t be the right answer.