by Steven Brill
Burt was the chief technology officer at Civis Analytics in Chicago. That’s the company, funded by Google’s Eric Schmidt, whose data analysts and technologists had achieved rock star status in the Obama 2012 campaign, and whose data Enroll America’s Anne Filipic was already using to target the uninsured young invincibles.
Park, the White House CTO, had connected to Civis via the White House political office. Only now, the White House was interested in the other side of the house at Civis—its technology whiz kids, not its analysts who had been crunching demographic targeting data to feed potential enrollees to Filipic’s now-frustrated field-workers.
On October 11, when Park had called from the White House, Mikey Dickerson—who worked at Google as a “site reliability manager”—had been in town visiting Burt and Civis CEO Dan Wagner. They were friends and Obama 2012 campaign alums. Dickerson, who was thirty-four at the time and favored cartoon-decorated T-shirts over suits and ties, had taken a leave from Google in 2012 to help with scaling the Obama campaign website and to create its Election Day turnout-reporting software.
“I consider Mikey a mentor,” Burt would later tell me. “We were picking his brain about our company when Dan got a call about the healthcare site.… We all wanted to do something.” Dickerson agreed that weekend to join Burt in a rescue squad.
GEEKS IN A VAN
By the afternoon of October 18, 2013, Burt had left his room at the Doubletree and was at the Maryland headquarters of QSSI, one of the contractors that had been hired by CMS to build and run the website that was now eighteen days into becoming a national debacle. Of the many contractors that had worked on HealthCare.gov, QSSI—which was responsible for, among other things, the successful connection of the website to the IRS database—was thought to have performed the least badly, and certainly better than CGI.
That same afternoon, Mikey Dickerson, the Google engineer, who was in California preparing to fly east the following Monday to join Burt, jumped on what he later described as a “really bizarre conference call.”
The call was led by the now-energized Todd Park. Park was riding in a White House van around D.C., Maryland, and Virginia with the beginnings of a hastily assembled team trying to assess the damage.
Also in the van was Paul Smith, another Chicago-based techie whom Burt had recruited. Smith, thirty-six, had been deputy director of the Democratic National Committee’s tech operation. He immediately put fund-raising for a start-up he was planning on hold to join the group.
Another passenger was Ryan Panchadsaram, twenty-eight, whose title was senior adviser to Park. He had come to the White House as part of a program called Presidential Innovation Fellows, which was launched by Park to bring high-tech achievers into government to work on specific projects that they design. (The program was already responsible for a series of innovations in making government data and healthcare records more available electronically.)
“I decided we should all go introduce ourselves to the people we were going to help,” Park later recalled, explaining the van ride.
The team started by driving from the White House to meet Marilyn Tavenner, the CMS administrator, at her Washington office. She was thrilled to see them.
They then drove off to Baltimore to meet other senior CMS officials. It was during that drive that Park decided to loop Dickerson and some others into a rolling conference call. “We were passing around an iPhone with a speaker so we could all talk,” Park said. “I wanted us to get to know each other.”
“I had no idea who this guy leading the call was, and couldn’t hear a lot of it,” recalled Dickerson. “Finally, I jumped in and asked, ‘Who am I talking to? Who is leading this call?’ And the guy says, ‘I’m Todd Park.’ So I Googled him and saw he’s the chief technology officer of the country and had founded two healthcare technology companies. Oh, I figured. Not bad.”
Park’s van continued on from Baltimore, stopping at the two main contractors working on the website. It turned out that the engineers at both QSSI and even CGI were not as defensive or hostile as Park and the others had feared.
“These guys want to fix things. They’re engineers, and they were embarrassed,” recalled one of the members of Park’s gathering band. “Their bosses might have been turf conscious, but by then the guys in the suits really didn’t want to have anything to do with the site, so they were glad to let us take over.”
When the meetings with the contractors ended at a CMS outpost in Herndon, Virginia, at about 7 P.M., the rescue squad already on the scene decided they had more work to do. One of the things that had shocked Burt and Park’s team most—“among many jaw-dropping aspects of what we found” is how one rescue squad member later put it—was that the people running HealthCare.gov had no “dashboard,” no quick way for engineers to measure what was going on on the website, such as how many people were using it, what the response times were for various click-throughs, or where traffic was getting tied up.
So late into the night of October 18, Burt and the others spent about five hours coding and putting up a dashboard.
What they saw, recalled Park, was a site with wild gyrations. “It looked awfully spikey,” Ryan Panchadsaram, the White House innovation fellow, remembered. “The question was whether we could ride that bull. Could we fix it?”
The team went home at about 2:30 A.M. on Saturday, October 19.
CALLING SILICON VALLEY
At the White House, the decision had still not been made whether to save or scrap HealthCare.gov. Jeffrey Zients wanted still more eyes from Silicon Valley on the problem. At about six in the morning on Saturday, October 19, 2013, he emailed John Doerr, a senior partner at Kleiner, Perkins, Caufield & Byers, the Menlo Park–based venture capital powerhouse whose investments include Amazon, Google, Sun, Intuit, and Twitter. Could Doerr call him when he awoke to talk about the healthcare website? Zients asked.
When Doerr quickly called back, Zients said, “We’re pulling together this surge of people to do this assessment to see if the site’s fixable or not. We’ve got to do it incredibly quickly. Do you know anyone?” Doerr recommended a relatively new Kleiner, Perkins partner named Mike Abbott, the former chief technology officer at Twitter.
“Mike saved Twitter’s technology when it was failing,” Doerr told me later, referring to the days when the Twitter “fail whale” error message icon was ubiquitous. “His being there gave me the confidence to make the largest investment we had ever made—over $100 million.… He had also worked at Microsoft and led the team at Palm that rebuilt their system.… Yet he’s really low-key and well-liked.”
Abbott, who was forty-one at the time, spoke to Zients the next day, Sunday, October 20. He flew to Washington on October 21.
That day, Obama offered what The New York Times called “an impassioned defense of the Affordable Care Act” in a Rose Garden statement, “acknowledging the technical failures of the HealthCare.gov website, but providing little new information about the problems with the online portal or the efforts by government contractors to fix it.”
However, Obama did mention, without elaborating, that “we’ve had some of the best IT talent in the entire country join the team. And we’re well into a ‘tech surge’ to fix the problem.”
The president did not volunteer that his new team’s first job was to decide whether to kill the website altogether and start over.
“The first red flag you look for,” explained Abbott, the former Twitter CTO, “is whether there is a willingness by the people there to have outside help. If not, then I’d say it’s simpler to write it new than to understand the code base as it is if the people who wrote it are not cooperating. But they were eager to cooperate.
“The second thing, of course, was, what were the tech problems,” Abbott continued. “Were they beyond repair? Nothing I saw was beyond repair. Yes, it was messed up. Software wasn’t built to talk to other software, stuff like that.… A lot of that was because they had made the most basic mistake you can ever make. The gove
rnment is not used to shipping products to consumers. You never open a service like this to everyone at once. You open it in small concentric circles and expand”—such as one state first, then a few more—“so you can watch it, fix it and scale it.”
What Abbott could not find, however, was leadership. He was never able to figure out who was supposed to have been running the HealthCare.gov launch. Instead he saw multiple contractors bickering and no one taking ownership for anything. Someone would have to be put in charge, he told Zients. Beyond that, Abbott recalled, “there was a lack of urgency” despite the fact that the website was becoming a national joke and crippling the Obama presidency.
By then, Mikey Dickerson—the Google reliability guru and Burt mentor—had arrived. “I knew Mikey by reputation,” Abbott recalled. “He was a natural fit to lead this team.”
Looking over the dashboard that Park, Burt, and the others had rigged up the prior Friday night, Abbott and the group discovered what they thought was the lowest hanging fruit—a quick fix to an obvious mistake that could improve things immediately. HealthCare.gov had been constructed so that every time a user had to get information from the website’s vast database, the website had to make what was called a query into that database. Well-constructed, high-volume sites, especially e-commerce sites, will instead assemble the most frequently accessed information, such as the summaries of the insurance plans, a layer above the entire database, in what is called a cache. That way, the query can be faster and not tie up the connections to the overall database. Not doing that created a huge, unnecessary bottleneck. It was like slowing down traffic on an entry ramp to an otherwise empty highway.
The team began almost immediately to cache the data. The result was encouraging: The site’s overall response time—the time it takes a page to load—dropped on the evening of October 22 from eight seconds to two. That was still terrible, but it represented so much of an improvement that it cheered the engineers. They could see that HealthCare.gov could be saved instead of scrapped.
Also weighing in by this time on the phone and through chat lines was another Silicon Valley legend recruited by Zients, who also happened to be named Abbott. Marty Abbott had been the CTO of eBay and now ran a consulting business that offered high-tech crisis management and evaluation. Venture funds paid him “tens of thousands of dollars a day,” said Zients, to kick the tires, hard, of potential companies seeking their money. The companies themselves hired him when their websites or other technology crashed.
“It was pretty obvious from the first look that the system hadn’t been designed to work right,” Marty Abbott later told me. “It was not really managed at all, and wasn’t architected to scale.”
Marty Abbott volunteered his time, which would be limited to participation in multiple conference calls in the first few weeks of the salvage effort. Mike Abbott was also a volunteer; he ended up staying in the D.C. area until October 23, then would participate through December on conference calls, sometimes doing two or three a day.
Mikey Dickerson, Gabriel Burt, and the others—who had arrived for what they thought would be a few days, but would end up staying eight to eleven weeks—were told that government regulations did not allow them, even though they offered, to be volunteers if they worked for any sustained period. So they were put on the payroll of contractor QSSI as hourly workers, making what Dickerson said was “a fraction” of his Google pay.
On Wednesday, October 23, the day after their first breakthrough with the caching, Abbott, Dickerson, and the rest of the team gave Zients and Park their verdict: They could fix the site by the end of November, six weeks away, so that “the vast majority” of visitors could go on and enroll.
“I was, like, never worried,” Dickerson later explained. “It’s just a website. We’re not going to the moon.”
A few hours later on the afternoon of October 23, Zients and chief of staff McDonough told the president the news.
There was one additional, ironic piece of the plan: The general contractor Zients and Park had chosen to coordinate things, they told the president, was QSSI, which had handled some of the more successful functions of the ailing website. QSSI’s parent company was UnitedHealth Group, the giant insurer. Which meant that the largest player in an industry that had opposed Obamacare in 2010 was now about to take a lead role in saving it. And profiting from it.
That same day, another insurer, WellPoint, told stock analysts during a conference call that it was raising its earnings estimates for the year and that interest in the Obamacare exchanges appeared “robust.”
STAND-UPS AND HICCUPS
It was above a “Smile Center” dental clinic in a nondescript office building in Columbia, Maryland, in a four-thousand-square-foot room rented by QSSI—its walls lined with Samsung TV monitors showing various dashboard readings—that the rescue of Barack Obama’s healthcare reform law would be attempted.
All the years of Ted Kennedy’s crusading, all of Liz Fowler’s drafting, all of the days wrangling votes on Capitol Hill in 2009 and 2010, all of the backroom deals and furious lobbying by all of those industry players, all of the frantic efforts to game the CBO scoring process, all of the millions of hours and billions of dollars spent writing regulations and building the almost junked website—the fate of all of it—now depended on Mikey Dickerson’s stand-ups.
Stand-ups, which became a standard part of Mike Abbott’s playbook at Twitter, are Silicon Valley–style meetings where everyone usually stands, rather than sits, and works through a problem, or a set of problems, fast. The group then disperses, acts, and reports back at the end of the day, during a second stand-up.
Put differently, they were not like any of the meetings held in the CMS war rooms.
Dickerson held the first of his stand-ups on October 24, 2013. Until the end of November he would convene them every day, including weekends, at 10 in the morning and 6:30 in the evening.
Each typically ran about forty-five minutes (“causing some of us to sit down,” Dickerson conceded). An open phone line connected people working on the website at other locations; in fact, the open line remained live twenty-four hours a day so that everyone could instantly talk to the others if an issue suddenly came up.
Dickerson immediately established the rules, which he posted on a wall just outside the control center.
Rule 1: “The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame.”
Rule 2: “The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it.” Explained Dickerson later, “If you can get the managers out of the way, the engineers will want to solve things.”
Rule 3 was equally blunt: “We need to stay focused on the most urgent issues, like those that will hurt us in the next 24–48 hours.”
A CONGRESSIONAL INQUISITION
As Mikey Dickerson convened that first stand-up, Congressman Fred Upton, the Republican chairman of the House Energy and Commerce Committee, began what would become a series of brutal congressional hearings on what to everyone (except those in the room with Dickerson, above the Smile Center) had become the hopeless website debacle.
With Democratic committee members, including healthcare reform champion Henry Waxman, left with little more to say than that the law was already helping millions of Americans because of provisions unrelated to the website, Upton and his Republican colleagues were unsparing.
“Over the past several months leading up to the October first launch, top administration officials and lead contractors appeared before this committee, looked us in the eye, and assured us repeatedly that everything was on track,” Upton began. “Except that it wasn’t, as we now know too well. So, why did they assure us that the website would work? Did th
ey not know, or did they not disclose?”
Upton’s witnesses were representatives of four of the contractors, including CGI as well as QSSI, the UnitedHealthcare subsidiary that Zients and Park had just chosen, unbeknownst to the committee, to coordinate the rebuild being done by Dickerson and his crew.
Each witness testified that the decision to launch the site on October 1, 2013, had been made by their client, CMS, not by them.
“Prior to October first,” Upton asked the group, “did you not know that the HealthCare.gov website was going to have crippling problems, or did you know about these problems and choose not to disclose them to the administration?”
Cheryl Campbell, CGI’s leader on the website project, offered the first answer: “So, Chairman, from a CGI perspective, our portion of the application worked as designed. People have been able to enroll. Not at the pace—not at the experience we would have liked, but the end-to-end testing was the responsibility of CMS.… Our portion of the system … was ready to go live, but it was not our decision to go live.”
The three others provided much the same answer.
At the same October 24 hearing, Upton alluded to another problem that he said his constituents had begun contacting his office about: They were getting notices from their health insurance companies that their policies had to be canceled beginning January 1 because the new law no longer allowed them to be sold.
Readers of the Federal Register on June 17, 2010, would have known that, because of Jeanne Lambrew’s tough grandfathering rules, millions were going to have their policies canceled. The Federal Register notice had said so. But neither Upton nor his constituents were devotees of the Federal Register.
“The president promised Americans that they could keep their health plan if they liked them, no matter what,” Upton noted. “Yet, here we are, twenty-four days into open enrollment, and more people are receiving cancellation notices in just two states than the … Americans that the administration boasts have … begun applying in the entire country. This is a troubling fact, but we still don’t know the real picture, as the administration appears allergic to transparency, and continues to withhold enrollment figures.”