The Phoenix Project

by Gene Kim

  As we walk out together in silence, I ask her, “From up here, how does it look like the release is going?”

  She says noncommittally, “You know how these things go when we’re trying to be nimble, right? There’s always unforeseen things when it comes to technology. If you want to make omelets, you’ve got to be willing to break some eggs.”

  “I think it’s a bit worse than your usual rollout. I trust you saw my e-mail, right?”

  She merely says, “Yes, of course. And you saw my reply?”

  Shit.

  I say, “No. But, before you explain, I wanted to make sure you understood the implications and the risks we’re posing to the business.” And then I repeat almost word for word what I told Steve just minutes before.

  Not surprisingly, Sarah is unimpressed. As soon as I stop talking, she says, “We’ve all been busting ass getting Phoenix this far. Marketing is ready, Development is ready. Everyone is ready but you. I’ve told you before, but apparently, you’re not listening: Perfection is the enemy of good. We’ve got to keep going.”

  Marveling at this colossal waste of time, I just shake my head and say, “No, lack of competence is the enemy of good. Mark my words. We’re going to be picking up the pieces for days, if not weeks, because of your dumb decisions.”

  As I storm back into the noc, I read Sarah’s e-mail, which makes me even more furious. I resist the urge to reply and add fuel to the fire. I also resist the emotional desire to delete it—I may need it to cover my ass later.

  From: Sarah Moulton

  To: Bill Palmer, Steve Masters

  Cc: Chris Anderson, Wes Davis, Patty McKee, William Mason

  Date: September 12, 8:15 PM

  Priority: Highest

  Subject: Re: URGENT: Phoenix deployment in major trouble—my recommendation: 1 week delay

  Everyone is ready but you. Marketing, Dev, Project Management all have given this project their all. Now it’s your turn.

  WE MUST GO!

  Sarah

  Suddenly, I panic for a brief moment that I haven’t told Paige anything for hours. I send her a quick text message:

  Night keeps getting worse. Am here for at least a couple more hrs. Will catch u in am. Love you. Wish me luck, darling.

  I feel a tap on my shoulder and turn around to see Wes. “Boss. We’ve got a very serious problem.”

  The expression on his face is enough to make me scared. I quickly stand up and follow him to the other side of the room.

  “Remember when we hit the point of no return around 9 p.m.? I’ve been tracking the progress of the Phoenix database conversion, and it’s thousands of times slower than we thought it would be. It was supposed to complete hours ago, but it’s only ten percent complete. That means all the data won’t be converted until Tuesday. We are totally screwed.”

  Maybe I’m more tired than I thought but I’m not following him. I say, “Why is this a problem?”

  Wes tries again, “That script needs to complete before the pos systems can come up. We can’t stop the script and we can’t restart it. Apparently, there’s nothing we can do to make it go faster. I think we can hack Phoenix so that it can run, but I don’t know about the in-store pos systems—we don’t have any to test with in the lab.”

  Holy crap.

  I think twice before I ask, “Brent?”

  He just shakes his head. “I had him look at it for a couple of minutes. He thinks that someone turned on database indexing too soon, which is slowing down the inserts. There’s nothing we can do about it now, though, without screwing up data. I put him back on the Phoenix deployment.”

  “How is everything else going?” I ask, wanting a full assessment of the situation. “Any improvement on performance? Any update on the database maintenance tools?”

  “Performance is still terrible,” he says. “I think there’s a huge memory leak, and that’s even without any users on it. My guys suspect we’re going to have to reboot a bunch of the servers every couple of hours just to keep it from blowing up. Damned developers…”

  He continues, “We’ve scrounged up fifteen more servers, some of them new and some yanked from various corners of the company. And now, believe it or not, we don’t have enough space in the data center racks to deploy them. We have to do a big recabling and racking job, moving crap around. Patty just put a call out and brought in a whole bunch of her people to help with that.”

  I feel my eyebrows hit my hairline in genuine surprise. And then I bend forward, laughing. I say, “Oh, dear God. We finally find servers to deploy, and now we can’t find space to put them in. Amazing. We just can’t get a break!”

  Wes shakes his head. “You know, I’ve heard stories like this from my buddies. But this may turn out to be the mother of all deployment failures.”

  He continues, “Here’s the most amazing part: We made a huge investment in virtualization, which was supposed to save us from things like this. But, when Development couldn’t fix the performance problems, they blamed the virtualization. So we had to move everything back onto physical servers!”

  And to think that Chris proposed this aggressive rollout date because virtualization would save our asses.

  I wipe my eyes and force myself to stop laughing. “And how about the database support tools the developers promised us?”

  Wes immediately stops smiling. “Absolute garbage. Our guys are going to have to manually edit the database to correct all the errors Phoenix is generating. And we’re going to have to manually trigger replenishments. We’re still learning about how much of this type of manual work Phoenix is going to require. It’s going to be very error-prone and take a ton of people to do.”

  I wince, thinking about how this will tie up even more of our guys, doing menial work that the broken application should be doing. Nothing worries auditors more than direct edits of data without audit trails and proper controls.

  “You’re doing a great job here. Our top priority is finding out what the effect of the incomplete database conversion will be on the in-store pos system. Find someone who knows those things inside and out, and get their thoughts. If necessary, call someone on Sarah’s team who handles day-to-day retail operations. Bonus points if you can get your hands on a pos device and server we can log into to see what the impact is ourselves.”

  “Got it,” Wes says, nodding. “I know just the person to put on this.”

  I watch him head off and then look around, trying to figure out where I can be the most useful.

  The morning light is starting to stream in from the windows, showing the accumulated mess of coffee cups, papers, and all sorts of other debris. In the corner, a developer is asleep under some chairs.

  I had just run to the bathroom to wash my face and wipe the grime from my teeth. I feel a little fresher, but it’s been years since I’ve pulled an all-nighter.

  Maggie Lee is the Senior Director of Retail Program Management and works for Sarah. She is kicking off the 7 a.m. emergency meeting, and there are nearly thirty people packed into the room. In a tired voice, she says, “It’s been a night of heroics, and I appreciate everyone doing what it takes to hit our Phoenix commitments.

  “As you know, the reason for this emergency meeting is that something went wrong in the database conversion,” she continues. “That means all the in-store pos systems will be down, which means that the stores will not have working cash registers. That means manual tills and manual card swipes.”

  She adds, “The good news is that the Phoenix website is up and running.” She gestures at me and says, “My thanks to Bill and the entire it Operations crew for making this happen.”

  Irritated, I say, “I’d far rather have those pos systems up instead of Phoenix. All hell is breaking loose in the noc. All our phones have been lit up for the past hour, because people in the stores are all screaming that their systems aren’t responding. It’s like the Jerry Lewis Telethon down there. Like all of you, my voicemail has already filled up from the staff in our 120 stor
es. We’re going to need to pull in more people just to man the phones.”

  A phone vibrates somewhere on the table, as if to punctuate my point.

  “We need to get proactive here,” I say to Sarah. “We need to send out a summary to everyone in the stores, as quickly as possible outlining what’s happened and more specific instructions on how to conduct operations without the pos systems.”

  Sarah momentarily looks blank, and then says, “That’s a good idea. How about you take a first cut at the e-mail, and we’ll take it from there?”

  Dumbfounded, I say, “What? I’m not a store manager! How about your group takes the first cut, and Chris and I can make sure it’s accurate.”

  Chris nods.

  Sarah looks around the room. “Okay. We’ll get something together in the next couple of hours.”

  “Are you kidding me?” I shout. “Stores on the East Coast start opening in less than an hour—we need to get something out there now!”

  “I’ll take care of it,” says Maggie, raising her hand. She immediately opens up her laptop and starts typing.

  As I squeeze my head between my hands to see if I can make my headache hurt less, I wonder how much worse this rollout could get.

  By 2 p.m. Saturday, it’s pretty clear that the bottom is a lot further down than I thought possible.

  All stores are now operating in total manual fallback mode. All sales are being processed by those manual credit card imprint machines, with the carbon paper imprints being stored in shoeboxes.

  Store managers have had employees running to the local office supply stores to find more carbon paper slips for the card imprint machines, as well as to the bank, so they could give out correct change.

  Customers using the Phoenix website are complaining about how it is either down or so slow as to be unusable. We have even managed to turn into a Twitter trending topic. All of our customers who had been excited to try our service started complaining about our big it fail after seeing our tv and newspaper ads.

  Those customers who were able to order online had a rude awakening when they went to the store to pick up their order. That’s when we discovered that Phoenix seemed to be randomly losing transactions, and in other cases, it was double- or triple-charging our customers’ credit cards.

  Furious that we’ve potentially lost integrity of the sales order data, Ann from Finance drove in and her team has now set up another war room across the hallway, fielding calls from the stores to handle problem orders. By noon, there were piles of papers from hundreds of pissed off customers that were being faxed in from the stores.

  To support Ann, Wes brought in even more engineers to create some tools for Ann’s staff to use, in order to process the ever-growing backlog of screwed up transactions.

  As I walk past the noc table for the third time, I decide that I’m too exhausted to be of use to anyone. It’s almost 2:30 p.m.

  Wes is arguing with someone across the room, so I wait until he’s done. I say to him, “Let’s face up to the fact that this is going to be a multiday ordeal. How are you holding up?”

  He yawns and replies, “I managed to get an hour of sleep. Wow, you look terrible. Go home and get a couple of hours yourself. I’ve got a handle on everything here. I’ll call you if anything comes up.”

  Too tired to argue, I thank him and leave.

  I wake with a start when I hear my cell phone ring. I bolt up and grab my phone. It’s 4:30 p.m. Wes is calling.

  I shake my head to gain some semblance of alertness and then answer, “What’s up?”

  I hear him say, “Bad news. In short, it’s all over Twitter that the Phoenix website is leaking customer credit card numbers. They’re even posting screenshots. Apparently, when you empty your shopping cart, the session crashes and displays the credit card number of the last successful order.”

  I’ve already jumped out of bed and am heading to the bathroom to get showered. “Call John. He’s going to have kittens. There’s probably some protocol for this, involving tons of paperwork and maybe even law enforcement. And probably lawyers, too.”

  Wes replies, “I already called him. He and his team are on the way in. And he is pissed. He sounded just like that dude from Pulp Fiction. He even quoted the line about the day of reckoning and striking people down with great vengeance and furious anger.”

  I laugh. I love that scene with John Travolta and Samuel L. Jackson. It’s not how I would have typecast our mild-mannered ciso, but as they say, you always have to watch out for the quiet ones.

  I take a quick shower. I run into the kitchen and grab a couple of sticks of the string cheese our son loves to eat. I take these with me in the car and start my drive back into the office.

  When I get on the highway, I call Paige. She answers on the first ring, “Darling, where have you been? I’m at work and the kids are with my mom.”

  I say, “I was actually at home for an hour. I fell asleep the instant I crawled into bed, but Wes just called. Apparently, the Phoenix application started showing the entire world people’s credit card numbers. It’s a huge security breach, so I’m driving back in right now.”

  I hear her sigh disapprovingly. “You’ve been there for over ten years and you’ve never worked these kind of hours. I’m really not sure I like this promotion.”

  “You and me both, honey…” I say.

  CHAPTER 13

  • Monday, September 15

  By Monday, the Phoenix crisis is a public fiasco. We made it onto the front-page news of all the technology sites. There are rumors that someone from The Wall Street Journal was trying to get Steve for an on-the-record interview.

  I start with a jolt when I think I hear Steve mention my name.

  Completely disoriented, I look around and realize that I’m at work and that I must have fallen asleep while waiting for the Phoenix status meeting to start. I sneak a peek at my watch. 11:04 a.m.

  I have to look at my phone to figure out that it’s Monday.

  For a moment, I wonder where my Sunday went, but seeing Steve red-faced, and addressing the entire room makes me pay attention.

  “—don’t care one bit whose fault this is. You can bet your ass that this won’t ever happen again on my watch. But right now, I don’t give two shits about the future—we are massively screwing our customers and shareholders. All I want to hear about is how we’re going to dig ourselves out of this hole and restore normal business operations.”

  He turns and points at Sarah saying, “And you are not off the hook until every one of your store managers says that they can transact normally. Manual card swipes? What are we, in some Third World country?”

  Sarah replies calmly, “I totally understand how unacceptable this is. I’m making sure my entire staff knows that they are accountable and responsible.”

  “No,” Steve responds quickly and gravely. “You are ultimately accountable and responsible. Do not forget that.”

  My heart actually lightens for a moment, as I wonder whether Steve has broken free of Sarah’s spell.

  Turning his attention back to the entire room he says gravely, “When the store managers say that we’re no longer operating on life support, I need fifteen minutes from each and every person who had a hand in this. I expect you to clear your calendar. No excuses.

  “That means you, Sarah, Chris, Bill, Kirsten, Ann. And even you, John,” he says, pointing at people as he names them.

  Way to go, John. You picked a great time to finally get noticed by Steve.

  He continues, “I’ll be back in two hours after I get on a phone call with another journalist because of this mess!”

  His door slam shakes the walls.

  Sarah breaks the silence. “Well, you all heard Steve. Not only do we need to get the pos systems up, but we must also get the Phoenix usability issues fixed. The press is having a heyday with the clunkiness of the ordering interface and everything timing out.”

  “Are you out of your mind?” I say, leaning forward. “We are keepi
ng Phoenix alive by sheer heroics. Wes wasn’t joking when he said that we’re proactively rebooting all the front-end servers every hour. We can’t introduce any more instabilities. I propose code rollouts only twice a day and restricting all code changes to those affecting performance.”

  To my surprise, Chris immediately chimes in, “I agree. William, what do you think?”

  William nods. “Absolutely. I suggest we announce to the developers that all code commits must have a defect number that corresponds to a performance problem. Anything that doesn’t will get rejected.”

  Chris says, “That good enough for you, Bill?”

  Pleased with the solution, I say, “Perfect.”

  Although Wes and Patty seem simultaneously pleased and taken aback by this sudden cooperation from Development, Sarah is not pleased. She says, “I don’t agree. We’ve got to be able to respond to the market, and the market is telling us that Phoenix is too hard to use. We can’t afford to screw this up.”

  Chris replies, “Look, the time for usability testing and validation was months ago. If we didn’t get it right the first time, we’re not going to get it right without some real work. Have your product managers work on their revised mockups and proposals. We’ll try to get it in as soon as we can after the crisis is over.”

  I affirm his position, saying, “I agree.”

  “You raise some good points. I approve,” she says, apparently realizing that she wasn’t going to win this argument.

  I’m not sure Sarah is actually in a position to approve anything. But, luckily, the discussion turns quickly to how to regain pos functionality.

  I revise my opinion of Chris upward a couple of notches. I still think he was a willing accomplice of Sarah’s, but maybe I’ll give him the benefit of the doubt.

  Leaving the Phoenix war room, I see the room across the hallway where Ann and her team handle problem orders. I’m overcome by a sudden curiosity, genuinely wanting to see how they’re doing.

  I knock and walk in, still chewing a stale bagel from the meeting. Since Saturday, there has been an endless supply of pizzas, pastries, Jolt colas, and coffee to keep all the troops at their tasks.