The Phoenix Project

by Gene Kim

  Despite how tired I am, I’m grateful to be at home early enough to resume my nighttime ritual with my older son.

  “Yes, we need to find out, Daddy,” Grant demands. I smile at him, and I dig my phone out from my pocket, intending to do a Google search for “tank engine tender car.”

  But first, I quickly scan my phone for any new updates on the customer invoicing problem. I’m amazed at the difference two weeks can make.

  During the last Sev 1 incident that hit our credit card processing systems, the conference call was full of finger-pointing, denials, and, most importantly, wasted time when our customers couldn’t give us money.

  Afterward, we did the first of a series of ongoing blameless postmortems to figure out what really happened and come up with ideas on how to prevent it from happening again. Better yet, Patty led a series of mock incident calls with all hands on deck, to rehearse the new procedures.

  It was terrific to watch. Even Wes saw the value.

  I’m pleased to see all the e-mails indicating a lot of good information and effective discussion among the teams working the problem. They’ve kept the telephone conference bridge and a chat room open for people working the issue, and I plan on calling in at 10 p.m. to see how it’s going.

  That’s forty-five minutes from now. Plenty of time to spend with Grant, who should be falling asleep soon.

  He nudges me, obviously expecting more progress on the research front.

  “Sorry, Granty. Daddy got distracted,” I say as I open up the browser. I’m surprised by how many of the search results are all about Thomas the Tank Engine. It’s the book series that spawned a multibillion dollar franchise of toy trains, clothing, videos, and coloring books. With two sons, we seem destined to own two of every item soon.

  I’m reading a promising Wikipedia entry on trains when my phone starts vibrating and the screen displays “Call from Steve Masters.”

  I groan and double-check my watch. It’s 9:15 p.m.

  I’ve had way too many meetings and phone calls with Steve lately. In my head, I wonder how many of these meetings I can take.

  On the other hand, after the Phoenix debacle, every outage and incident is trivial in comparison, right?

  I say gently, “Hang on, Grant. Daddy has to take a phone call. I’ll be right back.” I jump out of his bed and walk into the dark hallway.

  I’m glad I had just scanned through all the e-mail traffic on the outage just seconds before. I take a deep breath before I hit the button to answer the call.

  I say, “Bill here.”

  Steve’s loud voice booms in my ear. “Evening, Bill. I’m glad you’re there. Of course you know about the customer invoicing problems from Dick?”

  “Yes, of course,” I reply, surprised at his tone. “My team declared a major incident early this afternoon and we’ve been working this issue ever since. I’ve been sending out status reports every hour. Dick and I spent twenty minutes on the phone earlier this evening. I know the problem is serious, and my team is following the process we’ve created after the payroll failure. I’m completely satisfied that the process is working.”

  “Well I just got off the phone with Dick, and he tells me that you’re dragging your feet,” says Steve, clearly very angry. “Obviously, I’m not calling you at night because I want to chitchat. Do you understand how intolerable this is? Yet another it screwup jeopardizing everything. Cash is the lifeblood of the company, and if we can’t invoice customers, we can’t get paid!”

  Falling back on old training on handling frustrated people, I calmly reiterate what I already stated. “As I said, I talked with Dick earlier today. He very much impressed upon me all the implications. We’ve activated our new incident process, and we’re methodically looking into what could have caused the failure. They’re doing exactly what I want them to, because with so many moving pieces, it’s way too easy to make things worse by jumping to conclusions—”

  “Are you in the office?” Steve demands, cutting me off before I could finish.

  His question genuinely catches me off guard.

  “Uh… No, I’m at home,” I answer.

  Is he worried that I’ve delegated the problem away? To reinforce my role in handling the crisis and what my expectations from my team are, I say, “I will be calling into the war bridge line at ten o’clock. As always, we have a duty officer on site, and those on my staff who need to be in the office are there already.”

  Finally, I ask bluntly, “Steve, want to tell me what’s on your mind? I’m on top of this situation. What do you need that you aren’t getting right now?”

  He responds hotly, “What I need from you is some sense of urgency. Dick and his team are burning the midnight oil trying to figure out how our quarter will end up in six working days. But I think I already know what the answer will be.”

  He continues, “We’ll probably miss almost every target that we’ve promised the board: revenue, cash, receivables—everything. In fact, every measure we’ve promised the board is going the wrong way! This screwup may confirm the board’s suspicion that we’ve completely lost control of managing this company!”

  Steve is almost snarling now as he says, “So, what I want from you, Bill, is to stay sufficiently on top of things, so that I don’t have my cfo saying that you’re dragging your feet. The house is burning down, and all I hear from you is about drawing pictures and timelines. What in the hell is wrong with you? You afraid to get people out of bed?”

  I start again, “Steve, if I thought it would help, I’d have everyone pull all-nighters in the data center tonight. For Phoenix, some people didn’t go home for nearly a week. Trust me, I know the house is on fire, but right now, more than anything, we need situational awareness. Before we send the teams crashing through the front door with fire hoses, we have to have someone at least quickly walk the perimeter of the yard—otherwise, we’ll end up burning down the houses next door!”

  I realize that I’ve raised my voice in the relative quiet of our house as we’re trying to get the kids to sleep. I resume, more quietly, “And just in case you forgot, during the payroll outage, we made the outage worse by our own actions. We probably could have completed the payroll run during the business day if someone hadn’t started screwing with the san. Because of that, we added another six hours to the outage, and we nearly lost payroll data!”

  My hopes that the calm voice of reason is reaching him are dashed when I hear him say, “Oh, yeah? I don’t think your team agrees with you. What was the name of that smart guy who you introduced me to? Bob? No, Brent. I talked with Brent earlier today, and he’s very skeptical of your approach. He thinks what you’re doing is separating people who actually do the work from what needs to get done. What is Brent doing right now?”

  Shit.

  I like transparency. I always try to make my team totally accessible to my boss and the business. But there’s always risk in doing this.

  Like having Brent spout off his crazy theories to the ceo.

  “I hope Brent is at home, because that’s exactly where he should be,” I respond. “Until we know for sure exactly what went wrong, that’s where I want him. Look, it’s rocket scientists like him that often cause the problem in the first place. Every time we escalate to Brent, we perpetuate our reliance on him, and make it that much less likely we can fix things without him!”

  Suspecting that I may be losing Steve, I start again. “The chaotic way we currently work, Brent is having to fix the punctured hulls almost every day. I’m pretty sure, though, that Brent is one of the main reasons the hull is punctured in the first place. It’s not malicious, of course, but it’s just a side effect of the way we work and fix outages here.”

  There is a pause. Then he says slowly and decisively, “I’m glad you’re being so professorial about this, but we’ve got a wildfire that’s out of control. Up until now, we’ve done it your way. And now we’re going to do it my way.

  “I want you to call Brent in, and I want him to roll
up his sleeves and help fix this outage. And not just Brent. I want all eyeballs on screens and all hands on keyboards. I’m Captain Kirk. You’re Scotty. And I need warp speed, so get your lazy engineers off their asses! Do you understand me?”

  Steve is yelling so loudly by now that I’m holding the phone away from my ear.

  Suddenly, I’m furious. Steve is going to screw this up again.

  Recalling my days in the Marines, I finally say, “Permission to speak freely, sir?”

  I hear Steve on the other end of the line snort dismissively in response. “Yes, dammit.”

  “You think I’m being overly cautious, and that I’m hesitating to do what needs to be done. But you are wrong. Dead wrong,” I say adamantly. “If you do what I think you’re suggesting, which is basically ‘all hands on deck,’ I predict that we’re going to make things much worse.”

  I continue, “I tried to advise you of something very similar before the Phoenix launch. Up until now, we have not been sufficiently disciplined in how we work outages. Given all the complexity and moving pieces, there’s too much likelihood of causing another problem. I may not know exactly what caused the customer invoicing issue, but I know enough to absolutely conclude that what you’re proposing is a very bad idea. I recommend continuing along the lines I am currently prosecuting.”

  I hold my breath, waiting to hear how he reacts.

  He says slowly, “I’m sorry you feel that way, Bill, but the drawers open on my side of the desk. I’m telling you that it’s now defcon 1, so go get the smartest people working on this problem. And I want status updates on this it failure every two hours until it’s fixed. Understood?”

  Before I can think about what to say, I find myself saying, “I don’t know why you need me to do that. You’re talking directly to my people, and you’re calling all shots on the ground. Do it yourself. I can’t be held responsible for the results of this fubar situation.”

  And before I hang up on him, I say with finality, “And expect my resignation in the morning.”

  I wipe the sweat off my forehead, and look up from my phone to see my wife Paige staring at me wide-eyed.

  “Are you insane? You just quit? Just like that? How are we going to pay the bills now?” she asks, her voice rising.

  I turn the ringer off on my phone and put it back in my pocket, saying, “Honey, I’m not sure how much of that conversation you heard, but let me explain…”

  Part 2

  CHAPTER 17

  • Monday, September 22

  In the four days since quitting, Paige has been fretting endlessly. On the other hand, I’m amazed at how much better I’m sleeping at night, as if some huge, hidden weight has been lifted from my shoulders.

  Uninterrupted by e-mails or emergency pages, the weekend was incredibly peaceful. I was still receiving them on Thursday, but I just deleted the e-mail accounts and blocked the text messages.

  It felt great.

  I tell Paige not to take Grant to her mother’s. Instead, I’m taking him on an adventure. Paige reacts with a bemused smile and helps me pack his Thomas the Tank Engine backpack.

  By 8 a.m., we’re out of the house and heading happily to the train station, where, for months, I’ve been promising to take him. For an hour we watch trains go by, and I’m continually amazed at Grant’s unabashed joy. Despite the uncertainty around what I’ll be doing next, I feel blessed that I can share this moment with Grant.

  As I’m taking pictures of Grant screaming with delight and pointing at the diesel trains going by, I realize how few pictures I’ve taken of either of my kids in the last month.

  We’re still watching the trains when my phone rings. It’s Wes. I let it roll to voicemail.

  He calls several more times, and each time he leaves another voicemail.

  Then Patty calls, which I let roll to voicemail, too. After three more calls, I mutter in exasperation, “Come on, guys…”

  “Palmer,” I answer the phone.

  “Bill, we just heard the news from Steve,” I hear Patty say, sounding like she’s on a speakerphone. With surprising anger in her voice, she continues, “I’ve got Wes here, and we’re both completely shocked. We knew something wasn’t right when you didn’t show up for our regular cab meeting on Friday. I just can’t believe you resigned during this outage—and after everything we’ve achieved!”

  “Look, guys, it has nothing to do with you,” I explain. “Steve and I just had some irreconcilable differences about how to resolve the big invoicing failure. I’m sure you guys will do fine without me.”

  As I say the last part, I feel slightly disingenuous.

  “Well, we’ve pretty much screwed the pooch since you’ve left,” Wes says, sounding genuinely abashed, confirming my worst fears. “Steve insisted that we bring in all the engineers, including Brent. He said he wanted a ‘sense of urgency’ and ‘hands on keyboards, not people sitting on the bench.’ Obviously, we didn’t do a good enough job coordinating everyone’s efforts, and…”

  Wes doesn’t finish his sentence. Patty picks up where he left off, “We don’t know for sure, but at the very least, the inventory management systems are now completely down, too. No one can get inventory levels in the plants or warehouses, and they don’t know which raw materials we need to replenish. All the finance guys are about to jump from window ledges, because they may not be able to close the books for the quarter on time. With all these systems down, no one has the data they need to compute cost of goods sold, gross profit, and net margin.”

  “Holy shit.” Speechless for a moment, I finally say, “That’s incredible.”

  Grant grabs at my phone, demanding my attention. I say, “Look, guys, I’m with my son, and we’re in the middle of something important. I can’t talk for very long. But rest assured that I’m really proud of everything that we’ve done together, and I know that you guys can get through this crisis without me.”

  “That’s a load of junk, and you know it,” Patty says. “How can you leave us in the lurch like this? We have so many things that we planned on fixing together, and you’re leaving it all completely unfinished! I never figured you as someone who would quit like this!”

  “I agree. Leaving now is pretty shitty, if you ask me,” Wes says, chiming in.

  I sigh. I’m never going to tell them about all the frustrating and absurd meetings I’ve had to put up with Steve. That’s between him and me.

  “I’m sorry to let you down, but it’s something that I had to do,” I say. “You’ll do just fine. Just don’t let Steve or anyone else micromanage you. No one knows the it systems like you guys do, so don’t let anyone try to call the shots, okay?”

  I hear Wes mutter, “Too late for that.”

  By now, Grant is trying to hang up my phone. “Guys, I’ve got to run. We’ll catch up later, okay? Over beers.”

  “Yeah, sure,” Wes says.

  “Gee, thanks for everything,” Patty says. “Catch you around.”

  With that, the line disconnects.

  I let out a long sigh. Then, looking at Grant, I put away my phone and give him my full attention again, intent on recapturing our moment of happiness before it was interrupted.

  My phone rings again on our drive home. Grant is asleep in the backseat. This time, it’s Steve.

  Having no interest in talking with him just yet, I let it go to voicemail. Three times.

  I pull into our garage and get out of the car, trying to get Grant out of his car seat without waking him up. As I walk through the house with him, I see Paige. I point to Grant, silently mouthing to her, “Asleep.” I pad softly up the stairs, at last transferring him to his bed and taking off his shoes.

  With a sigh of relief, I close the door behind me and walk back downstairs.

  When Paige sees me, she says, “That bastard Steve called me this morning. I almost hung up on him, but he gave me a long story about doing all this soul-searching with some guy named Erik. He says he has a proposition for you. I told him I’d pass a
long the message.”

  When I roll my eyes, she says in a suddenly concerned voice, “Look, I know you resigned because you felt it was the right thing to do. But you know as well as I do that there aren’t many other companies in town that pay as well as Parts Unlimited. Especially after your promotion. I don’t want to move away from my family.”

  She looks levelly at me. “Honey, I know he’s a bastard, but we both still need to earn a living. Promise me that you’ll listen to what Steve has to say and keep an open mind, okay? Bill? Okay?”

  I merely nod and step into the dining room, hitting the speed dial for Steve.

  Steve answers his cell phone on the first ring. “Good afternoon, Bill. Thanks for calling me back. I had the pleasure of talking with your wife, telling her all about what a jackass I’ve been.”

  “Yeah, she said something to that effect,” I respond. “She said that you really wanted to talk.”

  I hear him say, “Look, I wanted to apologize for the way I’ve behaved since you graciously accepted my request to become our vp of it Operations. Dick thought I was crazy when I told him that I was going to have it report to me. But I told him about how, when I first became a plant manager, many decades ago, I worked on the assembly line for a month, just to make sure that I understood the ins and outs of daily life of everyone who worked there.

  “I promised Dick that I would get my hands dirty and not just delegate the problem away. I’m angry with myself that I haven’t lived up to that promise. And delegating all the it issues to Sarah was a total screwup.

  “Listen, I know I haven’t been fair to you, especially when you’ve fulfilled your end of the bargain. You’ve been a straight shooter, and you’ve genuinely tried to prevent bad things from happening.”

  He pauses for a couple of moments. “Look, I just got kicked in the ass by Erik and by the entire audit committee. He held my feet to the fire until I finally understood something. It made me realize that I’ve been doing something really wrong for many years, and I want to make it right.