The Phoenix Project

by Gene Kim

  It is apparently a convincing argument, as Wes says, “Okay, he’s all yours. I almost feel sorry for him now.”

  I suddenly regret my choice of words. This isn’t a witch hunt, and I’m not looking for retribution. We still need a timeline of all relevant events leading up to the failure.

  Jumping to inappropriate conclusions caused the san failure last night. We won’t make these kinds of mistakes again. Not on my watch.

  As Patty and I call John, I squint at the phone number on Patty’s screen, wondering if it’s time to heed my wife’s advice to get glasses. Yet another reminder that forty is just around the corner.

  I dial the number, and a voice answers in one ring, “John here.”

  I quickly tell him about the payroll and san failure and then ask, “Did you make any changes to the timekeeping application yesterday?”

  He says, “That sounds bad, but I can assure you that we didn’t make any changes to your midrange systems. Sorry I can’t be of more help.”

  I sigh. I thought that by now either Steve or Laura would have sent out the announcement of my promotion. I seem destined to explain my new role in every interaction I have.

  I wonder if it would be easier if I just sent out the announcement myself.

  I repeat the abridged story of my hasty promotion yet again. “Wes, Patty, and I heard that you were working with Max to deploy something urgent yesterday. What was it?”

  “Luke and Damon are gone?” John sounds surprised. “I never thought that Steve would actually fire both of them over a compliance audit finding. But who knows? Maybe things are finally starting to change around here. Let this be a lesson to you, Bill. You Operations people can’t keep dragging your feet on security issues anymore! Just some friendly advice…

  “Speaking of which, I’m suspicious about how the competition keeps getting the jump on us,” he continues. “As they say, once is coincidence. Twice is happenstance. Third must be enemy action. Maybe our salespeople’s e-mail systems have been hacked. That would sure explain why we’re losing so many deals.”

  John continues to talk, but my mind is still stuck at his suggestion that Luke and Damon may have been fired over something security related. It’s possible—John routinely deals with some pretty powerful people, like Steve and the board as well as the internal and external auditors.

  However, I’m certain Steve didn’t mention either John or Information Security as reasons for their departure—only the need to focus on Phoenix.

  I look at Patty questioningly. She just rolls her eyes and then twirls her finger around her ear. Clearly, she thinks John’s theory is crazy.

  “Has Steve given you any insights on the new org structure?” I ask out of genuine curiosity—John is always complaining that information security was always prioritized too low. He’s been lobbying to become a peer of the cio, saying it would resolve an inherent conflict of interest. To my knowledge, he hadn’t succeeded.

  It’s no secret that Luke and Damon sidelined John as much as possible, so he couldn’t interfere with people who did real work. John still managed to show up at meetings, despite their best efforts.

  “What? I have no clue what’s going on,” he says in an aggrieved tone, my question apparently striking a nerve. “I’m being kept in the dark, like usual. I’ll probably be the last to find out, too, if history is any guide. Until you told me, I thought I was still reporting to Luke. And now that he’s gone, I don’t know who I’m reporting to. You got a call from Steve?”

  “This is all above my pay grade—I’m as much in the dark as you are,” I respond, playing it dumb. Quickly changing the subject, I ask, “What can you tell us about the timekeeping app change?”

  “I’ll call Steve and find out what’s going on. He’s probably forgotten Information Security even exists,” he continues, making me wonder whether we’ll ever be able to talk about payroll.

  To my relief, he finally says, “Okay, yeah, you were asking about Max. We had an urgent audit issue around storage of pii—that is, personally identifiable information like ssns—that’s Social Security numbers, obviously, birthdays, and so forth. European Union law and now many US state laws prohibit us from storing that kind of data. We got a huge audit finding around this. I knew it was up to my team to save this company from itself and prevent us from getting dinged again. That would be front-page news, you know?”

  He continues, “We found a product that tokenized this information, so we no longer have to store the ssns. It was supposed to be deployed almost a year ago, but it never got done, despite all my badgering. Now we’re out of time. The Payment Card Industry auditors, that’s pci for short, are here later this month, so I fast-tracked the work with the timekeeping team to get it done.”

  I stare at my phone, speechless.

  On the one hand, I’m ecstatic because we’ve found the smoking gun in John’s hand. John’s mention of the ssn field matches Ann’s description of the corrupted data.

  On the other hand: “Let me see if I’ve got this right…” I say slowly. “You deployed this tokenization application to fix an audit finding, which caused the payroll run failure, which has Dick and Steve climbing the walls?”

  John responds hotly, “First, I am quite certain the tokenization security product didn’t cause the issue. It’s inconceivable. The vendor assured us that it’s safe, and we checked all their references. Second, Dick and Steve have every reason to be climbing the walls: Compliance is not optional. It’s the law. My job is to keep them out of orange jumpsuits, and so I did what I had to do.”

  “‘Orange jumpsuits?’”

  “Like what you wear in prison,” he says. “My job is to keep management in compliance with all relevant laws, regulations, and contractual obligations. Luke and Damon were reckless. They cut corners that severely affected our audit and security posture. If it weren’t for my actions, we’d probably all be in jail by now.”

  I thought we were talking about a payroll failure, not being thrown in jail by some imaginary police force.

  “John, we have processes and procedures for how you introduce changes into production,” Patty says. “You went around them, and, once again, you’ve caused a big problem that we’re having to repair. Why didn’t you follow the process?”

  “Ha! Good one, Patty,” John snorts. “I did follow the process. You know what your people told me? That the next possible deployment window was in four months. Hello? The auditors are on-site next week!”

  He says adamantly, “Getting trapped in your bureaucratic process was simply not an option. If you were in my shoes, you’d do the same thing.”

  Patty reddens. I say calmly, “According to Dick, we have fewer than four hours to get the timekeeping app up. Now that we know there was a change that affected ssns, I think we have what we need.”

  I continue, “Max, who helped with the deployment, is on vacation today. Wes or Brent will be contacting you to learn more about this tokenization product you deployed. I know you’ll provide them with whatever help they need. This is important.”

  When John agrees, I thank him for his time. “Wait, one more question. Why do you believe that this product didn’t cause the failure? Did you test the change?”

  There’s a short silence on the phone before John replies, “No, we couldn’t test the change. There’s no test environment. Apparently, you guys requested a budget years ago, but…”

  I should have known.

  “Well, that’s good news,” Patty says after John hangs up. “It may not be easy to fix, but at least we finally know what’s going on.”

  “Was John’s tokenization change in the change schedule?” I ask.

  She laughs humorlessly. “That’s what I’ve been trying to tell you. John rarely goes through our change process. Nor do most people, for that matter. It’s like the Wild West out here. We’re mostly shooting from the hip.”

  She says defensively. “We need more process around here and better support from the top, includin
g it process tooling and training. Everyone thinks that the real way to get work done is to just do it. That makes my job nearly impossible.”

  In my old group, we were always disciplined about doing changes. No one made changes without telling everyone else, and we’d bend over backward to make sure our changes wouldn’t screw someone else up.

  I’m not used to flying this blind.

  “We don’t have time to do interrogations every time something goes wrong,” I say, exasperated. “Get me a list of all the changes made in the past, say, three days. Without an accurate timeline, we won’t be able to establish cause and effect, and we’ll probably end up causing another outage.”

  “Good idea,” she nods. “If necessary, I’ll e-mail everyone in it to find out what they were doing, to catch things that weren’t on our schedule.”

  “What do you mean, ‘e-mail everyone?’ There’s no system where people put in their changes? What about our ticketing system or the change-authorization system?” I ask, stunned. This is like Scotland Yard e-mailing everyone in London to find out who was near the scene of a crime.

  “Dream on,” she says, looking at me like I’m a newbie, which I suppose I am. “For years, I’ve been trying to get people to use our change management process and tools. But just like John, no one uses it. Same with our ticketing system. It’s pretty hit-or-miss, too.”

  Things are far worse than I thought.

  “Okay, do what you need to do,” I finally say, unable to hide my frustration. “Make sure you hit all the developers supporting the timekeeping system as well as all the system administrators and networking people. Call their managers, and tell them it’s important that we know about any changes, regardless of how unimportant they may seem. Don’t forget John’s people, too.”

  When Patty nods, I say, “Look, you’re the change manager. We’ve got to do better than this. We need better situational awareness, and that means we need some sort of functional change management process. Get everyone to bring in their changes so we can build a picture of what is actually going on out there.”

  To my surprise, Patty looks dejected. “Look, I’ve tried this before. I’ll tell you what will happen. The Change Advisory Board, or cab, will get together once or twice. And within a couple of weeks, people will stop attending, saying they’re too busy. Or they’ll just make the changes without waiting for authorization because of deadline pressures. Either way, it’ll fizzle out within a month.”

  “Not this time,” I say adamantly. “Send out a meeting notice to all the technology leads and announce that attendance is not optional. If they can’t make it, they need to send a delegate. When is the next meeting?”

  “Tomorrow,” she says.

  “Excellent,” I say with genuine enthusiasm. “I’m looking forward to it.”

  When I finally get home, it’s after midnight. After a long day of disappointments, I’m exhausted. Balloons are on the floor and a half-empty bottle of wine sits on the kitchen table. On the wall is a crayon poster saying, “Congratulations Daddy!”

  When I called my wife, Paige, this afternoon telling her about my promotion, she was far happier than I was. She insisted on inviting the neighbors over to throw a little celebration. Coming home so late, I missed my own party.

  At 2 p.m., Patty had successfully argued that of the twenty-seven changes made in the past three days, only John’s tokenization change and the san upgrade could be reasonably linked to the payroll failure. However, Wes and his team were still unable to restore san operations.

  At 3 p.m., I had to tell Ann and Dick the bad news that we had no choice but to execute plan B. Their frustration and disappointment were all too evident.

  It wasn’t until 7 p.m. when the timekeeping application was back up and 11 p.m. when the san was finally brought back online.

  Not a great performance on my first day as vp of it Operations.

  Before I left work, I e-mailed Steve, Dick, and Ann a quick status report, promising to do whatever it takes to prevent this type of failure from happening again.

  I go upstairs, finish brushing my teeth, and check my phone one last time before going to bed, being careful not to wake up Paige. I curse when I see an e-mail from our company pr manager, with a subject of “Bad news. We may be on the front page tomorrow…”

  I sit on the bed, squinting to read the accompanying news story.

  Elkhart Grove Herald Times

  Parts Unlimited flubs paychecks, local union leader calls failure ‘unconscionable’

  Automotive parts supplier Parts Unlimited has failed to adequately compensate its workers, with some employees receiving no pay at all, according to a Parts Unlimited internal memo. The locally headquartered company admitted that it had failed to issue correct paychecks to some of its hourly factory workers and that others hadn’t received any compensation for their work. Parts Unlimited denies that the issue is connected to cash flow problems and instead attributes the error to a payroll system failure.

  The once high-flying $4 billion company has been plagued by flagging revenue and growing losses in recent quarters. These financial woes, which some blame on a failure of upper management, have led to rampant job insecurity among local workers struggling to support their families.

  According to the memo, whatever the cause of the payroll failure, employees might have to wait days or weeks to be compensated.

  “This is just the latest in a long string of management execution missteps taken by the company in recent years,” according to Nestor Meyers Chief Industry Analyst Kelly Lawrence.

  Parts Unlimited CFO Dick Landry did not return phone calls from the Herald Times requesting comment on the payroll issue, accounting errors and questions of managerial competency.

  In a statement issued on behalf of Parts Unlimited, Landry expressed regret at the “glitch,” and vowed that the mistake would not be repeated.

  The Herald Times will continue to post updates as the story progresses.

  Too tired to do anything more, I turn off the lights, making a mental note to myself to find Dick tomorrow to apologize in person. I close my eyes and try to sleep.

  An hour later I’m still staring at the ceiling, very much awake.

  CHAPTER 4

  • Wednesday, September 3

  I drink my coffee as I open up my laptop at 7:30 a.m., hoping to get through my e-mails and voicemails before my 8 a.m. meeting. I stare at the screen. In the twenty-two hours since I was promoted, 526 new e-mails have arrived in my inbox.

  Holy crap.

  I skip all the messages about yesterday’s failure and am startled by all the congratulatory notes from vendors, wanting to meet for lunch. How did they find out? I’m pretty sure most of my organization still doesn’t know.

  I read an e-mail from Ellen, my former boss’ assistant, who is now assigned to support me, congratulating me and asking when we can meet. I reply, telling her I’d like to take her out for coffee this morning. I send a note to the it service desk, requesting that Ellen be granted access to my calendar.

  A blinking red light on my desk phone catches my attention. It reads, “7:50 a.m. 62 new voicemails.”

  My jaw drops. It would take an hour I don’t have just to listen to them. I e-mail Ellen again, asking her to go through all my voicemails, transcribing any that require action.

  Before I hit send, I quickly add, “If there are any messages from Steve or Dick, please call me right away on my cell phone.”

  Grabbing my clipboard, I hurry toward my first meeting when my phone vibrates. It’s an urgent e-mail:

  From: Sarah Moulton

  To: Bill Palmer

  Cc: Steve Masters

  Date: September 3, 7:58 AM

  Priority: Highest

  Subject: Latest Phoenix slip

  Bill, as you know, Project Phoenix is the most important project this company is undertaking. I’ve heard disturbing rumors that you are holding up the release.

  I don’t need to remind you that our com
petition isn’t standing still. Each day that goes by, our market share goes down. I need everyone to have a sense of urgency. Especially from you, Bill.

  We have an emergency project management meeting at 10 AM today. Please join us, and be prepared to explain these unacceptable delays.

  Steve, I know how important this project is for you, given the commitments you’ve made to the board. Please feel free to attend. We’d love your perspective.

  Regards,

  Sarah

  Oh no.

  I forward the e-mail to Wes and Patty, flagging it as high priority. Something seems wrong in a world where half the e-mail messages sent are urgent. Can everything really be that important?

  I call Wes’ cell phone. “I just got your e-mail from Sarah,” he says. “What utter bullshit.”

  “What’s this all this about?” I ask.

  He says, “I’m pretty sure it’s about Brent not finishing up that configuration work for the Phoenix developers. Everyone is chasing their tails because the developers can’t actually tell us what the test environment should look like. We’re doing our best, but every time we deliver something, they tell us we did it wrong.”

  “When did they tell us about it?” I ask.

  “Two weeks ago. It’s the typical bullshit with Development, but worse. They’re so freaked out about hitting their deadlines, they’re only now starting to think about how to test and deploy it. Apparently, they’re making it our problem. I hope you’re wearing your asbestos underwear like me. Sarah is going to be at that meeting with torches, wanting to throw us onto the bonfire.”

  It’s amazing to me how handoffs between Development and it Operations always get screwed up. But given the perpetual tribal warfare between the two groups, maybe I shouldn’t be surprised.

  I reply, “I get the picture. Look, make sure you dig into this Dev specification issue personally. We’ve got to get this nailed down—grab everyone involved, whether they’re in Dev or Ops, and lock them in a room until they come up with a written specification. Phoenix is so important, we can’t afford to screw this up.”