The Phoenix Project

by Gene Kim

  “Furthermore, your group was also unable to produce any change management meeting minutes, which is supposed to meet weekly, according to your policy.”

  I try not wince visibly, recalling that no one even showed up at the cab meeting yesterday, and during the payroll incident, we were so oblivious to John’s tokenization change that we ended up bricking the san.

  If we were clueless about those changes, I sincerely doubt that we’d notice if someone disabled a control that would enable a minor, say, $100 million fraudulent transaction.

  “Really? That’s unbelievable! I’ll look into that.” I say with what I hope is the right amount of surprise and moral outrage. After I pretend to take detailed notes on my clipboard, circling and underlining random words, I nod, prompting Tim to continue.

  “Next, we found numerous instances where developers have administrative access to production applications and databases. This violates the required segregation of duty required to prevent risk for fraud.”

  I look over to John. “Really? You don’t say. Developers making changes to an application without an approved change order? That certainly sounds like a security risk. What would happen if someone coerced a developer, say Max, into doing something unauthorized? We’ve go to do something about that, right, John?”

  John turns bright red, but says politely, “Yes, of course. I agree and would be happy to help.”

  Tim says, “Good. Let’s move onto the sixteen significant deficiencies.”

  A half hour later, Tim is still droning on. I stare glumly at the huge stack of findings. Most of these issues are just like the huge, useless reports we get from Information Security, which is another reason why John has such a bad reputation.

  It’s the never-ending hamster wheel of pain: Information Security fills up people’s inboxes with never-ending lists of critical security remediation work, quarter after quarter.

  When Tim finally finishes, John volunteers, “We must get these vulnerable systems patched. My team has a lot of experience patching systems, if you require assistance. These audit findings are an awesome opportunity to close some big security holes.”

  “Look, both of you guys have no idea what you’re asking for!” Wes says to John and Tim, clearly exasperated. “Some of the servers that those manufacturing erp systems run on are over twenty years old. Half the company will grind to a halt if they go down, and the vendor went out of business decades ago! These things are so fragile that if you even look at them at the wrong time of day, they’ll crash and require all sorts of voodoo to get them to successfully reboot. They’ll never survive the changes you’re proposing!”

  He leans over the table, putting his finger in John’s face. “You want to patch it yourself, fine. But I want a signed piece of paper from you saying that if you push the button and the entire business grinds to a halt, you’ll fly around and grovel to all the plant managers, explaining to them why they didn’t hit their production targets. Deal?”

  My eyes widen with amazement when John actually leans forward into Wes’ finger and says angrily, “Oh, yeah? How about when we’re on the front page of the news because we lost consumer data that we’re responsible for protecting? You’ll personally apologize to the thousands or millions of families whose data are now being sold by the Russian Mafia?”

  I say, “Settle down, everyone. We all want to do what’s right for the company. The trick is figuring out what we have time to do and what systems can actually be patched.”

  I look at the stack of papers. Wes, Patty, and I can assign people the task of investigating each issue, but who will actually do the work? We’re already buried with Phoenix, and I fear that this new massive project might be the straw that breaks the camel’s back.

  I say to Nancy, “I’ll get with my team right away, and we’ll come up with a plan. I can’t promise you that we’ll have our response letter completed by then, but I can promise you that we’ll get you everything we can. Will that be adequate?”

  “Quite so,” Nancy says amicably. “Going through the preliminary audit findings and identifying next steps were the only objectives for this meeting.”

  As the meeting adjourns, I ask Wes to stay behind.

  Noticing this, John remains behind, as well. “This is a disaster. All my objectives and bonuses are tied to getting a clean compliance report for the sox-404 and pci audits. I’m going to fail because you Ops guys can’t get your shit together!”

  “Join the club,” I say.

  To get him off my back, I say, “Sarah and Steve decided to move up the Phoenix deployment date to next Friday. They want to skip all the security reviews. You probably should talk to Chris and Sarah right away.”

  Predictably, John swears and storms out, slamming the door behind him.

  Exhausted, I lean back in my chair and say to Wes, “This is just not our week.”

  Wes laughs humorlessly. “I told you that the pace of things around here would make your head explode.”

  I gesture at the audit findings. “We’re supposed to protect all our key resources for Phoenix, but that’s sucking in everybody. We don’t have a bunch of people just sitting on the bench we can throw at the audit findings, right?”

  Wes shakes his head, his face uncharacteristically pinched with tension.

  He flips through his stack of papers again. “We’re definitely going to need to bring the technology leads into this. But as you said, they’re already assigned to the Phoenix team. Should we reassign them here?”

  I honestly don’t know. Wes stares at one of the pages for a moment. “By the way, I think a bunch of these will require Brent.”

  “Oh, come on.” I mutter. “Brent. Brent, Brent, Brent! Can’t we do anything without him? Look at us! We’re trying to have a management discussion about commitments and resources, and all we do is talk about one guy! I don’t care how talented he is. If you’re telling me that our organization can’t do anything without him, we’ve got a big problem.”

  Wes shrugs, slightly embarrassed. “He’s undoubtedly one of our best guys. He’s really smart, and he knows a lot about almost everything we have in this shop. He’s one of the few people who really understand how all the applications talk together at an enterprise level. Heck, the guy may know more about how this company works than I do.”

  “You’re a senior manager. This should be as unacceptable to you as it is to me!” I say firmly. “How many more Brents do you need? One, ten, or a hundred? I’m going to need Steve to prioritize all this work. What I need from you is what resources we need. If I ask Steve for more resources, I don’t want to have to crawl back, begging for more later.”

  He rolls his eyes. “Look, I’ll tell you right now what’s going to happen. We’ll go to management and present our case. Not only will they say no, they’ll cut our budget by another five percent. That’s what they’ve done for the past five years. In the meantime, everyone will continue to want everything at the same time, and keep adding to our list of things to do.”

  Exasperated, he continues, “And just so you know, I have tried to hire more Brents. Because I never got the budget, I eliminated a bunch of positions just so I could hire four more very senior engineers at the same level of experience as Brent. And you know what happened?”

  I merely raise my eyebrows.

  Wes says, “Half quit within a year, and I’m not getting anywhere near the productivity I need from the ones who stayed. Although I don’t have data to prove it, I’m guessing Brent is even more behind than ever. He complains that he had to spend a bunch of time training and helping the new guys, and is now stretched thinner than ever. And he’s still in the middle of everything.”

  I respond, “You said that people ‘add stuff to our list.’ What does the list look like right now? Where can I get a copy? Who owns the list?”

  Wes replies slowly, “Well, there are the business projects and the various it infrastructure projects. But a lot of the commitments just aren’t written down.”
/>   “How many business projects? How many infrastructure projects?” I ask.

  Wes shakes his head. “I don’t know offhand. I can get the list of business projects from Kirsten, but I’m not sure if anyone knows the answer to your second question. Those don’t go through the Project Management Office.”

  I have a sinking feeling in the pit of my stomach. How can we manage production if we don’t know what the demand, priorities, status of work in process, and resource availability are? Suddenly, I’m kicking myself that I didn’t ask these questions on my first day.

  Finally, I’m thinking like a manager.

  I call Patty. “Wes and I just got hammered by audit and they need a response one week from Monday. I need your help to figure what all our work commitments are, so I can have an intelligent discussion with Steve about resourcing. Can you talk?”

  She says, “That’s right up my alley. Come on over.”

  After Wes briefs Patty on the implications of the mammoth audit report that he thumped down on the table, she whistles.

  “You know, I really wish you were at that meeting with the auditors,” I say. “Most of the biggest issues were around the absence of a functional change management process. I think you could end up being the auditors’ best friend.”

  “Auditors have friends?” she laughs.

  “I need you to help Wes estimate the work to fix the audit findings by Monday,” I say. “But right now, let’s talk about a higher level issue. I’m trying to get the list of what all our commitments to the organization are. How big is that list and how do things get on it?”

  After hearing what Wes told me, Patty replies, “Wes is right. Kirsten owns the official business project list, almost all of which have something that we’re on the hook for. We have our own it Operations projects, which are typically managed by the technology budget owner—there is no centralized list of those projects.”

  Patty continues, “We also have all the calls going into the service desk, whether it’s requests for something new or asking to fix something. But that list will be incomplete, too, because so many people in the business just go to their favorite it person. All that work is completely off the books.”

  I ask slowly, “So, you’re saying that we have no idea what the list of our commitments is? Really?”

  Wes says defensively, “Until now, no one ever asked. We’ve always hired smart people and tasked them with certain areas of responsibility. We’ve never had to manage things beyond that.”

  “Well, we need to start. We can’t make new commitments to other people when we don’t even know what our commitments are now!” I say. “At the very least, get me the work estimate to fix the audit findings. Then, for each of those resources, tell me what their other commitments are that we’re going to be pulling them off of.”

  Thinking for a moment, I add, “For that matter, do the same thing for every person assigned to Phoenix. I’m guessing we’re overloaded, so I want to know by how much. I want to proactively tell people whose projects have been bumped, so they’re not surprised when we don’t deliver what we promised.”

  Both Wes and Patty look surprised. Wes speaks up first, “But…but we’d have to talk with almost everyone! Patty may have fun grilling people on what changes they’re making, but we can’t go around wasting the time of our best people. They’ve got real work to do!”

  “Yes, I know they have real work to do,” I say adamantly. “I merely want a one-line description about what all that work is and how long they think it will take!”

  Realizing how this might come across, I add, “Make sure you tell people that we’re doing this so we can get more resources. I don’t want anyone thinking that we’re outsourcing or firing anyone, okay?”

  Patty nods. “We should have done this a long time ago. We bump up the priorities of things all the time, but we never really know what just got bumped down. That is, until someone screams at us, demanding to know why we haven’t delivered something.”

  She types on her laptop. “You just want a list of organizational commitments for our key resources, with a one-liner on what they’re working on and how long it will take. We’ll start with all Phoenix and audit remediation resources first, but will eventually cover the entire it Operations organization. Do I have it right?”

  I smile, genuinely happy that Patty has framed it so succinctly. I know she’s going to do a great job. “Exactly. Bonus points if you and Wes can determine which resources are most overutilized and how many new resources we need. That would be the basis of an ask to Steve for more staffing.”

  Patty says to Wes, “This should be pretty straightforward. We can put together fifteen-minute interviews, pull data from our service desk and ticketing system, get Kirsten’s project list…”

  Surprisingly, Wes agrees, adding, “We could also look in our budgeting tools to see how we’ve coded personnel and hardware requests.”

  I stand up. “Great thinking, guys. Get a meeting set up for us to go over what you find, no later than Friday. I want to have a meeting with Steve on Monday, armed with some real data.”

  She gives me the thumbs-up. Now we’re getting somewhere.

  CHAPTER 6

  • Friday, September 5

  In another one of the endless Phoenix status meetings, I realize that the developers are even more behind than we feared. As Wes had predicted, more and more work is being deferred to the next release, including almost all of the testing.

  This means that we’ll be the ones finding the problems when they blow up in production, instead of the Quality Assurance (qa) Department.

  Great.

  During a lull in the discussion, I look down at my phone and see an e-mail from Patty. She wants to meet about resourcing, promising some eye-opening surprises.

  I open the attached spreadsheet, seeing an encouraging level of detail, but on my minuscule phone screen, I can’t make heads or tails of it. I reply to Patty that I’m on the way and ask her to have Wes meet me there.

  When I arrive, I’m surprised to see that Wes has set up a projector, displaying a spreadsheet on the wall. I’m excited that we’re meeting to analyze the situation, instead of just reacting to the daily fires.

  I grab a seat. “Okay, whatcha got for me?”

  Wes starts. “Patty did a great job putting this together. What we found was—well, it was interesting.”

  Patty explains, “We did our interviews, collected the data, and then did our analysis. Right now, these numbers are only for our key resources. We’re already seeing something troubling.”

  She points at a row in the spreadsheet. “First, we have a lot of projects. Kirsten says she’s officially managing about thirty-five major business projects, each of which we have commitments to. Internal to it Operations, we’ve already identified over seventy projects, and that number keeps growing with each person we interview.”

  “Wait,” I say, genuinely startled, sitting upright in my chair. “We have 150 it Operations people, right? If you’ve already found over 105 projects, that’s 1.5 people per project. Doesn’t that seem like a lot to you?”

  Wes replies, “Totally. And we know that the project count is low. So by the end, it’ll probably be more like one person per project. That’s insane.”

  I ask, “How big are these internal projects?”

  Wes switches tabs on the spreadsheet, showing the list of projects they’ve inventoried, along with the estimated number of man-weeks. “Consolidate and upgrade e-mail server,” “Upgrade thirty-five instances of Oracle databases,” “Install supported Lemming database server,” “Virtualize and migrate primary business applications,” and so on.

  I groan. While some projects are small, most seem like major undertakings, estimated at three man-years or more.

  When Patty sees the expression on my face, she says, “That was my reaction, too. We’re on the hook for a huge number of projects. So, let’s look at what our capacity is. This is a little harder, since we can’t just ass
ign random people to any given project.”

  She continues, “When we looked at who was assigned to each project and what their other commitments and availability were, here’s what we found.”

  When Wes flips to another spreadsheet tab, my heart drops.

  “Grim, huh?” says Wes. “Most of our resources are going to Phoenix. And look at the next line: Compliance is the next largest project. And even if we only worked on compliance, it would consume most of our key resources for an entire year! And that includes Brent, by the way.”

  Incredulous, I say, “You’re kidding. If we put all our projects on hold except for the audit findings, our key resources would be tied up for an entire year?”

  “Yep,” Patty says, nodding. “It’s hard to believe, but it just shows you how much work is in that stack of audit findings.”

  I look down at the table, speechless.

  If someone had shown me these figures during my first conversation with Steve, I would have run from the room, screaming like a little boy.

  It’s not too late, I think, smiling at the image.

  With practiced calm, I say, “Okay, knowing is always better than not knowing. Keep going.”

  Wes looks back at the spreadsheet. “The third largest item is incident and break-fix work. Right now, it’s probably consuming seventy-five percent of our staff’s time. And because these often involve critical business systems, incidents will take priority over everything else, including Phoenix and fixing audit findings.

  “By the way, did you know that yesterday, when we were talking with Brent, we had to reschedule the interview twice because he had to go help fix an outage? So there we were interrupting him from Phoenix work, only to be interrupted by an outage!” he says, laughing.

  I start to laugh, but then stop abruptly. “Wait. What outage? Why didn’t I hear about it? We can’t keep running our organization like this!”

  “Well, it was another san issue, but nothing critical,” Wes replies. “A drive went bad a couple of months ago, so the san was running with no redundancy. When another drive failed, the entire volume went down. Brent had to help restore some of the databases when we got the san back up.”