by Gene Kim
They’re telling analysts about the Phoenix roadmap? With so many features being delayed to the next release, I question the wisdom of making insufficiently informed promises to the market.
Steve just nods and replies happily, “Let’s see if it changes their impressions of us. Good job scheduling the call. Catch you later today for the next one.”
Sarah gives me a smile and says, “Hey, Bill. You’re up and at it early today, aren’t you?”
Gritting my teeth, I just ignore her comment. “Good morning, everyone.” Trying to show an interest, I say, “Sounds like you had a good call.”
Sarah smiles even more broadly. “Yes, they’re really excited about our vision and agree that it’s going to be a real game changer for us. This is what we need to change how we’re perceived by the broader market and Wall Street.”
I look levelly at her, wondering whether these briefings that we give the outside world might be what is causing such pressure on Chris’ team to release features so prematurely.
I grab a seat across from Steve. I can’t quite turn my back on Sarah, but I do my best.
I don’t want to give Steve my handouts until Sarah has left the room, but she continues to talk with Steve, recounting their meeting and how to change the talk track for their next analyst call.
As they talk, all I can think of is how she’s eating into my time with Steve.
Eleven minutes later, Steve laughs at a joke from Sarah, and she finally leaves the office, closing the door behind her. Steve turns to me and says, “Sorry about going over—our next Phoenix analyst briefing is in twenty minutes. So, what’s on your mind?”
“You’ve impressed upon me from the very beginning that I need to help maximize the probability of success of the Phoenix rollout,” I start. “Based on my observations of the past week, we are stretched dangerously thin, to the point that I believe Phoenix is in considerable jeopardy.
“I’ve had my staff establish what our levels of demand and capacity truly are,” I continue. “We’ve started to inventory everything we’re being asked to do, regardless of how big or small. Based on the analysis so far, it’s clear to me that the demand for it work far exceeds our ability to deliver. I’ve asked them to make more visible what the pipeline of work looks like, so we can make more informed decisions about who should be working on what and when.”
With as much gravitas as I can muster, I say, “One thing is very clear, though. We are definitely understaffed. There’s no way that we can deliver everything we’ve promised. Either we need to cut down the project list, or we’ve got to staff up.”
Trying to replicate the reasoned and logical argument I’ve spent all weekend rehearsing, I continue, “The other big problem is that we have too many different projects competing for our attention. You’ve been consistent and clear that Phoenix is the most important, but we can’t seem to keep resources dedicated to it. For instance, last Thursday, internal audit delivered to us a set of findings that we must investigate and assemble a response letter for in one week. Doing so will impact Phoenix.”
I’ve been watching Steve as I talked, and so far, he’s remained expressionless. I look at him calmly and ask, “What I’d like to get out of our meeting is an understanding of the relative priority of Phoenix versus the audit findings and to talk about the number of projects and how to adequately staff them.”
In my mind, I’ve done a good job of being the competent and passionate manager who is dispassionately struggling to decide how to best serve the business, without making moral judgments.
Steve replies in an exasperated voice, “What kind of bullshit prioritization question is this? If I went to my board and told them that I need to do either sales or marketing, and asked them which of those I should do, I’d be laughed out of the room. I need to do both, just like you need to do both! Life is tough. Phoenix is the top company priority, but that doesn’t mean you get to hold the sox-404 audit hostage.”
I count to three before saying, “Obviously, I’m not being clear. Both Phoenix and the compliance project share certain key resources, such as Brent. The compliance project alone would tie up these people for a year, but we need them focused on Phoenix. On top of that, our infrastructure is so fragile, that we have daily failures, which often require these same resources to restore normal operations. If a similar outage to the payroll run failure happened today, we’d likely have to pull Brent off both the Phoenix and compliance work to figure out what went wrong.”
I look at him dead-on and say, “We’ve looked at different resourcing options, including hiring and moving people around, but none of them will have any effect fast enough to make a difference. If Phoenix is really the top priority, we need to put some of the compliance work on hold.”
“Out of the question,” he says, before I can even finish. “I’ve seen that big pile of audit findings, and we will be in very hot water if we don’t get those issues fixed.”
This is definitely not going according to plan. “Okay…” I say slowly. “We’ll do our best, but let me state for the record that we’re too understaffed to do a good job on either one, let alone both.”
I wait for him to acknowledge my point. Seconds go by before he finally nods.
Realizing this is probably the best I’m going to get, I indicate to the first page of the handout I gave him. I say, “Let’s zoom up and talk about project demand and capacity. We’re currently supporting over thirty-five business projects through Kirsten’s Project Management Office, and at current count, seventy-plus other smaller business projects and internal initiatives. And there are others out there that we just haven’t counted yet. With our 143 it Operations people, nothing is getting done as promised.”
I point him to the second page of the handout, saying, “As you can see, my team and I have come up with a request for six additional resources that we’re most short-handed on.”
I go for the close, saying, “My goal is to increase our throughput so we won’t be in this position again, and to get as many of these projects done as we can. I’d like your approval to get these openings immediately, so we can start our search. Talent like Brent is not easy to find, and we need to start sooner rather than later.”
In my rehearsals, this is when Steve would run his finger down the figures, ask me some questions, and we’d have a meaningful discussion about how to make the best trade-offs. Maybe he’d even pat me on the back and compliment me on the quality of my analysis.
But Steve doesn’t even pick up my handouts. Instead, he looks at me and says, “Bill, Phoenix is already $10 million over budget, and we must get cash-flow positive soon. You have some of the most expensive resources in the entire company. You have to use what you’ve got.”
He crosses his arms and continues, “Last year, we had some it analysts come in and benchmark our company against our peers. They told us that we’re spending way more on it than our competitors.
“You may think that with three thousand employees, six more employees won’t make a difference. But, trust me, every expense is under scrutiny. If I can’t close the profitability gap, I’ll have to do another layoff. Your math of adding another $2 million in labor costs just doesn’t work.”
He continues in a more sympathetic voice. “My suggestion to you? Go to your peers and make your case to them. If your case is really valid, they should be willing to transfer some of their budget to you. But let me be clear: Any budget increases are out of the question. If anything, we may have to cut some heads in your area.”
I spent hours role-playing worst-case scenarios over the weekend. Apparently, I’m going to have to practice being more pessimistic.
“Steve, I don’t know how I can be more clear about this,” I say, feeling a little desperate. “This stuff isn’t magic. All this work being piled on us is done by real people. Commitments like the compliance work are made without any regard for what’s already on people’s plates, like Phoenix.”
Realizing I have little to lose,
but trying to shock some sense into him, I say, “If you really care about closing the gap with the competition by having Phoenix succeed, you sure aren’t acting like it. To me, it seems like you’re just being suckered to rush to the gunfight late, showing up with only a knife.”
I expected some kind of reaction but he merely leans back, crossing his arms in front. “We’re all doing our best. So you better go back and do the same.”
Just then, the door opens as Sarah walks in. “Hi, Steve. Sorry to interrupt, but we have the next analyst call in two minutes. Shall I dial us in?”
Shit. I look down at my watch. 9:27.
She even robbed me of my last three minutes.
Utterly defeated, I finally say, “Okay, got it. Keep pushing. I’ll keep you posted.”
Steve nods in thanks, and then turns to Sarah as I close the door behind me. On my way out, I toss the presentation I had worked on all weekend into Stacy’s recycling bin.
I try to wave away the stench of failure as I walk to the cab meeting. I’m still thinking about how I’m going to break the bad news to Wes and Patty when I walk into the conference room that Patty has coined the Change Coordination Room.
All thoughts of Steve disappear when I see what’s there.
Almost every area of the wall is now covered in whiteboards. Index cards cover nearly every inch of the whiteboards on two of the walls. It’s not just one deep—in some places there are hooks attached to the board, with ten cards hanging off them.
On the conference room table are twenty, maybe even thirty, more piles of cards.
On the far side of the table, two guys who work for Patty have their backs to us, studying a card. After a moment, they tape it between two other cards in front of them.
“Holy crap,” I say.
“We’ve got a problem.” Patty says from behind me.
“Not enough space for more whiteboards?” I say, only half joking.
Before Patty answers, I hear Wes enter the room. “Holy shit!” he says. Where did all these cards come from? Are they all for this week?”
I turn to ask him, “Are you surprised? Most of these are coming from your group.”
He looks around at all the boards and then at the cards on the table, “I knew that my guys were really busy, but, there must be a couple of hundred changes here.”
Patty turns her laptop around to show us the spreadsheet she has open, “Since last Friday afternoon, there have been 437 changes submitted for this week.”
Wes for once is speechless. He finally shakes his head and says, “And now we’re supposed to go through and approve all of them? This meeting was only scheduled for an hour—we’d need days to go through all of these!”
He looks at me. “Mind you, I’m not saying that we shouldn’t, but if we’re going to do this every week…”
Again, Wes stops speaking, overwhelmed at the task in front of us.
Quite honestly, I feel the same way. Apparently, getting all the managers to submit their changes for the week was just the first step. I didn’t expect that the process would fall apart as we went beyond collecting the data and actually set out to process and authorize the changes.
I force myself to say cheerfully, “This is a great start. Like most things, things get worse before they get better. We’ve got enthusiastic support from the technical managers, so now we’ve got to figure out how to get these changes scrutinized and scheduled on an on-going basis. Any ideas?”
Patty is first to speak up. “Well, no one says that we have to be the ones reviewing all the changes—maybe we can push down some of these to delegates.”
I listen to Wes and Patty trade ideas back and forth before I say, “Let’s go back to our goals: get the left and right hands to know what the other is doing, give us some situational awareness during outages, and give audit some evidence that we’re addressing change control.
“We need to focus on the riskiest changes,” I continue. “The 80/20 rule likely applies here: Twenty percent of the changes pose eighty percent of the risk.”
I stare again at piles of cards in front of us and pick up a couple at random, searching for some inspiration.
Holding up a card that has a big frowny face drawn on it, I ask, “What’s puccar?”
“That worthless app,” Wes says with disgust, “is the Parts Unlimited Check Clearing and Reconciliation application that someone implemented almost two decades ago. We call it ‘pucker’ because every time we change it, it blows up, and no one knows how to fix it. The vendor went out of business during the dot-com boom, but we’ve never gotten funding to replace it.”
I ask, “If we know it’s that prone to crashing, why do we need to change it?”
Wes says quickly, “We try not to. But sometimes the business rules change, and we also have to keep it patched. It’s running an operating system that’s out of maintenance, so it’s always dicey…”
“Good! It’s a risky change. What other types of changes are being submitted like puccar?” I ask.
We make a pile of nearly fifty cards proposing changes to the Rainbow, Saturn, and Taser applications, and also changes to the network and certain shared databases, which could impact a significant portion, or even all, of the business.
“Even looking at those cards makes my heart palpitate,” Wes says. “These are some of the dangerous changes we make around here.”
He’s right. I say, “Okay, let’s mark all of these as ‘fragile.’ These are high risk and must be authorized by the cab. Patty, changes like this should be at the top of the pile during our meetings.”
Patty nods, taking notes saying, “Got it. We’re predefining high-risk categories of change that not only must have change requests submitted, but must have authorization before being scheduled and implemented.”
We quickly create a list of the top ten most fragile services, applications, and infrastructure. Any change request that could affect any of these will be immediately flagged for cab scrutiny.
Patty adds, “We need to create some standard procedures around these changes—like when we’ll want them implemented—and have key resources not only aware of them but also standing by, just in case things go wrong—even the vendors.”
She adds with a half smile, “You know, like having firefighters and ambulances lined up on the runway, ready to spray safety foam when the airplane lands in flames.”
Wes laughs and adds wryly, “Yeah, in the case of puccar, have the coroner stock up on a bunch of body bags, too. And a pr person ready to handle the angry phone calls from the business, saying that some customers were allergic to the foam we used.”
I laugh. “You know, that’s an interesting idea. Let’s let the business choose the foam. There’s no reason why all the responsibility should rest on our shoulders. We can send an e-mail out to the business ahead of time and ask when the best implementation time would be. If we can give them data on the outcomes of previous changes, they may even withdraw the change.”
Patty is typing away. “Got it. For these types of changes, I’ll have my staff generate some reports on the changes’ success rates and any associated downtime. This will help the business make more informed decisions around the changes.”
I’m extremely pleased with Patty’s idea and am confident that we’re on the right track. “Okay, that still leaves four hundred cards to go. Any ideas?”
Wes has been going through the cards methodically, creating two big stacks of cards next to him. He picks a card from the bigger pile, “This pile has changes we do all the time. Like this one about the monthly tax table upload to the pos systems. I don’t think we should suspend any of these changes.
“On the other hand, these changes are stuff like ‘increasing the Java application server thread pool size,’ ‘installing the Kumquat vendor application hotfix to resolve performance issue,’ and ‘resetting the Kentucky data center load balancer to default duplex settings.’
“What the hell do I know about these things?” Wes says.
“I just don’t know enough about the context to have an actual opinion. I don’t want to be like a seagull, flying in, crapping on people, and then flying away, you know?”
Excited, Patty says, “Excellent! The first ones are the low-risk changes that itil calls ‘standard changes.’ For changes we’ve done many times before successfully, we just preapprove. They still need to be submitted, but they can be scheduled without us.”
When everyone nods, she continues, “That leaves about two hundred changes that are medium-risk changes that we still need to look at.”
“I agree with Wes,” I respond. “For these, we need to trust that the manager knows what he or she is doing. But I’d like Patty to verify that people have appropriately informed anyone they could affect, and gotten the ‘okay to proceed’ from all of them.”
I think for a moment and say, “Take John’s tokenization application. Before that change request would even come to us, I would expect him to get the nod from the application and database owners, and also the business. If he’s done that, that’s good enough for me. I view our role as making sure that he’s dotted the i’s and crossed the t’s. At this level, I care more about the integrity of the process, not so much about the actual changes.”
Patty is typing away. “Let me see if I’ve got this right: For the ‘messy middle changes,’ we’re deciding that the change submitter has responsibility and accountability for consulting and getting approval from people potentially affected. Once they do that, they submit their change card for us to review and approve for scheduling.”
I smile and say, “Yep. Work for you, Wes?”
At last, he says, “I think it’ll work. Let’s give it a shot.”
“Good,” I say. Then I say to Patty, “You can help make sure the change requesters are actually doing all the work beforehand?”
Patty smiles and says, “With pleasure.”