by Gene Kim
He writes something down on a Post-it note, putting it on his monitor next to two other Post-it notes. Exasperated, he answers his cell phone, “Yeah, Brent here… The what service is down? Have you tried rebooting it? Look, I’m really slammed right now with Phoenix—I’ll get back to you later today?”
I’m in the middle of silently congratulating him when I hear him say, “Uh-huh…I don’t even know who that is. The vp of what? Okay, let me have a look.”
I sigh, taking a seat in an empty cubicle to watch today’s episode of A Day in The Life of Brent.
He’s on the phone for another five minutes, hanging up only after some critical production database is back up and running.
I appreciate how Brent seems to genuinely care that everyone relying on it systems can get their work done, but I’m dismayed that everyone seems to be using him as their free, personal Geek Squad. At the expense of Phoenix.
Brent grabs one of the Post-it notes off of his monitor and picks up his phone. Before he can dial, though, I stand up and say, “Hi, Brent.”
“Agh!” he shouts, startled. “How long have you been there?”
“Only for a couple of minutes,” I say, putting on my most friendly smile, grabbing a seat next to him. “Long enough to see you fix two people’s problems. That’s admirable, but I just came from Kirsten’s daily Phoenix stand-up. There are five tasks that have been assigned to you, which are now late.”
I show him the five tasks from the project management meeting. He says quickly, “I’m half-done with all of these already. I just need a couple of hours of quiet working time to get this done. I’d do this from home if I could, but the network connection is too slow.”
“Who’s been calling you, and what do they want?” I ask, frowning.
“Usually it’s other it people who are having problems fixing something,” he replies, rolling his eyes. “When something goes down, I’m apparently the only person who knows where to go looking.”
“I thought Wes hired a bunch of people to take over some of these escalations from you.” I say.
Brent rolls his eyes again. “That was the idea. But most had other responsibilities and were never available when we needed them. Others were let go during the downsizings because they weren’t busy enough. Trust me. That was no big loss. I ended up handling most of those issues anyway.”
“How many calls are you getting each day? Are you logging these calls anywhere?” I ask.
“You mean, like in our ticketing system? No, because opening up a ticket for each of those calls would take longer than fixing the problem.” Brent says dismissively. “The number of calls depends on the day. The last week has been worse than normal.”
I get it now. I bet if anyone called right now and yelled loud enough or name-dropped someone scary enough, Brent could be dragged into fixing someone else’s problem for hours on end.
“You tried to push back on the last person who called. What made you decide to work the issue, instead of telling them to go pound sand?” I ask.
He replies, “She told me that the vp of Logistics was screaming that replenishment orders weren’t getting created, and that if it didn’t get fixed right away, our stores were at risk of stocking out on fast-moving products. I didn’t want to be the person being called out for single-handedly allowing stock-outs to happen in the stores.”
I purse my lips. Company executives strong-arming my engineers into doing their bidding is total bullshit. But jeopardizing Phoenix is above their pay grade.
Standing up, I say, “Okay, from here on out, you’re working only on Phoenix. Steve Masters has said that this is everyone’s top priority. Now more than ever, the project needs you. I’m expecting you to reject any task that anyone tries to assign you.”
Brent looks simultaneously relieved and concerned. Maybe he’s thinking about that vp of Logistics.
I add, “If anyone contacts you about anything besides Phoenix, send them to Wes. Let him deal with all the jackasses.”
He says skeptically, “Look, I appreciate this, but I really don’t think this is going to work in the long run. Our guys around here just don’t seem up-to-speed with how all our systems work. In the end, they keep coming to me.”
“Well, they’re going to have to learn. When they call, send them to Wes. If anyone’s got a problem with that, then send them to me. In fact, put a vacation message on your e-mail, saying that you’re not responding to anything except for Phoenix and to instead contact…”
At my prompting, Brent says with a small smile, “Wes.”
“See? You’re already getting the hang of it.” I smile in return.
I point to his desk phone, “Do whatever it takes to break people of the habit of going directly to you. You’ve got my permission to turn off your phone ringer and change your voicemail greeting to say you’re not available and to contact Wes instead. Whatever it takes.”
Realizing that I’m distracting Brent from Phoenix by just standing here, I say quickly, “No, I’ll have my assistant Ellen change your voicemail greeting for you.”
Brent smiles again, and says, “No, no, no. I can do that. Thanks for the offer, though.”
I write my cell phone number on a Post-it note, and hand it to him, “Ellen will do it. We need you on those Phoenix tasks. Call if you need anything from me.”
When he nods, I start heading back to Building 9, but then turn around to ask, “Hey, let me buy you a beer sometime next week?”
He agrees, his expression brightening.
As I leave the building, I immediately call Patty. When she picks up, I say, “Grab Wes and meet me outside the Phoenix war room. We need to change the way we’re managing escalations to Brent. Right now.”
We all sit down in the conference room across the hallway from the Phoenix war room.
“How’d it go with Brent?” Wes asks.
When I tell him that Brent wasn’t able to work on Phoenix because of all the break-fix work, he blanches. “He’s been in all these emergency meetings! How can he possibly think anything is more important than Phoenix!”
I say, “Good question. Why would Brent drop Phoenix to work on something else?”
Wes’ bluster disappears for a couple of moments. “Probably because someone like me was screaming at him, saying that I absolutely needed his help to get my most important task done. And it’s probably true: For way too many things, Brent seems to be the only one who knows how they actually work.”
“If it were me, I’d try to justify it by saying that it would only take a couple of minutes…” Patty says. “Which could be true, but it’s like death by a thousand cuts.”
“Processes are supposed to protect people. We need to figure out how to protect Brent,” I say. I then describe how I already told Brent to send everyone wanting anything to Wes.
“What? You want me to micromanage all of his time? I don’t have time to be Brent’s secretary or be some sort of help desk person!” he shouts.
“Okay, what’s on your plate that’s more important than making sure your resources are getting critical Phoenix work done?” I ask.
Wes looks back at me for several moments stonily and then laughs. “Okay, you got me. Look, Brent is a smart guy. But he’s also one of the worst people I’ve ever met at writing anything down. Let me tell you a real story of how impossible this is going to be: Several months ago, we were three hours into a Sev 1 outage, and we bent over backward not to escalate to Brent. But eventually, we got to a point where we were just out of ideas, and we were starting to make things worse. So, we put Brent on the problem.”
He shakes his head, recalling the memory, “He sat down at the keyboard, and it’s like he went into this trance. Ten minutes later, the problem is fixed. Everyone is happy and relieved that the system came back up. But then someone asked, ‘How did you do it?’ And I swear to God, Brent just looked back at him blankly and said, ‘I have no idea. I just did it.’”
Wes thumps the table and says, “And t
hat is the problem with Brent. How the hell do you document that? ‘Close your eyes and go into a trance’?”
Patty laughs, apparently recalling the story. She says, “I’m not suggesting Brent is doing this deliberately, but I wonder whether Brent views all his knowledge as a sort of power. Maybe some part of him is reluctant to give that up. It does put him in this position where he’s virtually impossible to replace.”
“Maybe. Maybe not,” I say. “I’ll tell you what I do know, though. Every time that we let Brent fix something that none of us can replicate, Brent gets a little smarter, and the entire system gets dumber. We’ve got to put an end to that.
“Maybe we create a resource pool of level 3 engineers to handle the escalations, but keep Brent out of that pool. The level 3s would be responsible for resolving all incidents to closure, and would be the only people who can get access to Brent—on one condition.
“If they want to talk with Brent, they must first get Wes’ or my approval,” I say. “They’d be responsible for documenting what they learned, and Brent would never be allowed to work on the same problem twice. I’d review each of the issues weekly, and if I find out that Brent worked a problem twice, there will be hell to pay. For both the level 3s and Brent.”
I add, “Based on Wes’ story, we shouldn’t even let Brent touch the keyboard. He’s allowed to tell people what to type and shoulder-surf, but under no condition will we allow him to do something that we can’t document afterward. Is that clear?”
“That’s great,” Patty says. “At the end of each incident, we’ll have one more article in our knowledge base of how to fix a hairy problem and a growing pool of people who can execute the fix.”
Wes doesn’t look completely convinced, but he eventually laughs. “I like it, too. We’ll treat him like Hannibal Lecter—when we need him, we’ll put him into a straightjacket, tie him to a wheelchair, and cart him out.”
I laugh.
Patty adds, “To prevent another Brent escalation, we should log every keystroke and record the terminal session. Maybe even have someone follow him around with a video camera and turn on audit logging so we know exactly what he changed.”
I like it, although it sounds a bit extreme. However, I suspect that it will take extreme measures to get us out of this situation.
I venture, “Maybe we take away his production access, so the only way the work can get done is him telling the level 3s what to do.”
Wes guffaws. “He might quit if we did that right away.”
“So, who do we have that’s available to put into this level 3 resource pool?” I ask.
He hesitates. “Well, we have the two hires we made a year ago that were meant to help shore up Brent. One is working on creating server build standards, but we can take her off of that temporarily. There are two other engineers that we identified for cross-training years ago, but we never had the time to pursue it further. So, that’s three people.”
“I’ll define the new Brent procedures,” Patty says. “I like gating all access to him through you and Wes. But how will we discourage people like that vp of Logistics from going directly to Brent?”
I reply immediately, “We’ll collect the names of the people who do, and I’ll call each of their bosses to tell them to cease and desist. And then I’ll let Steve know how they’re disrupting Phoenix.”
“Okay, let’s give it a try,” she says. “You know, we’ve got the ‘stick’ approach covered, but what about the ‘carrot’? How can we motivate Brent and the engineers to follow this new process?”
“Maybe we send them to whatever conference or training they want. When senior engineers get to the level of Brent, or aspire to be Brent, they want to learn and share what they’ve done. As for Brent, how about we make him take a week off, completely free of any escalation duties?” Wes suggests.
“My God,” Wes continues, shaking his head. “I don’t think Brent’s even been able to take a day off without a pager in about three years. You know, he’ll burst into tears when we offer that to him.”
“Make it so, guys,” I say, smiling as I imagine that scene.
Before I forget, I add, “Wes, I want a timesheet from Brent every day, and I want every escalation Brent works in the ticketing system. We need that documented so we can analyze it later. Anyone using Brent’s time will need to justify it to me. If it’s not justified, I’ll escalate it to Steve, and that person and his manager will have to explain to Steve why they think their project or task is so important.”
“This is amazing,” Patty says. “We’ve gotten more change, incident, and escalation processes going in the last week than we have in the last five years!”
“It’s probably just in the nick of time,” Wes says, sounding relieved. “Do me a favor, and don’t tell anyone I said that. I’ve got a reputation to protect.”
CHAPTER 11
• Thursday, September 11
Later that day during lunch, I curse loudly. I was trying to use my precious few unscheduled minutes during my break to get caught up on e-mails but forgot that my crappy laptop crashes if I turn it on while it’s in the docking station. It’s the third time I’ve done it this week.
I’m already eating late and half my lunch break will be gone by the time I can log on.
Looking around, I find a blank Post-it note on my desk and write in large letters, “do not insert laptop until powered on!!!” and put it on the docking station to avert my next act of time-wasting stupidity.
I’m smiling at my countermeasure when Patty calls me on my cell phone. “You have a minute to talk? I’m seeing something very odd on the change calendar. You need to see this.”
When I walk into the conference room, I see the now familiar change cards hanging on the wall. The inbox basket is full of cards and more are neatly stacked in piles on the table. Patty is scrutinizing something on her laptop, chewing a fingernail.
Looking exhausted, she says, “I’m starting to think this entire change process is a total waste of time. Organizing all these changes and managing all the stakeholder communication is taking up three people full-time. Based on what I’m seeing now, it may be useless.”
To see her suddenly disparaging the processes she has championed for years is genuinely alarming.
“Whoa,” I say, waving both my hands in front of her. “Catch me up, because I think you’ve done a fantastic job, and I don’t want us to go back to the old ways. What has you so concerned?”
She points to the Monday and Tuesday change boxes. “At the end of each day, my people start closing out the scheduled changes. We wanted to make sure that any changes that weren’t completed were flagged so they can be rescheduled and to make sure that our change calendar was tracking what was happening in reality.”
She points to the corner of one card. “We put a check on the change cards that have been verified as completed and then indicate whether it caused a service incident or outage. Since last Friday, sixty percent of the scheduled changes didn’t get implemented! Which means we’re doing all this work to authorize and schedule these changes, only to find that they’re not even getting done!”
I can see why Patty is alarmed.
“Why aren’t they being completed? And what do you do with the incomplete change cards?” I ask.
She scratches her head. “I’ve called a bunch of the change requesters, and their reasons are all over the board. A couple people said that they couldn’t get all the people they needed to start their change. Someone else discovered halfway through his change that the storage guys didn’t finish expanding the san like they had promised, so he had to back out his change, two hours into the procedure.”
I groan, thinking about the wasted time and effort. I keep listening as Patty continues, “Someone else said that she couldn’t implement her change because there was an outage in progress. And a bunch of other people said, um…”
She looks uncomfortable, so I prompt her to continue. “Well, they said they needed Brent for a port
ion of their changes, and he wasn’t available,” she says reluctantly. “In some cases, Brent’s involvement was planned. But in other cases, they discovered they needed his help only after they started implementing and had to abort when Brent wasn’t available.”
Before Patty is even finished speaking, I’m seeing red.
“What? Brent again? What is going on? Just how has Brent managed to wedge himself into everyone’s path?
“Oh, shit!” I exclaim when it hits me what’s happening. “Did we create this problem by focusing Brent solely on Phoenix? Is this new policy a mistake?”
She says after a moment, “You know, that’s an interesting question. If you genuinely believe that Brent should only be working on the most important projects, then I think the new policy is correct, and we shouldn’t change it back.
“I think it’s also important to note that until recently, Brent was helping people implement their changes, without that dependency recorded anywhere. Or rather, he’d try to. But he’d invariably be too busy to help everyone, so many of these changes wouldn’t have been completed, even in the old way.”
I pick up my phone and speed-dial Wes, telling him to join us.
When he arrives a couple of moments later, he takes a seat and then looks at my old laptop, saying, “Jeez. You still carrying that thing around? I’m sure we have a couple of newer eight-year-old laptops that you could use.”
Ignoring his comment, Patty quickly brings him up to speed. His reaction to her revelation isn’t much different than mine.
“You’ve got to be kidding me!” he says angrily, slapping his palm on his forehead. “Maybe we should allow Brent to help people make changes?”
I quickly say, “No, that can’t be the answer. I suggested that, too. But Patty pointed out that this would imply that the blocked changes are more important than Phoenix. Which they aren’t.”
I think aloud, “Somehow, just like we’re breaking the habits of people asking Brent to help with break-fix work, we need to do the same with change implementation. We’ve got to get all this knowledge into the hands of people actually doing the work. If they can’t grok it, then maybe we have a skills problem in those teams.”