by Gene Kim
All the firefighting displaced all the planned work, both projects and changes.
Ah… Now I see it.
What can displace planned work?
Unplanned work.
Of course.
I laugh uproariously, which earns me a look of genuine concern from Patty, who even takes a step back from me.
That’s why Erik called it the most destructive type of work. It’s not really work at all, like the others. The others are what you planned on doing, allegedly because you needed to do it.
Unplanned work is what prevents you from doing it. Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it. Like Phoenix.
So much of what I’ve been trying to do during my short tenure as vp of it Operations is to prevent unplanned work from happening: coordinating changes better so they don’t fail, ensuring the orderly handling of incidents and outages to prevent interrupting key resources, doing whatever it takes so that Brent won’t be escalated to…
I’ve been doing it mostly by instinct. I knew it was what had to be done, because people were working on the wrong things. I tried to take all necessary steps to keep people from doing wrong work, or rather, unplanned work.
I say, cackling and pumping my arms as if I had just scored a game-winning, sixty-yard field goal, “Yes! I see it now! It really is unplanned work! The fourth category of work is unplanned work!”
My ebullient mood is tempered when I look at Patty, who looks puzzled and genuinely concerned.
“I promise to explain later,” I say. “Just what is it that you wanted me to see on the change board?”
She’s taken aback, but points again at the void of completed changes for the past week. “I know you were concerned about when sixty percent of the changes weren’t getting completed. So, I thought you’d really flip your lid when one hundred percent of the changes didn’t complete. Right?”
“Yep. This is great work, Patty. Keep it up!” I say agreeably.
And then I turn around and head out the door, reaching for my cell phone. There’s someone I need to call.
“Hey!” Patty calls out. “Aren’t you going to fill me in?”
I yell over my shoulder, “Later! I promise!”
Back at my desk I search everywhere for that slip of paper that Erik gave me. I’m pretty sure I didn’t throw it away, but I honestly didn’t think that I would ever use it.
I hear Ellen say from behind me, “Need help with anything?”
And soon, both of us are scrounging all over my desk to find that little piece of paper.
“Is this it?” she asks, holding up something that she retrieved from my inbox.
I look more closely, and yes! It’s the crumpled two-inch strip of paper that Erik gave me. It looks like a gum wrapper.
Taking the piece of paper from her and holding it up, I say, “Great! Thank you so much for finding this—believe it or not, this may be the most important piece of paper I’ve gotten in years.”
I decide to sit outside while I talk. In the bright autumn sunlight, I find a spot on a bench near the parking lot. As I sit down, there’s not a cloud in the sky.
I call Erik, who answers on the first ring. “Hey, Bill. How are you guys doing after Phoenix crashed and burned so spectacularly?”
“Yeah, well… Things are improving,” I say. “You may have heard that our pos systems went down, and we also had a small credit card number breach.”
“Ha! ‘Small credit card breach.’ I like that. Like ‘small nuclear reactor meltdown.’ I’ve gotta write that one down,” he says, snorting.
He’s chuckling as though he predicted this level of calamity would occur, which, come to think of it, I suppose he did, in the conference room when I first met him. Something about “clearing the calendar.”
Just like clearing the change board, I realize. I kick myself for not picking up on his clue sooner.
“I trust you can tell me now what the four categories of work are?” I hear him ask.
“Yes, I think I can,” I say. “At the plant, I gave you one category, which was business projects, like Phoenix,” I say. “Later, I realized that I didn’t mention internal it projects. A week after that, I realized that changes are another category of work. But it was only after the Phoenix fiasco that I saw the last one, because of how it prevented all other work from getting completed, and that’s the last category, isn’t it? Firefighting. Unplanned work.”
“Precisely!” I hear Erik say. “You even used the term I like most for it: unplanned work. Firefighting is vividly descriptive, but ‘unplanned work’ is even better. It might even be better to call it ‘anti-work,’ since it further highlights its destructive and avoidable nature.
“Unlike the other categories of work, unplanned work is recovery work, which almost always takes you away from your goals. That’s why it’s so important to know where your unplanned work is coming from.”
I smile as he acknowledges my correct answer, and am oddly pleased that he validated my antimatter notion of unplanned work, as well.
He says, “What is this change board that you mentioned?”
I tell him about my attempts to get some sort of change process going and my attempt to elevate the discussion above how many fields there were on the change form, which then resulted in getting people to put their intended changes on index cards and our need to juggle them on the board.
“Very good,” he says. “You’ve put together tools to help with the visual management of work and pulling work through the system. This is a critical part of the First Way, which is creating fast flow of work through Development and it Operations. Index cards on a kanban board is one of the best mechanisms to do this, because everyone can see wip. Now you must continually eradicate your largest sources of unplanned work, per the Second Way.”
Until now, having been so wrapped up in defining what work is, I had forgotten about Erik and his Three Ways. I dismissed them before, but I’m now listening closely to his every word.
And over the next forty-five minutes, I find myself telling him the entire tale of my short tenure. I’m interrupted only by Erik’s loud laughs and guffaws as I describe the calamities and my attempts to curb the chaos.
When I’m finished, he says, “You’ve come much further than I thought: You’ve started to take steps to stabilize the operational environment, you’ve started to visually manage wip within it Operations, and you’ve started to protect your constraint, Brent. You’ve also reinforced a culture of operational rigor and discipline. Well done, Bill.”
I furrow my brows and say, “Wait. Brent is my constraint? What do you mean?”
He replies, “Ah, well if we’re going to talk about your next steps, you definitely need to know about constraints because you need to increase flow. Right now, nothing is more important.”
Erik assumes a lecturing voice as he starts, “You say you learned about plant operations management when you were in business school. I hope as part of your curriculum, you read The Goal by Dr. Eli Goldratt. If you don’t have a copy anymore, get another one. You’re going to need it.”
I think my copy of that book is in my office at home. As I jot a quick reminder to look for it, he continues, “Sensei Goldratt taught us that in most plants, there are a very small number of resources, whether it’s men, machines, or materials, that dictates the output of the entire system. We call this the constraint—or bottleneck. Either term works. Whatever you call it, until you create a trusted system to manage the flow of work to the constraint, the constraint is constantly wasted, which means that the constraint is likely being drastically underutilized.
“That means you’re not delivering to the business the full capacity available to you. It also likely means that you’re not paying down technical debt, so your problems and amount of unplanned work continues to increase over time,” he says.
He continues, “You’ve identified this Brent
person as a constraint to restore service. Trust me, you’ll find that he constrains many other important flows of work, as well.”
I try to interrupt to ask a question, but he continues headlong, “There are five focusing steps which Sensei Goldratt describes in The Goal: Step 1 is to identify the constraint. You’ve done that, so congratulations. Keep challenging yourself to really make sure that’s your organizational constraint, because if you’re wrong, nothing you do will matter. Remember, any improvement not made at the constraint is just an illusion, yes?
“Step 2 is to exploit the constraint,” he continues. “In other words, make sure that the constraint is not allowed to waste any time. Ever. It should never be waiting on any other resource for anything, and it should always be working on the highest priority commitment the it Operations organization has made to the rest of the enterprise. Always.”
I hear him say encouragingly, “You’ve done a good job exploiting the constraint on several fronts. You’ve reduced reliance on Brent for unplanned work and outages. You’ve even started to figure out how to exploit Brent better for the three other types of work: business and it projects and changes. Remember, unplanned work kills your ability to do planned work, so you must always do whatever it takes to eradicate it. Murphy does exist, so you’ll always have unplanned work, but it must be handled efficiently. You’ve still got a long way to go.”
In a more stern voice, he says, “But you’re ready to start thinking about Step 3, which is to subordinate the constraint. In the Theory of Constraints, this is typically implemented by something called Drum-Buffer-Rope. In The Goal, the main character, Alex, learns about this when he discovers that Herbie, the slowest Boy Scout in the troop, actually dictates the entire group’s marching pace. Alex moved Herbie to the front of the line to prevent kids from going on too far ahead. Later at Alex’s plant, he started to release all work in accordance to the rate it could be consumed by the heat treat ovens, which was his plant’s bottleneck. That was his real-life Herbie.”
“Fully two decades after The Goal was published,” he continues, “Sensei David J. Anderson developed techniques of using a kanban board to release work and control wip for Development and it Operations. You may find that of interest. You and Penelope are close with your change board to a kanban board that can manage flow.”
“So, here’s your homework,” he says. “Figure out how to set the tempo of work according to Brent. Once you make the appropriate mapping of it Operations to work on the plant floor, it will be obvious. Call me when you’ve figured it out.”
“Wait, wait,” I say, hurriedly before he hangs up. “I’ll do the homework, but aren’t we missing the entire point here? What caused all the unplanned work is Phoenix. Why are we focusing on Brent right now? Don’t we need to address all the issues with Phoenix inside of Development, where all the unplanned work actually came from?”
“Now you sound just like Jimmy, complaining about things you can’t control,” he sighs. “Of course Phoenix is causing all the problems. You get what you design for. Chester, your peer in Development, is spending all his cycles on features, instead of stability, security, scalability, manageability, operability, continuity, and all those other beautiful ’itties.
“On the other end of the assembly line, Jimmy keeps trying to retrofit production controls after the toothpaste is out of the tube,” he says, scoffing. “Hopeless! Futile! It’ll never work! You need to design these things, what some call ‘nonfunctional requirements,’ into the product. But your problem is that the person who knows the most about where your technical debt is and how to actually build code that is designed for Operations is too busy. You know who that person is, don’t you?”
I groan. “Brent.”
“Yep,” he says. “Without solving your Brent problem, you’ll just be inviting him to design and architecture meetings with Development, but he’ll never show up because…”
Being prompted again, I respond, “Unplanned work.”
“Good!” he says. “You’re getting better at this. But before you get a big head, I’ll tell you that there’s still a big piece of the First Way that you’re missing. Jimmy’s problem with the auditors shows that he can’t distinguish what work matters to the business versus what doesn’t. And incidentally, you have the same problem, too. Remember, it goes beyond reducing wip. Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever.”
He continues, “Remember, outcomes are what matter—not the process, not controls, or, for that matter, what work you complete.”
I sigh. Just when I think I have a concrete enough understanding of constraints, once again Erik becomes illusive.
“Don’t get distracted. Call me when you know how to throttle release of work to Brent,” he says and hangs up.
I can’t believe it. I try calling him back twice, but it rolls immediately to his voicemail.
Sitting down on the bench, I lean back, take a deep breath, and force myself to enjoy the warm morning. I hear birds chirping and the noise of traffic from the road.
Then, for the next ten minutes, I capture as much as I can remember on my clipboard, trying to piece together what Erik covered.
When I’m done, I head inside to call Wes and Patty. I know exactly what I need to do and am excited to get started.
CHAPTER 16
• Thursday, September 18
I’m at my desk, tying up some loose ends when Ellen runs up to me, holding an e-mail printout. It’s from Dick, raising the alarm with all company executives that something has gone terribly wrong with the company invoicing systems. Earlier today, one of the clerks discovered that no customers had been invoiced for three days. Among other things, this means that customers haven’t been paying on time, which means the company will have less cash in the bank at the end of the quarter than projected, which will raise all sorts of uncomfortable questions when the company earnings are announced.
It’s clear from Dick’s string of e-mails that he’s livid, and apparently, his whole accounts receivable staff and controller have been chain smoking and doing damage control at all levels.
From: Dick Landry
To: Steve Masters
Cc: Bill Palmer
Date: September 18, 3:11 PM
Priority: Highest
Subject: ACTION NEEDED: Potential $50MM cash shortfall due to IT failure
ALL CUSTOMER INVOICES ARE STILL STUCK OR MISSING IN THE SYSTEM. WE CAN’T EVEN RETRIEVE THEM TO MANUALLY SEND INVOICES BY E-MAIL!
We’re trying to figure out how we can resume normal business operations. There’s likely $50MM of receivables stuck in the system, which will be missing from our cash account at end of quarter.
Get your IT guys to fix this. The hole this blows in our quarterly numbers will be impossible to hide, and maybe even impossible to explain away.
Call me, Steve. I’ll be on the window ledge.
Dick
We’re all gathered in the noc conference room. I’m pleased that when Patty finishes describing the incident, she quickly presented all the relevant changes for the last seventy-two hours.
After she’s done, I say firmly to the entire team, “First and foremost on my mind is the risk of losing transactions. Ladies and gentlemen, I need to be very clear about this: do not touch anything without getting approval from me. This is not an outage we’re dealing with here. We’re in a situation where we could accidentally lose order entry or accounts receivable data. This terrifies me. And that should absolutely terrify you.
“As Patty said, we need timelines and hypotheses for what might have caused the invoicing system to fail,” I say. “This is our Apollo 13 moment, and I’m Gene Kranz in Houston Mission Control. I don’t want guesswork. I want hypotheses backed up with facts. So get b
ack to your screens, assemble timelines and data, and I want to hear your best thinking on cause and effect. Failure is not an option.”
By 6 p.m., Patty’s team has documented over twenty different potential failure causes that have been proposed. After further investigation, eight remain as likely possibilities. An owner has been assigned to look into each.
Realizing that there’s little more we can do as a group until they complete their research, we agree to reconvene at 10 p.m. tonight.
On the one hand, I’m frustrated that once again, we’ve been plunged into a crisis and our day is dominated by unplanned incident work. On the other hand, I feel a deep sense of satisfaction at the orderly nature of our incident investigation and quickly text Paige that I’ll be joining the family for dinner shortly.
“Daddy,” I hear, as I’m sitting in bed with Grant, trying to put him to sleep while keeping the thoughts of outages out of my head. “Why doesn’t Thomas the Tank Engine have a tender car? Why?”
Smiling down at him, I marvel at the questions my three-year-old son comes up with. We’re going through our nighttime ritual of reading books. I’m glad to be doing this again, which I do every night. Or did, that is, until the Phoenix recovery effort.
Most of the lights are off, but one lamp is still dimly lit. There is a pile of books on Grant’s bed, and we’re on the third one of the night.
I’m starting to get a little dry-mouthed from reading. The idea of taking a little break and doing some research on the Internet on train tender cars sounds pretty appealing.
I love how inquisitive my kids are and how much they love books, but there are nights when I’m so exhausted that I’ve actually fallen asleep during our nightly ritual. My wife will walk in, find me asleep with one of Grant’s books lying on top of my face and Grant asleep beside me.