The Phoenix Project

by Gene Kim

Erik seems aggravated that I named an actual person, suggesting that Brent was a piece of equipment.

  I look again at the heat treat oven. And then I see them. There are two people wearing coveralls, hard hats, and goggles. One is in front of a computer screen, punching in something, while the other is inspecting a pile of parts on a loading pallet, scanning something with his handheld computer.

  “Oh,” I say, thinking out loud. “The heat treat oven is a work center, which has workers associated with it. You asked what work centers are our constraints, and I told you that it was Brent, which can’t be right, because Brent isn’t a work center.

  “Brent is a worker, not a work center,” I say again. “And I’m betting that Brent is probably a worker supporting way too many work centers. Which is why he’s a constraint.”

  “Now we’re getting somewhere!” Erik says, smiling. Gesturing broadly at the plant floor below, he says, “Imagine if twenty-five percent of all the work centers down there could only be operated by one person named Brent. What would happen to the flow of work?”

  I close my eyes to think.

  “Work wouldn’t complete on time, because Brent can only be at one work center at a time,” I say. Enthusiastically, I continue, “That’s exactly what’s happening with us. I know that for a bunch of our planned changes, work can’t even start if Brent isn’t on hand. When that happens, we’ll escalate to Brent, telling him to drop whatever he’s doing, so some other work center can get going. We’ll be lucky if he can stay there long enough for the change to be completely implemented before he’s interrupted by someone else.”

  “Exactly!” he says.

  I’m slightly dismayed at the warm feeling of approval that I feel in response.

  “Obviously,” he continues, “every work center is made up of four things: the machine, the man, the method, and the measures. Suppose for the machine, we select the heat treat oven. The men are the two people required to execute the predefined steps, and we obviously will need measures based on the outcomes of executing the steps in the method.”

  I frown. These factory terms are vaguely familiar from my mba years. But I never thought they’d be relevant in the it domain.

  Looking for some way to write this down, I realize I left my clipboard in my car. I pat my pockets and find a small crumpled index card in my back pocket.

  I hurriedly write down, “Work center: machine, man, method, measure.”

  Erik continues, “Of course, on this plant floor, you don’t have one quarter of the work centers dependent upon one person. That would be absurd. Unfortunately for you, you do. That’s why when Brent takes a vacation, all sorts of work will just grind to a halt, because only Brent knows how to complete certain steps—steps that probably only Brent even knew existed, right?”

  I nod, unable to resist groaning. “You’re right. I’ve heard my managers complain that if Brent were hit by the proverbial bus, we’d be completely up the creek. No one knows what’s in Brent’s head. Which is one of the reason I’ve created the level 3 escalation pool.”

  I quickly explain what I did to prevent escalations to Brent during outages to keep him from being interrupted by unplanned work and how I’ve attempted to do the same thing for planned changes.

  “Good,” he says. “You’re standardizing Brent’s work so that other people can execute it. And because you’re finally getting those steps documented, you’re able to enforce some level of consistency and quality, as well. You’re not only reducing the number of work centers where Brent is required, you’re generating documentation that will enable you to automate some of them.”

  He continues, “Incidentally, until you do this, no matter how many more Brents you hire, Brent will always remain your constraint. Anyone you hire will just end up standing around.”

  I nod in understanding. This is exactly as Wes described it. Even though he got the additional headcount to hire more Brents, we never were able to actually increase our throughput.

  I feel a sudden sense of exhilaration as the pieces fall into place in my head. He’s confirming some of my deeply held intuitions and providing an underpinning theory for why I believe them.

  My elation is short-lived. He looks me over disapprovingly, “You’re asking about how to lift the project freeze. Your problem is that you keep confusing two things. Until you can separate them in your head, you’ll just walk around in circles.”

  He starts walking and I hurry after him. Soon, we’re standing over the middle of the plant floor.

  “You see that work center over there, with the yellow blinking light?” he asks, pointing.

  When I nod, he says, “Tell me what you see.”

  Wondering what it would take to have a normal conversation with him, I resume my dumb trainee role. “Some piece of machinery is apparently down—that’s what I’m guessing the blinking light indicates. There are five people huddled off to the side, including what looks like two managers. They all look concerned. There are three more people crouched down, looking into what I’m guessing is the machine inspection panel. They have flashlights and—yeah—they’re also holding screwdrivers—definitely a machine down…”

  “Good guess,” he says. “That’s probably a computerized grinder that is out of commission, and the maintenance team is working on getting it back online. What would happen if every piece of equipment down there needs Brent to fix it?”

  I laugh. “Every outage escalated immediately to Brent.”

  “Yes.” He continues, “Let’s start with your first question. Which projects are safe to release when the project freeze is lifted? Knowing how work flows through certain work centers and how some work centers require Brent and some do not, what do you think the answer is?”

  I slowly repeat what Erik just recited, trying to piece together the answer

  “I got it,” I say, smiling. “The candidate projects which are safe to release are those that don’t require Brent.”

  I smile even wider when he says, “Bingo. Pretty simple, yes?”

  My smile disappears as I think through the implications. “Wait, how do I know which projects don’t require Brent? We never think we actually need Brent until we’re halfway through the work!”

  I immediately regret asking the question as Erik glares at me. “I’m supposed to give you the answer to everything that you’re too disorganized to be able to figure out for yourself?”

  “Sorry. I’ll figure it out,” I say quickly. “You know, I’ll be so relieved when we finally know all the work that actually requires Brent.”

  “Damn right,” he says. “What you’re building is the bill of materials for all the work that you do in it Operations. But instead of a list of parts and subassemblies, like moldings, screws, and casters, you’re cataloging all the prerequisites of what you need before you can complete the work—like laptop model numbers, specifications of user information, the software and licenses needed, their configurations, version information, the security and capacity and continuity requirements, yada yada…”

  He interrupts himself, saying, “Well, to be more accurate, you’re actually building a bill of resources. That’s the bill of materials along with the list of the required work centers and the routing. Once you have that, along with the work orders and your resources, you’ll finally be able to get a handle on what your capacity and demand is. This is what will enable you to finally know whether you can accept new work and then actually be able to schedule the work.”

  Amazing. I think I almost get it.

  I’m about ask some questions, but Erik says, “Your second question was whether it was safe to start your monitoring project. You already established it doesn’t require Brent. Furthermore, you say that the goal of this project is to prevent outages, which prevents Brent escalations. More than that, when outages do occur, you’ll need less of Brent’s time to troubleshoot and fix. You’ve already identified the constraint, exploited it to squeeze the most out of it, and then you’ve subordinated the flow o
f work to the constraint. So, how important is this monitoring project?”

  I think for a moment. And then groan at the obvious answer.

  I run my fingers through my hair. “You said that we always need to be looking for ways to elevate the constraint, which means I need to do whatever is required to get more cycles from Brent. That’s exactly what the monitoring project does!”

  I say with some disbelief that I didn’t see this before, “The monitoring project is probably the most important improvement project we have—we need to start this project right away.”

  “Precisely,” Erik says. “Properly elevating preventive work is at the heart of programs like Total Productive Maintenance, which has been embraced by the Lean Community. tpm insists that we do whatever it takes to assure machine availability by elevating maintenance. As one of my senseis would say, ‘Improving daily work is even more important than doing daily work.’ The Third Way is all about ensuring that we’re continually putting tension into the system, so that we’re continually reinforcing habits and improving something. Resilience engineering tells us that we should routinely inject faults into the system, doing them frequently, to make them less painful.

  “Sensei Mike Rother says that it almost doesn’t matter what you improve, as long as you’re improving something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss.”

  Suddenly, it’s so obvious and evident. I feel like I need to call Patty right away to tell her to start the monitoring project immediately.

  Erik continues, “Sensei Rother calls this the Improvement Kata,” he continues. “He used the word kata, because he understood that repetition creates habits, and habits are what enable mastery. Whether you’re talking about sports training, learning a musical instrument, or training in the Special Forces, nothing is more to mastery than practice and drills. Studies have shown that practicing five minutes daily is better than practicing once a week for three hours. And if you want to create a genuine culture of improvement, you must create those habits.”

  Turning back to the plant floor, he continues, “Before we leave, turn your attention from the work centers to all the space between the work centers. Just as important as throttling the release of work is managing the handoffs. The wait time for a given resource is the percentage that resource is busy, divided by the percentage that resource is idle. So, if a resource is fifty percent utilized, the wait time is 50/50, or 1 unit. If the resource is ninety percent utilized, the wait time is 90/10, or nine times longer. And if the resource is ninety-nine percent utilized?”

  Although I’m not quite understanding the relevance, I do the math in my head: 99/1. I say, “Ninety-nine.”

  “Correct,” he says. “When a resource is ninety-nine percent utilized, you have to wait ninety-nine times as long as if that resource is fifty percent utilized.”

  He gestures expansively, “A critical part of the Second Way is making wait times visible, so you know when your work spends days sitting in someone’s queue—or worse, when work has to go backward, because it doesn’t have all the parts or requires rework.

  “Remember that our goal is to maximize flow. Here at mrp-8, we had a situation many years ago where certain components were never showing up at final assembly on time. Was it because we didn’t have enough resources or because certain tasks were taking too long?

  “No! When we actually followed the parts around on the plant floor, we found that for the majority of time, the parts were just sitting in queues. In other words, the ‘touch time’ was a tiny fraction of ‘total process time.’ Our expediters had to search through mountains of work to find the parts and push them through the work center,” he says incredulously.

  “That’s happening at your plant, too, so watch for it,” he says.

  I nod and say, “Erik, I’m still stuck on releasing the monitoring project. People always insist that their special project is urgent, and needs to be worked at the expense of everything else. Where do all the urgent audit and security remediation projects that John is screaming for fit in?”

  Erik looks intently at my face and finally says, “Have you heard a single word I’ve been saying in the last two weeks?”

  He looks at his watch and says, “Gotta go.”

  Startled, I watch him as he walks quickly to the catwalk exit. I have to run to catch him. He’s a big guy, probably a little over fifty years old. Despite the extra pounds he’s carrying, he moves fast.

  When I catch up to him, I say, “Wait. Are you saying that audit issues aren’t important enough to fix?”

  “I never said that,” he says, stopping in his tracks and turning to face me. “You screw up something that jeopardizes the business’ ability to maintain compliance with relevant laws and regulations? You better fix it—or you should be fired.”

  He turns around and resumes his pace, saying over his shoulder, “Tell me. All those projects that Jimmy your ciso is pushing. Do they increase the flow of project work through the it organization?”

  “No,” I quickly answer, rushing to catch up again.

  “Do they increase operational stability or decrease the time required to detect and recover from outages or security breaches?”

  I think a bit longer. “Probably not. A lot of it is just more busywork, and in most cases, the work they’re asking for is risky and actually could cause outages.”

  “Do these projects increase Brent’s capacity?”

  I laugh humorlessly. “No, the opposite. The audit issues alone could tie up Brent for the next year.”

  “And what would doing all of Jimmy’s projects do to wip levels?” he asks, opening the door that takes us back into the stairwell.

  Exasperated, I say as we descend the two sets of stairs, “It would go through the roof. Again.”

  When we reach the bottom, Erik suddenly stops and asks, “Okay. These ‘security’ projects decrease your project throughput, which is the constraint for the entire business. And swamp the most constrained resource in your organization. And they don’t do squat for scalability, availability, survivability, sustainability, security, supportability, or the defensibility of the organization.”

  He asks deadpan, “So, genius: Do Jimmy’s projects sound like a good use of time to you?”

  As I start to answer, he just opens the exit door and walks through it. Apparently, it was a rhetorical question.

  CHAPTER 21

  • Friday, September 26

  Despite breaking every speed limit on the way, I’m twenty minutes late to the audit meeting in Building 2. When I step into the conference room, I’m stunned at how packed it is.

  It’s immediately obvious that this is a high-stakes meeting, fraught with political nuance. Dick and our corporate counsel are at the head of the table.

  Opposite them are the external auditors who are legally liable for finding financial reporting errors and fraud, and yet they still want to keep us as clients.

  Dick and his team will try to show that everything the auditors have found is all a genuine misunderstanding. Their goal is to appear earnest, but indignant that their precious time is being wasted.

  It’s all political theater but high-stakes political theater that is definitely above my pay grade.

  Ann and Nancy are also here along with Wes and some other folks who look familiar.

  Then I see John and do a double take.

  My God, he looks terrible—like someone on his third day of quitting an addiction. He looks as if he thinks that the entire room will turn on him at a moment’s notice and tear him to shreds, which may not be that far from the truth.

  Sitting next to John is Erik, who is the picture of composure.

  How did he get here so quickly? And where did he change into those khaki pants and denim shirt? In the car? While he was walking?

  As I sit down next to Wes, he leans toward me. He gestures at a s
tapled set of papers and whispers, “The agenda for this meeting is to go through these two material weaknesses and the sixteen significant deficiencies. There’s John, looking like he’s in front of the firing squad, waiting for the bullet.”

  I see the sweat stains under John’s arms, and think to myself, Good grief, John. Pull yourself together. I’m the operational manager where all those it deficiencies reside, so I’m actually the one on the firing line, not you.

  But unlike John, I’ve had the benefit of having Erik’s constant reassurances that everything will work out.

  Then again, Erik doesn’t have his ass on the line and for a brief moment, I wonder whether I should be as nervous as John.

  Five hours later, the conference table is covered with marked-up papers and empty cups of coffee, the room smelling stale and rank from all the tension and heated arguments.

  I look up at the sound of the audit partner closing his briefcase.

  He says to Dick, “Given this new data, it does appear that for the two potential material weaknesses, the it controls may indeed be out of scope and thus can be resolved very quickly. Thank you in advance for making yourselves available to get us the documentation we need to close out these issues as expeditiously as possible.

  “We will take all this under advisement and send you something in the next day or two,” he continues. “Most likely, we’ll want to schedule further testing of these newly documented downstream controls to make sure they were in place and operating—to support the financial statement assertions you’re making.”

  As he stands up, I stare in disbelief at the audit partner. We really dodged the bullet. Looking around the table, the Parts Unlimited team looks equally surprised.

  One exception is Erik, who just nods approvingly, obviously irritated that it took so long to finally have the auditors on the run.

  The other exception is John. He looks extremely distraught, sitting with his shoulders slumped over that I’m suddenly concerned about his well-being.

  I’m about to get up to check on John when the audit partner shakes Dick’s hand and, to my surprise, Erik gets up to give him a hug.