by Gene Kim
CHAPTER 7
• Thursday, September 18
The owner of the familiar voice is, to Maxine’s surprise, the bartender the last time she was at the Dockside.
He sets a tray of drinks down next to Maxine and gives Kurt a friendly pat on the back. Then he turns to Kirsten, saying, “Oh, ho—if it isn’t Ms. Fingle! Long time no see! Welcome to the Dockside, headquarters of the budding Rebellion.”
“Holy cow,” Kirsten says, staring.
“Uh, you know each other?” Kurt asks, his usual confident tone missing.
Kirsten laughs. “This is Dr. Erik Reid. You may not know this, but Steve and Dick have been trying to recruit him to serve on the board of Parts Unlimited for months. He’s worked with the company for decades. In fact, Erik was part of the initial MRP rollout in the ’80s, and then he helped the manufacturing plants adopt Lean principles and practices. We were one of the first companies to have an automated MRP system, and he’s a genuine hero among the manufacturing ranks.”
“Him?” Kurt says in disbelief, pointing his thumb toward the bartender.
Maxine is surprised too. After all, she took over continuing development and operations of the amazing homegrown MRP system years ago. She’s always been impressed at how it codified not only a wonderful way of working that led to fantastic flow but also enabled continual learning, for both line workers and plant managers.
“Don’t believe everything you hear,” Erik says, snorting.
Maxine quickly sizes him up. He appears to be in his mid- to latefifties, about the right age to be the progenitor of the MRP system. He has the build of someone large who used to be in great shape. He has shoulder-length, graying hair, reminding her of The Dude from The Big Lebowski. But instead of being mellow and cool, Erik is clearly sharp and attentive.
He turns to Maxine with a sly smile. “On behalf of everyone in manufacturing operations, thanks for taking such good care of the MRP system. You’ve helped create and sustain software that is a masterpiece of simplicity and locality. You’re not only magnificently meeting the business objectives, you’ve also created a system where small teams of engineers are able to work productively and independently of each other, with components painstakingly and splendidly isolated from each other, instead of being complected into a giant, ugly, knotty mess.
“A truly magnificent feat of engineering and architecture!” he says, beaming. “The developer productivity you’ve enabled is a beautiful testament to elegant simplicity. And even more impressive is your ruthless eradication of technical debt as a part of your daily work. I’m pleased to finally meet you!”
Maxine stares at Erik. It’s not every day that a bartender compliments you on the code you’ve painstakingly written and shepherded for years, she thinks.
“Thank you—I’ll make sure to pass that on to the team,” she says, perplexed, but unable to hide her pride.
“Uh, what does ‘complected’ mean?” Kurt asks.
Erik answers, “It’s an archaic word, resurrected by Sensei Rich Hickey. ‘Complect’ means to turn something simple into something complex.
“In tightly coupled and complected systems, it’s nearly impossible to change anything, because you can’t just change one area of the code, you must change one hundred, or even a thousand, areas of the code. And even the smallest changes can cause wildly unpredictable effects in distant parts of the system, maybe in something you’ve never even heard of.
“Sensei Hickey would say, ‘Think of four strands of yarn that hang independently—that’s a simple system. Now take those same four strands of yarn and braid them together. Now you’ve complected them.’ Both configurations of yarn could fulfill the same engineering goal, but one is dramatically easier to change than the other. In the simple system, you can change one string independently without having to touch the others. Which is very good.”
Erik laughs, “However, in the complected system, when you want to make a change to one strand of yarn, you are forced to change the other three strands too. In fact, for many things you may want to do, you simply cannot, because everything is so knotted together.
“And when that happens,” he continues, “you’ve trapped yourself in a system of work where you can no longer solve real business problems easily anymore—instead, you’re forced to merely solve puzzles all day, trying to figure out how to make your small change, obstructed by your complected system every step of the way. You must schedule meetings with other teams, try to convince them to change something for you, escalate it to their managers, maybe all the way up the chain.
“Everything you do becomes increasingly distant from the real business problem you’re trying to solve,” he says. “And that, Dwayne, is what everyone discovered when they switched out the routers in those manufacturing plants. Before, you had three independent strands, with teams able to work independently but at the cost of having to maintain three networking switches.
“When you put them all on one switch, you complected their value streams, all now having dependencies on each other that didn’t exist before. They must constantly communicate, coordinate, schedule, marshal, sequence, and deconflict their work. They now have an extremely high cost of coordination, which has lengthened lead times, decreased quality, and, in your story, led to a week-long catastrophe that significantly impaired the business, going all the way up to Steve,” Erik says with glee.
“The importance of lead times in software delivery is tantamount, as Senseis Dr. Nicole Forsgren and Jez Humble have discovered in their research,” Erik says. “Code deployment lead time, code deployment frequency, and time to resolve problems are predictive of software delivery, operational performance, and organizational performance, and they correlate with burnout, employee engagement, and so much more.
“Simplicity is important because it enables locality. Locality in our code is what keeps systems loosely coupled, enabling us to deliver features faster. Teams can quickly and independently develop, test, and deploy value to customers. Locality in our organizations allows teams to make decisions without having to communicate and coordinate with people outside the team, potentially having to get approvals from distant authorities or committees so far removed from the work that they have no relevant basis to make good decisions,” he says, clearly disgusted.
“You should be able to create value by changing one file, one module, one service, one component, one API call, one container, one app, or whatever! Which is why putting cross-cutting concerns in one place is so great, like logging, security, or retry policies. You change it there, and you’ve changed it everywhere,” he says. “Isn’t it absurd that when you build a feature, changes sometimes have to be made by the UI team, the front-end team, the back-end team, and the database team?”
“Interesting,” Maxine says. “Locality in our code and organization is so desirable, as opposed to what we have now, which is code scattered everywhere!”
“Yes, exactly. Scattered!” Erik says. “And achieving this greatness is never free. It requires focus and elevation of improvement of daily work, even over daily work itself. Without this ruthless focus, every simple system degrades over time, increasingly buried under a tundra of technical debt. Just look at the disaster that is the Phoenix build system.”
Maxine furrows her brow. “You’re saying that Phoenix used to be simple, but now it has become complected beyond recognition. That Phoenix used to have a great build process, but over the years it has become neglected, taking a backseat to features, and eventually bumped out of the car entirely.”
“Precisely,” says Erik. “Build responsibility moved from Dev to QA to interns. Tech giants like Facebook, Amazon, Netflix, Google, and Microsoft give Dev productivity responsibilities to only the most senior and experienced engineers. But here at Parts Unlimited, it’s the exact opposite.”
Dwayne laughs, “At least our builds aren’t outsourced anymore. Not too long ago, it cost $85 each time a build was performed.” Everyone, including Maxine, guffa
ws in disbelief.
Kirsten says, “I hear engineers complain all the time about technical debt? But what exactly is it, besides being something bad?”
Erik laughs. “There are many definitions, but my favorite is how it was originally defined by Ward Cunningham in 2003. He said, ‘technical debt is what you feel the next time you want to make a change.’ There are many things that people call technical debt, but it usually refers to things we need to clean up, or where we need to create or restore simplicity, so that that we can quickly, confidently, and safely make changes to the system.
“Sometimes it’s a build and test system that doesn’t give fast feedback to developers, or when it stops working entirely,” he continues. “Sometimes it’s when simple components become complected, and you can no longer reason about it or change it without immense effort or risk of catastrophe. Sometimes it’s when decision-making processes or the organizational structure loses locality, forcing even small decisions to be escalated—your infamous ‘Square.’
“I’ve started calling all of these things ‘complexity debt,’ because they’re not just technical issues—they’re business issues. And it’s always a choice,” he says. “You can choose to build new features or you can choose to pay down complexity debt. When a fool spends all their time on features, the inevitable outcome is that even easy tasks become difficult and take longer to execute. And no matter how hard you try or how many people you have, it eventually collapses under its own weight , forcing you to start over from scratch.”
He looks at Maxine and says, “Which is why what you’ve done with the MRP system is so remarkable. Your teams are able to add features at a rate that the entire Phoenix team should envy. And that is only possible because you pay down technical debt as a part of daily work. It’s a magnificent example of the First Ideal of Locality and Simplicity in our code and organizations. Well done, Maxine.”
Erik stands up. “I’m a little short-staffed tonight. I’ll catch you later, and great to see you, Kirsten!
“Oh, one more thing,” he says, turning around. “Think about the engagement scores of the technology employees versus the rest of the business and ponder the differences, especially on the Phoenix Project.”
As Maxine watches Erik head back to the bar, she hears everyone burst into conversation.
Maxine says, “I have no idea what just happened.” Looking at both Kirsten and Kurt, she asks, “What was that all about? And what did he mean by the First Ideal?”
“I have no idea,” Kurt says, shaking his head. “I’ve known Erik for over a year. I had absolutely no idea he had some connection to the company …”
Dwayne says to Kurt, “I never bothered to tell you because, you know, it didn’t seem that important. But one evening he asked me whether I knew anything about configuring Kubernetes clusters. That was pretty strange.”
“That’s odd,” Shannon says. “Now that I think about it, I once had a debate with him about how completely you should or shouldn’t isolate the cardholder data environment in order to comply with the PCI Data Security Standard. He even sent me links to the specific subsections in the standard. He seemed very knowledgeable. An expert, even. I thought it was just because this bar took credit card payments …”
“I’ve heard he’s been having many conversations with Bill Palmer, the new VP of IT Operations,” Kirsten adds. “Bill told me about how Erik is teaching him something called the Three Ways and the Four Types of Work.”
“I’ve never heard of those,” Maxine says. “He only mentioned the First Ideal … I wonder how many other Ideals there are?”
“And what did he mean by engagement scores?” asks Kurt.
“I don’t know,” Kirsten says. “But I do know that we have some of the highest employee satisfaction scores in our industry … except for the IT department … which I think is negative twenty-seven.”
“Is that bad?” Dwayne asks.
Kirsten looks embarrassed. “Very bad.”
Maxine is not surprised. And yet, something bothers her. In the Town Hall, Steve talked about how much he cares about employee engagement. What does he think when he sees that the department responsible for the most strategic program in the company is miserable? Shouldn’t that worry him?
When Erik walks by with a full beer glass, Maxine gets up and rushes to catch up with him. “Thanks again for the kind words, Erik. You mentioned the First Ideal—How many of them are there and what are they?”
“Ha! That’s not the way it works,” Erik says, laughing. “In fact, I’ve got Bill Palmer running hither and yon, trying to find all the Four Types of Work, watch. But … perhaps I can give you all a head start.”
Erik and Maxine walk back to the table. “There are Five Ideals,” Erik begins. The whole table turns their attention to him. “I’ve already told you about the First Ideal of Locality and Simplicity. We need to design things so that we have locality in our systems and the organizations that build them. And we need simplicity in everything we do. The last place we want complexity is internally, whether it’s in our code, in our organization, or in our processes. The external world is complex enough, so it would be intolerable if we allow it in things we can actually control! We must make it easy to do our work.”
Maxine sits back down, opens her laptop (pleased she remembered it this time), and starts taking notes.
“The Second Ideal is Focus, Flow, and Joy. It’s all about how our daily work feels. Is our work marked by boredom and waiting for other people to get things done on our behalf? Do we blindly work on small pieces of the whole, only seeing the outcomes of our work during a deployment when everything blows up, leading to firefighting, punishment, and burnout? Or do we work in small batches, ideally single-piece flow, getting fast and continual feedback on our work? These are the conditions that allow for focus and flow, challenge, learning, discovery, mastering our domain, and even joy.”
He looks around the table with a smug expression on his face. “And that’s all you get for now. I’ll share with you the other three Ideals when you’re ready.”
“You’re kidding me,” Maxine says. “You’re pulling some sort of Yoda or Mr. Miyagi routine on us? Come on, at least tell us the names of the other Ideals!”
“Lucky for you, Young Grasshopper, I don’t have time to argue, as there’s a line at the bar I need to take care of,” he says. “In its briefest form: The Third Ideal is Improvement of Daily Work. Reflect upon what the Toyota Andon cord teaches us about how we must elevate improvement of daily work over daily work itself. The Fourth Ideal is Psychological Safety, where we make it safe to talk about problems, because solving problems requires prevention, which requires honesty, and honesty requires the absence of fear. In manufacturing, psychological safety is just as important as physical safety. And finally, the Fifth Ideal is Customer Focus, where we ruthlessly question whether something actually matters to our customers, as in, are they willing to pay us for it or is it only of value to our functional silo?”
Erik finishes his beer and says with a smile, “Good luck to you all. See you next week.”
“Wait, wait, that’s it?” Maxine says, but Erik is already gone. Maxine looks down at her quickly typed notes:
The First Ideal—Locality and Simplicity
The Second Ideal—Focus, Flow, and Joy
The Third Ideal—Improvement of Daily Work
The Fourth Ideal—Psychological Safety
The Fifth Ideal—Customer Focus
Maxine stares at the list—all of the Ideals sound nice, but how in the world are they supposed to use them to change the trajectory of the Phoenix Project?
“That was so strange,” Kurt says, saying what everyone is thinking.
Cranky Dave adds, “That bit about the Fourth Ideal hit home. A culture of fear where everyone is afraid to share bad news? That’s us.
“Erik is right,” Adam says. “No one talks about the real problem. Most people aren’t brave enough to say what they think or to do the
right thing. They just say ‘yes,’ whether they agree or not. But maybe this creates an opportunity. There are some big, gaping holes in the org chart now,” he says to Kurt. “You should throw your name into the hat for one of them. Maybe even for William’s position?”
Silence descends upon the table as everyone turns to look at Adam and Kurt.
“That’s a pretty good idea, Kurt. You could make a huge difference in the QA organization. You know all of us would be pretty happy about that,” says Shannon, with everyone around the table murmuring assent.
“Maybe,” says Kurt, nodding slowly. “But you know, if we really want to make a difference, there’s another move. I’m thinking about telling Chris that I want Peter’s position.”
Maxine hears some gasps around the table, followed by Cranky Dave’s loud laugh. “You’re right, Kurt. You would definitely make a much bigger difference by taking over a Dev team. We all know we need to change how QA does testing, but the best place to start is by changing how Dev does testing. And that requires being a Dev manager … but that brings up a teeny, tiny, little problem … they’ll never give you that position, Kurt,” he says. “You know, because you’re ‘just a QA manager.’”
Maxine winces. Cranky Dave is voicing an all-too-popular prejudice that developers have about QA people, which embarrasses her. QA is often viewed as an underclass, but at least they’re above Ops. All of which is crap, Maxine thinks. After all, she started her Ops career in high school, rotating backup tapes, and later, before graduate school, QA—if it weren’t for that background, she wouldn’t have become the person she is today. Technology is still too often a caste system.
Adam says to Kurt, “You know I’m a big fan and I love working with you—you’re a fantastic leader—but I agree with Dave. There’s no way that a bunch of Dev managers are going to let a QA manager take that spot. Maybe you should just settle for William’s old role. After all, someone has to lead QA out of the Stone Age and bring automated testing to the rest of the Phoenix Project.”