by Gene Kim
“I have to agree with your friends, Kurt,” Kirsten says. “You and I both know that William was never a big fan of yours. He never spoke very highly of you in meetings. They’re probably going to bring in someone from the outside.”
Kurt grins, seemingly not bothered by Kirsten’s observation. In his great William impersonation, Kurt says, “‘Yes, Kirsten, you are right. Although Kurt shows some potential, it’s clear to me that he doesn’t understand the testing game. Maybe in a couple of years he’ll have the maturity to run the QA organization.’”
Everyone laughs. Kurt continues in his normal voice, “Folks, here’s an opportunity for us to make a difference. But I don’t think we can do it from anywhere in the QA organization—QA as we know it is changing. We can’t keep being the people who test after the fact. We need to get into the game, and that means finagling our way into the development teams that are actually responsible for shipping features and the quality of their outcomes. Anything else is a waste of our time.”
He continues, “In fact, if we can take over Peter’s team, my goal will be to show that we can out-perform every other Dev team in the Phoenix Project. Gathered around this table is some of the best technical talent in the company, and we’ve already created the infrastructure that can bring some great technical practices to the game.”
Kurt leans forward. “If I can get Chris to give me that chance, would you all be willing to join the team and show that we can change the trajectory of the Phoenix Project?”
“Hell yes, Kurt. Count me in!” says Cranky Dave. Maxine is surprised that he’s the first to volunteer.
Maxine follows. “And me. This is what I want to work on. And I know we can run circles around all the other teams. I’ve seen the competition up close,” she says with a smile.
Everyone around the table chimes in, excited at the potential opportunity. Cranky Dave says, “Okay, we’re all in, Kurt. But frankly, I’m not holding my breath. Adam is right—you getting a Dev team is a long shot.”
Kirsten says, “Kurt, I agree with your instincts. If you want, I’ll write a letter of recommendation to Chris.”
“That would be fantastic, Kirsten,” says Kurt, beaming and obviously genuinely surprised and grateful for Kirsten’s offer. At that moment, Maxine realizes that Kurt has been operating this entire time without any real leadership air cover. He could get fired for going rogue, she realizes.
“Happy to help,” says Kirsten. “But let me be clear. I’m willing to write a letter to support Kurt’s ideas, but I really can’t be seen publicly with you all. At least, not yet. People need to see me as impartial.”
“Oh, you’re willing to give us a chance to take a risk and get fired, but you want to stay safely on the sidelines?” says Cranky Dave, halfway joking. Kirsten merely raises her glass to Dave.
PART TWO
September 23–November 9
CHAPTER 8
• Tuesday, September 23
On Tuesday of the following week, Maxine arrives at work to see Kurt beaming. “I got the job,” he says exuberantly.
“Really? The Dev job?” Maxine asks.
“Yes, the Dev job!” he says, as if he can’t quite believe it himself. “It couldn’t have happened without Kirsten’s support. I’m joining the Data Hub team, and you’re coming with me.”
“That’s awesome!” Maxine says, jubilant. “How’d you get Randy to approve my reassignment?”
“Well, he wasn’t happy about losing you. He kept going on about how you’re the best thing to happen to this place since sliced bread, but . . . well, I have my ways,” Kurt says with a sly smile.
Maxine gives him a high five.
He looks around and whispers, “All the managers are talking about something very strange happening. Apparently, the technology executives had an off-site with Steve earlier this week, and one of the things they agreed upon was a one-month feature freeze. Apparently, they’re actually hitting the brakes on feature delivery to pay down all the technical debt we’ve built up over the years!”
“Really?!” Maxine is shocked.
“They realize they need to fix all the crap that’s been built,” he says. “Ops is halting all work not related to the Phoenix Project so they can pay down technical debt and automate things. And Dev and QA will halt all feature work to pay down their technical debt too.
“This is our moment to shine. Here’s our chance to show people what engineering greatness looks like,” Kurt exclaims.
Later that day, an email goes out announcing Kurt’s new role. Maxine doesn’t want to hurt Kurt’s feelings, but she’s pretty sure that the real reason he got the job was that absolutely nobody else in Development wanted it. Data Hub is being widely touted as the “root cause” of the catastrophic crashes during and after the Phoenix release. Chris even called them out by name during one of the meetings Maxine was in, which she thought was quite unfair.
Blaming the Data Hub team for the smoking crater in the ground that was the Phoenix deployment was like blaming an airline crash on the passenger in the back of the plane who didn’t fasten his seatbelt tight enough.
She knows why blaming Data Hub is so easy. It’s one of the least glamorous technology areas in the company. Data Hub is part of a big, boring message bus system, which Maxine already loves because it’s how most of the major applications and systems of record talk to each other: the product database, pricing database, inventory management systems, order fulfillment systems, sales commissioning systems, company financials, and almost a hundred other major systems, many of them decades old.
Maxine has never liked that there are actually three inventory management systems—two for the physical stores (one was inherited by an acquisition and never retired) and another for the e-commerce channel. And there’s at least six order entry systems—three supporting the physical stores, one for e-commerce, another for OEM customers, and another for service station channel sales.
Maxine loves byzantine processes like pediatricians love sick children, but even Maxine is taken aback by just how many systems Data Hub has to talk to.
The more Maxine studies up on what Data Hub does, the more perplexed she becomes. Data Hub just didn’t seem like it should be part of Phoenix at all. After all, the majority of Data Hub was written over twenty years ago, which was long before Phoenix was even a concept.
Apparently, Data Hub used to be a collection of smaller applications that were scattered across the company. Some resided in finance with the ERP systems, some inside the manufacturing business units, and others within the Development group under Chris.
As the Phoenix juggernaut started rolling, an incredible number of new demands were put on those teams, and those teams just weren’t staffed to deal with it. Tons of new Phoenix functionality was blocked because of competing business priorities in Data Hub, and soon Phoenix features were being delayed, month after month.
Finally, as part of a re-org, all those components were rolled up into a new group called Data Hub and put under the Phoenix Project, making sure Phoenix priorities always came first. And now, everyone is blaming Data Hub for what went wrong.
On Wednesday morning, Maxine and Cranky Dave join Kurt for the first meeting with the Data Hub engineers. Maxine’s surprised to see that Cranky Dave was able to join Data Hub so quickly too. She asked him how he managed to swing that.
Cranky Dave merely smiled, saying, “One of the many benefits of my winning personality—no manager passes on the opportunity to give me to a different team. It allows me to go wherever I want.”
She stands next to Cranky Dave as the other five Data Hub engineers assemble in the central meeting area.
They’re all either her age or fresh out of college, no one in between. She suspects the senior developers have been on the team since the beginning, and the younger engineers will quickly leave for more interesting work and be replaced with other new college grads.
Chris clears his throat and addresses the room. “Good morning,
everyone. Please welcome Kurt Reznick, who will be taking over for Peter.”
Kurt seems surprised by the short introduction but says cheerily, “Hello, everyone. As you may know, this is the first Dev team I’ve managed. I believe my job my job is very simple: listen, do whatever you need me to do to help make you successful, and remove any obstacles in your way.” It’s clear from everyone’s unimpressed looks that they’re well aware of Kurt’s lack of experience.
Kurt continues, “I’ve talked with our numerous internal customers, and they told me how important Data Hub is. But they also told me about how we’re often the bottleneck for changes needed across the enterprise, as well as for the Phoenix Project. And as we all know, when our service goes down, so does Phoenix. I’ve scheduled a session later this week for us to brainstorm how we can make our service more reliable and resilient.”
“Blaming Data Hub and Peter for Phoenix going down is bullshit,” says one of the senior developers.
“I totally agree with you, Tom,” says Kurt. “And rest assured that I’ll be working to correct that perception.”
Kurt continues, “I really appreciate that Peter was willing to meet with me before I started. He told me that he’s been asking for additional headcount for senior developers for years because the business needs have kept growing, especially around the Phoenix integration. He recommended that I keep trying.”
Kurt gestures at Chris. “And I promise you that I’ll keep lobbying Chris for more headcount.”
“And I’ll keep lobbying Steve,” Chris replies with a tight-lipped smile. Kurt laughs. “So, in the meantime, I’ve brought with me two senior developers who have volunteered to join the team. Maxine is the most senior developer from the MRP team, and Dave is a senior developer from the Phoenix back-end server team. They’re two of the developers I trust the most.”
The Data Hub developers look at them, surprised but genuinely happy that she and Cranky Dave are here.
“There will be a directive coming out soon from Chris about a feature freeze, so we can work on fixing defects that impact our customers and fix problematic areas of our code,” Kurt says. “But don’t wait for the announcement. The top priority is to fix things you think should be fixed and, for that matter, anything that you think will help you be more productive or make Data Hub more stable. I’ll handle any complaints that come our way.”
Maxine smiles at the expressions of grudging approval from the Data Hub engineers.
As the new engineers, Cranky Dave and Maxine integrate themselves into the daily rituals of the Data Hub team. They attend the stand-ups and are quick to volunteer to help with things.
Maxine pairs up with Tom, the older developer who had commented on the unfairness of being scapegoated for the Phoenix failure. Tom is in his late forties and wears glasses, jeans, and a T-shirt. She sits at his desk with her laptop open as he explains what he’s currently working on.
As he talks, Maxine sees that Data Hub is a mishmash of technologies built up over the decades, including a big chunk that runs on Java servlets, some Python scripts, and something that she thinks is Delphi. There’s even a PHP web server.
She doesn’t judge or dismiss any of the technology stacks—after all, it’s been successfully serving the enterprise for decades. It may not be the most elegant piece of software she’s seen, but things that have been in production for twenty years rarely are. Software is like a city, constantly undergoing change, needing renovations and repair. She will, however, acknowledge that Data Hub is not the hippest neighborhood. It’s undoubtedly difficult to recruit new college grads who want to learn and use the hottest, most in-demand languages and frameworks.
But at least Data Hub is in much better shape than the Phoenix build systems, which were like uninhabitable, radioactive Superfund sites, or the shelled-out remains of a war zone.
Maxine is sitting at Tom’s desk as he explains what he’s working on, “I’m working on an urgent defect. Data Hub is occasionally generating incorrect message transactions and is crashing under high loads. It sometimes happens when in-store employees mark customer repair work as complete in the service station application,” he says. Looking embarrassed, he continues, “I’ve spent days working on this. I’ve finally created a semi-reproducible test case—it happens about one out of ten times. I’m pretty sure it’s because of a race condition.”
Talk about being thrown in the deep end, Maxine thinks. But she relishes the challenge and is sure that when they solve the problem, it will make a very positive impression on the entire team. After all, race conditions are one of the toughest categories of problems in all of distributed systems and software engineering. If working with the middle-school girls was a yellow belt challenge in karate, what Tom is describing can drive even the most experienced tenth-level black belts to despair and madness.
Maxine is impressed that Tom can even reproduce the problem at all. Someone once called these problems “heisenbugs,” referring to the quantum physics phenomena where the act of observation changes the nature of reality itself.
This type of work is very different than how coding is portrayed in the movies: a young, male programmer is typing away furiously, wearing a hoodie, of course, but curiously, also wearing sunglasses (which she has never actually seen a developer do in real life). He has many different windows open, text quickly scrolling by in all of them. Behind him, a crowd of people is watching over his shoulder, waiting anxiously. After a couple of seconds, the coder cries out, “I got it!” and everyone cheers. The solution is created, the feature is delivered, or the world is saved. And the scene ends.
But in reality, when developers work, they’re usually staring at the screen, deep in concentration, trying to understand what the code does so they can safely and surgically change it without breaking something else as an unintended side-effect, especially if they’re working on something mission-critical.
Tom walks her through the problem. “When there are multiple repair transactions being processed concurrently, sometimes one of the transactions gets the wrong customer ID, and sometimes Data Hub completely crashes,” he says. “I’ve tried putting a lock around the customer object, but it slowed down the entire application so much, it’s just not an option. We have enough performance problems as it is.”
Maxine nods, because Tom is confirming her long-held belief that multi-threading errors are at the very limits of what humans can reason about, especially since most mainstream programming languages like Java, C#, and JavaScript encourage mutation of shared state.
It is almost impossible to predict how a program will behave if any other part of the program can change data that you’re depending on at any time, Maxine thinks. But she’s pretty sure she knows how to fix this problem.
“Can we walk through the code path again?” Maxine asks. As they do, Maxine goes through a mental checklist to confirm her hypothesis. There’s a thread pool that handles incoming messages. Check. Service records can be handled by multiple concurrent threads. Check. The threads pass around objects, which are being mutated when its methods are called. Check.
Hypothesis confirmed. The problem is almost definitely state mutation going wrong, she thinks. Just like at the middle school.
“You’re right, it’s definitely a race condition,” Maxine says. “And I’m pretty sure we can solve this problem without putting a lock around the entire customer object. Can I show you what I’m thinking?”
When he nods, just as Maxine did with the middle-school girls, she proposes rewriting the code path using functional programming principles. Tom’s test case has a lot of mocks and stubs to simulate the production environment: a configuration server, a database, a message bus, a customer object factory . . . .
She jettisons all of them, because those are not areas of the system she wants to test. Instead, she pushes all that input/output and side-effects to the edges and creates unit tests around how an incoming repair order message is processed, how customer data is transformed, and w
hat outgoing messages are sent.
She has each thread make its own copy of the customer object. They rewrite each object method into a series of pure functions—a function whose output is completely dependent upon its inputs, with no side-effects, mutations, or accesses to global state.
When Maxine shows Tom a unit test that reproduces the problem 100% of the time, as well as the completely thread-safe fix that now works 100% of the time, Tom stares at her, eyes wide with wonder. “That’s … that’s … incredible.”
She knows why he’s impressed. Her code is so simple that it’s easy to understand and test for correctness. Eventually, marveling at the screen, he says, “I just can’t believe how much you simplified it. How can that achieve the same thing as the complex mess that we had before?” For the rest of the afternoon, he asks questions, obviously trying to prove to himself that Maxine’s test case captured the problem and that the rewrite is correct. At last he says, “I can’t believe it, but I think you’re right. This will definitely work!”
Maxine grins at Tom’s reaction. Another testament that functional programming principles are better tools to think with. And they’ve already made the code way better than when they found it—it’s definitely safer, easier to test, and way easier to understand. This is so much fun, she thinks. And a great example of the First Ideal of Locality and Simplicity.
“Okay, let’s get this fix merged in!” he says, opening up a terminal window, typing in some commands. He turns to Maxine. “Congratulations! You just fixed your first defect and checked in your first change!”
Maxine gives him a huge high-five, a big grin on her face. Vanquishing a race condition error on her first day is freaking awesome. “That’s great! So, let’s get this thing tested and pushed into production.” Maxine is excited at the thought of a grateful store manager thanking them.
“Errrr … Uhhhh …” Tom says, pausing. “Testing doesn’t start until Monday.”