by Gene Kim
Maxine stares at Sarah in disbelief, noting that she looks tired, not her usual immaculate self. Even Sarah isn’t escaping the Phoenix carnage completely unscathed, despite her Teflon-like ability to avoid getting blamed for nearly anything in her three-year tenure at Parts Unlimited.
Wes throws up his hands. “What can we do about it? Nothing. The entire application support team is out sick. Brent just went home sick. The DBAs are out sick. Even though we’ve got the supremely competent Maxine here, she’s like me—we don’t know enough about the service to do anything except reboot the systems, which is what the support teams are already doing.”
Maxine sees that Wes is sick too—he’s congested and looks terrible. Bags underneath his red eyes, hoarse voice … she suddenly wonders if she looks as bad as he does.
“This is not acceptable, Wes,” says Sarah. “The business depends on us. The store managers depend on us. We need to do something!”
“Well, these were the risks we warned you about when you proposed proceeding with the Phoenix launch—but you emailed saying that we ‘need to break some eggs to make omelets,’ right? We’re doing everything we can, but unless you want to help reboot some servers, I’m telling you there’s nothing we can do.”
Wes continues, “But here’s something that we should talk about: How do we keep our people healthy enough so they can actually do their jobs? And how do we keep them happy enough so they don’t quit? Chris says two of his key engineers quit in the last week. I’ve lost two people on the Ops side too, and there’s a good chance I may lose three more. Who knows how many more are actively looking?
“And when that happens, we will truly be up shit creek, because then we’ll have empty meetings like this all the time,” Wes says with a halfhearted laugh that turns into coughs.
He grabs his laptop and starts walking out the door. Before he leaves, he says, “Sarah, I know you think it’s strange that we have no one left on the bench to solve this important problem, but that’s the way it is. If you want to help, learn to be a doctor or learn some middleware. In the meantime, just stay out the way because we’re doing our best.”
Maxine likes the way Wes rolls—he’s fearless and he always says what he thinks.
She makes a mental note to ask the Rebellion about recruiting Wes.
Thinking about the Rebellion, she realizes how important that group is. To her, it’s a beacon of hope. Maxine knows she may be manic and loopy from lack of sleep, but the Rebellion has assembled some of the best engineers in the company. And they could liberate everybody from … from … all of this.
We need to keep the Rebellion together and keep this important work going, she thinks.
She texts Kurt right away:
No matter what, we cannot cancel our Dockside meeting on Thursday.
His reply shows up right away.
Great minds think alike. In fact, I have a surprise for everyone. See you in two days!
By Thursday, things have stabilized substantially. The most glaring defects and performance problems in Phoenix have been fixed. And it helps that customer traffic is way, way down. Who wants to go to a store or website that can’t take orders? The result is that it’s no longer necessary for everyone to work all night. Maxine slept in until ten this morning. As she was driving into work, she realized how much she was looking forward to the Dockside meeting that evening.
As promised, Kurt texted everyone in the Rebellion:
I’ll be a little late. Dwayne and Maxine, please run the standard agenda, including the Phoenix environment build. I will be bringing a very special guest.
Maxine is pretty sure everyone will be there tonight.
But despite getting some sleep, she doesn’t feel well. She desperately hopes she is not getting whatever illness decimated her fellow co-workers. Despite that, she is very glad to be working on the Phoenix builds again.
That evening, when she arrives at the Dockside, Maxine’s excited to see everyone. She wants to find out how to get a Rebellion sticker for her laptop and to trade war stories. She’s surprised to see that everyone looks angry and dejected.
Throwing her jacket over the back of a chair, she says cheerily, “Hi, everyone! What’s got everyone so grouchy?”
Dwayne looks at her. “Read the email that was just sent out. They fired William.”
From:
Chris Allers (VP Development)
To:
All IT Employees
Date:
4:58 p.m., September 18
Subject:
Personnel changes
Effective immediately, Peter Kilpatrick (Front-End Dev Manager) will be leaving the company, and William Mason (QA Director) will be on a leave of absence. We especially appreciate all their contributions.
Please direct all front-end Dev emails to Randy and all QA-related emails to me.
Thank you, Chris
Maxine slumps as she reads the message. The witch hunt has begun. Adam shakes his head angrily. “I wasn’t a huge fan of William,” he says, “but to blame him for everything is wrong.”
In Chris’ email there’s no mention about his own culpability in the Phoenix disaster. And even though Maxine doesn’t believe in punishment or scapegoating, it’s doubly unfair that all the blame is being put on the technology organization, and no one from the business or product side is being held accountable.
Cranky Dave looks up from his phone, disgusted. “Ditto for Peter—he was just doing what the business managers demanded. What a complete shit show.”
“This is so wrong,” Shannon mutters. “I don’t suppose it would help to write a petition or anything, right? You know, lodge our protest about their firing?”
Adam says, “No one who matters is being held responsible! We should …”
He suddenly stops talking, staring slack-jawed at something behind Maxine. “Holy shit …” he finally says. Everyone next to Adam is also looking shocked at whatever is behind her.
Maxine turns around and sees Kurt walking through the entrance.
Next to him is Kirsten, the director of project management.
“My God,” Maxine hears Adam say. He looks frightened, closing his laptop and standing up, as if he’s going to flee the scene.
“Oh, for Chrissakes, sit back down, Adam,” says Maxine. “This isn’t like the secret police showing up. Not one of us has done anything wrong—have some dignity.”
Cranky Dave laughs nervously, but like everyone else, he’s already closed his laptop, as if he has something to hide.
Kirsten is wearing a fancy blazer, two steps up from Maxine’s usual casual business garb and a full four steps up from the hoodies, T-shirts, and bowling shirts worn by the other engineers around the table. People in the bar are staring, clearly wondering who invited the management suit here.
Maxine knows that she looks slightly out of place at the Dockside, but wow, Kirsten looks way out of place, like she was on the way to an event for senior law partners but had a flat tire while driving by with a dead cell phone and had to come in to find help.
Looking around, Kurt smiles and says, “For those of you who don’t know Kirsten, she leads Project Management, which is undoubtedly the most trusted organization at Parts Unlimited, despite their association with us technology people.” Kurt laughs. “All of the most important company initiatives go through Kirsten and her project management clerics, and she routinely briefs Dick Landry, our CFO, on how they’re going.”
This is true, Maxine thinks. Kirsten is truly the high priestess of order and discipline. She assigns the score of red, yellow, or green to each major initiative of the organization, which can have career-catapulting or career-ending consequences for the people involved. Besides Sarah and the VP of sales, Kirsten is the person most mentioned by the CFO in his Town Halls.
Sitting, Kirsten pours herself a beer from the pitcher on the table and then pours a glass for Kurt. Kurt introduces everyone to Kirsten and then gestures at Maxine, “Maxine is the latest
addition to our elite group of rebels. She was exiled to the Phoenix Project as punishment for the payroll outage, and of course, her vast talents have been completely wasted ever since. That is, until we recruited her to help overthrow the uncaring, ancient, powerful existing order … oh, um …” Kurt suddenly looks embarrassed, realizing Kirsten is part of that order. “Present company excepted, of course,” he finishes.
Kirsten merely raises her glass in response.
Kurt continues, “It turns out that Maxine, in her boredom and search for meaning, began working on creating repeatable Phoenix builds, something that has eluded the Phoenix teams for well over a year. We believe in many great and virtuous things, but one thing we all agree on is that getting builds going again is one of the most urgent and important engineering practices we need right now. Once we get continuous builds going, we enable automated testing. We get automated testing, we can make changes quicker, with more confidence, and without having to depend on hundreds of hours of manual testing. And that, I believe, is the critical first step for how we can deliver better value, safer, faster, and happier.
“Without continuous builds, we are like a car manufacturer without an assembly line, where anyone can do whatever they want, independent of the plant goals,” he continues. “We need to discover problems only when we are in the build or testing process, not during deployment or production.
“I’ve wanted to own this for a year, but my boss, uh, rather, my recently departed ex-boss, didn’t think it mattered. So, I’ve been taking people off my team to work on it in secret and seeking out the best engineers in the company who are willing and able to help. And Maxine has been a tremendous help in an amazingly short amount of time,” he adds.
Kurt pauses. “Uh, let’s all raise a glass to William—he and I had our differences, but he certainly didn’t deserve to take the blame for the entire Phoenix fiasco.”
Maxine raises her glass, as everyone else does the same. She takes the time to clink glasses with everyone around the table.
Looking at Kirsten, she says, “It sounds crazy, Kirsten, but I really think this group can make a big difference. I’ve seen developers wait for months to get a Dev environment. The lack of environments and centralized builds slow us down in countless ways. In fact, most Dev teams eventually stop waiting for environments or builds and just write code in isolation, without caring whether it actually works with the system as a whole.”
Maxine continues, “Look at what happened last week with the Phoenix release. Better engineering practices would have prevented so much of that. What a waste …”
“We all agree with Maxine,” Cranky Dave says. “But, Kirsten, uh, what in the world are you doing here?”
Kirsten laughs. “I’ve long harbored a suspicion that how we manage technology at this company is not working. And it’s not just the Phoenix release catastrophe. Look at all the things we need from Phoenix that are still years away on the project plan.
“Kurt has been telling me for months about the work the Rebellion is doing. But my aha moment was when Kurt pointed out that we’ve somehow created a system where hundreds of engineers are unable to get simple things done without an incredible amount of communication and coordination,” she explains. “Sure, it’s our job to safeguard the most important projects in the company. But ideally, everyone should be able to get what they need done without any help from us. Somehow, I think Project Management has turned into an army of paper pushers, being dragged into every single task because of all the dependencies.
“We track the work of nearly three hundred people working on the various parts of Phoenix. But, the real effort is even larger,” she continues. “You’d think we have thirty teams of ten people, with each team able to get things done independently. But at times, it’s like we have only one team of three hundred people … Or maybe three hundred teams of one. In either case, something is very wrong …”
She turns to Kurt. “What was that term you used? Watermelon projects? Green on the outside, but red on the inside? That’s what every one of our IT projects is these days,” Kirsten observes wryly.
She continues, “I’ve been here for fifteen years, and we’ve been playing this game of outsourcing and insourcing IT the whole time. The last time around, the CIO proclaimed that Parts Unlimited was ‘no longer in the people business,’ if you can believe that, and outsourced everything. We eventually brought most of it back in-house, but everything we got back was in worse shape than ever. And we’ve lost the capability to do some of even the most basic things ourselves. Last year, we had to make a simple schema change for our data warehouse. We put out the request to our normal list of outsourcing partners. It took them about three weeks to get an estimate back to us. They said the work would take about ten thousand hours to complete,” she says. “Before we outsourced IT, this was something we could have done in a couple of hours.”
Maxine does the math in her head. From her consulting days, she knows one fully loaded engineer works about two thousand hours per year—that’s forty hours per week, fifty-two weeks per year, if they don’t take any vacation. She bursts out laughing. “That’s five engineers working full-time for a year, just to make a database column change?! That’s something I could do in fifteen minutes!”
“Yep,” Kurt says, with a sad smile. “The data warehouse change requires work from two or three different outsourcers. You’d need to pull together meetings from the account managers from each of those teams. Each account manager would require a change fee and a feasibility study. It takes weeks to get all the technical people to agree upon a change plan, and even then, the tickets bounce back and forth for weeks. It takes a super-heroic effort to actually get the change made.”
Dwayne laughs loudly. “You think that’s bad? That’s nothing! We used to have three networking switches in all of our manufacturing plants. One for internal plant operations, one for employees and guest WiFi, and one for all our equipment vendors that need to phone home to their mothership.
“A couple of years ago, probably during budgeting season, some bean counter looked at those three networking vendors and decided to consolidate them down to one switch. Sort of makes sense, right?” he continues.
“So without asking anyone, they went ahead and did it. And not just in one plant, but in a bunch of the plants. They replaced the three switches with one bigger, beefier switch, and then moved all plant traffic onto it,” Dwayne says. “But what they didn’t know was that they had three separate outsourcers managing the three different networks. So now all three outsourcers who used to work on their own separate switches had to work on one switch and were suddenly stepping on each other’s toes all the time.
“Within a week, one of the manufacturing plants had their entire network knocked offline—absolutely nothing from inside the plant could talk with the outside world. No one could get plant scheduling information, no one could send out replenishment orders, equipment couldn’t get maintenance updates … All interfaces were dead!” Dwayne continues, still clearly in awe of the scale of the outage.
“The only thing that worked was the fax machine. Everyone from every department had to wait in line to send out things like weekly production reports to management, orders for raw materials …” Dwayne says.
Maxine bursts out laughing. “I remember that—it was incredible. We had to buy some USB printers from the local office supply store for a couple of systems that couldn’t connect to the network printers. It was like going back to the 1970s for almost a week.”
Adam mutters from across the table, “Yeah, just like we did to the in-store systems this weekend.”
Dwayne takes another drink of beer and leans back, enjoying having everyone’s attention. “You’re probably wondering why it took a week to restore service. Well, that entire time, no one took responsibility for what happened. All three outsourcers denied that it was them, even when we presented them the log files that clearly showed that one of the them had disabled everyone else’s acc
ounts. Apparently someone got tired of having their changes trampled on by the other two, so they just locked them out.”
Everyone roars in laughter, but Maxine’s jaw drops.
Dwayne continues, “That entire week all three outsourcers kept blaming each other, and the network stayed down for days. It escalated all the way to Steve. Yep. The CEO. Even after he got all the CEOs of all three outsourcers on the phone together, it still took almost twenty-four hours for the network to be restored.”
As everyone jeers, Maxine says slowly, “That’s so interesting. Consolidating network switches isn’t inherently a bad idea. Before, three teams were able to work independently on their own networks. And when they were all put on one network switch, suddenly they were coupled together, unable to work independently, having to communicate and coordinate in order to not interfere with each other, right?”
With awe in her voice, she continues, “You know, after they got put onto one switch, I bet those teams needed to create a master schedule with all of their work on it. And I’m even betting that they needed to bring in project managers where they probably didn’t need them before.
“Holy cow,” Maxine continues, on a roll. “They did it to reduce costs, but surely, in the end, it was more expensive for everyone all around. And I bet it took everyone longer to do their work, with everyone having to communicate, coordinate, get approvals, with project managers shuffling and deconflicting all the work.
“Oh, my God. It’s just like the Phoenix Project!” she exclaims.
Silence falls upon the table as everyone stares at Maxine in a mix of horror and dawning realization.
“You mean everything that’s wrong with the Phoenix Project we did to ourselves?” Shannon asks.
Kirsten looks rattled, brow furrowed, but says nothing. “Yes,” says Maxine. “I think we did it to ourselves.”
“You are correct, Maxine. You are truly on the cusp of understanding the magnitude and scale of the challenges that await you,” a voice says from behind Maxine.