by Gene Kim
Flanking him are a woman and a younger man. The way they snicker makes Maxine immediately think of Crabbe and Goyle, the two mean-spirited goons who were best friends with Malfoy, Harry Potter’s rival from Slytherin house.
“Uhh …” says one of the Dev managers. “Actually, there is one component that can only run on Kumquat servers. It’s an extension we had to build off of the existing message bus. But it’s only a small modification. It shouldn’t cause any problems, and it should add negligible load …”
Maxine hears groans from around the room, and not just from the Slytherins on the opposing side of the table. The younger man sitting next to the large man, whom Maxine is already thinking of as Big Malfoy, sighs. “Technically, there’s nothing wrong with Kumquats—we have over a decade of experience running production workloads on them, and we understand their characteristics pretty well. The problem is that the reboot time for that cluster is almost eight hours. We just need to be careful about anything that could involve restarts, like security patching. I’m concerned that certain changes will require multiple reboots, which could mean a day of downtime … or that they’ll never come back at all …”
These are all the Ops people, Maxine realizes. No wonder she hasn’t seen them around.
“Wes, trust me, we are as afraid of that scenario as you are,” the Dev manager replies from across the table. “We’ve been trying for three years to get this application re-platformed, but it always takes a backseat to something more important.”
“Yeah, you developers always make sure features take priority, and you never clean up all the technical debt you create … typical crap,” Big Malfoy says, gesturing angrily.
Bill says to Big Malfoy, without even turning his head, “Stow it, Wes. Work the problem. Stay focused.”
“Yeah, yeah. Got it, boss,” Wes (Big Malfoy) says. “Handful of Linux servers, handful of Windows servers, and one Kumquat server. Got it. Now who can define a ‘handful?’”
Maxine watches all the Dev managers put their heads together, tabulating the compute needs for each of their components. It’s clear they’re just going on gut, not any sort of thoughtful capacity-planning exercise.
Maxine realizes that this release is in even more trouble than she thought. The developers still haven’t merged all their code together. And they haven’t defined the production environment that the application needs to run in—describing your environment in “handfuls” definitely does not cut it.
Raising her voice, she asks, “How many transactions per second are we expecting for product displays and orders? And how many transactions per second are the current builds capable of handling right now? That will tell us how many servers we need for the horizontally scalable portions, as well as how far we’re off for the vertically scaled components, like the database.”
The room falls silent. Everyone turns toward Maxine. They seem startled by her common sense question. The woman sitting to the left of Wes says, “Thank you! That is precisely what we need to know!”
Maxine gives a small nod and winks.
Chris stands up. “This is the highest publicity release in the entire company’s history. Marketing has pulled out all the stops. They’re going to spend almost a million dollars getting the word out about the Phoenix launch. All the store managers have been given instructions to tell every customer to download the app and hit the website Saturday—they’re even having contests to see which stores register the most new mobile customers. They’re hitting all the industry and business press. They’re trying to get either Sarah or Steve on all the news shows—even Good Morning America.
“Here are the best calculations that I’ve been able to get from Marketing,” Chris continues, flipping through his notebook. “Expect one million people to come banging on the Parts Unlimited website and mobile apps. If all goes well, we should be prepared to sustain at least two hundred orders per second.”
Maxine hears mutters and curses from all around the room.
Wes scans the room and finally turns to Chris, all sense of jocularity gone. “Okay, that’s good to know.” He gestures at Maxine, “Our smart architect just asked how many transactions Phoenix can handle right now. Well?”
Chris looks to William, who pulls out a printout. “Hot off the press from this morning. In our tests, Phoenix currently handles about five transactions per second. Anything over that causes the database clients to start crashing due to timeouts, including the mobile apps … I think we’re missing a bunch of database indexes, but we haven’t figured out which ones yet …”
William looks up. “It’s very, very not good, Chris.”
Wes sits in stunned silence for a moment. And then in a blunt, world-weary voice, he says to Chris, “We’re not going to make it, are we?”
No one says anything. Eventually, Bill asks, “Wes, what help do you need?”
“… I don’t even know,” he replies. “Maybe just give the teams some air cover so they can stay focused.”
At that moment, Maxine hears a loud voice from the doorway. “For the survival of Parts Unlimited, we have to make this work, so of course we’re going to make it.”
Oh no, Maxine thinks. It’s Sarah Moulton.
She’s dressed in a bright, expensive looking yellow suit, and her face is so radiant that Maxine wonders how it’s even possible. The fluorescent lights in the office usually make people look ghoulish and devoid of color. Maxine wonders if maybe she adds radium to her makeup to make herself glow like a 1950s bedside clock. Sarah has a certain dangerous glamour about her, and everyone in the room seems similarly rapt.
“We are in a market that is shrinking, with fierce competitors taking market share away from us,” Sarah says. “Not to mention tech giants like Amazon and twenty new startups that are coming in to disrupt this entire category. As Steve said at the Town Hall, we’ve had three years to prepare for this. Now it’s time for us to go to war and defend what is rightfully ours.”
She looks around the room, scanning for signs of resistance or rebellion. “This is the strategy the executives of this company have decided on. Anyone have a problem with that?” she challenges.
Incredibly, Maxine hears herself laughing. Horrified, she covers her mouth. Keep it together, Maxine! Quickly, she wipes all expression from her face, like a student caught doing something bad in high school. Since when have you ever cared what people in authority think about you? she wonders.
Ever since Chris warned me to keep my head down, she realizes. Maxine forces herself to look calmly at Sarah with her best Lieutenant Saavik expression, radiating only cool, dispassionate logic.
“Something seem particularly funny to you … umm, sorry, what is your name?” Sarah asks, looking at Maxine coolly.
“Maxine,” she replies, calmly. “I was laughing because you were talking about why you think Phoenix is important. But in this room, we’re just trying to figure out how to get Phoenix deployed.”
“Which is not going terribly well, either,” Wes mutters loudly to some nervous laughter.
“I can see that some of you have not bought into our mission,” Sarah says, appraising everyone in the room. “Well, as I mentioned in the Town Hall, the skills that got us here are not necessarily the same skills that will take us to where we need to go. As leaders, we need to figure out if we have the right people on the bus. I’ll be sure to keep Steve apprised. I know this release is personally very important to him.”
Upon hearing Steve’s name, Chris looks at Maxine with disbelief and then covers his face with both hands. Nice job keeping a low profile, Maxine tells herself.
“Okay, Sarah, that’s enough,” Bill says, standing up. “Let’s go apprise Steve about some of these problems and let the team figure out how to execute the release. We’re only getting in the way here.”
“Yes, Steve needs to hear about this,” she says. Sarah turns to leave, but then looks back at Maxine. “I like that you say what you think. If you’re available sometime this week, let�
�s get lunch. I’d like to get to know you better.”
What the … Maxine freezes like a deer in headlights.
“As women, we really need to stick together, don’t we?” Sarah says with a wink.
With a frozen smile, Maxine says, “Uhh … thank you—I … I’d love to.” Immediately, she hates herself, embarrassed that so many people just witnessed her lying so baldly.
“Let’s make it so,” Sarah replies with a warm smile. “And if you need someone to mentor you, I’d be happy to.” She looks at her phone and says, “That’s Steve. He needs something from me. I’ll leave you to it. Remember, we all need optimism.”
When Sarah is gone, Maxine lets out a long breath, not quite believing what happened. She knows how important it is to have a great network, able to find people who can help get important things done. But she’s not terribly excited to be associated with Sarah, no matter how influential she is. Maxine is very picky about who she associates with.
For the next hour, Maxine drifts between the various groups as the huge release team tries to fully understand what is required to support the Phoenix launch. There are at least twelve different technology stacks that need to be deployed, more than Maxine had estimated during her build archaeology.
She knew of the various application servers on Windows and Linux, and the front-end applications that run on the web, but she totally forgot about the two mobile applications (one for the iPhone and one for Android), and those all collectively hit at least ten different back-end systems from across the business, all of which required changes in order to accommodate Phoenix.
She had also forgotten that when you throw the Operations teams into the mix, the number of teams involved more than doubles, because getting all those applications running in production requires the server administration teams, virtualization teams, cloud teams, storage teams, networking teams …
All this reminds Maxine of why production deployments are some of the most complex activities of any technology organization, because they require so much coordination between so many different parts of the organization. And Phoenix wasn’t just any deployment—it was designed to change how almost every part of the organization interacts with the customer.
The more Maxine hears, the worse she feels. It seems impossible that they can get everything right the first time with this many moving parts. Getting an environment required Maxine to open up scores of tickets, and she still wasn’t successful. She’s guessing that deploying Phoenix will require hundreds, or maybe even thousands, of tickets.
The project manager in the group she’s sitting with says, “Won’t we need a bunch of firewall changes too? Not just to external traffic. I don’t think some of these systems have ever talked to each other …”
Maxine raises an eyebrow. She hears more groans around the group. “Oh, great. The firewall teams usually need at least four weeks to get change requests through,” says the woman who Maxine learns is Patty. “You think our change management process can be slow? We’re speed demons compared to Information Security.”
Suddenly, Maxine hears a door slam open behind her, and Patty looks up. “Well, speak of the devil. Here’s John, our chief information security officer. This should be fun …” she says.
John is in his late thirties. He’s about twenty pounds overweight, but his clothes are still baggy on him. Like in an old Western, John is flanked by two people—one male and one female engineer, who looks vaguely familiar. “At last, I’ve found you all,” John sneers, looking around as if he were a sheriff who had hunted down a group of outlaws. “I’m here about this mad plan to deploy the Phoenix application. This deployment will only happen over my dead body.”
The woman behind John suddenly looks embarrassed, as if she’s seen John say this before. John continues, “The Phoenix Project has millions of lines of new code, and we cannot responsibly deploy it without my team testing it for vulnerabilities. We just came out of a very, very interesting meeting with the auditors, and trust me, they aren’t going to take it very kindly if we put something into production that jeopardizes our compliance posture.
“I have it on pretty good authority that the CIO and VP of IT Operations were just fired over some compliance audit findings that were no longer tolerable,” John continues. “Let that be a warning to you that compliance is not just a moral obligation or a set of contractual obligations … it’s also the law.”
Maxine wonders how many times John’s rehearsed that line. It’s a pretty good line, she acknowledges.
Kirsten says from the front of the room, “As you know, the decision to ship Phoenix came straight from the top—Steve Masters, the CEO, and Sarah Moulton, the SVP of retail operations. In fact, Sarah was just here, reminding us of that. The release is scheduled to start at five tomorrow so that everything is live when stores open on Saturday morning.”
“That’s what you think, Kirsten,” John says. “I’m going to go speak with Steve right now. Rest assured that I will stop this madness.”
He turns to Wes. “You were there at the meeting with the auditors—tell them how serious this is and why there is no way the production release can happen tomorrow!”
Wes quickly replies, “No—leave me out of this, John. That train has left the station, and you can’t put the toothpaste back in the tube. The only thing we can do is figure out how to keep this rocket from blowing up on the launch pad and killing us all. Pardon the mixed metaphor,” he says with a loud laugh, looking around the room to see who’s with him.
“Or was that a simile?” Wes asks suddenly, with a puzzled look on his face.
The woman behind John says in a deadpan voice. “It’s a metaphor, Wes. When you say something ‘is a pile of crap,’ that’s a metaphor. When you say that something ‘is like a pile of crap,’ that is a simile.”
“Thanks, Shannon,” he says with a big smile. “I’ve always gotten those confused.”
John glares at Shannon, and then says angrily to Wes, “I will not leave you out of this, Wes. It is your moral responsibility to stop this release!” He turns to the whole room. “It is all of your moral responsibility to stop this release! You all know where I stand—as I said, this release will be deployed over my dead body.”
Wes mumbles, “We can always hope.”
Maxine hears some nervous giggles as John and his posse leave. Kirsten stands up, looking a bit uncomfortable. “Well, I should take a moment to say that we made a commitment to deploy Phoenix on Friday. But if any of you feel you have a, umm, moral obligation to not participate in this release, please let me know.”
Wes chortles. “Kirsten, going down this path is almost certainly the stupidest thing I’ve seen in my entire career … but to support the team, I promise we’ll all do what we can.” With an air of weary exhaustion and resignation, he continues, “Let’s just get it over with.”
Maxine looks around, thinking about the sudden, surreal appearance of Sarah and then John. She’s reminded of Redshirts by John Scalzi and Wil Wheaton, a funny book loosely based on a Star Trek-like universe. It’s written from the perspective of one of the redshirts, the nameless low-ranking characters wandering in the background of the show, who learns that interacting with any of the bridge officers is bad news. Whoever is chosen to beam down to the planet with the officers is doomed to die in bizarre ways: an Alteran blood worm, mind virus, carnivorous plants, an errant Klingon disruptor blast. In the book, the redshirts plant sensors everywhere to detect when the equivalents of Captain Kirk or Commander Spock come below deck so that they can hide.
She is disheartened by how Parts Unlimited executives, the bridge officers, are so disconnected from the daily work of the “redshirts” in the technology organization. It was not helpful for Sarah to remind everyone of how “saving the universe” depends on Phoenix. And it was not helpful for John to appeal to their “moral sense of correctness.”
We all know the threat the company faces is real, she thinks. The job of the bridge crew is to ensu
re the company strategy is viable, not to remind them of the strategy or to micromanage everyone to death. Their job should be to ensure everyone can get their work done.
How did this all come to be?
Maxine drags herself back to her desk with a sandwich, exhausted from the endless Phoenix release meetings, surrounded by everyone who has similarly been sucked into the launch vortex. Oddly, she also sees some people happily working at their desks, as if it were just a regular day.
Curious, she asks one of the them why he doesn’t seem very worried. He replies with a puzzled look, “I’m a developer—I work on features. I give them to QA and Ops to test and deploy. Then I work on the features for the next release. That keeps me plenty busy.”
Maxine leaves, boggling at what he said. She has never in her career abdicated all testing and deployment to someone else. How can you create anything of value if you don’t have feedback on how it’s used? she thinks.
When she gets to her desk, Kurt is there with a black three-ring binder. Seeing her, he flashes a big smile. “I have a present for you!”
It’s an eighty-page document full of tabs. Just scanning the section headings makes her heart leap—they’re the painstakingly assembled Phoenix build instructions, complete with links to documents, license keys, step-by-step tutorials, and even links to a bunch of videos. One is titled “Getting your uberjar to run in our (very) crazy, screwed up production web cluster (8 min),” and another is “How to monitor your apps despite our Ops groups (12 min).”
She sees twenty-character hexadecimal strings of activation codes and license keys. She sees user names and temporary passwords for network shares. Best of all, there’s a link to a four-node virtual machine cluster with administrative access! That means Maxine will be able to do whatever she wants without having to fill out another service desk ticket!
She’s speechless. She feels her eyes tearing up. Over license keys?