by Gene Kim
And most of the writers didn’t read the memo from the executive producers that stated they were making the story a horror movie with giant undersea monsters due to changing market tastes.
Merging code is equally difficult. Editing code isn’t like editing in Google Docs, where all the developers can see each other’s changes. Instead, like the scriptwriters, they create private working branches of the source code, their own private copy. Like those scriptwriters, developers may work in isolation for weeks, sometimes even months.
All modern source control systems have tools to automate the merge process, but when there are many changes, their limitations become all too evident. Someone may discover that someone else has overwritten their changes, or that they changed or deleted something that everyone else depended upon, or that multiple people made conflicting changes to the same parts of the code … just to name a few things that could go wrong.
Maxine loves it when everyone merges their changes frequently to the ‘master branch,’ such as once per day. That way, the size of the changes being merged are never allowed to get too large. Small batch sizes, like in manufacturing, create a smooth flow of work, with no jarring disruptions or catastrophes.
On the other hand, you have what Phoenix developers do—a hundred developers work for weeks at a time without merging, and from what Purna says, it usually takes at least three days to merge. Maxine thinks, Who would ever want to work that way?
Maxine walks with Kurt and Purna back to Building 5 to the “merging war room,” which she thinks is a very appropriate name. The moment she walks into the room, she’s hit with a wall of humid air caused by too many people being packed into a hot, crowded room. Looking around, she says to Kurt with certainty, “I don’t care what Kirsten says. There is no way we’re going to get a release branch today.”
Purna walks to the front of the room and takes out her laptop. On the walk over, Maxine learned that Purna is the integration manager responsible for making sure that all the promised features and defect fixes make it into the QA release branch. Everyone affectionately calls her the “merge boss.”
Maxine looks at the printed spreadsheet that Purna gave her. There are 392 Dev tickets to be merged. Each row has a ticket number from the Dev ticketing tool, a description of the issue, a checkbox showing whether it’s been merged, a link to the QA test plan, the QA ticket number, and on and on …
Purna is responsible for getting all these changes merged so QA can finally test them as a whole, find and report any problems, and make sure that any reported defects are fixed. It’s a big and often thankless job.
Maxine grabs a seat at the back of the room with Kurt. Gathered around the table are about twenty-five developers and managers representing each team with changes to merge. They’re bunched up in groups of two or three, with at least one laptop in front of them. Typically, one person is typing at their laptop, with the others looking over their shoulder.
There is a low buzz of frustrated conversation. “Sounds like developers merging,” Tom says, grabbing the seat next to her.
“You know the joke—what’s the plural of ‘developer’?” says Maxine. “A ‘merge conflict.’”
Tom laughs, opening his laptop. “I might as well merge all of our changes into the release branch now. I usually don’t do it right away, because what’s the rush? I mean, look around … It usually takes days for everyone to get their changes in.”
He opens up the source code management application, drags and drops a couple of things, clicks here and there, and types something. He says, “Done!” and closes his laptop.
“I usually don’t stick around,” he says. “We barely have any shared code with the rest of the Phoenix teams. I can’t remember ever having a merge issue …”
Maxine nods, thinking again about how strange it is that Data Hub is part of Phoenix.
“Do you think these people will be done merging today?” Maxine asks.
Tom laughs, pointing at the large TV in front of the room connected to Purna’s laptop. “Four changes have been merged. Five, when you include the one that we just did. That’s 387 more to go. At this rate, I think it’d be a miracle if they’re done tomorrow. Three days, I’m guessing. At least.”
Over the next hour, as people have problems with their merges, more developers enter the room to help. When people no longer have room to stand, they create a second war room across the hallway. One of the managers gripes, “I don’t know why we don’t just pre-reserve two or three conference rooms. This happens every time.”
Maxine sees a Dev lead type into a terminal window on his laptop “git pull.” He immediately gets a long error message showing forty-three merge conflicts. Maxine actually recoils from his screen in shock. She wonders how long it will take them to untangle that mess.
Later, when she hears another team talk about bringing printouts of the source code for everyone so they can manually reconcile each change, she almost spits out her coffee.
There’s a group of ten people crowded around the TV at the front of the room, studying a code diff from four different sets of changes to the same part of a file.
Seeing the expression on her face, Kurt asks, “What’s wrong?”
Speechless, she gestures at all the chaos and disruption around her. “Developers should be solving business problems … Not … this … This is madness.”
Kurt just laughs. “For sure. All the Dev managers are complaining about how much of a hassle this is. Some are lobbying to do these merges less frequently—instead of once per month, maybe once per quarter.”
Maxine blanches. “You’re kidding, right?”
“Nope,” Kurt says, genuinely amused by Maxine’s reaction. “If it hurts, do it less often. That’s their reasoning.”
“No, no, no,” Maxine says, dismayed. “They’ve got it all wrong. It hurts because the merge sizes are so large. To make it hurt less, they need to do it more frequently, so that the merges are small and create fewer conflicts.”
Kurt laughs again. “Yeah, well,” he says, shrugging his shoulders, gesturing around the room.
Maxine doesn’t laugh, because she doesn’t see anything funny. She looks at her watch. It’s nearly four thirty. She looks at Purna’s laptop screen. Only thirty-five changes are merged, with 359 more to go. They were only ten percent complete.
At this rate, Maxine thinks, it’ll take them another forty hours of work—a full week away.
The next day, Maxine is slumped in a chair in the lunchroom, surrounded by pizza. It’s almost the end of the second day of the code merge. She stares at the big posted signs everywhere, admonishing, “For Merge Teams ONLY.”
Maxine doesn’t know why they bother. Over the past day and a half, she’s guessing that every developer has been in one of the merging war rooms.
“Maxine, there you are,” Kurt says, interrupting her reverie. “Holy cow. Uhh, you look like hell, if you don’t mind me saying.”
Maxine just gives Kurt a tight-lipped smile. She just doesn’t have it in her to explain what she’s seen and how much it bothers her.
Maxine knows that code merges are never anyone’s idea of a fun time, but she wasn’t prepared for what she saw over the last two days.
She’s seen managers copy source files from computer to computer on USB drives because their teams didn’t want to use the same version control system as everyone else.
She’s seen people trying to resolve merge conflicts that were over one thousand lines long, splattered across scores of files.
She’s seen people forget to merge in their changes, caught only when Purna reconciled her spreadsheet.
She even saw two teams grapple with a genuine semantic merge conflict—a rare occurrence, usually only found in stories that developers tell to scare each other. It’s the result of an automated merge that compiles correctly but wildly changes how the program functions. The worst part was that it was a near-miss. They discovered it almost by accident. Frankly, she’s amazed tha
t they caught it when someone said, “That doesn’t look quite right.” Otherwise it would have escaped into production, where it undoubtedly would have wreaked havoc.
She keeps wondering how many similar errors weren’t caught that are now in production, sitting there like ticking bombs ready to explode when that code path is finally executed.
Looking back at Kurt, she says, “I’ve seen things. Unspeakable things, Kurt. Such waste and needless suffering … No developer should have to go through … this … this … madness!” Again, she’s at a loss for words.
“Ah,” Kurt says, suddenly looking concerned. “Throw that pizza away and come join the rest of the Rebellion. Shannon just reported out the results of her interactions with the Phoenix Dev teams, and she has a great idea.”
Maxine looks down at her hands and sees that she’s holding a cold, half-eaten slice of pizza, the cheese completely hardened into a white, greasy slab. She didn’t remember eating it.
She throws it away and follows Kurt without saying a word.
Kurt brings Maxine to another conference room, far away from the ongoing merge madness. She sees Tom, Brent, Shannon, and Dwayne gathered around the table. They all smile and wave at her. Shannon stares at her for a moment, but unlike Kurt, politely says nothing about her haggard appearance.
“Maxine, I think you’re going to love this,” Shannon says. “We’ve all been thinking about how all the Data Hub changes have been merged. But in order for them to be tested, we have to wait for everyone else to be done merging too.
“We’re thinking about what it would it take to get Data Hub decoupled from Phoenix, so that we can test independently,” Shannon continues. “If we can, we have QA people ready to work on our changes right now.”
It takes several moments for Maxine to understand what Shannon suggested, her mind still shell-shocked and ravaged from the merge. Then it hits her.
“Yes!” Maxine exclaims. “Yes, that’s a great idea. We can’t do much for the rest of the Phoenix teams right now, but that doesn’t mean that we have to suffer along with them.”
Kurt says, “I’ve been talking with Purna and Kirsten. They’ve assigned two people to help us get Data Hub tested and certified. As long as the Phoenix merge is still going on, they’re ours. In fact, I bet we could get all our changes tested before then.”
Frowning, Maxine says, “But we still need a test environment to run Data Hub in.” She thinks for a moment. “I wonder if we could create a Data Hub test environment to run in our cluster. It would be so much smaller and simpler than the Phoenix environments. That way, the QA group could use them anytime instead of the scarce environments everyone is always fighting for.”
“The environments team won’t be happy about that,” Kurt says, with a smile. “What do you need?”
She looks around. “If I had Brent and Adam’s help for two or three days, I think we could get at least a simplified environment running by Monday. I know Brent is on a Phoenix lockdown, but, hey, technically Data Hub is still part of Phoenix, right?”
Suddenly, Maxine is excited again. The idea of liberating Data Hub from the Phoenix Project morass is thrilling.
“The First Ideal,” says Brent with a smile.
The next day, Maxine, Brent, and Adam are furiously working around the clock to create a slimmed down environment that they can use to run and test Data Hub. It’s a race against the Phoenix merge.
Purna gave a thumbs-up on the plan. Kirsten did as well, saying, “We created all these rules, so we can break them too. Especially if this permanently eliminates these damned dependencies. Any project manager would jump for joy.” That was good enough for Kurt, telling them to go for it, without bothering to get any approvals from further up the chain.
“I’ll ask for forgiveness later if we need to,” Kurt had said with a smile.
At the moment, Brent is trying to reproduce the environment build that currently only works on Tom’s laptop. Meanwhile, Maxine is working with Adam, trying to get the last Data Hub release to work in their trimmed-down Phoenix environment.
She’s delighted that they’re cracking yet another piece of the build puzzle that has vexed her ever since she was exiled. They’re both watching a scrolling terminal window as Data Hub boots, hoping they resolved the last error. They’re still watching the log messages scroll by when she hears commotion from the merging war room.
One of the Dev managers yells out, “I need everyone’s attention! We’ve been having intermittent production issues for the last two hours on the e-commerce site. Something in Phoenix is causing incorrect or incomplete promotion pricing to be displayed when users are in the shopping cart. Does anyone know what could be going wrong?”
Good timing for an outage, Maxine thinks as she walks back to the war room. Virtually all the Dev managers are already in there, so it should be pretty easy to figure out which part of the code is causing the problem. It’s like having a heart attack at a cardiologist convention—lots of qualified doctors are around.
As she watches, Maxine approves of their disciplined problem-solving. They are efficient, logical, and there’s no hint of any blame as they try to replicate the problem on their laptops and systematically think through what could be going wrong.
Ten minutes later, the middleware Dev manager takes the lead. She makes a convincing case that the problem has to be in her area of the code. In takes only another fifteen minutes for her team to generate a fix. “It’s a one-line change. We can just push the change to the current release branch,” she announces. “Oh, dammit, no we can’t … Only the SCM manager can push to this old release branch. We need Jared. Anyone know where he is?”
“I’ll go find him,” yells someone, who runs out of the room.
“Who’s Jared?” Maxine asks Kurt. Kurt rubs his eyes, trying not to laugh.
The middleware Dev manager next to them says in a tired voice, “Jared is the source code manager. Developers aren’t allowed access to production. The only time developers can push changes to the release branch is for P1 issues. This is only a P3 issue,” she explains. “So, either we need to ask Ops to change it to a P1 issue, which is never going to happen, or Jared needs to grant me temporary access so I can check in our fix.”
“And if Jared were here, what would he do?” Maxine asks, knowing what’s coming.
The middleware manager says, “He’d take the commit ID of our fix, manually copy it into the release branch, and promote it into production.”
“That’s it?” Maxine asks.
“Yep,” she replies.
Maxine curses under her breath. To her surprise, she’s actually angry. Like, actually mad.
A couple of minutes ago, she thought the timing of the outage was fortunate. That lucky patient, she had thought. All the best experts in heart trauma who could correctly diagnose the problem and administer emergency treatment just happened to be in the room.
But here at Parts Unlimited, doctors aren’t allowed to touch the patient. Well, except if there’s a P1 ticket open. But if the patient isn’t on the verge of death, like right now, apparently only Jared can touch the patient. And then Jared just does whatever the doctors tell him to, because, you know, doctors can’t touch the patient. Jared isn’t a doctor. He’s probably just an administrator, just adding and removing users and making sure things are backed up.
“No one can find Jared. I think he might still be at lunch,” says the guy who searched for him.
“Oh, for Chrissakes,” Maxine mutters under her breath. It’s happening again, she thinks, remembering how wrecked she felt in the lunchroom yesterday.
Everyone tries to come up with a backup plan because no one can find Jared. Twenty minutes later, Randy shows up declaring nothing can be done, but he’s still working on finding Jared.
Everyone nods, going back to merging their changes.
“How is this okay?!” Maxine says loudly, addressing the entire room, no longer able to just watch. “Why aren’t developers able to pu
sh their own changes into production? Why do we need Jared to push the changes? I mean, I’m sure he’s really good at what he does, but why can’t we just do it ourselves?”
The entire room falls silent. Everyone stares at her, looking shocked. Like she had just belched loudly at a wedding or a funeral. Finally, someone says, “Compliance.” And another person chimes in, “And Information Security.”
Around the room, she hears people utter other reasons.
“ITIL.”
“Change management.”
“SOX.”
“PCI.”
“Regulators.”
She looks around. All of these people are capable and responsible. And yet … “Come on, everybody. Those reasons don’t make sense at all. I think I know the real reason we aren’t allowed to push our changes … they don’t trust us. Doesn’t that bother you?! How can Jared know more about making changes than the developers who wrote them?”
Scanning the room, Maxine sees that only about ten people are remotely troubled by her epiphany.
“Do they think we’ll deliberately sabotage the changes? That someone copying and pasting our changes can do a better job than we can?” Maxine knows she’s pretty far out on a limb here, but she can’t stop herself. “We’re almost all developers in this room. Doesn’t it bother anyone that we’re not trusted enough to push our own changes into production?”
A couple of people just shrug their shoulders. Several others stare back at her as if she’s crazy or hopelessly naive.
Maxine knows she hasn’t delivered a stirring Henry V Saint Crispin Day rallying speech, but she’s dumbfounded that people aren’t more bothered by this situation. She was hoping someone would yell out, “Hell yes, that bothers me, and we’re not going to take it anymore!”
But instead, there’s just silence.