by Gene Kim
She sees another pattern just like this, but for winter tires and chains, chains and windshield de-icing fluid, and many more. There’s another string of tickets for “Thanksgiving Promotion.” Each of these campaigns requires two Data Hub deployments—one to create the new product in the products database, and another to create special promotional pricing in the pricing database.
That means each discounted bundle always takes two months to create. Feeling like she’s onto something, she scans the other requested features in the promotions category. One immediately catches her attention. The ticket was created seven months ago, and the title reads, “Create in one step: new product bundle SKU with associated discount.”
Opening the ticket, Maxine reads about how Marketing wants the ability to create and price new SKUs entirely self-service, without having to go through the Data Hub team.
Yes! Exactly as I had thought. The feature description points out that the current process requires almost ninety days for a newly created discount to be available to customers.
The author of the ticket is Maggie Lee, the senior director of products. Suddenly Maxine suspects that Data Hub is sitting on an organizational constraint! She emails Kurt and Maggie. In about five minutes she gets a call from Kurt.
She croaks, “You saw my email?”
“Yep,” Kurt says. “I checked out the links you sent me. That’s definitely interesting. While you’ve been gone, I’ve been trying to figure out who our most important customers are. And I’m also on the lookout for who the heavy hitters are who can give us some air cover as we move Data Hub out of Phoenix. Maggie’s name has come up over and over again.
“She works for Sarah, and all the product owners for in-store and e-commerce report to her,” Kurt continues. “I’ll send you the org chart I dug up. I’ve already met her admin, and we have a meeting scheduled with her soon.”
“Fantastic, Kurt!” Maxine says, but when she smiles, she groans in pain. She’s excited to get back to work … when she’s healthy again.
Groaning, she hangs up, crawls back into bed, and goes to sleep.
On Monday, Maxine is back at work, discussing Data Hub with Dwayne, Tom, and Kurt in a conference room. Tom is displaying his laptop on the screen in the front of the room. “We worked on this all weekend, making sure it’s stable enough to show. Holy cow, I’m excited. We now have the Data Hub environment running entirely in a Docker image, so absolutely anyone can use it. Brent and I based it all off the work Maxine did before she was out sick. Thank you, Maxine!
“Now, instead of waiting weeks to get access to one of the scarce QA environments, you can just run this Docker image on your laptop. It takes a couple minutes to download, but only a couple of seconds to start up. It’s incredible …” Tom says, typing into a terminal window. “With Brent’s help, I got these environments wired into our CI server so it can run Data Hub tests. We are finally in the build and test business! We’re using it with QA to test the four features we got completed since the last release.”
Looking at Maxine, he says, “We have enough capacity on our CI server for anyone who wants to use it. We couldn’t have done it without your great work, Maxine.”
Tom smiles, shaking his head. “We’ve been wanting to do something like this for years, but we never had the time. I’m so excited because it will completely change how Data Hub developers spend their time. Everyone can be more productive—they’ll be able to develop and test so much faster. And maybe, if a miracle occurs, we can even get these features into production faster too.”
Kurt cheers, raising his fists in the air. “Now this is an amazing success story! We can finally start showing people the value we can deliver.”
Maxine is impressed. This is an awesome accomplishment, and she’s proud that Brent and Tom, who she met only weeks ago, were able to get so much done without her.
Kurt frowns. “Actually, I take what I said back. This is a Dev and QA success story. We still have angry business stakeholders who don’t have their features. How do you get these features into production?”
“Now that is a totally different enchilada,” Dwayne says, shaking his head and drumming his fingers on the table. “Maxine is right. There’s a long history of not letting developers push things directly into production. There’s entire institutions whose only purpose is to prevent that from happening.”
“Who’s the most powerful opposition?” Kurt asks.
“Security, most definitely,” Dwayne says. “They’re going to want to do a security review of the code before it goes into production—that’s enterprise policy. And Operations won’t be so keen on this, either. And for that matter, there are so many people in the business that have been screwed over by bad changes that most of them won’t be jumping for joy when you propose this … So, yeah, basically what I’m saying is that everyone is against developers deploying directly into production,” he says with a humorless smile.
Maxine nods. “Security is already very familiar with Data Hub. It’s not like we’re popping an entirely new application on them. We just need them to schedule their review of Data Hub separately from Phoenix.”
“Let’s meet with them. The worst they can say is no, and it’s not like we haven’t heard that before, right?” Kurt says. “So besides Security, what is the official process to get approval with Operations?”
Dwayne sighs, not responding for several moments. Finally, he says, “We probably need to go through TEP-LARB.”
“Oh,” Maxine says. Kurt flinches as if something just stung him. Tom looks around the table, confused. “Is that bad?”
“Well, there are certainly easier things to get through than the TEP-LARB …” Kurt says, staring at an empty spot on the table in front of him.
Dwayne says, “Actually, that’s a bit disingenuous, Kurt. The truth is there is nothing more difficult than getting through TEP-LARB. Nothing gets through TEP and LARB. And I should know, I’m on the LARB.”
“He’s right, Kurt,” Maxine says. “In all my years here, I’ve never been able to get anything through. It’s a ton of work to even fill out their forms, and I’ve never seen them actually approve anything. They’re the Grand Pointless Council of No.”
She looks at Dwayne and says, “No offense.”
He quickly replies with a smile, “None taken.”
“What is TEP-LARB? And why do they always say no?” Tom asks.
“‘LARB’ stands for Lead Architecture Review Board,” Dwayne explains. “It was a committee created decades ago after a whole bunch of bad things happened in technology, long before I joined the company. Someone decided to create a bunch of rules to make sure anything new was ‘properly reviewed,’” Dwayne says, air quoting with his hands.
“It’s a committee of committees. There are seven Ops architects, seven Dev architects, two Security architects, and two Enterprise architects. It’s like they’re frozen in time, still acting like it’s the 1990s,” he says. “Any major technology initiative needs their sign-off.
“And to pitch anything to them, you first have to fill out the Technology Evaluation Process form, or the TEP,” he explains. “Maxine is right. It’s a lot of effort. It’s about fifty pages these days.”
Maxine’s eyes widen. Assembling all the information to fill out the TEP was an incredible ordeal the last time she attempted it. And it was only about half that size then. She asks, “If you’re on the LARB, why haven’t you made the process easier?”
Dwayne says, “It’s a committee. They all think their job is to say no. I’m the lone radical voice in the whole group, and without more kindred spirits, it’s impossible for me to drive a yes vote or bring in younger committee members. Trust me, I’ve been trying.”
Kurt drums his fingers for a moment. “Dwayne is right. Any major technology initiative needs to go through TEP-LARB. If we don’t, they’ll kill our effort before we even get started.”
He takes a deep breath. “It pains me to say this, but I think we need to pr
epare a TEP and pitch the LARB. Just like we’re going to have to ask Security, even though we already know they’re going to say no too.”
Maxine replies, “You know, we could just run Data Hub ourselves. Like, run it completely without any help from Operations, similar to how we ran our own MRP system in my old group. Besides, whenever something goes wrong with Data Hub, it’s not like we’re not being escalated to eventually.”
Everyone looks at Maxine, shocked. In particular, Dwayne and Shannon look scandalized, as if Maxine has just proposed doing something illegal or maybe even immoral. Brent says, “But how? What about Information Security? And compliance? And the TEP-LARB?”
Maxine snorts, recalling that these were exactly the same reasons she heard for why only Jared could deploy code.
She watches Dwayne switch between nodding and shaking his head, as if two wildly opposing views were violently battling inside of his head. “Oh, wow, that would be great. But they’d never let us run this type of enterprise-class service ourselves. It’s not like we don’t have the skills on the team … we’d just have to be responsible for all the data, make sure it’s backed up and all. It would be amazing, because we could run it the way we want …”
His voice trails off. Maxine acknowledges his concerns, “That’s right. We run our own MRP system, which all our manufacturing plants rely on. That’s about as mission-critical as you can get. We do all the backups, preventive maintenance, patching … It’s not easy or simple, but it’s not exactly rocket science, either. But we’ve got some of the best Ops people in the company in this room. We can do it.”
Brent says, “Hell, yes. I fear nothing in production.”
Dwayne slowly nods. “Okay, I’m in. We need this badly, and of course, I know we can run everything ourselves just fine.”
Kurt smiles wide. “Okay, we have a Plan B. If all else fails, we operate Data Hub ourselves. This will require getting Chris on board, of course.”
Maxine chokes on her coffee, but nods with everyone in agreement.
Tom is clearly excited at the idea that everything that he’s helped build could soon be running in production. Suddenly, he frowns. “Wait, wait, wait. Does that mean we’re all going to have to wear a pager?”
“Yes,” says Brent, adamantly. “You build it, you run it.”
Tom’s excitement visibly fades.
Maxine laughs.
Even Maxine is stunned by how quickly the entire Data Hub Dev team starts using the new environments. Everyone is using one in some fashion or another. They’ve spread like wildfire. Some are just using the Docker images on their laptops, some are using environments in vagrant, Git, or terraform configurations, simulating both Dev and test environments.
More importantly, Purna and the QA teams are using the Data Hub environments as well; once features are flagged as “Ready to Test,” they’re tested within hours. And because tests are being checked in with the code themselves, it’s easy for the developer to quickly reproduce and fix the problem.
This new way of working means that many defects and even a couple of features are completely implemented and tested in one day. Because of some reporting requirements that Maxine doesn’t quite understand yet, they’re still having to use two separate ticketing tools. But the Dev and QA teams are coordinating more closely than ever. In fact, many of the QA teams are sitting side by side with developers each day. Some in Building 5 and some in Building 7.
Watching how the teams work reminds Maxine of her startup days, where everyone was working together toward a common goal. She’s amazed at how quickly attitudes changed in Data Hub.
Over the next three days, they close out more fixes as “shippable” than they used to in most months, and everyone’s energy is high and enthusiastic. More importantly, Maxine knows that everyone is having fun.
Maxine and Tom finish another issue, marking it as done in the ticketing system. Within a minute, two engineers in the chat room say that they’ll review and test it within the hour.
Maxine stands up. “Unless you need me, I’m heading over to visit Cranky Dave and Purna to see how they’re doing.”
“Hey, I’m coming with you,” says Tom, grabbing his laptop. “I’m going to help them test our fixes.”
They find Purna with Cranky Dave and another Data Hub developer, all looking closely at something on her monitor. “Whatcha doing?” Maxine asks.
“We’re finally testing the surplus inventory functionality,” Purna says.
Cranky Dave adds, “It’s to support one of the biggest Phoenix initiatives. It’s the critical glue that enables Promotions to scan the in-store inventory systems for products that have been sitting on the shelves gathering dust and ship them to one of the regional warehouses, making them eligible to be promoted on the e-commerce site.”
“This is the first time we’ve been able to get it running,” Purna says. “This feature was completed over six months ago, but in the last two releases, we couldn’t get it to work. The first time it wouldn’t connect with the inventory and customer profile systems. The next time, it couldn’t connect with the purchasing history systems. Both times there was some environment or configuration problem, but there just wasn’t time to figure it out.
“We had to yank this feature out of the release, otherwise it would have made all the other features late too,” Purna says.
This is one of the great things about using Docker containers, Maxine thinks. Containers are immutable, unable to be changed after they’re created, so if it works in Dev, it will almost certainly work in Test.
Immutability is another concept from functional programming that is making the world a more predictable and safer place, Maxine thinks, smiling.
“We’re on step twenty of the eighty-step test,” Cranky Dave says, with no trace of crankiness at all, Maxine observes. “I have a good feeling about this. We found one problem earlier today, but I fixed it in less than five minutes, and we kept marching down the list. This is great!”
Even Cranky Dave can’t be that cranky when his features are working, Maxine thinks.
He continues, “Every developer knows that in the next interval, they need to write automated tests as they write the feature, not afterward. Which reminds me, we should really have some of the QA team permanently co-located with us. It seems so silly that we have to walk to a different building to pair on small problems.”
“Great idea,” Maxine says. “Let’s give that one to Kurt—dealing with all the politics of office space and facilities is definitely in his bailiwick. But I think it would be terrific.”
“By the way, you should check out what Adam and Shannon are doing over in the conference room. I think it’ll make you smile,” Cranky Dave says, obviously trying very hard not to spill the beans.
Maxine sees Adam and Shannon at a large table with six other Dev and QA engineers around them, laptops open. Adam is projecting his laptop screen on the TV.
“Holy cow, is that what I think it is?” Maxine asks, stopping midstride and staring at the screen.
“If you mean, does this look like a continuous integration server that is doing code builds and automated tests on Data Hub for every check-in, running in the environments that you helped build? If so, then you would be absolutely right,” says Adam, a huge smile on his face.
Maxine recognizes the CI tool immediately. Everyone thinks that Data Hub is so archaic and backward, and yet it’s now running under continuous integration. They now have better technical practices than most of Phoenix.
“This is incredibly beautiful,” Maxine says, feeling misty-eyed. “Every-one from Data Hub has access? When will other teams be able to use this?”
Shannon looks up from her laptop and says, “Everyone from Data Hub is in. And as you know, getting their code into CI was one of the top requests from the Phoenix teams. Adam and I are onboarding the first teams and getting them trained. We’re going to do whatever it takes to make sure they’re successful. When they’re up and running, we
’ve got a line a mile long to be the next onboard,” she says.
Maxine savors the moment. This is what she had been hoping since her first day on the Phoenix Project. Every developer deserves to have this infrastructure to make them productive and a team of experts to help them get up and running.
She looks at the screen and sees that in the last four hours, five Data Hub developers have checked in code changes, and in two cases, the tests failed but were corrected within ten minutes.
Erik would be proud, she thinks. This fast and frequent feedback is such a big part of achieving the Second Ideal of Focus, Flow, and Joy. And all of this was enabled by properly elevating the improvement of daily work over daily work itself, as dictated by the Third Ideal.
“I love the idea of co-locating QA and Dev,” Kurt says, addressing everyone assembled at Dockside. “Although when I brought this up with some of the other Dev managers, they found the idea quite scandalous,” he adds with a smile.
“Right before I came over, I showed some proposed floor plans to the director of Facilities,” he continues. “When he saw them, he almost hit the ceiling. I actually think he wanted to confiscate them.” Kurt laughs. “He started telling me about all the rules they have about conforming to the space guidelines that HR came up with. Apparently, there are rules about how large the spaces can be based on job titles …”
“Sounds like the USDA rules about the sizes of cow pens,” says Cranky Dave. Everyone looks at him. “What? I came from a family of farmers. I had to deal with the occasional USDA audit.”
“Great,” Shannon says. “He’s calling engineers livestock now.”
“What’s the timeline, Kurt?” asks Adam.
“Nine months,” Kurt says.
Maxine hears several people repeat back, “Nine months?!” Some just guffaw.
“Yeah, well …” Kurt says, looking at his notes. “Anything that Facilities does will take forever. We’d have to order the officially supported chairs and desks through Purchasing and schedule the furniture installation with the Facilities staff …”