B00AZRBLHO EBOK

by Kim, Gene

  Steve replies, “This quarter is make or break. We promised the world that we’d get Phoenix out last month, but because of all the features we delayed, we’re not getting the sales benefits that we hoped for. Now we’re over a month through the quarter, with the holiday buying season in fewer than thirty days. We are out of time.”

  Thinking this through, I force myself to accept that Chris is stating the reality he sees, and that it is based on facts. And the same goes for Steve.

  I say to Chris, “If you say that the Phoenix team needs to slow down, you won’t get any argument from me. In the Marines, when you have a company of a hundred men with a man wounded, the first thing you lose is mobility.

  “But we still need to figure out how to achieve what Steve needs,” I continue. “As Erik suggested, if we can’t do that inside the Phoenix framework, maybe we can do it outside of Phoenix. I propose we form a SWAT team by detaching a small squad from the main Phoenix team, telling them to figure out what features can help us hit our revenue goals as soon as possible. There’s not a lot of time, so we’ll need to select the features carefully. We’ll tell them that they’re allowed to break whatever rules required to get the job done.”

  Chris considers this for a moment and finally nods. “Phoenix is all about helping customers buy things from us, faster and in larger quantities. The last two releases have all been putting down the groundwork to make that happen, but the features to really increase sales are still bogged down. We need to focus on generating good customer recommendations and enable Marketing to create promotions to sell profitable products that we have in inventory.”

  “We have years of customer purchasing data and because of our branded credit cards, we know our customer demographics and preferences,” Steve interjects, leaning forward. “Marketing assures me that we can create some really compelling offers to our customers, if we could only get those features shipped.”

  Chris, Wes, and Patty dive in to discuss this further, while John looks dubious. Eventually Wes says, “You know, this just may work.” When everyone nods, including John, I feel there’s a sense of excitement and possibility that was missing just minutes ago.

  Chapter 30

  • Monday, November 3

  An hour after the meeting with Steve adjourned, I’m still mulling over Erik’s cryptic comments. I feel like we’re on the verge of something big, but I have too many questions. I finally decide to call him.

  “Yeah?” he answers.

  “Bill here,” I say. “I need some more clues about what the hell we’re supposed to be doing…”

  “Meet me outside the building,” he says, hanging up.

  When I get outside, the wind is gusty and fierce. I look around for a couple of moments, when I hear a horn honk. Erik is in an expensive-looking red BMW convertible, with the top down. “Come on in. Hurry!”

  “Nice ride,” I say, climbing into the passenger seat.

  “Thanks,” he says. “My friend insisted that I borrow this while I’m in town.”

  As he floors the accelerator, I grab the armrest and hurriedly buckle my seatbelt. I see a purse on the floor, and immediately wonder who this “friend” is.

  “We’re heading back to MRP-8,” he says.

  When I ask him to raise the convertible top, he looks over at me and says, “I thought there was no such thing as an ‘ex-Marine.’ Maybe they made you guys softer than in my day.”

  “You were in the service?” I ask, trying to hide my chattering teeth.

  He laughs. “Over twenty years.”

  “You retired as an officer, I suppose?” I asked.

  “Major, Special Forces, US Army,” he replies, looking at me. I keep hoping he’ll keep his eyes on the road, given how fast we’re going. Instead, he continues, “Same branch as Steve, but he joined as an officer. I joined as an enlisted grunt, just like you.”

  He doesn’t reveal any more, but he has already told me enough to understand his military career. He was obviously a senior NCO, like many I had to deal with on a daily basis, now recognizing his all-too-familiar demeanor and physical bearing. He must have been identified as one of those rare high-potential people by the higher-ups, who decided to invest in his future, sending him to college and Officer Candidate School, then rejoining the ranks as the oldest second lieutenant around, probably ten years older than everybody else.

  It takes a special person to go through that.

  We make it to the plant in record time and are now standing on the catwalk. He begins the speech that I’ve been expecting. “A manufacturing plant is a system. The raw materials start on one side, and a million things need to go just right in order for it to leave as finished goods as scheduled out the other side. Everything works together. If any work center is warring with the other work centers, especially if Manufacturing is at war with Engineering, every inch of progress will be a struggle.”

  Erik turns to me, pointing, “You’ve got to stop thinking like a work center supervisor. You need to think bigger, like a plant manager. Or better yet, think like the person who designed this manufacturing plant and all of the processes it relies upon. They look at the entire flow of work, identify where the constraints are, and use every possible technology and bit of process knowledge they have to ensure work is performed effectively and efficiently. They harness their ‘inner-Allspaw.’”

  I’m about to ask what he means by an “Allspaw,” when he just waves my question away. “In manufacturing, we have a measure called takt time, which is the cycle time needed in order to keep up with customer demand. If any operation in the flow of work takes longer than the takt time, you will not be able to keep up with customer demand.”

  “So when you run around screaming, ‘Oh no! We don’t have environments for Phoenix ready! Help, help! Oh, no! We can’t deploy, because someone broke the Phoenix environments again!’” he says in a high, girlish voice, “That means the cycle time of some critical operation in your area of responsibility is greater than the takt time. That is the reason you can’t keep up with customer demand.

  “As part of the Second Way, you need to create a feedback loop that goes all the way back to the earliest parts of product definition, design, and development,” he says. “Given the conversations you’re having with Dick, you may even be able to go earlier in the process.”

  Pointing at the floor, he says, “Look down at the long lane of equipment between the orange tape on the floor. That lane makes some of the highest profit items we have. But as fate would have it, that particular flow of work involves two operations that have the longest setup and process times: application of a paint powder coating and baking it in the heat treat oven.”

  He looks up, arms spread outward. “Back in the day, the cycle time for those two operations was so much larger than takt time, we were never able to keep up with customer demand. How can life be so unfair? The most profitable items used both of our constraints: the heat treat oven and the paint booths! What do we do?

  “Customers were even offering to throw money at us, begging us for more of these widgets, but we had to turn them away. The setup time for each job took hours or even days. We had to use enormous batch sizes to meet demand. We had these huge trays to paint and would bake as many units at a time as possible. We knew we had to reduce batch sizes to improve throughput, but everyone said that it couldn’t be done.”

  “How Toyota solved this problem is legendary,” he says. “During the 1950s, they had a hood stamping process that had a change-over time of almost three days. It required moving huge, heavy dies that weighed many tons. Like us, the setup times were so long that they needed to use large batch sizes, which prevented them from using one stamping machine to manufacture multiple different car models simultaneously. You can’t make one hood for a Prius and then one hood for a Camry if it takes you three days to do the changeovers, right?

  “What did they do?” he asks rhetorically. “They
closely observed all the steps required to do the changeover, and then put in a series of preparations and improvements that brought the changeover time down to under ten minutes. And that, of course, is where the legendary ‘single-minute exchange of die’ term comes from.

  “We studied all the works of Ohno, Spear, and Rother. We knew that we had to decrease our batch size, but we weren’t dealing with hood stamping dies. We were dealing with painting and curing,” he continues. “After weeks of brainstorming, investigation, and experimentation with Engineering, we had a crazy idea: Maybe we could do the painting and curing in a single machine. We cobbled together an oven that also applied the paint powder onto the parts, which were pulled through on a chain and gear that we took from a bicycle.

  “We combined four work centers into one, eliminating over thirty manual, error-prone steps, completely automating the work cycle, achieving single-piece flow, and eliminating all that setup time. Throughput went through the roof.

  “The benefits were enormous,” he says with pride. “First, when defects were found, we fixed them immediately and we didn’t have to scrap all the other parts in that batch. Second, WIP was brought down because each work center never overproduced product, only to sit in the queue of the next work center. But the most important benefit was that order lead times were cut from one month to less than a week. We could build and deliver whatever and however many the customer wanted and never had a warehouse full of crap that we’d need to liquidate at fire-sale prices.

  “So, now it’s your turn,” he says sternly, poking a finger into my chest. “You’ve got to figure out how to decrease your changeover time and enable faster deployment cycle time.

  “I think your target should be…” he says, pausing for a moment. “Ten deploys a day. Why not?”

  My jaw drops. “That’s impossible.”

  “Oh, really?” he says, deadpan. “Let me tell you a story. Back in 2009, I was a board director at a technology company, where one of our engineers went to the Velocity Conference and came back raving like a madman, full of dangerous, impossible ideas. He saw a presentation given by John Allspaw and his colleague Paul Hammond that flipped the world on its head. Allspaw and Hammond ran the IT Operations and Engineering groups at Flickr. Instead of fighting like cats and dogs, they talked about how they were working together to routinely do ten deploys a day! This is in a world when most IT organizations were mostly doing quarterly or annual deployments. Imagine that. He was doing deploys at a rate one thousand times faster than the previous state of the art.

  “Let me tell you,” he continues, “we all thought that this engineer had lost his marbles. But I learned that the practices that Allspaw and Hammond espoused are the inevitable outcome of applying the Three Ways to the IT value stream. It totally changed how we managed IT and it saved our company.

  “How did they do it?” I ask, dumbfounded.

  “Good question,” he replies. “Allspaw taught us that Dev and Ops working together, along with QA and the business, are a super-tribe that can achieve amazing things. They also knew that until code is in production, no value is actually being generated, because it’s merely WIP stuck in the system. He kept reducing the batch size, enabling fast feature flow. In part, he did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, just like the application that Development ships. That enabled him to create a one-step environment creation and deploy procedure, just like we figured out a way to do one-step painting and curing.

  “So, we now know that Allspaw and Hammond weren’t so crazy after all. Jez Humble and Dave Farley independently came to the same conclusions, and then codified the practices and principles that enable multiple deployments per day in their seminal book Continuous Delivery. Eric Ries then showed us how this capability can help the business learn and win in his Lean Startup work.”

  As Erik talks, he is as animated as I’ve ever seen him. Shaking his head, he looks sternly at me.

  “Your next step should be obvious by now, grasshopper. In order for you to keep up with customer demand, which includes your upstream comrades in Development,” he says, “you need to create what Humble and Farley called a deployment pipeline. That’s your entire value stream from code check-in to production. That’s not an art. That’s production. You need to get everything in version control. Everything. Not just the code, but everything required to build the environment. Then you need to automate the entire environment creation process. You need a deployment pipeline where you can create test and production environments, and then deploy code into them, entirely on-demand. That’s how you reduce your setup times and eliminate errors, so you can finally match whatever rate of change Development sets the tempo at.”

  “Hold on,” I say. “What is it exactly that I’m supposed to automate?”

  Erik looks at me sternly. “Go ask Brent. Get him assigned to that new team, and make sure that he doesn’t get distracted. Now more than ever, until you get your build process automated, he is your bottleneck. Get the things that are in his head encoded into the build procedures. Get humans out of the deployment business. Figure out how to get to ten deploys a day.”

  I can’t get over my skepticism. “Ten deploys a day? I’m pretty sure that no one is asking for that. Aren’t you setting a target that’s higher than what the business needs?”

  Erik sighs, rolling his eyes. “Stop focusing on the deployment target rate. Business agility is not just about raw speed. It’s about how good you are at detecting and responding to changes in the market and being able to take larger and more calculated risks. It’s about continual experimentation, like Scott Cook did at Intuit, where they did over forty experiments during the peak tax filing season to figure out how to maximize customer conversion rates. During the peak tax filing season!

  “If you can’t out-experiment and beat your competitors in time to market and agility, you are sunk. Features are always a gamble. If you’re lucky, ten percent will get the desired benefits. So the faster you can get those features to market and test them, the better off you’ll be. Incidentally, you also pay back the business faster for the use of capital, which means the business starts making money faster, too.

  “Steve is betting his entire survival on your ability to execute and deploy capabilities faster. So, get to work with Chris to figure out how at every stage of the agile development process, you not only have shippable code, but a working environment it can deploy into!”

  “Okay, okay,” I say. “But why did you drag me all the way over here in the freezing cold? Wouldn’t explaining it on a whiteboard have been enough?”

  “You think IT Operations is rocket-science compared to manufacturing. What absolute baloney,” he says dismissively. “From where I’m sitting, the people in this building have been far more creative and courageous than anything I’ve seen come from you IT guys so far.”

  Chapter 31

  • Monday, November 3

  It’s 12:13 p.m. when I walk into the SWAT team kick-off meeting. My hair is dripping wet and my shirt is soaked from my ride back in Erik’s convertible. Chris is talking. “—and so Steve has authorized this small team to deliver the promotion functionality and do whatever it takes to make a positive impact on the holiday shopping season.”

  Chris turns to me and points to the back of the room. “I went ahead and ordered lunch for everyone to kick this off. Go ahead and—Jesus—what happened to you?”

  I wave away his question. Looking at where he’s pointing, I’m pleasantly surprised to see a turkey sandwich lunch box still at the back. Grabbing it, I take a seat and try to gauge the temperature of everyone in the room, especially Brent.

  Brent responds, “Explain again why I’m here?”

  “That’s what we’re here to figure out,” Wes says earnestly. “You know as much as we do. One of the potential board memb
ers insisted that you be a part of this team. Quite frankly, he’s been right enough times that I trust him, even if I have no freaking idea why.”

  Patty chimes in. “Well, he gave us a couple of clues. He said that the problems we need to focus on are the deployment process and the way we’re building the environments. He seems to think we must be doing something fundamentally wrong because of all the chaos resulting from each Phoenix deployment.”

  As I unwrap my sandwich I say, “I just came back from a meeting with him. He showed me a bunch of stuff and explained how they do single-minute exchanges of die at Toyota. He thinks we need to build the capability to do ten deploys per day. He not only insists this is possible but also that it supports the feature deployment cycles the business needs, not just to survive, but to win in the marketplace.”

  Surprisingly, Chris speaks out the most fiercely. “What? Why in the world would we need to do ten deploys a day? Our sprint intervals are three weeks long. We don’t have anything to deploy ten times a day!”

  Patty shakes her head. “Are you sure? What about bug fixes? What about performance enhancements when the site grinds to a halt, like what’s happened during the last two major launches? Wouldn’t you love to do these types of changes in production routinely, without having to break all the rules to do some sort of emergency change?”

  Chris thinks for a couple of moments before responding. “Interesting. I would normally call those types of fixes a patch or a minor release. But you’re right—those are deployments, too. It would be great if we could roll out fixes more quickly, but come on, ten deploys a day?”

  Thinking about what Erik said, I add, “How about enabling Marketing to make their own changes to content or business rules or enabling faster experimentation and A/B split testing, to see what offers work best?”

  Wes puts both of his hands on the table. “Mark my words, folks. It can’t be done. We’re dealing with the laws of physics here. Forget about how long it currently takes, which requires over one week of preparation and over eight hours to do the actual deployment! You can only put bits down on the disk so fast.”