B00AZRBLHO EBOK

by Kim, Gene

  After discussions with William I recommend that we delay the Phoenix launch by one week to increase the likelihood that Phoenix achieves its goals and avert what I believe will be a NEAR-CERTAIN disaster.

  I think we’re looking at problems on the scale of the “November 1999 Thanksgiving Toys R Us” train-wreck, meaning multiday outages and performance problems that potentially put customer and order data at risk.

  Steve, I will be calling you in just a couple of minutes.

  Regards,

  Bill

  * * *

  I take a moment to collect my thoughts and call Steve, who answers on the first ring.

  “Steve, it’s Bill. I just sent out an e-mail to you, Sarah, and Chris. I cannot overstate how badly this rollout has gone so far. This is going to bite us in the ass. Even William agrees. My team is now extremely concerned that the rollout will not complete in time for the stores to open at 8 a.m. Eastern time tomorrow. That could disrupt the stores’ ability to take sales, as well as probably cause multiday outages to the website.

  “It’s not too late to stop this train wreck,” I implore. “Failure means that we’ll have problems taking orders from anyone, whether they’re in the stores or on the Internet. Failure could mean jeopardizing and screwing up order data and customer records, which means losing customers. Delaying by a week would just mean disappointing customers, but at least they’ll come back!”

  Steve breathes into the phone and then replies, “It sounds bad, but at this point, we don’t have a choice. We have to keep going. Marketing already bought weekend newspaper ads announcing the availability of Phoenix. They’re bought, paid for, and being delivered to homes across the country. Our partners are all lined up and ready to go.”

  Flabbergasted, I say, “Steve, just how bad does it have to be for you to delay this release? I’m telling you that we could be taking a reckless level of risk in this rollout!”

  He pauses for several moments. “Tell you what. If you can convince Sarah to postpone the rollout, let’s talk. Otherwise, keep pushing.”

  “Are you kidding me? She’s the one who’s created this kamikaze mess. Goddamnit!”

  Before I can stop myself, I hang up on Steve. For a brief moment, I consider calling him back to apologize.

  As much as I hate to, I feel like I owe the company one last try to stop this insanity. Which means talking to Sarah in person.

  * * *

  Back in the Phoenix war room it’s stuffy and rank from too many people sweating from tension and fear. Sarah is sitting by herself, typing away on her laptop.

  I call out to her, “Sarah, can we talk?”

  She gestures to the chair next to her, saying, “Sure. What’s up?”

  When I say in a lowered voice, “Let’s talk in the hallway.”

  As we walk out together in silence, I ask her, “From up here, how does it look like the release is going?”

  She says noncommittally, “You know how these things go when we’re trying to be nimble, right? There’s always unforeseen things when it comes to technology. If you want to make omelets, you’ve got to be willing to break some eggs.”

  “I think it’s a bit worse than your usual rollout. I trust you saw my e-mail, right?”

  She merely says, “Yes, of course. And you saw my reply?”

  Shit.

  I say, “No. But, before you explain, I wanted to make sure you understood the implications and the risks we’re posing to the business.” And then I repeat almost word for word what I told Steve just minutes before.

  Not surprisingly, Sarah is unimpressed. As soon as I stop talking, she says, “We’ve all been busting ass getting Phoenix this far. Marketing is ready, Development is ready. Everyone is ready but you. I’ve told you before, but apparently, you’re not listening: Perfection is the enemy of good. We’ve got to keep going.”

  Marveling at this colossal waste of time, I just shake my head and say, “No, lack of competence is the enemy of good. Mark my words. We’re going to be picking up the pieces for days, if not weeks, because of your dumb decisions.”

  As I storm back into the NOC, I read Sarah’s e-mail, which makes me even more furious. I resist the urge to reply and add fuel to the fire. I also resist the emotional desire to delete it—I may need it to cover my ass later.

  From: Sarah Moulton

  To: Bill Palmer, Steve Masters

  Cc: Chris Anderson, Wes Davis, Patty McKee, William Mason

  Date: September 12, 8:15 PM

  Priority: Highest

  Subject: Re: URGENT: Phoenix deployment in major trouble—my recommendation: 1 week delay

  Everyone is ready but you. Marketing, Dev, Project Management all have given this project their all. Now it’s your turn.

  WE MUST GO!

  Sarah

  Suddenly, I panic for a brief moment that I haven’t told Paige anything for hours. I send her a quick text message:

  Night keeps getting worse. Am here for at least a couple more hrs. Will catch u in am. Love you. Wish me luck, darling.

  I feel a tap on my shoulder and turn around to see Wes. “Boss. We’ve got a very serious problem.”

  The expression on his face is enough to make me scared. I quickly stand up and follow him to the other side of the room.

  “Remember when we hit the point of no return around 9 p.m.? I’ve been tracking the progress of the Phoenix database conversion, and it’s thousands of times slower than we thought it would be. It was supposed to complete hours ago, but it’s only ten percent complete. That means all the data won’t be converted until Tuesday. We are totally screwed.”

  Maybe I’m more tired than I thought but I’m not following him. I say, “Why is this a problem?”

  Wes tries again, “That script needs to complete before the POS systems can come up. We can’t stop the script and we can’t restart it. Apparently, there’s nothing we can do to make it go faster. I think we can hack Phoenix so that it can run, but I don’t know about the in-store POS systems—we don’t have any to test with in the lab.”

  Holy crap.

  I think twice before I ask, “Brent?”

  He just shakes his head. “I had him look at it for a couple of minutes. He thinks that someone turned on database indexing too soon, which is slowing down the inserts. There’s nothing we can do about it now, though, without screwing up data. I put him back on the Phoenix deployment.”

  “How is everything else going?” I ask, wanting a full assessment of the situation. “Any improvement on performance? Any update on the database maintenance tools?”

  “Performance is still terrible,” he says. “I think there’s a huge memory leak, and that’s even without any users on it. My guys suspect we’re going to have to reboot a bunch of the servers every couple of hours just to keep it from blowing up. Damned developers…”

  He continues, “We’ve scrounged up fifteen more servers, some of them new and some yanked from various corners of the company. And now, believe it or not, we don’t have enough space in the data center racks to deploy them. We have to do a big recabling and racking job, moving crap around. Patty just put a call out and brought in a whole bunch of her people to help with that.”

  I feel my eyebrows hit my hairline in genuine surprise. And then I bend forward, laughing. I say, “Oh, dear God. We finally find servers to deploy, and now we can’t find space to put them in. Amazing. We just can’t get a break!”

  Wes shakes his head. “You know, I’ve heard stories like this from my buddies. But this may turn out to be the mother of all deployment failures.”

  He continues, “Here’s the most amazing part: We made a huge investment in virtualization, which was supposed to save us from things like this. But, when Development couldn’t fix the performance problems, they blamed the virtualization. So we had to move everything back onto physical servers!”

  A
nd to think that Chris proposed this aggressive rollout date because virtualization would save our asses.

  I wipe my eyes and force myself to stop laughing. “And how about the database support tools the developers promised us?”

  Wes immediately stops smiling. “Absolute garbage. Our guys are going to have to manually edit the database to correct all the errors Phoenix is generating. And we’re going to have to manually trigger replenishments. We’re still learning about how much of this type of manual work Phoenix is going to require. It’s going to be very error-prone and take a ton of people to do.”

  I wince, thinking about how this will tie up even more of our guys, doing menial work that the damned application should be doing. Nothing worries auditors more than direct edits of data without audit trails and proper controls.

  “You’re doing a great job here. Our top priority is finding out what the effect of the incomplete database conversion will be on the in-store POS system. Find someone who knows those things inside and out, and get their thoughts. If necessary, call someone on Sarah’s team who handles day-to-day retail operations. Bonus points if you can get your hands on a POS device and server we can log into to see what the impact is ourselves.”

  “Got it,” Wes says, nodding. “I know just the person to put on this.”

  I watch him head off and then look around, trying to figure out where I can be the most useful.

  * * *

  The morning light is starting to stream in from the windows, showing the accumulated mess of coffee cups, papers, and all sorts of other debris. In the corner, a developer is asleep under some chairs.

  I had just run to the bathroom to wash my face and wipe the grime from my teeth. I feel a little fresher, but it’s been years since I’ve pulled an all-nighter.

  Maggie Lee is the Senior Director of Retail Program Management and works for Sarah. She is kicking off the 7 a.m. emergency meeting, and there are nearly thirty people packed into the room. In a tired voice, she says, “It’s been a night of heroics, and I appreciate everyone doing what it takes to hit our Phoenix commitments.

  “As you know, the reason for this emergency meeting is that something went wrong in the database conversion,” she continues. “That means all the in-store POS systems will be down, which means that the stores will not have working cash registers. That means manual tills and manual card swipes.”

  She adds, “The good news is that the Phoenix website is up and running.” She gestures at me and says, “My thanks to Bill and the entire IT Operations crew for making this happen.”

  Irritated, I say, “I’d far rather have those POS systems up instead of Phoenix. All hell is breaking loose in the NOC. All our phones have been lit up for the past hour, because people in the stores are all screaming that their systems aren’t responding. It’s like the Jerry Lewis Telethon down there. Like all of you, my voicemail has already filled up from the staff in our 120 stores. We’re going to need to pull in more people just to man the phones.”

  A phone vibrates somewhere on the table, as if to punctuate my point.

  “We need to get proactive here,” I say to Sarah. “We need to send out a summary to everyone in the stores, as quickly as possible outlining what’s happened and more specific instructions on how to conduct operations without the POS systems.”

  Sarah momentarily looks blank, and then says, “That’s a good idea. How about you take a first cut at the e-mail, and we’ll take it from there?”

  Dumbfounded, I say, “What? I’m not a store manager! How about your group takes the first cut, and Chris and I can make sure it’s accurate.”

  Chris nods.

  Sarah looks around the room. “Okay. We’ll get something together in the next couple of hours.”

  “Are you kidding me?” I shout. “Stores on the East Coast start opening in less than an hour—we need to get something out there now!”

  “I’ll take care of it,” says Maggie, raising her hand. She immediately opens up her laptop and starts typing.

  As I squeeze my head between my hands to see if I can make my headache hurt less, I wonder how much worse this rollout could get.

  * * *

  By 2 p.m. Saturday, it’s pretty clear that the bottom is a lot further down than I thought possible.

  All stores are now operating in total manual fallback mode. All sales are being processed by those manual credit card imprint machines, with the carbon paper imprints being stored in shoeboxes.

  Store managers have had employees running to the local office supply stores to find more carbon paper slips for the card imprint machines, as well as to the bank, so they could give out correct change.

  Customers using the Phoenix website are complaining about how it is either down or so slow as to be unusable. We have even managed to turn into a Twitter trending topic. All of our customers who had been excited to try our service started complaining about our big IT fail after seeing our TV and newspaper ads.

  Customers using the Phoenix website are complaining about how it is either down or so slow as to be unusable. We have even managed to turn into a Twitter trending topic. All of our customers who had been excited to try our service started complaining about our big IT fail after seeing our TV and newspaper ads.

  Those customers who were able to order online had a rude awakening when they went to the store to pick up their order. That’s when we discovered that Phoenix seemed to be randomly losing transactions, and in other cases, it was double- or triple-charging our customers’ credit cards.

  Furious that we’ve potentially lost integrity of the sales order data, Ann from Finance drove in and her team has now set up another war room across the hallway, fielding calls from the stores to handle problem orders. By noon, there were piles of papers from hundreds of pissed off customers that were being faxed in from the stores.

  To support Ann, Wes brought in even more engineers to create some tools for Ann’s staff to use, in order to process the ever-growing backlog of screwed up transactions.

  As I walk past the NOC table for the third time, I decide that I’m too exhausted to be of use to anyone. It’s almost 2:30 p.m.

  Wes is arguing with someone across the room, so I wait until he’s done. I say to him, “Let’s face up to the fact that this is going to be a multiday ordeal. How are you holding up?”

  He yawns and replies, “I managed to get an hour of sleep. Wow, you look terrible. Go home and get a couple of hours yourself. I’ve got a handle on everything here. I’ll call you if anything comes up.”

  Too tired to argue, I thank him and leave.

  * * *

  I wake with a start when I hear my cell phone ring. I bolt up and grab my phone. It’s 4:30 p.m. Wes is calling.

  I shake my head to gain some semblance of alertness and then answer, “What’s up?”

  I hear him say, “Bad news. In short, it’s all over Twitter that the Phoenix website is leaking customer credit card numbers. They’re even posting screenshots. Apparently, when you empty your shopping cart, the session crashes and displays the credit card number of the last successful order.”

  I’ve already jumped out of bed and am heading to the bathroom to get showered. “Call John. He’s going to have kittens. There’s probably some protocol for this, involving tons of paperwork and maybe even law enforcement. And probably lawyers, too.”

  Wes replies, “I already called him. He and his team are on the way in. And he is pissed. He sounded just like that dude from Pulp Fiction. He even quoted the line about the day of reckoning and striking people down with great vengeance and furious anger.”

  I laugh. I love that scene with John Travolta and Samuel L. Jackson. It’s not how I would have typecast our mild-mannered CISO, but as they say, you always have to watch out for the quiet ones.

  I take a quick shower. I run into the kitchen and grab a couple of sticks of the string cheese our son loves to eat. I take these with
me in the car and start my drive back into the office.

  When I get on the highway, I call Paige. She answers on the first ring, “Darling, where have you been? I’m at work and the kids are with my mom.”

  I say, “I was actually at home for an hour. I fell asleep the instant I crawled into bed, but Wes just called. Apparently, the Phoenix application started showing the entire world people’s credit card numbers. It’s a huge security breach, so I’m driving back in right now.”

  I hear her sigh disapprovingly. “You’ve been there for over ten years and you’ve never worked these kind of hours. I’m really not sure I like this promotion.”

  “You and me both, honey…” I say.

  Chapter 13

  • Monday, September 15

  By Monday, the Phoenix crisis is a public fiasco. We made it onto the front-page news of all the technology sites. There are rumors that someone from The Wall Street Journal was trying to get Steve for an on-the-record interview.

  I start with a jolt when I think I hear Steve mention my name.

  Completely disoriented, I look around and realize that I’m at work and that I must have fallen asleep while waiting for the Phoenix status meeting to start. I sneak a peek at my watch. 11:04 a.m.

  I have to look at my phone to figure out that it’s Monday.

  For a moment, I wonder where my Sunday went, but seeing Steve red-faced, and addressing the entire room makes me pay attention.

  “—don’t care a goddamned bit whose fault this is. You can bet your ass that this won’t ever happen again on my watch. But right now, I don’t give two shits about the future—we are massively screwing our customers and shareholders. All I want to hear about is how we’re going to dig ourselves out of this hole and restore normal business operations.”

  He turns and points at Sarah saying, “And you are not off the hook until every one of your store managers says that they can transact normally. Manual card swipes? What are we, in some Third World country?”