B00AZRBLHO EBOK

by Kim, Gene

  John looks around as Patty, Wes, and I stare blankly at him.

  I say, “I’m not following. How does this relate to reducing the audit workload?”

  “I’m rebuilding our compliance program from scratch, based upon our new understanding of precisely where we’re relying our controls,” John says. “That dictates what matters. It’s like having a magic set of glasses that can differentiate what controls are earth-shatteringly important versus those that have no value at all.”

  “Yes!” I say. “Those ‘magic glasses’ helped us finally see what matters to Dick for company operations. It was right in front of us for years, but we never saw it.”

  John nods and smiles broadly. He flips to the last page of the handout. “I’m proposing five things that could reduce our security-related workload by seventy-five percent.”

  What he presents is breathtaking. His first proposal drastically reduces the scope of the SOX-404 compliance program. When he verbalizes so precisely why it’s safe to do, I realize that John too is also mastering the First Way, having truly achieved a “profound appreciation of the system.”

  His second proposal requires that we find out how production vulnerabilities got there in the first place and that we ensure that they don’t happen again by modifying our deployment processes.

  His third proposal requires that we flag all the systems in the scope for compliance audits in Patty’s change management process—so we can avoid changes that could jeopardize our audits—and that we create the on-going documentation that the auditors will ask for.

  John looks around, seeing all of us staring at him in shocked silence. “Did I say something wrong?”

  “No offense, John…” Wes says slowly. “But…uh… You feeling okay?”

  I say, “John, I don’t think you’ll get any objections from my team on your proposals. I think they’re great ideas.” Wes and Patty vehemently nod in agreement.

  Looking pleased, he continues, “My fourth proposal is to reduce the size of our PCI compliance program by getting rid of anything that stores or processes cardholder data, which is like toxic waste. Losing or mishandling it can be lethal, and it costs too much to protect.

  “Let’s start with the goddamned cafeteria point of sale system. I never want to do another security review of that piece of crap. Frankly, I don’t care who takes it, even if it’s Sarah’s cousin Vinnie. It’s gotta go.”

  Patty has one hand covering her mouth, and even Wes’ jaw is on the table. Has John completely lost his mind? This proposal seems…potentially reckless.

  Wes thinks for a moment, and changes his mind. “I love it! I wish we could have gotten rid of it years ago. We’ve spent months securing that system for those audits. It even went into scope for the SOX-404 audits because it talked to the payroll systems!”

  Patty eventually nods. “I suppose no one would argue that the cafeteria POS is a core competency. It doesn’t help our business but can definitely hurt it. And it pulls scarce resources from Phoenix and our in-store POS systems, which are definitely part of our core competencies.”

  “Okay, John, let’s do it. You’re batting four out of four,” I say, decisively. “But do you really think we can get rid of it in time to make a difference?”

  “Yep,” John says, smiling confidently. “I’ve already talked with Dick and the legal team. We just need to find a suitable outsourcer and convince ourselves that they can be trusted to maintain and secure the systems and data. We can outsource the work but not the responsibility.”

  Wes interjects hopefully, “Can you do something about getting Phoenix out of scope of the audits, too?”

  “Over my dead body,” John says flatly, crossing his arms. “My fifth and last proposal is that we pay down all the technical debt in Phoenix, using all the time we’ve saved from my previous proposals. We know there’s a huge amount of risk in Phoenix: strategic risk, operational risk, huge security and compliance risk. Almost all of Dick’s key measures hinge on it.

  “As Patty said, our order entry and inventory management systems are a core competency. We’re relying on it to give us a competitive edge, but with all the shortcuts we’ve taken with it, it’s like a powder keg waiting to blow up.”

  Wes sighs, looking annoyed. Bad old John is back, his expression says.

  I disagree. This John is far more complex and nuanced than the old John. In the span of a couple of minutes, he’s been willing to take bigger, almost reckless, risks from outsourcing our cafeteria POS systems to his unyielding and categorical insistence that we secure and harden Phoenix.

  I like this new John.

  “You’re absolutely right, John. We’ve got to pay down technical debt,” I say firmly. “How do you propose we do it?”

  We quickly agree to pair up people in Wes’ and Chris’ group with John’s team, so that we can increase the bench of security expertise. By doing this, we will start integrating security into all of our daily work, no longer securing things after they’re deployed.

  John thanks everyone, indicating that we’ve covered everything on his agenda. I look at my watch. We’re done thirty minutes early. This must be a new world record for the shortest time required to agree on anything security-related.

  Chapter 28

  • Monday, October 27

  On my drive into work, I have to turn on my seat heaters months earlier than usual.

  I hope this winter won’t be as awful as last year. Paige’s relatives, the most skeptical people I’ve ever met, have started wondering whether there actually might be something to this global climate change thing, after all.

  When I get to my office, I take my laptop out of my bag, smiling at how quickly it powers on. As I write up a report for Steve on how far we’ve come in the last six weeks, I don’t put in anything about my new laptop, but I want to.

  To me, the laptop represents everything my team has achieved together. I’m incredibly proud of them. Life feels different now. The number of Sev 1 outages this month is down by more than two-thirds. Incident recovery time is down, too, probably by more than half.

  The insight we’ve gained from that first strange meeting with Dick and John tells me that we’re hot on the trail of understanding how we can really help the business win.

  Opening up my e-mail, I see a note from Kirsten. All her project managers are gushing about how projects are flowing so much faster. The number of tasks waiting for Brent and the rest of IT Operations is way down. In fact, if I’m reading the report correctly, Brent is almost caught up.

  On the project front, we’re in fantastic shape—especially with Phoenix.

  There’s another Phoenix deployment scheduled for Friday. It’s only a bunch of defect fixes, with no major functionality added or changed, so it should be much better than last time. We’ve completed all of our deliverables on time, but as usual, there are still a million details that still need to be worked out.

  I’m grateful that my team can stay so focused on Phoenix, because we’ve stabilized our infrastructure. When the inevitable outages and incidents do occur, we’re operating like a well-oiled machine. We’re building a body of tribal knowledge that’s helping us fix things faster than ever, and, when we do need to escalate, it’s controlled and orderly.

  Because of our ever-improving production monitoring of the infrastructure and applications, more often than not, we know about the incidents before the business does.

  Our project backlog has been cut way down, partially from eradicating dumb projects from our queue. And John has delivered. We’ve cut a bunch of unneeded security projects from our audit preparation and remediation work, replacing them with preventive security projects that my entire team is helping with. By modifying our development and deployment processes, we’re hardening and securing both the applications and production infrastructure in a meaningful and systematic way. And we’re gaining confidence that those defects will never happe
n again in the future.

  Our change management meetings are going more smoothly and regularly than ever. We not only have visibility into what our teams are doing, but work is really flowing.

  More than ever, people know exactly what they should be working on. People are getting satisfaction out of fixing things. I’m hearing that people are feeling happier and more upbeat, because they can actually do their jobs.

  It’s strange how much more clearly I see the IT world now and how differently it looks to me than even a couple of months ago.

  Patty’s experiments with establishing kanbans around Brent are a success. We’re also finding instances of work going backward to Brent, because we didn’t understand or didn’t sufficiently specify some task or outcome, requiring Brent to translate or fix it.

  When this happens now, we quickly jump on it to make sure that it doesn’t happen again.

  And it’s not just Brent’s work that we’re improving. By reducing the number of projects in flight, we’re keeping clear lanes of work, so work can go from one work center to the other quickly, getting completed in record time.

  We’ve all but emptied our ticketing system of outdated work. In one case, we even found a ticket that Wes put in over ten years ago as a junior engineer, referring to some task for a machine that has been long since decommissioned. Now we have confidence that all work in the system is important and actually has a prayer of being completed.

  We are no longer the Bates Motel of work.

  Against my staff’s expectations, we keep bumping up the number of projects we think we can handle concurrently. Because we have a better idea of what our flows of work are, and managing carefully which ones are allowed to go to Brent, we’re finding that we can keep releasing more projects without impacting our existing commitments.

  I no longer think of Erik as a raving madman, but he’s eccentric, for sure. Now that I’ve seen the results with my own eyes in my own organization, I know that IT Operations work is very similar to plant work. Erik has stated repeatedly that our improvements to date are only the tip of the iceberg.

  Erik says that we are starting to master the First Way: We’re curbing the handoffs of defects to downstream work centers, managing the flow of work, setting the tempo by our constraints, and, based on our results from audit and from Dick, we’re understanding better than we ever have what is important versus what is not.

  At the end, I led the retrospective portion, where we self-assessed how we did and the areas that we should improve. When someone mentioned that we should start inviting people from Development when we do our outage postmortem root cause analysis meetings, I realized that we are now also well on our way to understanding Erik’s Third Way, as well.

  As Erik keeps reminding me, a great team performs best when they practice. Practice creates habits, and habits create mastery of any process or skill. Whether it’s calisthenics, sports training, playing a musical instrument, or in my experience, the endless drilling we did in the Marines. Repetition, especially for things that require teamwork, creates trust and transparency.

  Last week, as I sat through our latest biweekly outage drill, I was very impressed. We were getting very good at this.

  I feel certain that if the payroll failure that happened on my first day of the job happened now, we could complete the entire payroll run—not just the salaried staff, but the hourly staff, as well.

  John quickly got the approval from Dick and Steve to have an outsourcer to take over the cafeteria POS systems and replace it with something commercially supported.

  It was a fascinating exercise for Wes, Patty, and me to work with John to put together the outsourcing requirements for the cafeteria POS systems. As part of the due diligence process, we were going to hear from all the prospective outsourcers all the dogmas we used to believe before all our interactions with Erik. It will be interesting to see if we can retrain them.

  It seems to me that if anyone is managing IT without talking about the Three Ways, they are managing IT on dangerously faulty assumptions.

  As I’m pondering this, my phone rings. It’s John.

  When I answer, he says, “My team discovered something troubling today. To prevent unauthorized black market IT activities from cropping up, we’ve started routinely reviewing all the proposed projects coming into Kirsten’s Project Management Office. We also search all the corporate credit cards for recurring charges that might be for online or cloud services—which is just another form of unauthorized IT. Some people are going around the project freeze. You have time to talk?”

  “Let’s meet in ten minutes,” I say. “Don’t leave me hanging. Who’s trying to backdoor the system?”

  I hear John laugh on the other end of the line. “Sarah. Who else?”

  * * *

  I invite Wes and Patty to the impromptu meeting but only Patty can make it.

  John starts presenting what he found. Sarah’s group has four instances of using outside vendors and online services. Two are relatively innocuous but the others are more serious: she has contracted a vendor for a $200,000 project to do customer data mining and another vendor to plug into all our POS systems to get sales data for customer analytics.

  “The first problem is that both projects violate the data privacy policy that we’ve given our customers,” John says. “We repeatedly promise that we will not share data with partners. Whether we change that policy or not is, of course, a business decision. But make no mistake, if we go ahead with the customer data mining initiative, we’re out of compliance with our own privacy policy. We may even be breaking several state privacy regulations that expose us to some liability.”

  This doesn’t sound good, but John’s tone of voice suggests there’s worse to come. “The second problem is that Sarah’s vendor uses the same database technology that we used for our cafeteria POS system, which we know is virtually impossible to secure and maintain support for in production, if and when it becomes a part of daily operations.”

  I feel my face get red hot. It’s not just about another cafeteria POS system that we’ll need to retrofit for production. It’s because applications like this contribute to our inaccurate sales order entry and inventory management data. We have too many cooks in the kitchen and no one accountable for maintaining the integrity of the data.

  “Look, I don’t care about Sarah’s project management and invoicing tools—if it makes them more productive, let them use it,” I say. “It’s probably safe as long as it doesn’t interface with an existing business system, store confidential data, affect financial reporting, or whatever. But if it does, then we need to be involved and at least confirm that it doesn’t impact any of our existing commitments.”

  “I agree,” John says, “Want me to take the first stab at that outsourced IT service policy document?”

  “Perfect,” I say. But with less certainty, I continue, “Although, what’s the right way to handle Sarah? I feel completely out of my league. Steve constantly protects her. How do we convey to him the potential mayhem she’s causing with her unauthorized projects?”

  Making sure John’s office door is closed, I say to John and Patty “Guys, help me out. What does Steve see in her? How does she get away with so much crap? Over the past couple of weeks, I see how hard-nosed Steve can be, but Sarah routinely gets away with murder. Why?”

  Patty snorts. “If Steve were a woman, I’d say that he’s attracted to dangerous men. A bunch of us have speculated about this for years. I’ve had a theory, which I must say, was pretty much validated in our last off-site.”

  When she sees John and me both conspiratorially leaning forward, she smiles. “Steve prides himself on being an operations guy, and he’s admitted several times in company meetings that he doesn’t have a flair for strategy. I think that’s why he loved working with his old boss and our new chairman Bob so much. For a decade, Bob was the strategy guy, and all Steve had to do was execute the v
ision.

  “For years, Steve searched for a strategy person to be his right-hand man. He went through quite a few people, even setting a couple executives against each other in this awful, drawn out competition. Pretty Machiavellian,” she continues. “And Sarah won. The word on the street was that there was a lot of backstabbing and underhanded tactics, but I suppose that’s what it takes to come out on top. Evidently, she has mastered how to whisper the right things in his ear, reinforcing his paranoia and aspirations.”

  Patty’s explanation is so much more sophisticated than anything I’ve come up with. In fact, it sounds strikingly similar to what Paige would speculate when I got that distant, angry look at dinnertime.

  John says awkwardly, “Umm, you don’t think there’s anything between them, do you? Like, anything…untoward?”

  I raise my eyebrows. I wondered about that, too.

  Patty just bursts out laughing. “I’m a pretty good judge of people. Both my parents were psychologists. I’d eat both of their diplomas if that were true.”

  Seeing the expression on my face, she laughs even harder. “Look, not even Wes believes that, and there’s no one better at manufacturing drama than him. Sarah’s scared to death of Steve! You ever notice that when someone is talking, Sarah is always still looking at Steve, trying to gauge his reaction? It’s freakish, actually.”

  She continues, “Steve has a blind spot for Sarah’s shortcomings, because she has something he needs and admires, which is the ability to come up with creative strategies, regardless of whether the strategy is good or bad. On the other hand, because Sarah is so insecure, she’ll do whatever it takes to not look bad.

  “She simply doesn’t care about the body count she leaves in her wake, because she wants to be the next CEO of Parts Unlimited.” Patty says. “And apparently, Steve does too. He’s been grooming her as his successor for years.”