What to Do When Things Go Wrong

by Frank Supovitz

  My projects are often staged in stadiums or arenas in which people who are accessing the Wi-Fi system are relatively densely packed together, which can impact connectivity and slows down the transfer of data for customers and people behind the scenes alike. For those who require constant access to the Internet, we install Ethernet cables for uninterrupted connectivity. Sometimes, mobile phone service is challenged for the same reason. We add landlines for teammates who are stationary, or walkie-talkies as backups for those who are on the go.

  Your business or project may not operate in a stadium, but connectivity issues that can interfere with communication between teammates and with our customers are still something to think about during contingency planning. Computer servers can fail under normal circumstances and may have even more devastating effects during more stressful project periods. Which teammates do you need to have on speed dial to restore the system? Is your project protected with a backup server? If there is a power failure, which of your communications systems will be functional, and for how long? Be sure your team knows how to communicate or receive information when their primary pathways are interrupted, and if all efforts to remain connected with the team fail, what you expect them to do.

  At events, I make sure that my production team has our run of show document, the minute-to-minute description of what is supposed to happen and when. If the only thing that goes wrong is me being cut off from being able to communicate during an event, the team can use that document to make sure that the event unfolds as it was supposed to. Prepare and circulate the documents that tell your project team what needs to happen, and when, if appropriate, even if you are not able to guide them.

  THE DANGERS OF HOARDED INFORMATION

  Our national-anthem-related foul-up at the 1997 NHL All-Star Game in San Jose started with no pathway for communication at all, and was made immeasurably worse by the human factor: the TV broadcast coordinator’s failure to use the equipment he did have to communicate with the TV truck. Although talking to the coordinator was the customary way to keep the TV broadcaster apprised, it was my error to assume that he was doing that. It was my responsibility to confirm that he was with a simple direct query like “are you keeping the truck informed?” That night, we discovered that a failure, at any critical link, to pass along information can be just as damaging as no communication at all.

  The TV producer, who was tucked away in a windowless mobile broadcast studio under the stands, was not aware that we were running behind schedule—or by how much. So our failure became his failure—an even greater, more nationally noticeable moment gone wrong. It also resulted in an ugly, well-deserved visit to the Commissioner’s Office when we got back to New York. He was an exceptional communicator and very clearly shared his extreme displeasure with the both of us.

  Not wanting to ever repeat the experience in San Jose, we learned the value of having methods and processes for constant communication and confirmation during events along the entire chain of responsibility. We also learned that the more complex and departmentally siloed an organization, the more vulnerable that chain is to having information fail to flow the way it should.

  I have never kept statistics on how often something goes wrong, or something goes from bad to worse, because someone simply didn’t think it was important enough—or not his or her job—to share the information he or she possessed. It’s most often not malicious. It’s just that people do not appreciate that the information they have is important to others on whom they rely to deliver a particular outcome. Changes to our plans, no matter how seemingly inconsequential and innocuous, can lead to unexpectedly poor results.

  During the week leading up to Super Bowl XLIII at Raymond James Stadium in Tampa, Florida, in 2009, we received a call from the loading dock. Approximately 70,000 “fun-size” Snickers bars were being off-loaded for free distribution to fans as they entered on game day. We knew about the promotion and had expected the shipment to arrive late on the day before the game, the ideal time to accept a delivery of delicious chocolate confections in the most likely climactic conditions of Tampa—sunny, hot, and humid. Instead, the truck arrived unexpectedly several days early, dropped the pallets of candy bars, and departed.

  According to one chocolatier, it is ideal to store chocolates in a cool, dry place at a consistent temperature between 65 and 68 degrees, at a humidity of less than 55 percent. Tampa that week, however, was a sticky 80 degrees with 80 percent humidity. We had requested a last-minute delivery because every available cool space was stocked to capacity with consumables for game-day concessions and catering, so it was very likely that tens of thousands of units of the fan-favorite candy bar were going to be in suboptimal, perhaps soupy conditions after a few days on the loading dock. We called the NFL’s sponsorship group to find out what had gone wrong. Apparently, our colleagues knew that the truck was going to arrive days earlier than scheduled and had not thought it was important to let us know. After all, they thought, an early delivery is better than a late one, right? It was now our problem and we had no good place to store tens of thousands of candy bars. So, as the temperature and humidity continued to provide conditions inhospitable to Snickers bars, I sadly imagined them slowly softening to an unintended redefinition of smooth, creamy chocolate, nougat, and caramel snuggling roasted peanuts.

  Just a few days before the premature delivery of the Snickers bars, the New York Times reported on “one of the largest food contamination scares in the nation’s history,” a story that was growing in prominence and ubiquity. Recalls had already been announced for more than 400 products containing peanuts “after eight people died and more than 500 people in 43 states, half of them children, were sickened by salmonella poisoning.” In fact, the recall was expanding daily, and although there were no indications that Snickers bars were ever going to be affected, M&M Mars decided to quietly abandon the Super Bowl fan giveaway and sent a truck back to pick up the shipment. As a result, no one opened a single gooey package of chocolaty, peanutty goodness on game day. Sometimes things go wrong and few people notice. Usually, I’m not that lucky.

  SHARE CHANGE

  Plans change all the time. It’s a natural outcome of the emergence of new information, continuing refinement, and the actions of outside forces during the planning process. As details change, there is almost always a need to share those shifts with project teammates and stakeholders. Without that flow of information, unnecessary work, anxiety, and confusion can reign, and desired outcomes can be subverted. Dispatching an early delivery of Snickers bars is a seemingly innocent occurrence, but it precipitated wasted effort during the very busiest of times. Had our coordinator let the TV producer know that we were running late back in San Jose, we would have avoided breaking into the broadcast halfway through The Star-Spangled Banner. Failure to communicate provides proof positive of the truth of Murphy’s First Corollary: “Left to themselves, things tend to go from bad to worse.”

  When overseeing a project team, it is essential to establish an environment in which sharing information is a mandatory expectation. That is why the “mantra” for my second year at the NFL shifted from “Assume Nothing, Double Check Everything” to “Communicate or Die.” A project as complex and visible as the Super Bowl had to be managed in a culture of personal responsibility for sharing information and an intolerance for carelessly failing to pass along important details as plans and schedules changed. When we need to make even a minor alteration in the scope of our projects, we must consider who needs to know so that they can fulfill their jobs effectively.

  Minor changes may affect only a small number of teammates, but more impactful changes to the plan may require wholesale communication to the entire team. Ready-made distribution lists for the dissemination of certain types of information can often be helpful, reducing the potential error of omitting someone who should have been informed.

  SURPRISES SUCK!

  Taking responsibility as a project leader to share pertinent changes with teammates d
oes more than buy back valuable time and productivity. It models the behaviors you wish to promote: an environment of swift and open notification in both directions that will avoid unnecessary, unproductive, and unwelcome surprises. Effective leadership is all about modeling desired behaviors. If we keep our teams informed and express the expectation that they will do the same for us, the likelihood of a surprise development that someone else in the organization anticipated, but didn’t share, can be greatly reduced. For our Super Bowl team, this mandate was expressed with another wry “mantra” one year—“Surprises Suck!” No one wanted to have to say it, but just as important, no one wanted to have to hear me say it because they withheld information.

  At what point does information flow become a case of over-communication? In my opinion, there is no such thing. It is up to us as project leaders to evaluate the importance of all information we receive and act on information that we believe is of the highest priority. That said, I would rather be over-informed than over-surprised.

  Cultivating a culture and expectation to communicate changes and threats on a timely basis can help lessen the likelihood of something going wrong or can reduce the severity of the impact if it does go wrong anyway. There are other important components to communications planning that will prove essential when the very worst impacts are unavoidable:

  • Define how we will make critical decisions.

  • Define how we will manage critical communications with the outside world after something goes horribly wrong.

  10

  COMMAND AND COLLABORATE

  It seemed like much more was going wrong than right. Snow and ice had been cascading off the roof since Friday, flattening the massive clear-span tents that were installed to serve as sheltered extensions to the stadium that contained additional rest rooms, concession counters, and merchandise kiosks. Instead, they had become a hazardous no-man’s-land, filled with mangled, misshapen steel beams, acres of tattered nylon, and scattered debris of destroyed furniture, fixtures, and equipment. None of the entrances to the stadium behind the wreckage were usable, and the area was still too dangerous to clear any of the damage before the fans were scheduled to arrive. Temporary grandstands inside the stadium were still being installed and the push to complete them before admitting the public necessitated a delay in opening the gates.

  This description is of the North Texas Super Bowl XLV in 2011, discussed earlier, during which thousands of fans were stranded in massive queues on the wrong side of the building, and it seemed as though one intractable problem piled up on the last one all day long. Our operations team huddled early at NFL Control. Reminiscent of NASA’s Mission Control Center, team members responsible for every facet of the operation sat at long rows of tables, chairs, and equipment, looking through enormous glass windows at the field instead of video feeds from outer space.

  It seemed very natural to have everyone at NFL Control facing the field, since the seats in the stadium generally faced in the same direction. As more and more things went wrong that day, however, it proved to be an extremely inefficient configuration for working as a team to respond to problems. Strategically sandwiched between the lead teammates responsible for security and broadcasting operations, I sat in the front row of NFL Control monitoring multiple radio channels. But I was largely unable to communicate with others in the command center without radioing, phoning, or texting them. That day, a great deal of coordinated effort was required to manage all that failed, and many of us spent hours standing, turning, and shouting across the stair-stepped rows stretching the width of the room. It was apparent very quickly that this was a great arrangement for everyone to be able to see the game while things were going well, but far from the best configuration for a command center requiring collaborative problem solving when things were not going well.

  Just a few years before, I visited the Pagoda Tower at the Indianapolis Motor Speedway (IMS) to see how they managed race day at the Indy 500. Given the vast footprint of the 2½-mile oval track, there is no spot where anyone could see everything, and the command center was no exception. From Race Control, all team members who had a need to see the action were seated at the windows overlooking the “Yard of Bricks,” the 36-inch strip that comprises the start and finish line, dating to when the entire Speedway track was paved with bricks in 1909. Stretching out in front of the team members was a bank of TV monitors displaying images covering every square foot of the oval track. Each teammate was on the lookout for crashes, car parts, and dangerous debris in their assigned section of the track. The race director stood at the corner windows watching the front stretch; beside him was a large TV screen displaying a swirling cascade of Global Positioning System (GPS) coordinates, representing each car on the track.

  As cool as the technology was, what impressed me the most was on the other side of the elevator bank: the large rectangular conference table located with no view of the race at all, which was decidedly low-tech. Seated around the table were the IMS team members with responsibilities that did not require a view of the track—security, transportation, facility operations, and more. As the Speedway filled with fans, this group worked swiftly and efficiently together to manage operations, share updates with one another, and solve problems collaboratively, all because they were seated around a rectangular conference table looking and talking with each other, rather than facing forward in a classroom-style configuration.

  It took the woefully inefficient and frustratingly ineffective experience of trying to manage multiple problems at Super Bowl XLV, in 2011, for us to consider redesigning NFL Control. For Super Bowl XLVI, held in Indianapolis in 2012, we applied the intelligence gathered during our visit to the Indy 500’s Race Control facility. We positioned the people who needed to see the field to best do their job—officiating, football operations, broadcasting, and media relations—facing the window. Right behind them, we installed a large rectangular conference table, where representatives of each area not directly related to the activities on the field—stadium operations, transportation, medical services, and social media monitors—could keep each other updated and work collaboratively to solve problems. For several of us facing the field—including myself, the head of security, and public safety officials—we could easily swivel our chairs around to face the conference table should our input be required. Although we installed additional TV monitors, game clocks, and time-of-day clocks for those seated at the conference table to keep tabs on the game, I’m sure it was a highly unpopular decision to take people away from their view of the field. But, we all understood that we were there to do our jobs to deliver the best outcome, and without question, this proved to be a better way to do it.

  YOUR COMMAND CENTER

  Setting up a single location where key constituents and stakeholders can gather and work together as a project rolls out—your own “NFL Control”—is a smart move and one that can significantly improve timely, informed decision making when something goes wrong. You don’t have to be managing a major event for this to make sense. You may be preparing for a potential threat, such as a negative news story, a court decision, or a labor dispute.

  We set up such a workspace as a deadline approached threatening the NHL with its first strike in 1992. Our “crisis communications center” was filled with banks of phones and TV monitors, enabling representatives of all league business units to stay consistently on-message and equipped with the most up-to-the-minute information. With details rapidly unfolding, information could be quickly, accurately, and simultaneously disseminated across the business. So, too, we could collect and compare market sentiment from our most important customers, clients, and the press, and then elevate consistent themes to the highest levels of management.

  We called it the “crisis communications center” because we were truly facing significant brand and economic risk. But you don’t need to be facing an existential crisis to establish a central location from which to monitor and manage a new project. A temporary installation—where the p
roject leader and representatives from all relevant support areas are housed—is as good an idea for gathering results and reactions from the marketplace as it is for the quickest coordinated response to when things go wrong. It is also the singular place where senior management can get a comprehensive real-time snapshot of a project’s performance.

  Forces operating outside of our organizations can often impact results and cause us to make midcourse corrections to our plans. In addition to installing televisions showing broadcast coverage of the Super Bowl, NFL Control monitored news feeds so we could be instantly apprised of any news or business developments that might require us to make adjustments or last-minute decisions. Do the same in your project office, keeping a close watch on news and other feeds most relevant to your company or industry. Our digital media team combed through social media platforms to inform us of conversation trends, observations, and complaints so we could respond to emerging issues. Your project control center might also benefit from dashboards displaying essential real-time performance metrics such as orders, sales, social media sentiment, stock prices, and news streams.

  Every company or project office can have a designated location—a room equipped with phones, teleconferencing equipment, computers, and other communications gear, white boards, and office supplies—where problem solvers know to get together when a crisis or challenge unfolds. Identifying a command center can be as simple as designating a company conference room that can be quickly activated when something unexpectedly goes wrong. There should also be a predesignated rallying alert to send word out that an emergency meeting or a coordinated response is needed. With just a little advance planning and at a manageable cost, getting the word out quickly to gather is easy.