by Paul Dye
Things really got cranking in MOD about nine to six months before the mission. By this time, the plan was pretty well set—the payload operations were well known and drafts of all of the flight specific procedures (and the flight plan) should be available to everyone. Reviews became intense and nit-picky as it became obvious that the flight was real and was going to happen. Simulations were still months away, but it was time for folks to start going through the plan with a fine-tooth comb, verify procedures in the simulator, and start seriously working on contingency plans for the cases where something might not work right or took longer than planned.
Mission Operations had developed a well-earned reputation for being able to make complex real-time decisions quickly, accurately—and correctly. But in actuality, most of these decisions aren’t made in real time—they are made months (or years) before, long before the spacecraft was hurtling around the planet at 5 miles per second. The planning process was where these decisions were made by way of long working sessions and meetings where options were discussed, potential solutions to potential problems were hammered out, and decisions were made on what to do when “X” happens. There is a big difference between making a decision and executing a plan, and executing was what we wanted to do in real time. In real time then, all you had to do was apply the premade decision to the situation that was happening in front of you. This made it look like the decisions were being made in an instant. In truth we were merely executing a plan.
This is of course the ideal way of doing business—just execute premade decisions when bad things happen. This was the way it worked much of the time. But the reality was that things happened that no one thought of in advance, and so there were real-time issues that had to be solved and decisions that had to be made. The fact that you have worked through the paths to take for many possible contingencies, however, leaves you with more time to handle the unexpected when it inevitably comes up. If you waited until the actual flight to work out all the potential problems, you’d be overwhelmed very quickly.
We all now remember the Apollo 13 mission, long before the Shuttle came along. This mission is still cited as the benchmark for how Mission Control operates and what it can do. There were so many problems to solve to safely bring home the crew of the crippled Apollo 13 spacecraft that most flight controllers didn’t go home for three days. There were work-arounds for damaged hardware to be developed, new ways of flying the spacecraft to be explored, and lots of number crunching to be done on the trajectory and the available consumables. When the crew made it home safely, it looked like they had pulled the proverbial rabbit out of the hat. This is because they did. But they didn’t do it from nothing. Flight controllers had been thinking about far-out contingency cases and failures for years, and while much of this work was done unofficially and kept in various “hip pockets,” all that time and work proved useful when the oxygen tank exploded and the “far-out” cases became real. I learned from my mentors, the folks who were actually there, that virtually everything that was done to bring Apollo 13 home had been thought of before the flight and was in someone’s head as a possibility before it was needed. They didn’t have to make it all up in real time; they had at least gone through the thought process before, which gave them a huge head start on solving the problem.
Planning for contingencies was therefore a huge part of the preflight preparation process, and it probably took more time than developing the primary plan for each flight. It was what Mission Control and Mission Operations did. It was their bread and butter, the secret to their success. You don’t have to be the fastest to get to the finish line first, sometimes it is just a matter of starting out earlier than everyone else.
Of course, a lot of the contingencies that were looked at before each flight weren’t the work of an individual or a group of flight controllers brainstorming what might go wrong. Many of the best contingency cases (and solutions) came from another venue, another department. Enter the world of flight specific mission training and simulations…
Train the Way You Fly—Fly the Way You Train
In theory (and mostly in practice), flight controllers who were assigned to a particular mission had first to be certified in their position through the normal process of studying, training, and generic sims. That meant that when we scheduled simulations for a particular mission, everyone was already good at what they did—we didn’t need to test them on routine operations or their judgment on handling cases that were common to all missions. What we needed to do was to rehearse the mission we were about to fly, or at least the parts of the timeline that were unique to that flight. We also needed to test the unique procedures and rules we developed for the mission. Our goal was to determine if we had actually thought of all the ways that things could go to worms, and discover the holes in our contingency planning.
In the early days of Shuttle operations, it was not uncommon to conduct two hundred hours of integrated simulations (which involved both the crew and the MCC) for a specific flight. This volume decreased over the years for a number of reasons. The first was budget pressure. It costs a lot of money to bring up an entire operations team, multiple buildings, communication links, and all the support facilities to run a sim. The second was that as we gained experience we had a better handle on the job of flying the Shuttle, and our deeper experience told us what we needed to train for and what we probably didn’t have to worry about. The third was that the simulators and MCC got better. Early mission simulations often collapsed because the simulator wouldn’t run right, or the communication network wasn’t working, or the software didn’t accurately predict the actual way the systems operated (or performed). Oftentimes, a full-up sim would grind to a halt as the sim team worked to troubleshoot the machinery we needed to proceed and, eventually, the session would be canceled.
One of the fun parts of early missions were the integrated long sims. These rehearsals were up to two days in length, and everyone ran in real time and worked shifts just like an actual flight. They were fun because we were truly immersed in a flight environment, and at times it was hard to tell the difference between a simulation and the real thing. In the middle years of the Shuttle program, we decided that these long sims took up too many resources that were needed for the multiple missions that were, by then, flying. So we cut them back and only used the technique for special cases of very new, very complex missions. By the end of the program, long sims were a thing of the past. We flew often enough that new folks could learn the techniques of handovers and round the clock operations by helping on real missions. It was a bit of a shame though. Many of those long sims truly showed us what we were lacking in our plans, and it was hard to develop really informative and interesting mission-unique sim cases that could be executed in just eight hours. Forgoing the long sims never truly harmed us in those later years, but they were experiences that the younger generation missed.
In developing a plan for mission-unique simulations, the Lead Flight Director and the Lead Simulation Supervisor (Sim Sup) would sit down and look for the key points in the mission that they felt could hang up the teams. Those would be the focal point for the orbit simulations for the primary teams. They would then pull out a fairly standard template for the ascent and entry sims, and then throw in a couple of sims for the nonprime teams (the second and third shifts that were not working with the Lead Flight Director). Once we entered the ISS construction era, the sims became pretty cookie-cutter. The core flight controllers sort of went into idle mode once the Shuttle was docked to the ISS. The Remote Manipulator System (RMS) and EVA (Extravehicular Activity) operators were just getting started at that point, of course, but then later in the assembly sequence even those tasks shifted over to the ISS side, which meant the ISS Flight Director and ISS team ran the operations and the Shuttle team mostly cheered them on.
Mission-specific simulations were not supposed to be used just to keep Shuttle flight controllers current. There were times when there was simply nothing for a Data Processing S
ystems (DPS) or Guidance, Navigation, and Control (GNC) Officer to do during a sim when the ISS was running an EVA along with cargo being pulled from our bay. Sure, the sim team could kill a computer in the DPS world or a black box in the GNC domain, but because the emphasis was on the payload operation it was generally just a matter of “if A fails, switch to B”—take two aspirin and call me in the morning. It could get slow in other words. That was when I encouraged good front room operators to let their back rooms watch over things so that they could pay attention to the big picture. We were always looking for new talent with the potential to be Flight Directors—and the ones who were willing to look outside of their discipline to spend time watching the big picture were the ones we wanted to find.
Training in the middle of the Shuttle program—back when we had significantly different payloads and weren’t just going up to assemble the ISS—was more interesting. We spent a lot of time learning to work with our payload partners and they learned a lot about what it was like to work with the Shuttle team. Large science payloads generally had a big team of supporting scientists. They worked around the clock from their own control center or in a room within our Mission Control Center that had been equipped to support them. These Payload Operations Control Centers (POCCs) were run by the Payload Operations Director (POD), the Flight Director’s counterpart and single point of contact for overall decision-making. I started working directly with PODs and POCCs in my early Spacelab days as a flight controller, and that experience gave me a leg up on many Flight Directors who grew up in a single, traditional Shuttle discipline that had little interface with payloads.
The Spacelab systems that I was responsible for as a flight controller were not Payloads but not exactly Shuttle—they were situated between the two. In essence, we provided Shuttle support to the experimenters. In that role I had to understand and work with the payload world at a level that most flight controllers did not. It was then that I learned just how important a simple little experiment could be and just how critical it was that we gave every one of them the maximum level of support so that they succeeded. It was easy to see how a simple container of fish eggs could be ignored in the big scheme of things—but this shortsightedness could ruin years of research in some scientist’s field. I always felt that if they had gone through the effort to get on board the Shuttle—a daunting bureaucratic mountain to climb if ever there was—then we owed them the best ride and the most attention we could give them. So I welcomed sims that tested all these little guys as well as the major payloads that were more visible. It was pleasing to help an experimenter succeed—and to show them that they would receive the same level of ingenuity that our regular flight controllers gave when it came to troubleshooting.
At the end of the Shuttle-Mir Program we flew the first Alpha Magnetic Spectrometer (AMS) payload into space. This was a large device designed to look for antimatter particles (among other things). It was actually the prototype for a larger, more capable device that would eventually be placed permanently aboard the ISS. As was typical of a major science experiment like this, before we flew I spent a considerable amount of time and effort getting to know the experiment, the science, and the scientists involved with the project. The head of the project was Dr. Sam Ting, a Nobel Prize winner and giant of particle physics. His team was spread around the world, as is true with most major science these days, but was headquartered in Geneva. The AMS itself was being built in Zurich. When we were just beginning the detailed mission-planning process, I took a quick trip to Switzerland to get to know the team and see the hardware.
My most vivid memory of the trip was of sitting down with Dr. Ting in his office and he asking me what I knew about particle physics. I was sad to report that the little I had learned back in freshman physics at the university had long been overwritten in my brain by more current and pressing matters. I admitted that what I did know these days was probably from educational television. Without missing a beat, he said, “Well then, we need to get you up to speed.” He then picked up a piece of chalk and began giving me a basic lecture that ended a couple hours later. In that time he taught me a good deal about just how the experiment worked, what it was looking for, and how this was going to benefit our overall understanding of the universe. There’s nothing like learning your particle physics from a Nobel Laureate in the field. It was kind of him to take on a simple engineer as a student, but it created a bond between us that kept us working together for the rest of the mission and let each of us know that we were committed to doing the best we could for the common good. Being a Lead Flight Director for a science mission was like that—it gave you great opportunities to step outside your own field and learn something about the contributions that the Shuttle was making to science.
It turned out that the sims for the AMS experiment were interesting. It was a huge system, and keeping a group of scientists all headed in the same direction while the clock is ticking and you’re going around the planet once every ninety minutes is, indeed, a challenge. By the time they recognized a problem and made everyone else aware of it, the time to activate the experiment oftentimes had already passed and the danger was in missing the next opportunity to take data. We often encouraged POCC teams to sim on their own, and Flight Directors would go to their facility to participate (as if a full MCC was involved)—before we got to the full-up sims that cost a lot and were limited in number.
It was good that the AMS guys did stand-alone training and all the integrated sims, because when we got that mission into orbit, we discovered a failure in the high-rate data system that was designed to give them real-time insight (on the ground) of the data that was being collected. After determining that we just weren’t going to be able to fix it, we recorded the data on board and then dumped small pieces of the data through our standard-rate telemetry system once every so often. This gave them glimpses of what was going on and allowed them to make system adjustments. It was like trying to watch a baseball game through a keyhole… you miss a lot of the action. They did remarkably well, however, because they took the training time seriously.
Of course, our own in-house training wasn’t all done in integrated sims. Shuttle crews did a considerable amount of stand-alone training before they even got into the integrated training, and most of the time they welcomed observers and participants from the flight control teams to these training sessions. This was especially true for the specialized flight controllers where the trainers and flight controllers were all the same people. The EVA, RMS, and rendezvous disciplines were like this, with the same people training the crew and then serving in the Control Center during the mission. Occasionally Flight Directors would drop in on these sessions. That always made for fun. Plus it was far better than sitting through a meeting or working in the office.
Because I personally had a significant background in underwater work (I worked my way through college as a diving instructor and technical diver), I was certified early on to dive in the Neutral Buoyancy Lab (NBL) and the smaller water tank that preceded it, the Weightless Environment Training Facility (WETF). As a flight controller, I helped develop some of the EVA procedures related to the Spacelab IPS because I was familiar with the hardware. This was in the old WETF, which really was just about the size of the Shuttle payload bay. When I became a Flight Director, I could go over to the WETF control room when I had a crew in training and watch the progress of the EVA on the TV monitors. The EVAs were confined to a small space (the payload bay), and so it was easy to keep track of what was happening and how close (or far) the crew was from the airlock, the tools, or the payload. I’d usually also take a quick dive to observe the general characteristics of the EVA tasks, and then just watch from the surface on subsequent days.
But when the ISS came along, we moved to the new facility—the NBL. It was the size of a football field and 40 feet deep. The ISS itself was (and is) immense, spreading over that proverbial football field. While they didn’t have the entire thing in the pool, they could simulate as muc
h of it as we needed. It was so massive that it was easy to lose track of just where crewmembers were working. Watching the feeds from the helmet cams and the occasional view from a TV camera could just make the landscape more confusing. This was because the crew’s helmet cams were close to the structure and didn’t reveal where they were in relation to much anything else. So as a Flight Director, I often liked to jump in the water with a crew as they went through the EVA for a mission just to make sure I had a good feel for the whole process. It was a nice way to spend six hours, floating in a crystal-clear pool as warm as bath water and not having to answer the phone.
We also took the chance to fly with our crewmembers in the simulators when they were trying a new rendezvous or docking technique. We had a mission to Mir early on in the program when the Mir lost several channels of its attitude control system just before our launch. This meant that if they had another failure, the Mir might well be drifting in some random attitude when we got there, meaning we wouldn’t be able to use our standard docking techniques. We sent the commander, Jim Wetherbee, over to the Shuttle Engineering Simulator. This simulator had a dome-shaped visual system; it gave you the best view of 3-D activities, which were necessary to try and dock with a slowly tumbling target spacecraft. We had incredibly small tolerances for errors in docking alignment (on the order of 3 inches) and rates, and it was challenging as heck for Jim to figure out where the Mir was going and how to match the Shuttle’s rates and attitudes with it.