by Paul Dye
Chapter 11
Thirty-Year Learning Curve
When you flight test a new aircraft design, the first many flights are all leaps of faith. You can analyze and calculate every possible piece of structure, you can simulate the aircraft’s flight characteristics, and you can test all the systems a hundred (or a thousand) times, but modern aircraft are so complex that you can’t relax until considerable time has been put on the airframe… and even then, you can (or will) be surprised by what it might do. So the fact that the Space Shuttle was declared operational after just four flights was always a cause for amusement for those of us who understood the system and what it really did. But really, we were still learning—about the spacecraft, and how to fly it—thirty-three years later on its 135th spaceflight. It was an experimental test vehicle throughout its life, and those of us who flew it knew that. Those who lived in a more political world might not have been fully aware—or willing—to accept the fact that there were always a great many unknowns that could bite you.
Of course, NASA went into the Shuttle program after successfully putting humans in space, and then sending them to the moon. They had built a space station (Skylab) and staffed it for long-duration missions three times. The men and (admittedly few) women at the Johnson Space Center had established a process by which they could send other men and women into Earth orbit and bring them home again. The Shuttle was easily an order-of-magnitude leap of complexity and aerodynamics beyond Apollo. But the techniques used to fly it were those developed during the first decade of human spaceflight, and in the days of the rocket planes flown over the high desert in the decade before that. A ground-based control center helping the airborne crew with the assistance of a stack of engineers was the way to keep eyes on all the systems and trajectories, and to help out when things didn’t go as planned.
Engineering a complex system in a ground-based environment is one thing—you have lots of margin when it comes to structure and weight. When you design something to fly, such as an aircraft, you must shave off weight in order to get performance, but, generally, you can still build things pretty stout and add systems redundancy when necessary. But getting a craft into orbit (or beyond) means shaving weight down to within a gnat’s eyelash of disaster in many cases. The performance difference between making it into space or not is often the difference in firing the main propulsion system for just a couple of extra seconds—a very tiny percentage of the overall burn. In the moon landing missions, the skin of the lunar module was so thin (to save weight) that if a technician dropped a screwdriver during ground preparations, it most likely would poke a hole in the cabin.
The Shuttle was built stouter than that, of course—but only enough to complete the mission for which it was designed. We were constantly working to stay within the design limits so as not to use up structural life. But it wasn’t just structural life that we had to worry about. We also had to keep track of how much time was put on systems and their components. Every thruster had a life limit, every piece of machinery that rotated or moved had a number of cycles assigned to it before overhaul. And those of us in Mission Control worried about how much life was used up every time things were checked out and cycled on the ground during postflight and preflight processing. It has often been speculated that with the amount of work done on the ground, we virtually wore the vehicles out in testing—and there was some truth to that. But we certainly didn’t want to forego any testing necessary to prove the systems ready for space either. In fact, we did specific testing on orbit that was designed to check things out while we were in space specifically so that they didn’t need to be tested again on the ground—a lesson learned from many missions worth of processing.
It was all part of the learning process, and thirty years of flying gave us lots of time to learn. The system matured in many ways from STS-1 through STS-135, and those lessons applied to the vehicles, the crews, and the thousands of support team members all along the way. Over the decades, we streamlined vehicle processing, mission preparation, training, and the teams themselves. All of these efforts were meant to make the system more efficient without compromising safety. It was a narrow path to walk, finding where we could shave costs or make up time, and we occasionally stepped out of bounds and had to walk back. When that happened we spent more time, money, or resources until we learned a better way forward. But the Shuttle program at the end was not the same as it was in the beginning—and most of that change was for the good.
The Road to Maturity
In the early days of Shuttle flying, an MCC (Mission Control Center) team consisted of about a dozen front room flight control positions, with each of those positions fielding a backroom support team of from three to ten flight controllers. The MCC building support team, under the control of the ground controller (GC) who sat in the front room, could easily have another fifty people spread around the building to manage everything from the Mission Operations Computer to the Communication systems. Peripheral to the flight control team was management support. There were rooms for first-line managers to sit and work interfaces between the flight control teams and engineering support. In addition to managing these requests for data and information, they kept upper management informed of what was going on, and they funneled management’s directions back to the folks flying the mission. The Mission Management Team (MMT) would meet once a day (or more if required) to keep all the organizations in sync with the program and deal with issues that needed management discussions, such as lengthening or shortening a mission, or doing more or less for particular payloads or customers.
In addition to all the people just mentioned, there was the Mission Evaluation Room (MER) that was owned and staffed by the Engineering Directorate. The MER staffing varied from flight to flight, and flight phase to flight phase, so it might house a few engineers looking at their systems performance or hundreds trying to solve a major problem. The MER was, for many years, situated in the bridge section of Building 30, between the office wing and the Mission Operations wing. Later, they moved to larger rooms in the Control Center itself, as back rooms shrank and space became available.
All told, the Mission Control Center was a busy, packed building with hundreds of people in place at all times, working their shift and then being replaced by a similar-sized group for the next shift. Backroom flight controllers watched over many different displays, recorded data on strip charts, kept track of the performance of all their system elements, and tracked the usage of their systems with procedures and rules. A lot of time was spent just scanning through data to make sure that everything was operating normally. Their front room operator, meanwhile, was doing less of the routine scanning so they could spend most of their time keeping up with the overall mission and coordinating with other disciplines on the way their system was interacting with everything else, as well as the mission. In general, we used to say that the back rooms were there to serve their system, and the front room was there to serve the mission.
One of the goals of the Shuttle program was to continually make space more accessible to humankind, and to do so at a reasonable cost and use of resources. This meant that throughout the three decades of operations, we were constantly looking for ways to save on overall effort. One of the most obvious ways for this to happen in the Mission Operations and MCC world was to cut down on the number of people supporting each shift and each flight. Early on, just about everyone in Mission Operations had some assigned role in every mission. This was done partly to have as many eyes and brains engaged, and partly for morale purposes. It was truly lonely in the office when everyone was working a mission and you were left out! But as time and missions wore on, the grind of constantly flying with everyone involved got old, so flight controllers themselves started looking for more efficient ways to do things. The first was that “hangers on” were generally dropped, because with an increasing flight rate these folks needed to be back in the office getting ready for the next mission. There were still legitimate spots for those doing
on-the-job training, but we no longer had assistants for the backup person for any system.
It was easy to reduce the count of people who were just there to watch, but we also needed to try and reduce the number of folks who were simply watching numbers change. This was done with the development of smarter monitoring software—a precursor to artificial intelligence (AI), which was not really that smart back then. Whereas the early MCC software basically decoded, calibrated, and displayed raw data to the flight controllers, later developments actually provided smart monitoring, alerting flight controllers to trends that were unusual for the systems. This freed controllers from a lot of boring and routine monitoring, which in turn allowed them to work on procedure and timeline updates—things that required higher level intelligence and innovation. The control teams began shrinking when these monitoring aids became available. By the end of the program less than half the flight controllers were present at any point in time in the Control Center than there had been at the beginning.
It’s worth noting that the ISS program eventually reached team-size reductions never achieved in the Shuttle days. This was done by actually combining front room positions at night or during slow mission times. The Shuttle program would occasionally release a front room position that had nothing to do on orbit, or when their systems (like EVA [Extravehicular Activity] or RMS [Remote Manipulator System]) had been stowed for the mission. But the Shuttle program wasn’t like ISS on a slow Saturday night, when only a Flight Director, ground controller, and one or two super systems controllers were on duty.
As flight control teams began to shrink, so too did their demand on facilities. Back rooms were consolidated until we basically had a single large back room for the systems folks, another for the flight dynamics people, and one more for the flight activities folks. Bringing these back rooms together added to the synergy between disciplines, because they didn’t have to talk to someone on the other end of a communications loop. Instead they could just lean over the console to the other controller, draw pictures, and discuss things face to face. These consolidations continued as the Shuttle team moved to the new Control Center (really an extension to the old building), and the entire architecture of the hardware changed. Gone were the P-Tubes, and we welcomed the ability to use office automation tools (PCs and the internet) to communicate. Documents that used to have to be duplicated and walked around by messengers now were available to everyone instantly and we no longer needed to have huge recycling bins to handle all the waste paper.
A good example of how automation helped was how we reviewed messages going up to the crew. In the early days, this was done via an actual TV camera in a back room. The message was typed up and put on a flat bed, and the camera was suspended vertically above it. The team would be called to a particular TV channel, which they’d bring up on their console. Then a person in the back room would grab a pen and mark up the document per conversations on the communications loop. We called this system Mister Hand because the person with the hand was simply marking up what they heard as it was said. Sometimes Mister Hand would mark up something, then another person would correct it, and then a third would come up with something altogether different… until the scribbles got so hard to read that the document would disappear, a clean copy would appear in its place, and we could start over. I don’t recall any obscene gestures being shown by Mister Hand to the team due to frustration, but I wouldn’t be surprised if this happened.
Later on, of course, we could simply have everyone look at the current version of the document on their PCs, the messages having been placed in a folder for everyone to see. You’d submit your markups digitally, the corrections would be made by the appropriate people, then the document would be redisplayed. If there were conflicting inputs (which were rare), then they would be reconciled on the Flight Director loop by the team.
The new technology cut back considerably on the support teams for the Flight Activities Officer (FAO) and helped streamline operations when the program had support teams around the globe. By the third decade of the Shuttle program, the teams were down to a reasonably small size, productivity was up due to automation, and more people were simply placed on call in case they were needed during a flight. Missions were no longer a case of all hands on deck, and creative engineers were busy working on future missions and spaceflight plans.
If It’s Exciting, You’ve Done Something Wrong
It’s ironic that while spaceflight is supposed to be exciting because it is exploration, doing brave new things, and advancing humanity beyond our planet, those of us who actually do it try to keep it as unexciting as possible. Now we say that with a little wink, a touch of tongue in cheek, but that’s just because there are different meanings to the word exciting. If spaceflight itself is exciting because of what it is and where you’re going, that’s great. But if it’s exciting because things are going wrong, or happening too fast, or you don’t know exactly what’s happening… that’s a bad thing. Mission Control has an unofficial motto: “Making the most exciting thing in the world as boring as possible for over fifty years!” We like high achievement, but we want everything to go as planned—and many people think that is boring. And that’s okay.
We like things to go as planned, we want to stay on the timeline, and even though we train for excitement, improvisation, and talented “genius” troubleshooting, things go much better on the real day if we don’t have to do any of that. You don’t want fires, leaks, or control failures. You want to make it look so easy that everyone else wonders why you need to spend your life training to make it happen.
Making things boring when you are ready for anything is a matter of keeping your margins high. It’s sort of like training to walk the tightrope but having a net below you so that if you do make a mistake, it’s not fatal. In fact, we like to make things so boring that sometimes it feels like the wire is just sitting on the ground and we’re just walking along it. There’s still the chance that you will trip, but at worst you’ll skin your knee.
Why is boring good? Why do we strive to make things as sedate and tranquil as possible? Well, our goal isn’t to go to work and have an adrenaline rush caused by the unknown. We aren’t daredevils, looking for a thrill. The thrill comes from executing the mission—getting done exactly what we set out to do. If what we are doing (the mission) is exciting, rewarding, and future-altering, well that is where the excitement comes from. Making a critical call that makes the mission succeed is like winning the Super Bowl for a flight controller. It’s all about making history—even when your name never appears in public. Knowing that you were part of it is satisfying enough.
I look around my office and see mementos of missions. The Contingency Control Panel for the Spacelab Instrument Pointing System—the training folks gave that to me. I have a little Lucite block with a tiny piece of Shuttle tile that came off the Columbia after STS-1—a memento from being a co-op during that momentous time. There are crew plaques from all the flights that I led—the names of all those astronauts bring back the memories of faces and the voices of folks who did good work in space for the betterment of humankind. All these things, and the many photographs, remind us that our teams did good—well enough to make history happen.
Flying the Shuttle was full of excitement in and of itself. We didn’t need to make it any more exciting by having things break. The goal was always to make the mission successful, not to come home with our hair on fire.
If You’re Not Ahead, You’re Behind!
Time is the single nonrenewable resource that we have. You can buy more spaceships, you can hire more people, you can get more funding—but once a moment is gone, you can never get it back. When you are hurtling along at 5 miles every second, time passes very quickly, as do opportunities to make things happen. You can’t ask for a rewind; you don’t necessarily get a chance for a do-over. You must be ready when the time comes, and you must make sure that you are ready to be right. And since most things that require precise timing require you
to be right on time, the only way to make them happen is to be ready in advance. Hence, what I have always told my teams: if you’re not early, then you are already late.
Spaceflight is horribly complicated, horribly expensive, and horribly unforgiving. It involves hundreds of thousands of people, and when you think about it, you often have only one chance to do something correctly in a mission. If you’re the one person who misses their cue, you could waste all that money, time, and effort—simply flush it down the drain. If you’re the kind of person who freezes up when faced with that reality, then being involved in such a position probably isn’t for you.
When we flew the SRTM mission to map the earth, the complete map depended on getting three passes over every bit of dirt that fell underneath the orbit of the vehicle in the remarkably short time of ten days. One hundred fifty-six orbits—that was how the geometry of the swath, the orbit, and the altitude worked out. When we came “feet dry” (crossed a coastline from water to land), we had to be mapping. The Orbiter had to be in attitude, with the attitude rates low enough that nothing would disturb the radar beam. At that same time, the payload had to be ready to start taking data—tapes had to be in the recorder, the recorder had to be activated, and all the many and various commands to activate the radar had to be sent. That was a lot of things that could trip you up. If any one of them didn’t get done on time, then you missed the pass and the map would be less than complete.
Because timing was often so critical, the only way to make sure that you were going to be ready (for anything) was to complete all the steps that could be done early. You’d front-load activities so that all you had to do was send one last command, or flick one last switch, when the time arrived. This was the way orbit maneuver burns were conducted—everything was set and ready to fire well before you got to the last ten seconds before the burn. At five seconds prior to the time of ignition, the computer would prompt the crew one last time with a message that didn’t actually say, but meant, “Are you sure?” The crewmember would then hit a single key to do the final arming of the burn software, and the computer would fire the engines on time. The full burn prep procedures could take five minutes to set up, but you did them all well in advance, just to make sure.