by Paul Dye
The answer, of course, was to have the appropriate patches to take out those limits preapproved and canned, ready to implement. We had well-thought-out criteria documented in the flight rules that everyone agreed to in advance. They defined the conditions under which the patches could be implemented, and everyone signed up to them. That meant that if we had a vernier temperature transducer issue, we could put the fix into work immediately and solve the problem quickly without consulting anyone for permission. Now I have always told flight controllers that the more failures you prepare for, the fewer will actually happen, because what you prepare for will never happen. But in this case—that wasn’t true.
There was one orbit each day that covered the longest stretch of water you could get in our inclination. You came off the Kamchatka Peninsula into the North Pacific, traversed the Pacific to the southeast, ducked almost under South America, and didn’t make landfall until you hit the equator on the western coast of Africa. It was something like a fifty-minute water pass. And on Flight Day 7, as we were coming off Kamchatka, I heard, “Flight, PROP—looks like we’ve had a transducer fail on F5R.” I looked up to see the message that denoted the failure of verniers, and sure enough, we had dropped into free drift—the Shuttle attitude was not being controlled. The mast was safe, we just weren’t going to be holding attitude. A quick discussion confirmed that we were in one of our preplanned patch cases, and I gave the DPS Officer a Go to put the patch to work. This, of course, required coordination between all the operators on the left side of the room—DPS, INCO, PROP, and GNC. There was little I could do but sit back and watch the clock, the world map, and the team.
When you know you have built a good team, the best thing you can do is trust them and stay out of the way. I made sure that they knew how much time we had, and then I watched as they went to work. Patching the software in the Shuttle GPCs was never something you did lightly. It amazes most technically knowledgeable people to find that the individual computers had only 256K of RAM—and that was double what they had on the first flights! The small internal memory meant that the software had to be incredibly compact and tightly written—there were no spare bits. When we’d do a patch like this one, we needed to change a single word. And we had to know exactly where that word was stored because we were going to address the memory directly. The patch had been simulated on the ground, of course—tested and verified that it would do what we wanted. But if we were off by one bit on the memory location, we could be writing to something else entirely and bring the entire machine to a halt.
Getting the patch right was all about checking, rechecking, and triple checking—and in this case, we had to do it with a tight timeline. But I was happy with the team, and they knew their job. They used all the time they needed, took no chances, and triple checked their work. Not only did we get the vehicle safed, the patch loaded, and the thrusters all reconfigured, we maneuvered back into attitude before hitting the dirt and didn’t miss a pass. It was incredible teamwork. All I had to do was sit back and watch the folks that I trusted do their jobs—perfectly.
Back when we did our first simulation for the SRTM mission along with the Payload Operations Control Center (POCC), we had a few problems. Well, actually, we had quite a few problems. Simulator problems, simulated problems—and problems caused by our own making. It seemed like every time we came “feet dry” to start a data pass over land, there was something wrong with the SRTM, or with the POCC’s commands, or the POCC’s plan—and as a result JPL was missing valuable observation time. It really can be overwhelming, to be honest, to find yourself in the middle of a complex Shuttle mission with brand-new hardware and software. Time is required to shake this stuff out. In fact, from the time we started playing with a new simulator or software load, it usually took hundreds (if not thousands) of hours to get it running properly and behaving the way the vehicle would. We flew the Shuttle over and over again, so our experience base built up mission after mission, and our simulator, procedures, and training kept getting better and better. Unfortunately, the POCC and SRTM team did not have this same background, and it showed.
When the sim was over, and the debriefings complete, I had a chat with the head of operations for the SRTM team at the POCC and expressed my concern that his folks kept missing their cues and not being ready. “Well,” he said, “you have to realize that we have a lot going on, what with getting the hardware and software ready for the mission. I can’t expect my folks to be perfect.”
Now I understood where he was coming from, and I understood his problems. I also understood the culture of JPL in the sense that they were used to taking time to do things with slow-acting spacecraft on the surface of (or in orbit around) some distant planet. But flying a Shuttle payload was different and I couldn’t afford to waste the limited sim time we’d have for the mission—or to come back with an incomplete map because we weren’t ready. So, I simply said, “The fact that you don’t expect them to be perfect is exactly why they won’t be! My folks know that I expect them to make no mistakes, so they don’t. Your team will live up to YOUR expectations. So set them high!”
He must have taken my comments to heart; either that or he was so angry with me that he did whatever he needed to do to get his team whipped up. Regardless, the teams improved as the simulations went on, and by the time we were ready to fly we were confident in our ability to get the entire map. The first flight day had a jam-packed timeline. We needed to get the mast deployed and checked out so that we could get the mapping started as quickly as possible. The deploy and checkout went fine, with only a few surprises. In any big experiment like this, unexpected things happen when devices are first turned on. The SRTM was no different, but the POCC was quick to recognize the difference between “I didn’t know it would work like that” and “It’s broken.” Fortunately, nothing (except that dang cold gas thruster) was broken. By the time I came in for my second day of the mission, the mapping was underway.
The teams in MCC (Mission Control Center) were mostly working to maintain the perfect orbit and the perfect attitude—everything we did was directed toward that specific goal. In addition, we were working hard to “make prop” (make propellant) and conserve cryo. Of course, we weren’t actually making any new propellant. Instead, we were constantly trying to find ways to save it so that we built a surplus as the mission progressed. The goal was to make more margin over what we had expected to use. The same thing was true of cryo—the stuff that made electricity. Orbiter systems were managed during the mission to draw as little power as they could so that more was available for the radar. Again, the goal was to get the extra day in the bank so that we could get the entire map.
The Trench and GNC folks were keeping track of the altitude and making sure that we stayed in the specific desired orbit at all times. To do this, they carefully planned the fly-casting burns for each day. They made sure that each one would get just the right amount of reboost to make up for the orbital decay we were seeing at the low altitude we were flying. Kevin and Dom had perfected their fly-casting procedures and techniques. As long as we got them good burn targets, they made sure that the orbit was perfectly maintained and the mast stayed in alignment.
Meanwhile, the POCC was busy—and the walls of MCC began to show it! I don’t know where they found the budget for large-scale flatbed color printers, but they must have bought a lot of them! By the third or fourth day of the mission, we were beginning to see 6-by-4-foot map printouts, in full color, popping up in the hallways. I don’t know if it had anything at all to do with the science, but it sure did inspire the troops to do a good job! There were false-color images, full color images, black-and-white images, and images done with an isometric view—essentially 3-D. The POCC even distributed 3-D glasses to all the MCC teams so they could see the amazing 3-D pictures hanging on the walls in all their glory.
In smaller scale, each shift prepared a map of the swaths that had been covered to date. They were cumulative, and we kept them in a stack on the Fl
ight Director console. A quick glance at the latest one showed what had been covered up until that point, and what had still to be done. Three passes over every bit of dirt were needed to get an accurate map, and there were only a couple of tiny spots under our ground track that we had missed. They were still available for future orbits though, so spirits remained high that we’d get everything we needed. (Eighteen years later, as I write this book, I still have that stack of maps here in my office—a reminder of the remarkable team, and the remarkable mission we flew.)
At each daily Mission Management team meeting, we showed our progress toward building sufficient propellant and cryo to add the extra day to the mission. About two-thirds of the way through the mission, it was apparent that we’d be successful. This allowed for the full mapping mission, with an extra day in case the mast didn’t stow. We weren’t terribly worried about that, of course, because the deploy had gone all right. Of course, the mast had been hanging out in the hot and cold vacuum of space for ten days, so we shouldn’t have felt too comfortable. Which brings us back to mast stow day…
LeRoy Cain was one of our newer Flight Directors at the time (and went on to be one of our experienced Ascent/Entry types later in the program). He was our Orbit 3 Flight Director for the mission, and the mast stow procedure fell on his shift. Technically speaking, I was done with the mission by this point; my shift would be taken over by the Ascent/Entry Flight Director, Wayne Hale, for the day before entry. I could sit back and relax in the glow of a job well done and all those maps spread around the building. Of course, that’s metaphorical; I never relaxed until everyone was safely on the ground. We also had all that data stored in tapes in the Orbiter—only a little bit of the entire mission data had been downlinked. It’s not over until it’s over.
When LeRoy’s team went to stow the mast, it retracted normally until it got to about the last 6 inches of travel. Motors pulled the mast in, and the mechanism collapsed the segments as they reached the mouth of the canister. When it was all the way in, latches held the lid closed, sort of like a jack-in-the-box toy. For some reason, upper management cringed when I described it that way, probably because we’d all seen jack-in-the-boxes spring open when we were children—not a pretty image if our payload bay doors were closed! Regardless, the procedure was to drive the mast all the way to the stow position, stop the motors, then turn on the latch motors to drive those closed.
The problems started on the first stowage attempt: the mast sucked almost all the way in—and then stalled. The lid wasn’t all the way closed, so the latches couldn’t grab it. How many of us have ever tried to get all our things back into the suitcase at the end of a long vacation? It can be a struggle. With cables and joints cold-soaked by a week and a half exposed to the space environment, in retrospect it was no surprise that they might be a little stiff. When I saw the stow process on the TV in my kitchen, it had reached this point, and there was a lot of head scratching going on. By the time I had reached the Control Center, pulled out my headset, and plugged in, they had already developed a plan. Basically, they were going to extend the mast a few feet, then engage the stow motors at full speed, essentially ramming the mast into the canister. Then they’d hit the switches for the latches to grab on before it could rebound. Imagine jumping on the bulging suitcase and then having someone ready to pull the zipper (or hit the latches) as soon as they saw it had closed. Same idea.
The good news was that it worked, and I didn’t need to worry about it coming home without the mast and outboard antenna. Of course, as we had pointed out to the Shuttle program in advance, the value of the mission was in those tapes stored away in the Mid-deck lockers, not in the mast and outboard antenna. The rest of the mission went fine, and Kregel brought the Orbiter in for a good landing at Kennedy Space Center to end a significant milestone in human exploration of the planet. It has been said that it is hard to explain your explorations when you can’t tell anyone exactly where you had been. For this reason, maps are vitally important artifacts of an expedition. And the very-high-resolution topographical maps from this mission are valuable for helping us understand the planet on which we live.
Topography drives the best location for cell phone towers, for instance. Picking the right high point can give you the best coverage, allowing rural areas to be served by fewer towers. I know of cave explorers who use topographical data for clues to locate massive sinkholes that mark possible underground passages. And since the radar penetrated much of the foliage of the land it surveyed, you could literally look through the trees for features that had been hidden from view—intentionally or unintentionally.
Time marches on, and missions come and go. I had quickly gone on to preparing for another mission when SRTM debriefs ended. There were congratulatory notes and meetings, and everyone appeared happy. The mapping organizations had their data and said that it would take several years to calibrate and reduce it all (post-processing, it is called) so that it would be available to anyone who needed it. I began training not only to fly Shuttles to build the space station but to become a Flight Director for the ISS as well. I lost track of my JPL contacts until we lost the Columbia in February of 2003. I was running the air search operations there, and we were using all sorts of sensor packages on aircraft to look for vehicle debris in the forests of East Texas.
One airplane had a radar sensor that looked down through the trees and brush, but it kept getting so many returns from old cars, rusted out water tanks, and even the nails in fences that the data wasn’t helping us much. All it told us was that metal-age humans had been living in the area for a long time. What I needed was a way to subtract out all of the metal that had been there before the pieces of the Columbia had rained down on the area—and I thought of SRTM. Perhaps their raw data showed much of the metal that was there in 2001, and we could use that as a baseline, looking for significant changes to indicate where pieces of the Orbiter might be. Everyone in our search headquarters agreed that it was an interesting idea. But, unfortunately, when I made contact with the radar experts from the SRTM mission at JPL, they said that the wavelengths were all wrong, and that they simply didn’t have data that would be useful. They knew that because they had had the same idea a couple of weeks before I did and had looked into it with hope—but it wasn’t to be.
The mast? I ran into that thing by surprise. I was visiting the Smithsonian’s Air and Space Museum annex in the Udvar-Hazey Center near Dulles International Airport in Washington, DC, a few years later. While approaching the Shuttle Enterprise, which was then on display where Discovery now resides, I happened to look up. And there, hanging from the rafters, was the mast and outboard antenna for museum visitors to see and wonder about. It was just a piece of old space hardware, of course, one that LeRoy and his team had saved from jettisoning that day in low-Earth orbit. I had always argued that the only reason that JPL would want the mast back was so that it could end up in a museum—and it turned out that I was right. I just didn’t know what museum it might be.
Chapter 9
Life in Mission Control
The Control Center is more than just a place where people go to work from nine to five—it has been a home away from home for thousands of flight controllers over the years. Starting with the Gemini and Apollo missions, and continuing into the Shuttle (and ISS) era, Building 30, as it is frequently referred to by the cognoscenti, has grown and changed considerably. The Shuttle era started in the old, original Mission Operations Wing (MOW) that was built in the early 1960s, and then transitioned over to the new wing of the MOW in the mid-1990s. Shuttle operations stayed there until the end. ISS operations have moved back and forth between the old and new wings as flight control rooms were refurbished and upgraded.
The various back rooms that make up the bulk of the square footage and house the majority of each flight control team, have always been spread around the building. Through the miracles of electronic communication, it made no difference if your back room was on the same floor as the front room
they were working with—or even in the same wing. Back rooms were variously referred to as Staff Support Rooms (SSRs) or Multipurpose Support Rooms (MPSRs), depending mostly on when you arrived to work in MCC (Mission Control Center). The two terms were created in different eras but referred to the same thing as far as most flight controllers were concerned. On any given day, flight controllers could be supporting multiple flights and simulations, or simply be supporting development work—and all of that could be going on at the same time.
Mission Control was originally built in the early 1960s. It was a marvel of its time. A three-story, windowless building with an attached office wing, it housed one of the most advanced computer complexes in the country. Five large mainframe computers chugged away on the first floor. Many subsidiary computing and communication boxes and racks filled out the floor, creating the Real Time Computing Complex (RTCC), the electronic heart of the Control Center. Surrounding the central space of the floor was a corridor that ran all the way around the roughly square building. On the outside of the corridor were yet more rooms filled with electronics and support facilities. The Communications Center was a long room filled with racks and racks of panels with patch cords connecting thousands of sockets that routed voice loops coming in from (and going out to) stations around the world—not to mention routing those voice connections throughout the building.
There was also the pneumatic tube room. It featured a central switching facility for the plumbing that routed the 4-inch-diameter pneumatic tube (P-Tube) carriers from room to room. Controllers used these carriers to send papers from one room to the next. The P-Tube system was copied from those found in large department stores, and now found almost exclusively in drive-through banks and pharmacies. This communication system became obsolete with the rise of email and the internet.