Chasing New Horizons

by Alan Stern

  The three approach phases, the three departure phases, and the Core itself would each be planned separately and with different levels of rigor. The phases closest to Pluto would be prepared farther in advance so that much higher levels of testing could be done, reflecting their greater importance to the overall science return.

  Each of the flyby’s phases was in turn then broken down into one to several long “command sequences,” each consisting of thousands of computer instructions to run the spacecraft, point it at the various targets in the Pluto system, operate the instruments, and store each data set. But before these command sequences could be written, Leslie’s PEP team designed over one hundred measurement techniques (MTs) to cover all the scientific objectives of the flyby. Each MT laid out where to point, what instrument and mode to use, at what distance from the target, which data recorder to store the results on, and so forth. Each MT was assigned a “champion”—a relevant expert on the science team who was responsible for designing it. Alan insisted that after each champion designed an MT, the MTs would be scrupulously reviewed by PEP to look for deficiencies and opportunities for improvements.

  The Core phase of the Pluto encounter would be when the actual flyby and closest approach to Pluto and all five of its moons would occur. During this nine-day-long beehive of onboard activity, the spacecraft would be placed in a special software mode called “Encounter Mode,” which would prevent onboard problems from ruining the encounter by stopping activities and calling home to Earth for help. If spacecraft engineers had a more playful sense of humor about such dire emergencies, Encounter Mode might have well been called the “Do not disturb!” or “Don’t bother me, I’m busy!” mode.

  In routine cruise flight, when the spacecraft detects an onboard problem, like the computer reboots described in Chapter 10, its autonomy software is designed to triage the problem, fixing the immediate danger (e.g., closing fuel valves if a fuel leak is detected), informing Earth of the problem, and then enter Safe Mode, stopping all further activities until new instructions arrive from the New Horizons Mission Operations Center. This kind of response is designed to keep the spacecraft from getting into any deeper danger until engineers on Earth have time to analyze the problem themselves and devise a complete response. But during the days closest to Pluto, such a process would be countereffective, because if the spacecraft shut down activities until help arrived—which would take a minimum of half a day from faraway Earth—whole swaths of irreplaceable science observations would be lost. So Encounter Mode was designed to handle problems differently when they matter most—at Pluto. In Encounter Mode, the spacecraft still triages its problem, but then it just goes back to the next step on its timeline, continuing to implement the observation plan. The logic of Encounter Mode is that while at Pluto it’s better to try to take data, even if the spacecraft has a problem, than to stop and wait for help from Earth. Encounter Mode had been planned ever since the proposal days, but never implemented. As flyby planning began, it was time for Chris Hersman and his spacecraft team to design and build the Encounter Mode software.

  At roughly the same time, the detailed planning of the flyby science sequences also began. The almost five hundred observations planned for the flyby involved all seven instruments aboard New Horizons, and addressed almost twenty separate top-level scientific objectives laid out by the New Horizons team and NASA. These objectives included mapping Pluto, imaging all of its satellites, measuring Pluto’s atmospheric properties, searching for moons and rings, measuring the temperatures of Pluto and its moons, and much more. Not only would each of the almost 500 observations be individually designed and then built as a sequence of software commands (turn on this instrument, select this mode, point in this direction, store the data, etc.), each would also be thoroughly tested on the New Horizons Operations Simulator (or NHOPS, pronounced “En-Hops”), a high-fidelity electronic replica of the spacecraft and its scientific instruments, housed at APL.

  With painstaking care, Leslie and her PEP team designed every single observation using flyby planning tools—software packages—they had built to let them check everything from the expected resolution and signal to noise of any given observation, to whether the instrument pointing was correct, to how forgiving the observation was to the spacecraft being a little off course or Pluto or its moons not being exactly where they had been mathematically predicted to be.

  PEP worked most closely with two other teams to do its job: SciOps (science operations), which planned all of the command sequences for the seven scientific instruments, and MOPS (or mission operations), which planned all of the spacecraft’s activities, from communications to data storage to interior temperature control to engine burns for course corrections. Together these three groups—PEP, SciOps, and MOPS—carefully choreographed every aspect of the flyby’s flight operations.

  One might at first think this would be as simple as just laying out where all the observations needed to be placed on the timeline, but the planning required was actually much more complex and nuanced. In effect, the PEP, SciOps, and MOPS teams were playing a kind of twenty-plus-dimensional chess game. For every observation, they had to make sure the spacecraft would never go over power budget, that time was always allocated to maneuver the bird to whatever direction the instruments needed to be pointed, that sufficient onboard data-recorder space was always set aside, and so forth. Literally dozens of factors had to be considered to plan each observation. Additionally, these three teams had to weave into the timeline a backup for each of the most important science observations, to ensure that if the spacecraft or one of the instruments malfunctioned during a critical data take, there was a second chance to get a similar data set.

  The teams also devised their plans to be resilient to possible instrument failures. For example, observations were planned to back up Ralph imaging with LORRI imaging, and vice versa. They also planned for each instrument to cycle, from observation to observation, between its prime and its backup electronics, and even to sprinkle instrument reboots throughout the timeline in case any instrument got stuck in a bad mode at any point. All of this had to be scripted down to the finest details, years in advance, because there was simply not time to do all this with a small team during the final year or two before flyby. For the New Horizons team, the brainstorming, planning, review, and testing of the entirety of the six-month flyby consumed most of the years from 2009 to 2012.

  If one was planning any analogous set of scientific investigations of Earth, such a detailed and hyper-scrutinized planning might seem neurotically compulsive. But at Pluto, if anything was improperly designed or was not fully thought through, there would not be another chance to get it right. So all this obsessively careful planning, all the checking and counterchecking, was the way to ensure that the mission would actually get the intended goods in humankind’s shot at exploring Pluto and its system of moons.

  SMASHING BUGS

  As leader of the mission, Alan felt as though it was part of his job to look for weaknesses in the flyby plan, ask a lot of questions of his teams, probe their assumptions, and ask for changes to fortify the planning. One of the many weaknesses he spotted and changes he asked for concerned NHOPS.

  About the time that the seven various flyby phases were being laid out and architected, Alan became concerned that the NHOPS spacecraft simulator, which was used to test all spacecraft command sequences to weed out bugs, could become a showstopper if it failed in 2015. He just wasn’t comfortable with the fact that an NHOPS failure in 2015, when there was little time for a repair, could risk the team’s ability to fully test the flyby sequences. A backup, called NHOPS-2 was already in place, but it was a stripped-down version of NHOPS that lacked much of the simulation capability and fidelity of the original. So at Alan’s direction, Glen put plans and budget in place to convert NHOPS-2 into a full-up spare to NHOPS, and to test it as thoroughly as had been done with NHOPS, to be sure it would be ready if there ever was a need. Little did he know then, this was a de
cision that would prove crucial in the final days of the approach to Pluto.

  As each of the dozens of command sequences that together comprised the entire flyby were designed and had passed their peer reviews, the MOPS team began running them on the NHOPS spacecraft simulator to see if they would work as expected. Bugs were often found, corrected, and then new NHOPS runs would be scheduled. MOPS repeated this again and again, until every sequence ran completely error free. For the Core load—the crucial nine-day-long chain of sequences that instructed the spacecraft how to execute all actions during the close flyby—that took eight tries. Each of these eight NHOPS runs of the Core load took the full nine days. Version 1 was called V-1, version 2 was called V-2, and so on. Every time the team found bugs it rewrote the errant parts of the sequence and started the NHOPS run again, from scratch. When the Core finally ran bug free on V-8, the eighth of these nine-day-long NHOPS runs, Alan celebrated by buying a couple cases of little cans of V-8 juice and handed the cans out for each team member to keep as souvenirs of the time-consuming battle to create a completely bug-free Core load.

  Once that error-free milestone was achieved, the Core sequence was “locked down” under a rigorous “no change without careful review and approval” process, called “configuration management” (or CM). CM’s job was to ensure that an extra level of scrutiny and testing rigor went into any change, no matter how minor. Weekly meetings of a group called the Encounter Change Control Board (or ECCB), were held to evaluate change requests to the Core load and half a dozen other sequences that would run on New Horizons during the period from May to July of 2015. The ECCB was chaired by Alan and staffed by chief engineer Chris Hersman, project manager Glen Fountain, MOPS lead Alice Bowman, senior project scientist Hal Weaver, PEP lead Leslie Young, and encounter manager Mark Holdridge.

  PREPARING FOR PROBLEMS

  At the same time that all the Pluto flyby command sequences were being developed, the project team also took a look at everything that could possibly go wrong during the encounter and how they or the spacecraft would have to react in order to fix any given situation. This kind of “malfunction procedures” development is common to space missions, and it was crucial for a one-shot opportunity like a Pluto flyby.

  The largest effort to prepare for potential problems was led by spacecraft chief engineer Chris Hersman. Hersman, incredibly sharp and meticulous in his attention to detail, and incredibly knowledgeable about all aspects of the bird, made plans for each of 264 potential spacecraft, ground system, and other problems that might arise. This wide-ranging effort, which took three years to flesh out and implement, reached into every aspect of the project. Hersman went far beyond what might go wrong on the spacecraft, looking also at what could go wrong with the team or in mission control itself. So for example, plans were made to train backups for every critical role, in case anyone was unavailable at the flyby due to health problems, car wrecks, or family emergencies. Chris also made sure that detailed checklists were put in place, peer reviewed, and then practiced, for flight controllers to handle dozens of different spacecraft and instrument malfunctions too complex for the onboard autonomy system to handle. Chris even made sure that, in the event that the Mission Operation Center suffered a fire or a terrorist attack (after all, New Horizons would be a high-profile target in 2015), the project had a fully capable, and fully checked-out second mission operations control center to use across the APL campus. Each of Hersman’s 264 backup plans was reviewed and critiqued in detail by engineers on the project, and then again by Glen and Alan, over a series of two dozen painstaking, multi-hour meetings that stretched across 2011–2014.

  PLANNING THE GROUND GAME

  Still another layer to flyby preparations began with the development of a plan for the “ground game” logistics of the flyby, which was going to eventually involve about two hundred people working in close concert over several months. Mark Holdridge, his deputy, Andy Calloway, Glen Fountain, Glen’s deputy Peter Bedini, and Alan’s assistant, Cindy Conrad, undertook this enormous planning effort. They began by mapping out where each of the almost two hundred people across the country would have to be, and what meetings each would attend, during every single day from January to July of 2015.

  This included mapping out every trip to APL by every person involved in flyby operations who did not live in Maryland, and even looking at shift schedules to identify periods when any given individual might not be getting sufficient sleep—owing to the pace of spacecraft activity taking place at odd hours back on Earth. It also included assessing all the office and conference room space needs at APL and reserving all the needed meeting and conference rooms for the 130 or so team members traveling to APL. Cindy Conrad and her assistant, Rayna Tedford, planned who needed what kind of APL access badges and what office supplies the travelers would need while in residence at APL. They even arranged for runners who could go out for meal orders when people didn’t have the time to leave APL, and they planned who among the APL staff lived too far away and would have to move into nearby hotels to be available at any time during the few weeks around closest approach.

  PRACTICE, PRACTICE

  As the arrival at Pluto neared, the project made plans to rehearse and practice as many aspects of the flyby as time, money, and individual schedules could afford.

  The centerpiece of this activity was the development of in-flight rehearsals in which New Horizons itself was loaded up with its actual Pluto flyby sequences and instructed to execute them in full. These flyby rehearsal runs took place out there in empty interplanetary space, running the spacecraft through all its paces to make sure that what had worked on the NHOPS spacecraft simulator would also work perfectly on the bird itself during the close flyby.

  The first of the spacecraft rehearsals took place in July of 2012 as a two-day “stress test” of the most intensive operations for the period right at closest approach. A second rehearsal was run in July of 2013. That one was a full-up, comprehensive, nine-day test of the entire close-approach Core sequence. This rehearsal even went so far as to involve the entire New Horizons ground control, science, and engineering teams, working the same shifts and performing all the same activities, holding all the same status and decision meetings, and even reporting “progress” to NASA, as they would during the real thing two years hence in 2015. Additionally, all of the remote elements of the flyby team, like the Deep Space Network tracking stations, were involved in this rehearsal. After each of the spacecraft rehearsals, the New Horizons team also undertook a detailed review to search for and repair even the tiniest discrepancies or flaws in what had occurred—on the bird out in space, or down on the ground.

  Yet an additional layer of practice for the flyby took the form of ground simulations, called Operational Readiness Tests, or ORTs. The ORTs were elaborate training drills in which various parts of the project undertook multi-hour to multi-day simulations of planned or malfunction-related flyby activities. The navigation team alone held almost a dozen of these as multi-day practice runs, led by Mark Holdridge, each with specific objectives and scoring of how well the team members, their processes, and software tools did. Each “Nav ORT” resulted in a formal action list of process or team or software improvements that needed to be made before the next Nav ORT. Other flyby ORTs ranged from mission operations and instrument-team malfunction scenarios, called “green card exercises,” to practice runs for the Deep Space Network critical operations at flyby, to the science team practicing spotting faint moons and rings in simulated imagery mimicking what it would be receiving on approach to Pluto. In all, across 2012–2014, more than forty project ORTs were designed, executed, and then forensically dissected in order to search for deficiencies that needed to be fixed, procedures that could be improved, or new kinds of training that was needed.

  The ORT phase culminated in 2014 and early 2015 with three science team “encounter dress rehearsals” involving the whole science team; the APL, SwRI, and NASA media teams; as well as a group of six prof
essional science communicators (dubbed “media embeds”), whom Alan had recruited to translate discoveries into real-time press releases, captioned image releases, and edgy “Pluto in a Minute” videos. In each of these, the science team worked with simulated Pluto images and spectra that John Spencer and a small cadre of others concocted (using, e.g., modified Cassini images of icy moons as stand-ins), replete with potential discoveries (like new moons, or puzzling surface features) for the team to practice on. No planetary mission had ever done this before.

  Was all this really necessary? Alan felt that despite the deep experience of the individual players who had been on other teams for other missions, they had to practice as a team rather than just wing it in real time when New Horizons arrived at Pluto. He knew that there would be no excuse for mistakes or delays in any aspect of how Pluto and its moons would be revealed to the world by New Horizons in the summer of 2015.

  The science team, many also busy on other space missions, agreed, but by the time the last of these massive science team ORTs was scheduled, with the spacecraft already on final approach in April of 2015, some science team members rolled their eyes at Alan’s intensity. The first two science team ORTs had worked, and many lessons had been learned—as intended. Did they really need to go through all this yet another time? Was Alan becoming a space-age Ahab, relentlessly and obsessively searching for the great white whale of some hidden problem that might still sink the effort? Or was he a fearless leader pressing his team onward, onward across the interplanetary seas toward their ultimate victory? For some team members, it was hard to tell. But one thing was certain: no one would second-guess the thoroughness of preparations that the New Horizons team had made for “showtime” when it arrived.