By that time Malin Space Science Systems had completed assembly of MARDI, but hadn’t completed the thermal testing that would be required for it to be included on MSL. It was validated enough for descent, but hadn’t been tested for surface operations, a detail that would become important after landing. Delivered in July, 2008, it became the first science instrument to be integrated on the rover, and the only one to participate in every major rover test.
1.5.8 Mastcam dezoomed
The longest-lasting impact of Stern’s science instrument descopes was on Mastcam. As originally conceived, Mastcam consisted of a pair of identical cameras. Each had a zoom/telephoto lens with up to 15x magnification, giving a field of view ranging from a wide 90° to zoomed 6°. They would have been capable of shooting color, high-definition, stereographic, cinematic video in stereo at a rate of 5 frames per second. For this reason, filmmaker James Cameron joined the Mastcam team as a co-investigator. The descoped version of the Mastcam consisted of two different cameras, both with narrow fields of view (one at 15°, one at 5.1°); they sacrificed same-focal-length stereo capability to maintain the ability to view the landscape at different scales. They could not capture the wide stereo video landscapes that the original design would have enabled. Cameron lost interest, because the new Mastcam was not a tool he could use for his art. The loss of public outreach value – high-definition stereo video from Mars, directed and distributed by a rich, well-connected Oscar-winning director – is incalculable.
Meanwhile, Malin Space Science Systems had to imagine, design, fabricate, test, and deliver a totally new optical design for MSL’s science cameras in barely more than one year. It wasn’t going to be easy, because the descope solutions had a major oversight: fixed-focus cameras wouldn’t be in focus at all the distances they were expected to cover. The Malin team considered focusing the narrower-angle camera at infinity and the wider-angle one closer to the rover, but then images in the middle ground would be out of focus for both cameras, and stereo imaging would be impossible. Mastcam engineers Mike Caplinger and Mike Ravine came up with an idea that might be cost-effective: use the already developed MAHLI instrument focus mechanism for the Mastcams. But this would not comply with the requirements of the descope. The Project Science Group recognized that being able to focus Mastcam would be crucial, and went back to NASA Headquarters to request permission to return focus but not zoom capability to Mastcam, because the camera would not be able to achieve its science goals without a focal mechanism. NASA eventually approved the request, and design of the new Mastcam optics began in December 2007.
1.5.9 Scarecrow’s debut
Back at JPL, they constructed a new outdoor Mars Yard at the top of the steep JPL campus for MSL mobility testing, and invited media to view “a parade of rovers” there on June 19, which I attended. One of the rovers on parade was Scarecrow, a prototype for MSL (Figure 1.9). Scarecrow has a rocker-bogie suspension system, motors, and wheels, but hardly any other hardware, which allows its wheels to exert the same ground pressure on Earth that the full-size MSL’s wheels would eventually do under Mars gravity. (It’s called Scarecrow because, like the character in The Wizard of Oz, the minimally instrumented rover has no brain.) Scarecrow and the Mars Yard are still in use for mobility testing, more than a decade later.
Figure 1.9. Scarecrow rolling over enormous rocks in the newly opened JPL Mars Yard on June 19, 2007. Note the “JPL” letters machined into the wheels’ treads. Photo by Emily Lakdawalla.
1.5.10 Budget balloons
The development problems all had financial implications. In December 2007, the MSL project requested another $91 million from NASA. Alan Stern doubted that this request reflected reality. He tasked Doug McCuistion with independently analyzing the MSL budget. McCuistion found that MSL had underestimated its likely needs by $40 million. Stern set aside $190 million to solve MSL’s budget woes, bringing the total cost to nearly $1.9 billion. Some of this money had newly become available with the delay of the second Mars Scout mission opportunity from 2011 to 2013. That would make 2011 the first Mars launch opportunity since 1996 to have no NASA mission slated for it.
The Mars budget had become a battleground. Mars scientists were worried about what they saw as Stern’s attacks on the Mars program, and lobbied hard for more money to be moved into the program. Stern was determined to keep the MSL overruns from damaging other areas of space science. He continued to look for places in the Mars program to find funds to pay for the overruns on MSL. On March 24, 2008, the Science Mission Directorate ordered the Mars Exploration Rover mission to cut $4 million from their $20 million budget that year, and another $8 million the following year. Principal investigator Steve Squyres responded that the cuts would require him to shut down Spirit. The public exploded with outrage over the threat to the charismatic rover. The next day, NASA Administrator Mike Griffin repudiated Stern’s letter. Hours later, Stern resigned.40
The public furor was symptomatic of Griffin and Stern’s incompatible management visions. They fundamentally disagreed about how to handle the perennial problem of budget-busting missions. Stern fought to contain the damage within programs, and to protect the small amounts of money supporting research and analysis of NASA data. He wanted to bring an end to what he saw as irresponsible fiscal management of missions and the collateral damage they wrought on other missions. But he acted unilaterally, without the concurrence of NASA leadership. Griffin replaced Stern with Ed Weiler, who had led the Science Mission Directorate from 1998 through 2004, during the overhaul of the Mars program in the wake of the twin disasters of 1999. Weiler would remain in the position through the rest of MSL’s development. Within months, Weiler delivered JPL the money they had requested. The MSL workforce increased from 700 to 800 people.41
1.5.11 Phoenix descends
On May 25, 2008, the Phoenix mission landed in Mars’ high northern latitudes. Although the landing went perfectly and NASA heralded it as a success, during its short, 5-month mission Phoenix would have frustrating problems attempting to sample Martian soil and ice and deliver it to laboratory instruments (Figure 1.10). Puffs of wind blew the samples away from the instrument doors. When sampled material did fall onto the instrument, the Martian soil tended to clump and stick, failing to fall through sieves that protected the instruments from large particles, even when the sieves were vibrated.
The difficulties on Phoenix were sobering news for MSL. Sample handling hadn’t yet been tested even under optimal conditions. Would wind blow away the drilled samples intended for SAM and CheMin? Would the powder stick to and clog the interior of the sample handling mechanism?
Figure 1.10. A mosaic of images of the Phoenix deck taken toward the end of the mission, after many attempts at sample delivery. The deck is covered and instrument funnels are clogged with clumpy soil. NASA/JPL-Caltech/UA/Texas A&M University release PIA12106.
1.5.12 Assembly begins
The MSL project finally began the assembly, test, and launch operations (ATLO) phase of the mission in May, 2008. Construction on cruise and descent stages and the rover mobility hardware proceeded rapidly. They were building two nearly identical sets of rover hardware. A testbed rover, under construction in JPL’s In-Situ Instrument Laboratory, would be used for testing of the rigors of landing and surface operations. The flight rover, along with the cruise stage, descent stage, and aeroshell, were all beginning to take physical form inside JPL’s High Bay, the clean room where white-garmented workers methodically assembled the spacecraft (Figure 1.11, Figure 1.12, Figure 1.13, and Figure 1.14). I visited the viewing galleries of both locations several times to watch the progress of construction.
Figure 1.11. Testbed rover hardware in the In-Situ Instrument Laboratory at JPL, August 25, 2008. The aluminum box at top center is the rover body. The mobility system is at left, with wheels behind red ropes at right. Arm hardware is at the bottom right. Photo by Emily Lakdawalla.
Figure 1.12. The mobility system was attached to the flight model of the rover i
n August 2008. NASA/JPL-Caltech release PIA11438.
Figure 1.13. The descent stage under construction in JPL’s High Bay, October 16, 2008. Photo by Emily Lakdawalla.
Figure 1.14. In the foreground, MSL’s gargantuan backshell, with its human-scale access port that would later be used for MMRTG installation, covered with a custom-built aluminum platform for work access. Behind it, an enormous (and enormously expensive) rotisserie-type rig upon which the spacecraft could be stacked and inverted. In the background, the cruise stage comes together. October 16, 2008. Photo by Emily Lakdawalla.
The engineers delightedly presented photos of assembly work to open the third landing site selection workshop on September 15, 2008. Watkins reported significant progress on spacecraft components, instruments, and software development, while acknowledging “lots of work to go, especially in system integration, the system level test program, and software development.” Work on incorporating heaters into the motors had relaxed the engineers’ concerns about far-southern sites. They assured workshop attendees that “All sites are currently acceptable to [the] project. Engineering [is] not a discriminator at this workshop.”
The third workshop yielded a list of four potential landing sites. Two were southern (Holden and Eberswalde craters, both of which appeared to contain ancient lake deltas). One was equatorial (Gale crater, which probably held an ancient lake and definitely had a central mountain containing layered rocks at its base). And one was northern (Mawrth Vallis, a site of uncertain geology but with fascinating chemistry). The orbiters refocused on these four locations.
1.5.13 Avionics problems
Work on the avionics and software was not proceeding as well as the more visible hardware assembly. Under schedule pressure, the avionics team began integrating the subsystems together before the individual boxes had been fully tested, which only added to the number of problems that cropped up during system testing. Tests often went poorly. Bad setup, operator error, and equipment problems meant that results were unusable or tests were unrepeatable.42 The situation was so bad that the partially redundant system might actually be less reliable than the original, single-string system would have been.43 Even worse, the growing complexity in the cross-strapping meant that the redundant systems might not actually be redundant:
The Rover Power Avionics Modules (RPAMs) were intended to be redundant boxes that cross-strapped power distribution as well as redundant analog and temperature telemetry across the system. As the number of spare power switches and telemetry channels eroded, adding additional cards to the RPAMs was considered. However the easily measured mass, volume, power, and cost implication of additional cards led to decision to instead make asymmetric connections amongst the existing cards. This asymmetry was justified by the use case where either string could be used for access to this telemetry and that in the event of a failure the loss of a non-redundant channel, the software or the ops team could find semi-graceful workarounds in flight base on inference from other channels and models. While feasible in principle, this asymmetric pattern was difficult to understand and led to confusion and testability shortcomings. It became very difficult to be able to say with certainty that loss of a redundant RPAM would be recoverable.44
Having redundant computers imposed a requirement of having a lot of test duplicates in addition to the flight hardware, and late in 2008 it looked like there would not be enough hardware to go around. If it couldn’t be tested, it couldn’t be flown. They made the decision to redesign the avionics again so that only one of the two computer systems would run at any given time.45 “Now the backup computer wouldn’t be monitoring the prime computer, so couldn’t take over immediately if the prime computer failed,” Manning recalls. So they also had to redesign the fault protection systems that would protect the rover if one of the computers failed. The fault protection redesign encompassed both software and hardware, requiring internal cables to be rerouted.46
JPL had three shifts working around the clock and on weekends to attempt to finish assembly on time. Working so many shifts was costly; the project requested another $300 million from NASA in September, 2007. When I spoke with engineers during this period, they told me they felt up against a wall, and that no amount of money or additional staff would help.
Any of these things could have caused a mission delay. The motors finally broke the schedule. After many delays, JPL actually sent engineers on long-term detail to the supplier, Aeroflex Corporation of Long Island, to help with their development and delivery, and had both Aeroflex and JPL staff working multiple shifts to complete the work. But Aeroflex discovered new issues late in the testing process that delayed delivery of the flight units again.47
1.6 A TWO-YEAR RESPITE (2009–2010)
1.6.1 Launch delay
On December 4, 2008, NASA announced that MSL would miss the 2009 launch opportunity. The next planetary alignment would not come until late 2011. The round-the-clock development and testing of the spacecraft came to an abrupt halt.
The delay brought relief to MSL and made its success possible, but it cost NASA dearly: an additional $400 million overrun brought the total price tag to $2.3 billion. Effects of the MSL cost overruns ripple across NASA’s planetary exploration program even today. Cancellations of technology development programs and lengthy delays in the announcements of new solar system mission proposal opportunities in the Discovery and New Frontiers programs, as well as NASA’s withdrawal from cooperation on ESA’s ExoMars project, can all be traced back to MSL’s cost overruns.48
The Department of Energy fueled the MMRTG just before NASA’s announcement, on October 28, 2008.49 They placed the MMRTG in cold storage at the Idaho National Laboratory. The plutonium was, of course, already decaying, so the rover would start Mars surface operations with less power than if it had launched in 2009.
NASA tried to minimize the budget impact of MSL’s delay by allocating very little money to the project in 2009, shifting development toward 2010.50 Richard Cook successfully fought for the project maintaining a team of engineers large enough to continue work on the major problems that had led to the delay. Motors and avionics were the two main ones, but smaller teams worked open issues on electrical systems, sample handling, test infrastructure, and software. The more visible pieces of the spacecraft, on which engineers had been laboring around the clock for months, were wrapped in plastic and moved to corners of the High Bay, not to be touched for a year (Figure 1.15). Most of the engineers who had been working on MSL were dispersed to other jobs, to be called back in mid-2010.
Figure 1.15. Panoramic view of the High Bay on March 16, 2009. The MSL hardware has been mothballed, and not an engineer is in sight. Photo by Emily Lakdawalla.
1.6.2 Becoming Curiosity
Despite the launch delay, NASA proceeded with a planned public contest to name the Mars Science Laboratory rover, running the contest from November 2008 to January 2009. More than 9,000 students (required to be between the ages of 5 and 18 and enrolled in a U.S. school) entered names and supporting essays into the contest. The winning entry, “Curiosity”, was submitted by 12-year-old Clara Ma of Lenexa, Kansas, and announced on May 27, 2009:
Curiosity is an everlasting flame that burns in everyone’s mind. It makes me get out of bed in the morning and wonder what surprises life will throw at me that day. Curiosity is such a powerful force. Without it, we wouldn’t be who we are today. When I was younger, I wondered, ‘Why is the sky blue?’, ‘Why do the stars twinkle?’, ‘Why am I me?’, and I still do. I had so many questions, and America is the place where I want to find my answers. Curiosity is the passion that drives us through our everyday lives. We have become explorers and scientists with our need to ask questions and to wonder. Sure, there are many risks and dangers, but despite that, we still continue to wonder and dream and create and hope. We have discovered so much about the world, but still so little. We will never know everything there is to know, but with our burning curiosity, we have learned so much.
1.6.3 Probl
em solving
With schedule pressure reduced, the sample handling team added elements to deal with the concerns raised by Phoenix’ problems handling samples. They mounted prongs and other tools to the front of the rover to allow it to poke out stubborn gunk, and a “sample playground” with a tray, funnel, and other devices where they could dump sample for visual inspection (see section 5.7). They added wind baffles around the sample inlets and across the sample portioning device. They modified the sample portioner from a straight tube to an inverted funnel shape, to make sure sample material would not get stuck as it had on Phoenix.51
The avionics team started over with one particularly challenging bit of the computer design: the asymmetric cross-strapping between the redundant main computers. They reduced the asymmetries, making the complex system slightly easier to understand and more straightforward to test.52
Data on the final four landing sites poured in from the three Mars orbiters. The United States Geological Survey used overlapping pairs of high-resolution images to develop highly detailed digital terrain models of large swaths of the landing ellipses. Engineer Paolo Bellutta took algorithms developed for the Mars Exploration Rovers and applied them to the digital terrain models, making maps of the “traversability” of the terrain and estimating drive times to reach likely science targets from likely landing spots.
The delay permitted the team to solve ChemCam’s operational problems as well. Thermal engineers designed a thermo-electric cooler that could be incorporated into the ChemCam body unit and used to cool its detectors (see section 9.2.1.2). ChemCam’s spectrometers and housing had originally been built out of beryllium and magnesium rather than aluminum in order to save mass, but now mass wasn’t a factor. The addition of the coolers doubled ChemCam’s mass.53 The end result was that ChemCam could actually be operated at almost any time of day, except for the hottest parts of summer afternoons.
The Design and Engineering of Curiosity Page 6