by The Design
By that time Malin Space Science Systems had completed assembly of MARDI, but
hadn’t completed the thermal testing that would be required for it to be included on
MSL. It was validated enough for descent, but hadn’t been tested for surface operations, a
detail that would become important after landing. Delivered in July, 2008, it became the
first science instrument to be integrated on the rover, and the only one to participate in
every major rover test.
1.5.8 Mastcam dezoomed
The longest-lasting impact of Stern’s science instrument descopes was on Mastcam. As
originally conceived, Mastcam consisted of a pair of identical cameras. Each had a zoom/
telephoto lens with up to 15x magnification, giving a field of view ranging from a wide 90°
to zoomed 6°. They would have been capable of shooting color, high-definition, stereo-
graphic, cinematic video in stereo at a rate of 5 frames per second. For this reason, film-
maker James Cameron joined the Mastcam team as a co-investigator. The descoped
version of the Mastcam consisted of two different cameras, both with narrow fields of
view (one at 15°, one at 5.1°); they sacrificed same-focal-length stereo capability to maintain the ability to view the landscape at different scales. They could not capture the wide stereo video landscapes that the original design would have enabled. Cameron lost interest, because the new Mastcam was not a tool he could use for his art. The loss of public
outreach value – high-definition stereo video from Mars, directed and distributed by a rich, well-connected Oscar-winning director – is incalculable.
Meanwhile, Malin Space Science Systems had to imagine, design, fabricate, test, and
deliver a totally new optical design for MSL’s science cameras in barely more than one year.
It wasn’t going to be easy, because the descope solutions had a major oversight: fixed-focus cameras wouldn’t be in focus at all the distances they were expected to cover. The Malin
team considered focusing the narrower-angle camera at infinity and the wider- angle one
closer to the rover, but then images in the middle ground would be out of focus for both
cameras, and stereo imaging would be impossible. Mastcam engineers Mike Caplinger and
Mike Ravine came up with an idea that might be cost-effective: use the already developed
MAHLI instrument focus mechanism for the Mastcams. But this would not comply with
the requirements of the descope. The Project Science Group recognized that being able to
focus Mastcam would be crucial, and went back to NASA Headquarters to request permis-
sion to return focus but not zoom capability to Mastcam, because the camera would not be
able to achieve its science goals without a focal mechanism. NASA eventually approved the
request, and design of the new Mastcam optics began in December 2007.
1.5.9 Scarecrow’s debut
Back at JPL, they constructed a new outdoor Mars Yard at the top of the steep JPL campus
for MSL mobility testing, and invited media to view “a parade of rovers” there on June 19,
32 Mars Science Laboratory
which I attended. One of the rovers on parade was Scarecrow, a prototype for MSL
(Figure 1.9). Scarecrow has a rocker-bogie suspension system, motors, and wheels, but hardly any other hardware, which allows its wheels to exert the same ground pressure on
Earth that the full-size MSL’s wheels would eventually do under Mars gravity. (It’s called
Scarecrow because, like the character in The Wizard of Oz, the minimally instrumented rover has no brain.) Scarecrow and the Mars Yard are still in use for mobility testing, more than a decade later.
Figure 1.9. Scarecrow rolling over enormous rocks in the newly opened JPL Mars Yard on June 19, 2007. Note the “JPL” letters machined into the wheels’ treads. Photo by Emily
Lakdawalla.
1.5.10 Budget balloons
The development problems all had financial implications. In December 2007, the MSL
project requested another $91 million from NASA. Alan Stern doubted that this request
reflected reality. He tasked Doug McCuistion with independently analyzing the MSL bud-
get. McCuistion found that MSL had underestimated its likely needs by $40 million. Stern
set aside $190 million to solve MSL’s budget woes, bringing the total cost to nearly $1.9
billion. Some of this money had newly become available with the delay of the second Mars
Scout mission opportunity from 2011 to 2013. That would make 2011 the first Mars launch
opportunity since 1996 to have no NASA mission slated for it.
1.5 The Cost of Complexity (2007–2008) 33
The Mars budget had become a battleground. Mars scientists were worried about what
they saw as Stern’s attacks on the Mars program, and lobbied hard for more money to be
moved into the program. Stern was determined to keep the MSL overruns from damaging
other areas of space science. He continued to look for places in the Mars program to find
funds to pay for the overruns on MSL. On March 24, 2008, the Science Mission Directorate
ordered the Mars Exploration Rover mission to cut $4 million from their $20 million bud-
get that year, and another $8 million the following year. Principal investigator Steve
Squyres responded that the cuts would require him to shut down Spirit. The public
exploded with outrage over the threat to the charismatic rover. The next day, NASA
Administrator Mike Griffin repudiated Stern’s letter. Hours later, Stern resigned. 40
The public furor was symptomatic of Griffin and Stern’s incompatible management
visions. They fundamentally disagreed about how to handle the perennial problem of
budget- busting missions. Stern fought to contain the damage within programs, and to pro-
tect the small amounts of money supporting research and analysis of NASA data. He
wanted to bring an end to what he saw as irresponsible fiscal management of missions and
the collateral damage they wrought on other missions. But he acted unilaterally, without
the concurrence of NASA leadership. Griffin replaced Stern with Ed Weiler, who had led
the Science Mission Directorate from 1998 through 2004, during the overhaul of the Mars
program in the wake of the twin disasters of 1999. Weiler would remain in the position
through the rest of MSL’s development. Within months, Weiler delivered JPL the money
they had requested. The MSL workforce increased from 700 to 800 people. 41
1.5.11 Phoenix descends
On May 25, 2008, the Phoenix mission landed in Mars’ high northern latitudes. Although
the landing went perfectly and NASA heralded it as a success, during its short, 5-month
mission Phoenix would have frustrating problems attempting to sample Martian soil and
ice and deliver it to laboratory instruments (Figure 1.10). Puffs of wind blew the samples away from the instrument doors. When sampled material did fall onto the instrument, the
Martian soil tended to clump and stick, failing to fall through sieves that protected the
instruments from large particles, even when the sieves were vibrated.
The difficulties on Phoenix were sobering news for MSL. Sample handling hadn’t yet
been tested even under optimal conditions. Would wind blow away the drilled samples
intended for SAM and CheMin? Would the powder stick to and clog the interior of the
sample handling mechanism?
1.5.12 Assembly begins
The MSL project finally began the assembly, test, and launch operations (ATLO) phase of
the mission in May, 2008
. Construction on cruise and descent stages and the rover mobility
hardware proceeded rapidly. They were building two nearly identical sets of rover hard-
ware. A testbed rover, under construction in JPL’s In-Situ Instrument Laboratory, would be
used for testing of the rigors of landing and surface operations. The flight rover, along with 40 Lawler (2008)
41 Manning and Simon (2014)
34 Mars Science Laboratory
Figure 1.10. A mosaic of images of the Phoenix deck taken toward the end of the mission, after many attempts at sample delivery. The deck is covered and instrument funnels are
clogged with clumpy soil. NASA/JPL-Caltech/UA/Texas A&M University release PIA12106.
the cruise stage, descent stage, and aeroshell, were all beginning to take physical form
inside JPL’s High Bay, the clean room where white-garmented workers methodically
assembled the spacecraft (Figure 1.11, Figure 1.12, Figure 1.13, and Figure 1.14). I visited
the viewing galleries of both locations several times to watch the progress of construction.
The engineers delightedly presented photos of assembly work to open the third landing
site selection workshop on September 15, 2008. Watkins reported significant progress on
spacecraft components, instruments, and software development, while acknowledging “lots
of work to go, especially in system integration, the system level test program, and software development.” Work on incorporating heaters into the motors had relaxed the engineers’
concerns about far-southern sites. They assured workshop attendees that “All sites are cur-
rently acceptable to [the] project. Engineering [is] not a discriminator at this workshop.”
The third workshop yielded a list of four potential landing sites. Two were southern
(Holden and Eberswalde craters, both of which appeared to contain ancient lake deltas).
One was equatorial (Gale crater, which probably held an ancient lake and definitely had a
central mountain containing layered rocks at its base). And one was northern (Mawrth
Vallis, a site of uncertain geology but with fascinating chemistry). The orbiters refocused on these four locations.
Figure 1.11. Testbed rover hardware in the In-Situ Instrument Laboratory at JPL, August 25, 2008. The aluminum box at top center is the rover body. The mobility system is at left, with wheels behind red ropes at right. Arm hardware is at the bottom right. Photo by Emily Lakdawalla.
Figure 1.12. The mobility system was attached to the flight model of the rover in August 2008.
NASA/JPL-Caltech release PIA11438.
36 Mars Science Laboratory
Figure 1.13. The descent stage under construction in JPL’s High Bay, October 16, 2008.
Photo by Emily Lakdawalla.
1.5.13 Avionics problems
Work on the avionics and software was not proceeding as well as the more visible hard-
ware assembly. Under schedule pressure, the avionics team began integrating the subsys-
tems together before the individual boxes had been fully tested, which only added to the
number of problems that cropped up during system testing. Tests often went poorly. Bad
setup, operator error, and equipment problems meant that results were unusable or tests
were unrepeatable. 42 The situation was so bad that the partially redundant system might actually be less reliable than the original, single-string system would have been. 43 Even 42 Devereaux and Manning (2012)
43 Cook (2011)
1.5 The Cost of Complexity (2007–2008) 37
Figure 1.14. In the foreground, MSL’s gargantuan backshell, with its human-scale access port that would later be used for MMRTG installation, covered with a custom-built aluminum platform for work access. Behind it, an enormous (and enormously expensive) rotisserie-type rig upon which the spacecraft could be stacked and inverted. In the background, the cruise stage comes together. October 16, 2008. Photo by Emily Lakdawalla.
38 Mars Science Laboratory
worse, the growing complexity in the cross-strapping meant that the redundant systems
might not actually be redundant:
The Rover Power Avionics Modules (RPAMs) were intended to be redundant boxes
that cross-strapped power distribution as well as redundant analog and temperature
telemetry across the system. As the number of spare power switches and telemetry
channels eroded, adding additional cards to the RPAMs was considered. However
the easily measured mass, volume, power, and cost implication of additional cards
led to decision to instead make asymmetric connections amongst the existing cards.
This asymmetry was justified by the use case where either string could be used for
access to this telemetry and that in the event of a failure the loss of a non-redundant
channel, the software or the ops team could find semi-graceful workarounds in flight
base on inference from other channels and models. While feasible in principle, this
asymmetric pattern was difficult to understand and led to confusion and testability
shortcomings. It became very difficult to be able to say with certainty that loss of a
redundant RPAM would be recoverable.44
Having redundant computers imposed a requirement of having a lot of test duplicates
in addition to the flight hardware, and late in 2008 it looked like there would not be enough hardware to go around. If it couldn’t be tested, it couldn’t be flown. They made the decision to redesign the avionics again so that only one of the two computer systems would run
at any given time.45 “Now the backup computer wouldn’t be monitoring the prime computer, so couldn’t take over immediately if the prime computer failed,” Manning recalls.
So they also had to redesign the fault protection systems that would protect the rover if one of the computers failed. The fault protection redesign encompassed both software and
hardware, requiring internal cables to be rerouted.46
JPL had three shifts working around the clock and on weekends to attempt to finish
assembly on time. Working so many shifts was costly; the project requested another $300
million from NASA in September, 2007. When I spoke with engineers during this period,
they told me they felt up against a wall, and that no amount of money or additional staff
would help.
Any of these things could have caused a mission delay. The motors finally broke the
schedule. After many delays, JPL actually sent engineers on long-term detail to the sup-
plier, Aeroflex Corporation of Long Island, to help with their development and delivery,
and had both Aeroflex and JPL staff working multiple shifts to complete the work. But
Aeroflex discovered new issues late in the testing process that delayed delivery of the
flight units again. 47
44 Welch et al (2013)
45 Devereaux (2013)
46 Manning and Simon (2014)
47 Cook R (2009)
1.6 A Two-Year Respite (2009–2010) 39
1.6 A TWO-YEAR RESPITE (2009–2010)
1.6.1 Launch delay
On December 4, 2008, NASA announced that MSL would miss the 2009 launch opportu-
nity. The next planetary alignment would not come until late 2011. The round-the-clock
development and testing of the spacecraft came to an abrupt halt.
The delay brought relief to MSL and made its success possible, but it cost NASA
dearly: an additional $400 million overrun brought the total price tag to $2.3 billion.
Effects of the MSL cost overruns ripple across NASA’s planetary exploration program
even today. Cancellations of technology development programs and lengthy delays in the
; announcements of new solar system mission proposal opportunities in the Discovery and
New Frontiers programs, as well as NASA’s withdrawal from cooperation on ESA’s
ExoMars project, can all be traced back to MSL’s cost overruns.48
The Department of Energy fueled the MMRTG just before NASA’s announcement, on
October 28, 2008.49 They placed the MMRTG in cold storage at the Idaho National Laboratory. The plutonium was, of course, already decaying, so the rover would start Mars
surface operations with less power than if it had launched in 2009.
NASA tried to minimize the budget impact of MSL’s delay by allocating very little
money to the project in 2009, shifting development toward 2010. 50 Richard Cook successfully fought for the project maintaining a team of engineers large enough to continue
work on the major problems that had led to the delay. Motors and avionics were the two
main ones, but smaller teams worked open issues on electrical systems, sample handling,
test infrastructure, and software. The more visible pieces of the spacecraft, on which
engineers had been laboring around the clock for months, were wrapped in plastic and
moved to corners of the High Bay, not to be touched for a year (Figure 1.15). Most of the engineers who had been working on MSL were dispersed to other jobs, to be called back
in mid-2010.
1.6.2 Becoming Curiosity
Despite the launch delay, NASA proceeded with a planned public contest to name the
Mars Science Laboratory rover, running the contest from November 2008 to January
2009. More than 9,000 students (required to be between the ages of 5 and 18 and enrolled
in a U.S. school) entered names and supporting essays into the contest. The winning entry,
“Curiosity”, was submitted by 12-year-old Clara Ma of Lenexa, Kansas, and announced
on May 27, 2009:
Curiosity is an everlasting flame that burns in everyone’s mind. It makes me get out
of bed in the morning and wonder what surprises life will throw at me that day.
Curiosity is such a powerful force. Without it, we wouldn’t be who we are today.