Red Rover

by Roger Wiens

  As the ChemCam project wore on, my family had to take things in stride. We were now about halfway into the project. The euphoria of building a Mars instrument had worn off. I would come home from the lab after eleven hours of struggling with all the things that were not working. The family was getting tired of me coming home so late and so weary. I didn’t share too many of the details of our woes, as my wife and kids were becoming less and less sympathetic to them. Sometimes, mentioning problems evoked the wrong kind of response. Thus the issue of fiber bundles became a favorite topic at our dinner table, but not in a good way.

  My boys, who were now thirteen and ten, had reached the age where their parents were no longer their heroes. I must have griped to them just a little too much about the fibers. Having no idea what they were, the boys somehow thought it was a funny joke. “Dad, how are you doing with the fiber bundles today? Are they still a disaster? Do we need to pray for them?” Carson, the older one, would guffaw in a voice that was just starting to change from boy to adolescent. “FIBER BUNDLES!” younger Isaac resounded in his higher pitched, squeaky voice, exploding the “B” in “BUNDLES.” He repeated it about seven times, giggling until he rolled off his chair and onto the floor. “FIBER BUNDLES, FIBER BUNDLES!” he would repeat a few more times, as he kicked his feet in the air, cackling some more. “We’re still working on them,” I would reply wearily, attempting to restore order at the dinner table. These instrument problems are not solved overnight. But it was hard to describe such slow progress to my boys, who just wanted more of my time.

  Eventually, the fibers did work very well, after we enlisted the help of NASA fiber experts at JPL and Goddard Space Flight Center.

  As the summer of 2007 rolled around, we took delivery of a prototype of the mast unit from France. This consisted of the laser, telescope, and camera. Our French colleagues had done their work in record time, delivering an engineering model in just two and a half years. Now ChemCam was going to get exciting! We could start zapping rocks and taking pictures.

  Shortly after delivery, we had a visit by some of the French team members, including my good friend Sylvestre. The purpose of their trip was to check out the electrical connections and software commands between the French and American parts of the instrument. We made a lot of progress, but we knew it would be a short week, as it was right before Labor Day weekend. By the end of day on Thursday, the camera images were transferring nicely from the French mast unit to the instrument’s computer in the Los Alamos–built unit and out to our rover simulator and laptop. But none of our technical staff would be around on Friday.

  Sylvestre and I spent the morning in discussions with other scientists on calibrations and on the Mars simulations we were running. However, by afternoon we were itching to get back with ChemCam. Sylvestre wanted to take pictures at longer range. In France, the assembly room had been too small to do this. Our cleanroom in Los Alamos, which was made to house small satellites, had a garage door on one end where the satellites could be transported directly into a large adjoining room. When we opened the garage door, we could take pictures at distances of up to 30 feet. We played with this for a bit, but we still wanted a longer range. After all, our camera was looking through a small telescope, somewhat like the long-range lenses that sportscasters use from their boxes high in the stands. Our cleanroom, and even our larger outer room, just wouldn’t do. Then we realized that if we had a mirror, we could effectively double the distance. ChemCam could take a “self-portrait” looking at itself in a mirror. Unfortunately, we didn’t have a mirror.

  I hunted around the lab for a while until Sylvestre suggested that we try the rest rooms. It was a silly idea, but a bathroom mirror might do the trick. The men’s room had a small one on the wall. We got a screwdriver and started to loosen it, but it didn’t come. Sylvestre ducked into the ladies’ room without even knocking. French people tend to be a little less modest in that regard. He invited me in to check out the hardware. The women’s room had the perfect mirror. It was floor length and could easily be removed from the wall with screwdrivers. There wasn’t a soul around, so we took the mirror and carried it down the hall to the lab, where we got it lined up by propping it between some boxes. Thirty feet away, we took turns crouching behind ChemCam to get our pictures taken in the mirror. At effectively 60 feet, the camera and telescope framed our faces nicely!

  By the end of summer it seemed as if the middle part of the project—the worst part—was over. We were nearly finished building the engineering model and we had started on the flight model. It had been a very difficult two years. I was looking forward to relaxing a little, as was the rest of our team. Little did we know that within a week we would be hit with the worst threat of all.

  chapter

  fourteen

  CANCELED

  THE MSL ROVER PROJECT GOT ALONG FINE FINANCIALLY FOR the first two years. Some areas grew in cost, but the program had more than enough in reserve to cover these areas. I thought the rover was doing very well for a major project of its size. Most of the large missions I had known ended up in financial trouble immediately. NASA’s usual response was to remove instruments from the payload. These removals, or descopes, could sometimes cripple the mission’s scientific return while yielding only a very small cost savings. But there are only two other options when cost runovers occur: letting the costs rise, or canceling a mission altogether. The problem is that the really big costs on every mission are the launch vehicle (the rocket), the software, and a lot of the engineering needed to make it all work. In short, the bulk of the costs are for essentials.

  In the case of the MSL rover, the payload had started at $75 million out of what was then a $1.4 billion mission. If the total mission costs increased by just 7 percent, or $100 million, NASA could conceivably wipe out the whole payload—the whole reason for the mission—and still not save enough money to contain the mission to the original cost.

  The $75 million figure for the payload had been inadequate from the beginning. Although all of the instruments had originally added up to $75 million, many things had to be changed on each one to meet the actual specifications of the rover. In the case of ChemCam, the wiring between the mast and the body was to be supplied by JPL. We were going to use voltage converters in the body to power the mast. However, the wires specified by JPL were so thin that the voltage would drop significantly going from the body to the mast, so we had to change our plans. Then there was the issue of the optical fibers, which was taking a lot of our resources.

  On top of that, JPL was getting worried about the mass of the rover. Every subsystem seemed to be gaining weight. To hold the line, JPL announced to the payload leaders that it would pay for us to find ways to lighten our instruments. We had planned to make our spectrometers out of titanium, which was very stable thermally, to accommodate our optics. John Bernardin, our systems engineer, suggested that we build these units out of beryllium.

  Beryllium is very expensive to manufacture and requires special controls because beryllium dust is a health hazard. Using beryllium for ChemCam would mean a longer time at the manufacturer, special training for team members, and a lot of restrictions on what we could do to modify the units. All of that costs money, but it would save about 2 pounds, almost cutting the weight of our body-mounted unit in half, and JPL was willing to pay several hundred thousand dollars for that. We agreed to lighten ChemCam in exchange for the increase in funding.

  The various changes to our instrument added up to a couple of million dollars. At the end of our first year of work, we signed an agreement with JPL that raised the ceiling on our costs from just under $7 million in the proposal to about $9 million. We were helped a lot by the fact that a good portion of the instrument, including the laser and the telescope, was being built in and paid for by France.

  The other MSL rover instrument teams were going through similar accommodation issues. In some cases they were finding out that the components and labor would simply cost more than planned. A couple of
instruments that started out four or five times the cost of ChemCam doubled their costs. In the end, the combined cost of the two most expensive instruments was more than $150 million. This is not too surprising, given that the much smaller payload of the previous Mars rovers cost well over $40 million, not adjusting for inflation.

  Another component that was costing a lot more than expected was the Sample Arm and Sample Preparation and Handling system. Dubbed SA-SPaH (“saw-spah”), this included a rock grinder, a drill, a rock crusher, and inlet systems for the two “analytical lab” instruments, which evaluated powdered samples that were dumped into inlets on the deck of the rover. The scientists decided they also wanted rock powder to be dumped onto an observation tray, where they could take close-up images of it. It was learned from the previous rovers that the rock grinder and the drill bits could become dull well before the end of the mission. The engineers were more than happy to design replacement bits and grinder heads that could be changed on the fly. But to do this robotically was very complicated and greatly increased the number of joints and other moving parts. The number of engineers working on the SA-SPaH increased and the completion dates were pushed back. Eventually, the whole thing became too complicated and the design was scaled back.

  Other parts of the rover were now facing problems, too. MSL was supposed to be the ultimate all-terrain vehicle that could operate at any temperature, down to the freezing point of the Mars atmosphere. NASA had planned some new technologies for its motors. One was to replace stainless steel bearings with titanium to save a couple hundred pounds.

  In the first round of testing, the new titanium gears failed. NASA ordered a second round, but the costs were mounting. In addition, the projected delivery date for the gears was getting later and later, which would require JPL to keep more people on the project longer, running up staffing costs.

  The rover was facing a cost crunch. But to NASA, this was only a part of a bigger picture of increases among all missions. In the NASA portfolio were missions at various stages. Some were almost ready to launch, while others, like the James Webb Space Telescope, were still in the planning and feasibility stages. Some missions, like the successful MER twins, had launched some time ago and were asking for mission extensions.

  All of these cost pressures were now brought up to a new NASA associate administrator for space science, Dr. Alan Stern. Dr. Stern promised to live up to his name in order to control the spiraling mission costs. So when the MSL management team went to Washington with its hat in hand in the summer of 2007, Dr. Stern told them to go back and find ways to cut back.

  Controlling costs is the most thankless and difficult task in the NASA administration. It is extremely difficult to estimate the cost of a new technology accurately. Pundits have concluded that in order to understand the cost of a new project, it must start into the development phase. You could say that new technology is like a tunnel—you can only see a little distance into the future. And if the future of some project turns out to be very expensive, you’ve already started into that tunnel when you find that out.

  In addition, each NASA administrator would like to receive the credit for starting as many new missions as possible. There is no political advantage to being fiscally conservative. Dr. Stern was saddled by fiscal problems started under a previous administration, and he wanted to come up with new missions under his watch. What NASA needs is some reserve funding that it can tap into when missions go over budget. However, the accounting arms of government won’t allow for a rainy day fund.

  With the rebuff from Washington, the MSL management was in a quandary. If the mission were still in the design phase, it would have been relatively easy to remove an instrument or support system. However, the instruments were mostly designed and in the process of being built, and the SA-SPaH had already been trimmed back.

  The instrument leaders first held an emergency session in Pasadena to review the list of options for cutbacks. It was a short list. There was a redundant software system to help with landing, a spare power pack, and then there were the instruments. The instruments were in the final construction phase, so all their contracts were in place, and much of the work was done. We left the Pasadena meeting with the feeling that very little could be done at this point other than canceling the mission out-right, wasting more than half a billion dollars already spent and leaving little in the way of plans for further Mars exploration.

  But NASA needed to take something out of the mission. I was slightly concerned for ChemCam because the laser technique was new. Political support for traditional analysis techniques had a couple of decades of advantage over LIBS, as graduate students had toiled over dissertations and learned every last detail of these techniques. They had grown up with this knowledge and betted their careers on their tried and proven methods. But while we were new on the block, we had the French on our side. NASA would surely think twice before killing an instrument with millions of dollars of contributions from a collaborating country. Such an act could mean the death of future collaborations, of which a number were in the works.

  The issue of descopes seemed to come and go. The emergency meeting in Pasadena had been the first week of August 2007. At the time, our ChemCam development had seemed at a standstill. Our electronics team was still coming up to speed on running the new detectors. But finally, in early September, our efforts turned to success. Things weren’t perfect, but they were working. We installed the detectors in our engineering model and went for our first complete test. Many of the engineers working on the project had never seen LIBS in action. We now called them to the cleanroom for our first try. Everyone put on their laser goggles. We commanded the laser. Pow! The plasma light erupted on the rock some distance away. Now all eyes turned to the computer monitor as the data was transferred from the instrument. There it came! Small but unmistakable blips. We knew we could tune things up from here. The team was really energized. ChemCam was going to work.

  I knew a lot of work was still ahead of us, but I was starting to relax just a little. Our family vacation that summer had consisted of going to a Mars conference at Caltech—certainly not much of a family event. In early September, Gwen and I began discussing getting away for our twentieth anniversary coming up that fall. We stayed up late planning a weekend cruise along Baja California. I figured I could at least afford to get away a little. We worked through all the details on the website, discussed it a little more, and then hit the “purchase” button. At first nothing happened, and then a window appeared saying we had been timed out. The evening was getting late so we decided to finish making reservations the next day.

  I went in to work the next morning just a few minutes later than usual. I turned on my work cell phone as I got out of the car. It had a message from the previous evening, 9/11. The recording said I was to call NASA Headquarters as soon as I could. My mind started churning. Could it be something about the rover descope?

  I got on the phone with Michael Meyer, NASA’s chief Mars scientist. “I have bad news for you, Roger,” were his first words. My heart sank. I let him go through his explanation that ChemCam was being canceled. When he finished I asked him the reason for the cancellation. His answer was that it was the instrument’s cost. “You’ve got to be kidding!” I replied. I explained that our costs were projected to be only about $1.5 million above the agreement with the rover team (in large part due to the change of detectors). We had some money in the bank, and we needed less than $2 million at this point to get the job done. To cancel it now would waste nearly $10 million, not to mention the French contribution. The French had invested close to twice as much money, and how would they take this? “Really?” was Michael Meyer’s reply. He seemed unaware of the French contribution. I went on for some time, but eventually the arguments grew tiring. Michael clearly agreed that the cancellation was a bad idea, but I would have to take it up with people above him. He asked us to arrange a meeting with the administrators.

  Visitors were knocking at my door as the phon
e call ended. A geology professor and her eager students had come to learn more about our exciting LIBS technique. In a choking voice, I squeaked to Sam Clegg, who ran our LIBS lab, that NASA called to say they were canceling ChemCam. He bustled the visitors off for the morning so I could deal with the issue.

  The next call came from France. Bruce Barraclough, our project manager, and Ed Miller, JPL’s payload manager, were there for a technical visit to the French ChemCam team. Everyone was incredulous. The discussion went on for some time. Most importantly, we learned that the director of the French space agency and his deputy were on their way to visit NASA Headquarters the very next week to discuss other collaborations. Our French counterparts passed the word along to make sure ChemCam’s cancellation would be brought up.

  In the meantime, the leaders of the other rover instruments were convening a meeting, and I drafted a quick memo to our science team. The rest of the day I was busy with our visitors and with our technical team. So far, the decision was not widely known, which was good. It couldn’t be real—it had to be a bad dream! Gwen and I quietly called off our anniversary plans.

  The next couple of days were a flurry of meetings. We talked to people at several different levels of the NASA chain of command. The person above Michael called to discuss the issue with our administrators. A colleague across the hall from me knew the next person up the chain at headquarters from graduate school. She contacted him, and her discussion seemed to confirm that not all the facts were straight about ChemCam—neither our financial agreements with JPL nor the French contributions to the project. The political reasons became a little clearer as well. It seemed that NASA was afraid to go after the instruments being built at two different NASA centers. That pair of instruments was ten times ChemCam’s cost, and they had ten times the cost overruns. But because ChemCam was not at a NASA center, we had been nailed. Meanwhile, Dr. Stern, who by now had publicly announced the decision, denounced our instrument and our institution to the press. Our administrators did not like this tactic.