Red Rover

by Roger Wiens

  I was waiting to hear the outcome of the French-American space agency summit in Washington. The French director had promised to bring up the issue. If there had been a misunderstanding, they would surely clear it up. Sylvestre was in touch with both the attaché and his space agency’s deputy director, and he called me very late in the evening French time. The meeting in Washington had taken place, and his contacts assured him that they had brought up the ChemCam issue. We waited another day, but we heard nothing. Finally, I called a colleague at JPL. A very good source told us he had heard Dr. Stern saying that the French directors never mentioned ChemCam in their meeting. What was going on? This was really crazy!

  The French directors continued on from Washington to JPL on the West Coast to attend an event celebrating the fiftieth birthday of the Space Age. We arranged to make sure the cancellation was discussed with the JPL director. Overall, JPL both loved and feared our instrument. Perhaps in the minds of many JPL engineers and administrators, ChemCam was a powerful death-ray gun that might destroy parts of their dear rover if its aim was off. And yet they had been thrilled to have this flashy instrument on their big, bold new vehicle. A laser gun was appropriate for their Mars battle tank. At any rate, the French directors received assurance from Charles Elachi, JPL’s director, that his institution would do its best to support ChemCam. From then on, both JPL and the French space agency worked hard to ensure its success. Director Elachi, who was later awarded the prestigious French Legion of Honor medal, specifically referred to ChemCam in his acceptance speech, joking that the zapping noise it made every time it hit a rock was really saying, very rapidly, “Vive la France! Vive la France!”

  With the French visit over and nothing clearly resolved, our science team decided to unleash a letter-writing campaign. A lot of misinformation had been leaked to the press about ChemCam, but we committed to keeping our response positive. We drafted a letter to inform the science community that ChemCam was a very inappropriate target for descope because of its relatively low cost. We clarified that the instrument was a joint project with the French space agency, which would be a desirable partner for future space collaborations, and we emphasized above all the instrument’s important contributions, which MSL would have serious difficulty doing without. We urged each of our colleagues to write to all of the NASA administrators responsible for the decision, asking them to deal constructively with the situation. I stayed up late every night writing hundreds of e-mails, and many of our science team members did the same. Everyone made phone calls as well as sending e-mail pleas. One scientist convinced most of the students at her small college to write to NASA. A week later at a meeting at JPL, I found myself sitting directly behind one of the targets of our campaign. The man had his laptop open and was trying to go through his e-mail as the meeting droned on. He finally turned around with a very exasperated look and said, “I’m getting hundreds of e-mails about ChemCam every day just from this one college alone! Could you please tell them to stop?” I felt sorry for him, but we didn’t stop.

  We followed up the letter writing with plans to work through NASA’s independent advisory groups. Eminent scientists meet regularly to advise NASA on Mars, Venus, the Moon, and the outer planets. These groups then report to the Planetary Science Subcommittee (PSS), which reports to the NASA Advisory Council of the National Academy of Sciences. There was to be a meeting of the PSS in early October, and, in fact, most of the NASA administrators would be there. Sylvestre made a trip to the United States so we could both attend. While there we met with several levels of NASA administrators. Sylvestre and I both met Dr. Stern. We were cordial to each other. It was clear that we were trying to do our jobs in the best way we knew how. However, our goals were at odds.

  The PSS meeting seemed to be the first time the NASA administrators really heard the issue. Afterward, I was told that all of them were on our side except the one who had made the decision in the first place. The planetary subcommittee sent a strongly worded recommendation to the NASA Advisory Council to find a way to get ChemCam back on the rover payload. As a final political move, we enlisted the help of some of the most eminent people in the field to use their influence on behalf of ChemCam.

  In the meantime, we scrubbed our budget to see if we could sweeten the deal for NASA. We cut out money for spare parts and for testing. The French team promised to send an engineer over to help where he was allowed to. I was hesitant to cut cost projections too far. Things often go wrong when you are working on one-of-a-kind instruments. We had to have money to cover inevitable failures. But it was do or die.

  We had amazing support from the rest of the rover team. Some of the scientists sacrificed their own funding to help support ChemCam. Dave Blake, leader of the CheMin XRD instrument, gave up half his salary for a year to help get us back on.

  Our technical team continued on even as we were supposedly going to be shut down. We had funds to last until the end of the year, and we had not received an official order to halt work. As a joke, someone had put a collection cup at the entrance to our electronics lab, and a few people had thrown in spare change. However, in spite of the lightheartedness, people were concerned deep down. John, our systems engineer, signed on with another project, part-time at first, but it was insurance against our eventual shutdown. And people were getting tired.

  One afternoon after more than a month of “cancellation,” Bruce came back from the lab looking more worn than usual. “Bad cable might have blown our system,” he mumbled, and he immediately dialed up JPL. They were responsible for all of the cables between the French and US parts of the instrument. They had just sent a new flight-like electrical cable for us to try out with our engineering model, which was just about ready to ship to JPL. Bruce had installed the cable and turned ChemCam on. Something went very wrong; the current spiked. He immediately shut everything off, but there was a sickening smell of smoke. Upon reinstalling the old cable nothing worked. The instrument was dead! Our electrical team went into troubleshooting mode.

  Both the French and US parts of ChemCam had been badly damaged. We briefly checked out the cable and sent it back to JPL. Later, we found out that it was originally designed as two cables, one for power and one for signals, but someone had decided to merge the two. Each individual cable originally had different electrical lines labeled by number. In the merged design, lines with the same number were all connected, so line number one of the power ended up going to line one of the signal as well as to line one of the power connector on the other side. Nobody caught the error. The contracting company that built the cable for JPL didn’t know any better than to build the design it had been given. Both JPL and Bruce had “pinned out” the cable, that is, they completed a simple check to see that all the connections were there and that adjacent lines were not shorted to each other. But once the cable was built, this diabolical error was nearly impossible to detect. The result was too much voltage on the signal lines, which burned out components on both sides of the instrument. JPL’s engineers and managers were horrified that they had contributed to ruining our instrument just when their director had promised to do everything possible to get it reinstated.

  The French electronics box was returned to France, where it was repaired and sent back to us within about a week. We tried to fix our part of the instrument. But even after we replaced the obviously damaged electronic parts, the unit was plagued by bugs. One moment it would work and the next it wouldn’t. A few minutes later, part of it would respond, but the signals from the spectrometers were garbled. Every day the problem would look different. Ralph Stiglich, our electronics lead, kept replacing more parts. Day followed fruitless day as the unit seemed to work for a while only to go down again. The bugs continued for two very long months. It was an endless succession of trying out the unit, shaking our heads, pulling out an electronics board and sending it back to the soldering shop, and taking it back up to the test lab, where the board was probed with voltmeters and ohmmeters and retried. Our engineers were wonde
ring if they would ever get it to work right again. We considered giving up on the engineering model and just finishing the flight instrument, but we knew we needed the engineering model to check out the software, both on our instrument and at the rover level. At last, more than eight weeks later, the electronics were bug free.

  In the meantime, by early November we had funds for only about one more month. Politically we had done all we could. Our letter-writing campaign had pretty much run its course. We had talked with administrators and committees at all levels and their recommendations had been made. I had visited the Planetary Society, talked with the Mars Society, and contacted every advocacy group I could find. We had cut our costs to the bone. There was nothing left to do. If something didn’t happen very soon we would go broke, our team would disperse, and it would become impossible to finish. I refused to think about this possibility. We were in bad shape—our engineering model’s electronic parts had been burned up, and we were at a standstill. The team was doing an amazing job under the circumstances, but how long could we hold out?

  I arrived at the office on November 8 to find an e-mail from NASA Headquarters congratulating us for cooperating to reduce our costs. There were no further details. Had we reduced our costs? We had shaved some things from the books, but with no contingencies for accidental problems, like the one that just happened. I learned later, mostly through the press, that NASA had decided to throw a small amount of money our way, along with the contributions from the other MSL scientists, and wish us good luck. I was numb. We were exhausted and the instrument wasn’t working, but somehow we would get there.

  chapter

  fifteen

  SOLDIERING ON

  THE TEAM WAS IN A FOUL MOOD. EVEN AFTER THE ENGINEERING model was working and delivered to JPL, the low spirits continued. Our systems engineer now had to also work on the other project he had signed up for. We were skimping on personnel and it affected everyone. NASA was expecting a miracle from us, but we were only human.

  JPL was doing its best to help. The payload manager promised to get our flight CCD detectors characterized for us, potentially saving us a lot of time. For the engineering model we had just slapped the detectors in place using the factory-suggested settings and hoped that they worked. They did work, but we could tell that the signals were not optimized. These units were very sensitive to the voltage supplied to them, and without spending a few days checking each detector for its optimal setting, getting it right was a shot in the dark. So we were glad JPL offered to do this for us. Unfortunately, the characterization done by JPL’s optical engineer was appropriate for 2-D operation used for taking pictures, not the 1-D mode we needed for spectrometers. The difference sent our electronics team off in the wrong direction for two months.

  While the CCDs we had originally wanted to fly operated on only one voltage, the new ones needed five different voltages. The recommendations that were sent to us from JPL were all correct except for one, as we found out later. When testing the CCDs we were supposed to see a nice, smooth and steady curve plotted on our computer screen showing the intensity at each wavelength provided by a light source we used for testing. Without the correct setting, we ended up getting the craziest images. We started making up terms like “jitterbug,” when the line on the screen would jump all over the place, or “hairball,” when the supposedly smooth line bristled with noise spikes. We had never experienced any of this on the engineering model. Why were we getting it now? The electronics engineer was pointing his finger at the person writing the code to operate the device, who was pointing the finger back at the electronics guy. Finally, after enough interactions with JPL, we learned that its optical engineer had given us instructions for operating the device in a different mode than the 1D mode we needed. With that clarified, in April we were finally able to change the one thing that was wrong and operate the CCDs correctly.

  In the meantime, our lack of spare parts bit us. One cost-saving measure had been not to order any spare parts. Now a delicate beryllium mounting frame broke and we were forced to use it anyway. Getting a replacement now would take months—and money that we didn’t have. The broken segment did not have to support anything, so probably it would work okay with tape and glue. Still, no one ever flies broken parts. Besides the obvious break, the structure was likely weakened in other places as well. We had to ignore that possibility.

  Our delivery date to JPL for the flight instrument was supposed to be May 2008, a year and a half before launch. However, the schedule was slipping on many fronts: most of the other instruments were about as far along as we were, and a number of parts of the rover were behind schedule. We wouldn’t be able to deliver ChemCam until several months later than originally planned. Once the CCDs were operating correctly, we still had to fix a few more snags with the fiber bundles, make some final changes on the electronics boards, complete the final assembly, mate it to the mast unit coming from France, start testing the unit, go through “shake and bake,” and run our final calibrations with rock samples. We knew there was some slack in the postdelivery schedule at JPL, so in that sense it wouldn’t hurt to deliver ChemCam a few months late. The problem was that the added time to complete the instrument would require more money.

  Our budget had originally included funds to support the necessary spacecraft-level testing after ChemCam was delivered to JPL. Although delivery is an important milestone, an instrument team has a lot to do at the spacecraft during its assembly, test, and launch operations (ATLO) phase. The instrument first goes through an incoming inspection. The unit is weighed and measured, and in the case of a Mars surface mission, everything is checked for contamination. The unit is placed in a vacuum chamber along with crystal microbalances sensitive enough to detect a weight change of nanograms or picograms on their surface. If the instrument is dirty, materials will volatilize when the chamber is evacuated, and some material will come to rest on the microbalances. Too much volatile material, usually including fingerprint grease, oils, or other organic material, is not good for the rest of the spacecraft. The rover had quite stringent requirements because it was going to Mars to look for organics, and scientists didn’t want to be fooled by contamination.

  After the incoming inspection, the instrument moves on to a test bed, where it is checked out from a signal and command perspective. Does it respond to all of the signals in an appropriate way, as defined by a document we had all agreed on? This was very important, and given the complexity of the instruments and interfaces, this test can take a long time. As an example of what could go wrong, when our French colleagues delivered the first laser to Los Alamos and we first tried it out, on a few occasions the laser started firing as soon as we powered it on instead of waiting for the proper command to be sent. This was a dangerous situation and we quickly mandated that all people in the room wear protective goggles until the software was completely checked out and corrected. On another occasion, this time at JPL, a command inadvertently turned on a heater on the telescope’s focus mechanism and left it on. As the instrument was at room temperature and not on Mars, the unit overheated and was thrown out of alignment.

  When the commands are all checked out, the unit is installed on the rover for more command and performance testing. The rover then goes through acoustic testing and electromagnetic interference tests. Different instruments are operated simultaneously to see if this causes problems. Finally, the rover undergoes the infamous shake-and-bake tests, this time spending several weeks in a thermal chamber while everything—all ten instruments and all the rover gear—is checked out. For many of these tests, a ChemCam engineer would need to be present, while for others, our team would need to watch the results from our computer monitors. This could be done remotely but still needed an expert eye from our team.

  We had dedicated a sufficient amount of funding for this ATLO testing. However, being under such a tight financial squeeze, and with further delays in ChemCam’s delivery, we now needed all the money we had just to get ChemCam finished
at Los Alamos and delivered. We decided to pull all of our funds forward to get to delivery, ignoring any needs that would come later. It was the equivalent of a “Hail Mary” pass lobbed down the field in the final seconds of a football game. JPL engineers would have to run our instrument on their own for all of the integration and prelaunch tests on the rover. This had never been done before, and particularly not on a complicated instrument like ours. It was making JPL and our team very nervous. It was also crazy, because the strategy didn’t save NASA any money. It would still take the same amount of funds, and in fact more, to have staff at JPL learn how to run our instrument and support all of the tests themselves. The only reason to do it was that we were told ChemCam would not receive any more funds, while JPL would probably still get additional money if it was necessary to finish the rover.

  In March 2008, the MSL payload managers at JPL wanted to know how we were going to actually complete ChemCam, given our limited funds. They contacted a panel of experts, and together we scheduled a review date. MSL had recently announced that more funding would be needed to complete the rover, and to pay for it NASA was threatening to cut off the successful MERs that were still operating on Mars. Against this backdrop the ChemCam cost review convened. Chairing the review team was Bill Gibson, who had been Dr. Stern’s boss before he moved up to NASA Headquarters. The review was held at Los Alamos to save us the cost of traveling to JPL.

  Bill started the meeting by questioning whether there was any chance of getting additional funding out of NASA, and if not, whether the review was useful at all. We discussed the situation, but we all knew that under the current administration we wouldn’t get another cent. We presented our cost plan, including the Hail Mary strategy of just getting to delivery with no money left over. The panel had rather strong words for the plan, but they had no better advice under the circumstances. The panel gave us less than a one-in-three chance of success unless something changed. In other words, they thought we would fail. The reviewers wished us luck and went on their way.