Red Rover

by Roger Wiens

  The challenge was how to correctly diagnose the problem without tearing the instrument apart. ChemCam was not built with disassembly in mind. We had cut corners on the design, so that many parts had been glued together rather than fastened properly with screws or nuts and bolts. In addition, the instrument had evolved over time. In one place some heaters had been added that were not part of the original plan. The heaters were actually epoxied across a joint we would have to open to take the unit apart. And we didn’t have spares, so if we ruined something we would be sunk. It was all part of keeping ChemCam as inexpensive as possible, reinforced in a big way by the cancellation. Now, of course, we were sorry for having cut corners.

  Rather than trying to open the unit, we attempted to replicate the problem with some “breadboard” equipment—that is, equipment used for early versions of assemblies that had been built and tested on the workbench. We first learned, surprisingly, that our breadboard did not have the problem. But that didn’t help us at all. Next, we ran some tests on the flight instrument that allowed us to look at the signals coming from the CCDs. But these tests were also inconclusive. As this was happening over a period of days, Ralph, our electronics engineer, became convinced that the detectors were the problem. He didn’t have much evidence on his side, so I didn’t take him very seriously. I had seen many situations in which engineers blamed the part they were not responsible for. The easiest thing was for Ralph to fault the detectors, which he didn’t build.

  Eventually, Steve Bender, our optics engineer, came up with a very good test. He remembered that we had some spare detectors that had been shipped along with the ones we had installed. The flight CCDs and the spares had both been tested at JPL after coming from the factory. We knew the detectors were fine when they were tested at JPL, and the spares, untouched after being delivered to us, should be fine, too, even if something had happened to the ones in ChemCam.

  Ralph and Steve went to the cleanroom to test the spare detectors. Lo and behold, the untouched sensors had the same problem as the flight ones. Now I knew Ralph was wrong. But Steve came to Ralph’s defense. Unknown to me, these sensors were not completely pristine. Steve had checked out each of the sensors as soon as they had arrived. He had a hunch that the check-out procedure might have done something to them. I was still unconvinced. Anyway, it is usually easier to go along with someone’s ideas until they are proven stupid than to try to convince him to pursue a different path. So I agreed to let Ralph and Steve pursue this lead. After all, with the launch delay, there was time.

  We had a few meetings with experts from the CCD manufacturer. Their experts agreed that there could be a failure mechanism that would result in the symptoms we were seeing, and the company announced that it was modifying its product line to make its CCDs less sensitive to electrostatic discharge or transient voltage spikes. When the new batch of CCDs was finished, experts from JPL flew to the manufacturing plant overseas to hand-carry the parts to Pasadena and then to our lab. We were able to replace and re-align the CCDs without destroying the heater or any other parts of the instrument.

  The detector replacement gave us an opportunity to fix another problem that had appeared over the past couple of years. There was a thermal mismatch in how the Curiosity rover was designed. The basic design was laid out on paper very early—well before the instruments were selected in 2004. The rover was to operate all of its electronics in a range between –40°C and 50°C (–40°F to 122°F). This was fine for most circuits, particularly those running the rover. However, the upper half of the range was much too hot to operate almost any kind of detector. Many detectors rely on various types of CCDs, the same devices used in commercial cameras. Their electronic noise doubles for every 12°F increase. People who might take pictures of relatively dark objects in a hot desert environment—for example, outdoors in Phoenix on a hot day—might notice the problem. Most scientific detectors are used to collect very low signal levels; they operate in a regime where the background noise acceptable to a photography enthusiast might totally wash out the data of interest. To make matters worse, the radiation damage experienced in space can multiply the noise levels by factors of fifty or more. To minimize this effect, many sensors on scientific instruments are operated below freezing, and sometimes at much colder, temperatures.

  As an additional caveat, the rover had to stay warm enough even in very windy conditions in the dead of a Martian winter to keep its instruments from being damaged by the extreme cold. The designers compensated by having the rover run hot at normal times. The result was that the Curiosity rover was poorly built to handle the cooler operating ranges of the scientific instruments it was to carry.

  The payload teams dealt with this problem in one of two ways. Those with bigger budgets designed chillers for their detectors. This was silly because the air temperature on Mars only gets up to room temperature on the warmest of summer days at the equator, and even at the equator the nighttime temperature plunges to the coldest arctic temperatures on our globe. A simple block of encapsulated ice would have kept the detectors cold throughout the Mars day. But while the rover had a very good heat source in the form of its RTG, the designers did not incorporate a cold source for the instruments to tap into.

  The ChemCam budget did not allow for a chiller; it would have to make its measurements early in the day while the CCDs were relatively cool. Actually, we were a long way into ChemCam’s development before we understood that the rover would run hot most of the time. At first we couldn’t believe that it would operate above room temperature on a cold planet. By the time the engineers convinced us, we had another problem: the rover motors would now only work at warmer temperatures than originally planned. These mechanisms would have to wait until later in the day, when it was warmer, to do their thing. The consequences for ChemCam were catastrophic. By the time the mast motors would be warm enough to point the instrument at a rock each late morning, ChemCam’s detectors would be too hot to make useful measurements!

  To understand the problem, a team of engineers drew up color-coded charts to show the times of the day the mast could point the laser and the times of the day our detectors could make measurements. The places where the colors overlapped would be ChemCam’s operating times. But when they finished the chart, the colors didn’t overlap at all! Not at any time of the day. Not in summer, not in winter, and not in spring or fall. In each season, the colors came near each other, but they never overlapped. The engineers showed the chart at an executive project meeting, and after a moment of staring at it the managers all shook their heads and laughed. No one could have put together a more perfectly designed snafu. We knew the problem would have to be solved or our instrument would only be dead weight.

  While the launch was still scheduled for 2009, a small team did what they could to passively cool the detectors. ChemCam’s body unit, the part where the sensors resided, was located next to an outside wall of the rover body. The wall would not be kept warm like the instruments, so it could be a source of cooling. The options were to connect a conductive copper strap to the wall, or to radiate the heat to it across a small gap. The engineers decided not to use a strap because it would warm up rapidly when the Sun shone on the wall. However, we were given a passive connection: both the inside wall of the rover and the box housing our sensors were painted black so they would radiate heat away, and our box was placed on insulating feet so it would get less heat from the rover. These measures partially solved the problem, but we still were left with only half an hour per day of guaranteed good operation—not enough to do all we wanted to do.

  Now that the launch was delayed and we had to replace our detectors, it was decided to add a chiller. The thermoelectric cooler, or TEC, would be designed and built by JPL. After all, the organization was hurting from the loss of jobs caused by the launch delay. It was a potentially messy affair for one organization to build an add-on to someone else’s instrument, and especially to retrofit it on an already-built, delicate device. JPL engineers had o
nly a few months in which to get the whole job done. But the team that JPL payload manager Ed Miller put together pulled it off on budget and on time. The engineers delivered the TEC to our door in early 2010. When we ran the thermal tests after installation, the temperature profiles matched JPL’s models so well that it was hard to tell which curve on the chart was the model and which was the test result. This improvement, along with the new detectors, made us thankful for the launch delay.

  But one more set of snafus awaited us. Earlier we had been experiencing difficulties with the “demux,” or optical demultiplexer—a box that took light from the telescope, split it into different color bands, and fed them to our three spectrometers. The optics were okay, but the mounts seemed to be shifting during mechanical stress tests. So we decided to redesign the box, taking advantage of the launch delay. The new plans were drawn up by October 2009, and we were ordering the parts, when Ed Miller decided it would be a good idea to have someone review the lenses as well. We had originally used a very simple optical design to keep our costs down. Many engineers know the KISS acronym and live by it: Keep It Simple, Stupid! Now the JPL optician reviewing the design thought we could eke out some improvements by using more complicated lenses. Since we were getting this help for free, we decided to take it. The main problem was that the new lenses would arrive a month later than we wanted them to fit the rest of our schedule. We decided the wait would be worth it. In hindsight, we should have kept it simple.

  The lenses came in mid-February, and by early March we had installed them and reassembled and reattached the demux. The unit now had LANL-designed lenses in one part and JPL-designed lenses in another part. Our engineers were starting a barrage of environmental tests—shaking and baking the assembly. The shake went fine this time, but when we tested ChemCam over the temperatures expected for flight, we noticed something really strange. The output color was shifting: it was bluer at one temperature and redder at the other extreme. The difference in color between cold and hot was nearly a factor of two! ChemCam depended on a completely stable output; otherwise it would misidentify the rocks on Mars. We put everything else on hold and started a long series of tests to figure out what was wrong. Once more we faced the dilemma of not knowing where the problem lay: Was it our detector (again!), the spectrometer, or the rebuilt demux? If it was an optical problem, then which lens or mirror was causing it? And once again, we didn’t want to tear the instrument apart to find the answer.

  After agonizing for a while, we separated the spectrometers from the demux and were able to show that the problem was in the latter, so we zeroed in on the optical parts in that box. Our JPL counterparts immediately suspected the Los Alamos lenses, although we had not seen this issue prior to installation of JPL’s new lenses. Part of their reasoning was that we had used cheaper components than they had. They recommended tearing our lenses out of the demux and sending them to JPL for testing. But Steve did something he wasn’t supposed to do: he removed the new JPL lens instead and inspected it under the microscope. What he saw seemed telltale: the doublet lens appeared to be delaminating—coming apart—along the edges. We immediately reported the finding. This was bad news, and somewhat surprising, because one of the other redesigned lenses was also a doublet. We didn’t yet know whether that one had a problem, too.

  We stopped work until we could determine if the other lens needed to be replaced as well. In the meantime, experts set to work diagnosing the bad one. It is a good thing we waited for the results, because both simulations and tests showed that the initial diagnosis was not the problem. It appeared that the glass itself was sensitive to temperature. The other JPL lens was made of another type of glass, so we didn’t have to remove it.

  In the meantime, the optical engineer came up with yet another lens design to replace the faulty one. It was simpler, but was still supposed to be an improvement over the original. We waited about five weeks for this lens to be ordered, fabricated, and delivered. At this point we were at the end of our leash: there was no more time before we had to deliver ChemCam and bolt it on the rover. The day the lens arrived we popped it in place, and Steve began the tedious job of alignment. The room lights were turned off so there was no stray light—only the illumination of the calibration lamp. Steve preferred to work alone in this environment. We left him there for eight hours. Always with steady hands and keen eyes, Steve would not leave a job until it was done right. We thought for sure he would be done by the end of a long day. Nope. Another day went by. We were getting impatient, but something still seemed wrong. Steve was not getting the expected improvement. In the meantime, the optician was checking his notes and calculations, finding nothing wrong. But Steve still wanted more time. After some negotiations, we agreed on one more day. That day, too, came and went, but the demux was still worse off than with its original lens.

  We had to move on. JPL had to install part of ChemCam on the rover immediately. We quickly finished our calibrations and sent the laser box on to JPL, where it was prepped for installation. However, Steve wasn’t at all happy. He was sure the original lens was better than the replacement. I didn’t want to change the configuration we had used for our final calibrations. But while I was away, Steve opened the instrument and switched the lenses. Sure enough, the original gave nearly 20 percent more light. We should have followed the KISS lesson and kept things simple!

  After a final alignment, we buttoned the unit up and gave this part of ChemCam its final calibrations and stress tests. We were done! The next part would be more interesting and a great change of pace—equipping Curiosity with its laser gun and enjoying the rover tests.

  chapter

  eighteen

  ON THE ROVER

  THE DAY OF OUR DELIVERY REVIEW FINALLY CAME. WE HAD been looking forward to this day since the beginning of the project over six years ago. It was the culmination of all the hard work we had put into building ChemCam. In the intervening time, team members had come and gone, babies had been born, children had grown, and politics and society had changed—but no new rovers had visited Mars.

  The ChemCam review was held on a Monday in late July 2010. A number of project members from France and JPL had come to review the final product and make sure it was ready for the rover. As always, something didn’t go quite right. Our secretary, who was supposed to provide the temporary badges for the visitors arriving for the review, called to say she couldn’t make it. The cows had gotten out next door to her place in the Rio Grande Valley, and she had to help get them back in the pen. It was only a minor inconvenience in the activities we had been anticipating for so long. Once everyone had their badges we started the meeting. The day was otherwise uneventful. The team was excited to hear of ChemCam’s performance and congratulated us on completing the job. We had waited so long to hear these words.

  After all was said and done, ChemCam really did perform well. Many times what is described in a proposal turns out to be too difficult to build. For ChemCam, we had started out with a LIBS instrument, and in the proposal process we had added a high-resolution imager. At times we discussed whether we really needed the Remote Micro-Imager (RMI). For the French team making the mast unit, it was difficult to balance the optical needs of the LIBS and the RMI functions. In the end we were very glad we kept the imager, as it proved its worth on Mars. On the LIBS side, the trade-off meant that ChemCam ended up with 14 millijoules of laser energy on the targets—somewhat less than we had hoped for—but we were still able to do accurate rock composition measurements out to a distance of 7 meters (23 feet). We were still learning to improve the accuracy of the technique. ChemCam would always be less accurate than an APXS instrument, given the distance at which the measurements were made and the small beam size, which is focused to less than a millimeter at the target, but overall we were close to meeting our goals.

  The total bill for ChemCam came out to almost exactly $15 million, not counting the French part and some help from JPL; the launch delay and modifications done during that time end
ed up causing the last $3 million of the total. ChemCam represented a very small fraction of the whole Curiosity rover cost, well under 1 percent. During and after our cancellation, some journalists had the impression that our instrument had caused hundreds of millions of dollars in mission overruns. Once a new NASA administrator was in charge we were able to set the record straight.

  Overall, the Curiosity mission ended up costing nearly $2.5 billion, with the launch delay representing over $400 million of that total. By comparison, the MERs had come in at around $850 million, not counting for inflation. Cassini, however, which explored the Saturn system with its rings and numerous moons, had cost significantly more, at $3.26 billion, including the foreign contributions but not counting for inflation in the intervening decade. I liked to explain that Curiosity cost the price of a movie ticket for every person in the United States. The public would get its money’s worth.

  With the ChemCam delivery review behind us, the installation on the rover began. The laser had already gone to JPL, and within a week of the review it was bolted on the mast. The spectrometers and data unit, which were destined to sit inside the body of the vehicle, were shipped as exclusive cargo on a Fed-Ex White Glove truck in August. In September the rover was turned upside down, and the last of ChemCam was installed from underneath. Software commands were developed to communicate with the instrument, and they were tested on a mock-up unit. By October we were ready to try out the real thing. A team of French and Los Alamos engineers converged on Pasadena for the event. We were given two shifts on the weekend, when no one else would be around, so that our laser would not accidentally harm anyone.

  At 7 A.M. on Saturday morning, a heavy marine cloud layer lay over the San Gabriel Valley in Southern California, with light drizzle slowly covering the roadways and cars. The city was lazily waking up in the semidarkness. I made my way from the hotel on old Route 66 to the Jet Propulsion Laboratory. From the outside the place looked nearly deserted. Betina Pavri let me into the Spacecraft Assembly Facility’s control room, located on one side of the tall assembly area housing the rover. The low-ceilinged control room was arrayed with electronics racks humming with cooling fans. These were interspersed between tables with rows of computer screens. Staff members were wearing headsets, and a loudspeaker broadcast the chatter from inside the assembly room. The rover commands and dialogue could be heard faintly above the hum of the fans. Along one side, a long series of windows looked in on the assembly facility. For today’s work the windows were covered with thick black fabric as a precaution against the laser beam. Above them in several locations were wide video screens projecting the activity inside.