Red Rover
Page 13
We tried to think of alternatives to the twist caps. How about running the cables up the middle of the mast tube? That wouldn’t work because the tube wasn’t hollow at the joint. Fortunately, as the project progressed, we were able to get a higher priority for the position of our spectrometers, bringing the total cable distance below 20 feet. We could live with that.
In the meantime, a conflict was brewing over the construction of the cable. The rover design team wanted it to have several connectors. That way, they could disconnect it during rover assembly and route it more easily through the rover body. We had never used optical connectors. For light that is traveling in an optical fiber to make it across a connection, the fiber on one side must be exactly lined up and touching the fiber on the other side. Any slight change, on the order of hundreds of thousandths of an inch, would reduce the amount of transmitted light. The rover engineers were used to electrical cables, where two pieces of metal simply have to touch each other across the joint—physical alignment didn’t play any role. They wanted to treat the optical fibers just like an electrical cable. We knew this just wouldn’t work.
In spite of our attempted explanations, the rover team wanted at least four connectors along the fiber and at the time they still had not moved the spectrometers any closer to the mast. The connector impasse was causing people from the two institutions to become quite uncivil to each other, so I took a trip to JPL to see why the engineers there insisted on the connectors. When I got there, I was handed a document detailing what they thought they could do. It suggested that with the connectors, between a tenth and a third of the light from the telescope would be transmitted to the spectrometers. But the fibers we used in the lab transmitted more than 98 percent of the light over most of the wavelengths. I was stunned that they would propose something like this. This team’s priorities were clearly different from mine. Their job, as it had been described to us, was to support our instrument, but this plan would certainly not maximize ChemCam’s output.
After a bit of discussion I tried a different tactic. I suggested that we just get rid of the cable altogether. Of course, this wouldn’t work at all, but I wanted to see the reaction. The team had just suggested that it would be acceptable to lose 70 to 90 percent of our signal—way more than half of our precious light. If we got rid of the cable entirely, we would lose 100 percent of the light—no light would be transmitted at all. But compared with losing 90 percent of the light, we would only be losing an additional tenth! I got some rather funny looks. As the discussion progressed, some of the engineers began to feel rather sheepish about what they were proposing to do, but others didn’t really seem to care. I left the meeting without a resolution.
The engineers had to first believe that we really needed these photons for our instrument. Typically, a first step in the design process is to determine the requirements. How much signal does the instrument need in order to perform its scientific functions? Once that is known, the requirements for the different parts can be determined: How much signal can the telescope receive at the front of the instrument? How much signal can the telescope afford to lose in the process of transmitting it to the optical fiber? How much can the fiber afford to lose on the way to the spectrometers? How efficient must the spectrometers be, and what are the requirements for the signal integrity in the detector and the electronic circuits that turn the signal into digital counts? We had worked all of these requirements out, and we didn’t have factors of five to spare—we could not be that inefficient with the optical fiber. How could we get these engineers to understand?
About the time I was ready to despair of solving the problem, we had a fortunate convergence of events. ChemCam was to undergo a review of our projected performance. During our meeting, we quite favorably impressed the audience. The audience heard our concerns about the optical cable and the low priority it was receiving with the rover. The reviewers decided the cable was now our instrument’s number one concern. As it turned out, three reviewers were also scheduled to attend a rover-wide meeting directly afterward. At that meeting, they brought our concerns to the head of the rover project, and immediately the tide began to turn. Within a week we were on the phone with JPL, listening to the description of a new plan that required the use of only one connector along the cable. Later, when they realized that they would be responsible for guaranteeing the performance of the cable, the engineers dropped the last remaining connector.
As the project progressed and everyone got to know everyone else better, the relationships improved significantly. We started seeing our common goals.
At first it was also a challenge to understand the different institutional and cultural expectations. Being a national defense installation, our lab expected us to complete a series of background investigations and associated paperwork on any foreigners with whom we were to hold technical meetings. Near the end of 2005, we had to schedule a review of the initial ChemCam design. Fourteen French colleagues were to come to represent their work, and about twenty JPL experts, including the review panel and a number of rover managers and engineers, were to judge our progress and our plans. The meeting was arranged for a location about 15 miles from Los Alamos. We had sent the French team members the paperwork to fill out to start the foreign visit process, but very few had responded in a timely way and some not at all. After all, the meeting wasn’t going to be in Los Alamos itself, and as the forms were in a foreign language to them, some just ignored them. As the meeting was about to start, our lab’s Foreign Visits office warned our managers that we needed to keep the unapproved visitors out of the meeting.
The paperwork issue was about to cause a big fiasco. At least one of the people without completed forms was a high-ranking director of the French space agency. The whole French contingent promised they would walk out of the meeting if he was barred. One of them muttered, “And we French people know how to go on strike!” We couldn’t hold the review with half the team thrown out or on strike! If our review was canceled at the last minute, with the two halves of the team bickering, the instrument itself might be in jeopardy. We quickly called our lab’s top administrators, pleading with them to let us hold the meeting. Fortunately, they found a way to deal with the paperwork, but they made us promise to pay more attention to protocol after that.
These events were frustrating to everyone involved. But by far the biggest technical challenge we faced was the failure of ChemCam’s detectors, which came in the second year of development. This was one place where our “keep it simple” policy went awry.
Every space mission must consider the effects of radiation on its instruments and components. Space is filled with cosmic rays—ultra-high-energy particles that can damage human cells and electronic circuits. The MSL mission was different from typical space missions in that it was going to a planet with at least a thin atmosphere to shield the rover from radiation. Additionally, the spacecraft itself was quite large, so it would shield its innards to a small extent during the ten months of travel time to Mars. On the downside, the rover used a radioisotope thermoelectric generator, or RTG, for power. This would end up producing about half of the rover’s radiation dose. But all in all, it would be a relatively low dose for a space mission.
When we first considered the electronic design, the engineers told us that the radiation level was well within the range that commercial electronic parts could survive. For the majority of the parts, we still planned to use military-grade components, mostly, in our minds, to keep the quality-control people happy. The detectors for the three spectrometers, however, were charge-coupled devices, or CCDs, and these could not easily be replaced by a military-grade component.
CCDs are the same devices used to record images in cameras of all types, from cell phones, to video cameras, to scientific cameras. They consist of arrays of pixels that convert light into electrical signals, which can be recorded on a memory chip. Many CCD models are unique, and as we looked into it, we found that the one we planned to use was clearly unique. No
other detector had such simple inputs or the same dimensions. And since the spectrometers were designed around this configuration, we were basically stuck with it unless we wanted to redesign the whole thing. Many spacecraft sensors used expensive, often custom-made, detectors, which could cost up to several million dollars each. Our budget would not allow this, and it flew in the face of how we liked to do things. I prefer to be pragmatic, to use whatever works.
On Genesis we had at one time scoured our town’s small hardware store for a cooking pot the right shape to use as a fixture to form a flexible part of the instrument. In the case of our spectrometer, the detector we had chosen was designed for use in grocery-store bar-code scanners. That didn’t matter to us as long as it did the job. And it did a good job in the lab. We had been using several commercial spectrometers with this detector for years. It was also easiest to stick with the commercial product, because we didn’t have any CCD experts in our part of the lab. Beyond that, this unit had passed the “shake and bake” tests for space flight. We knew we would have to do other tests before it could fly, but we were more worried about temperature effects than about the radiation dose.
After the MSL instrument selections were announced we had begun doing the additional tests on the CCDs. In the end they took far longer than they should have. The first step was finding a quality-assurance expert for our project. We chose a familiar person who was a contractor. Unfortunately, that meant we would have to go through a long contract approval process. During the six months after selection, the lab’s contract offices were moved three times, causing a big backlog. Overall, getting the contract took over four months, time we could very much have used in other ways.
Once that was complete, we had to find a company willing to put the CCDs through various tests: heat, cold, acceleration, high pressure, vacuum, moisture, pull-apart stress tests, and dissection under a microscope.
Right away we ran into problems. The first test was simply a careful analysis of the construction for workmanship quality and the materials used. The very first report warned us that the electrical leads were 100 percent tin, which was forbidden by NASA. Wires of this material had a very strange property in that they actually grew microscopic whiskers. The whiskers could come into contact with other components and short out the part. They could also act like miniature lightning rods in reverse, causing electrostatic discharges to their surroundings. Nobody really knew how the whiskers grew, but high-magnification images and failure reports abounded in the industry. It was proof enough that our CCDs could not fly in that condition. So we found a company in Massachusetts that would, for a high enough price, robotically dip the protruding wires into a vat of molten lead, which would get rid of the whiskers. We sent our large pile of detectors to all be hot-dipped.
When the lead-dipping was finished, the testing of our detectors dragged on. First, a batch of seventy-five CCDs (although only three would fly, we needed to test a large number of them) went through many of the nondestructive tests. A subset of those was then sent through another batch of even more strenuous ordeals, including high acceleration, pull tests, and acoustic tests. Everything took twice as long as it should have. It was now a year and a half since ChemCam had been selected for flight, and the testing was nearly finished. We had decided to do the radiation test at the end, thinking that it was a sure bet. We would run a single detector through this test first, then study the statistical variations between different units. When the first irradiated CCD came back, we were busy with other things, and our quality-assurance person was out of the country, so we let it sit for a couple of weeks. At the beginning of July 2006, we turned our attention to the detector.
It was a Tuesday at noon when I finally discovered the disastrous results—the unit was completely unusable! The background noise of the irradiated device was through the roof. I was shocked. What had gone wrong? I didn’t even completely believe the results. But as we had to give a monthly report to the JPL management the next day, I put the CCD results in our report, stating that they were tentative.
That night I stayed up late. In fact, I didn’t sleep well at all. I verified that the test had been done correctly—the problem was not an overdose of radiation. I also found that a similar test had been reported in the literature, but because I was not familiar with the conversion factors between different units of radioactivity, I had not recognized the implications. Finally, I noticed a construction analysis of the detector that warned that the CCDs might be quite susceptible to radiation. Now I began to wonder whether our project had a fighting chance or if the necessary changes would kill it. I prayed that they wouldn’t!
The next day we reported the bad news and promised to do all we could to examine other options. That night I stayed up late again, this time browsing the web to see if we could possibly find another CCD to fit our spectrometers. To my surprise, there were several scientific-grade options available that were designed to withstand radiation. These detectors were all rectangular, rather than configured in a single strip of pixels as we had planned, but that would have to do. One of our team members called the companies the following day to ask about suitability and availability. There was an eight-week wait for the one that seemed the most promising on paper. It looked like even getting an initial test done would be a long, drawn-out ordeal.
The following day, the JPL payload managers were due for a visit on another topic. Instead, the meeting was completely dedicated to the CCDs. The managers graciously promised that they would lend whatever expertise we could use. Over the next several days, we had phone meetings with various JPL experts, all of whom liked our candidate detector. Some of the accommodation changes to the rest of the instrument started not to seem quite so radical. Our engineers were figuring out ways to deal with them.
Then, even more amazing things happened. We learned that the manufacturer actually had a couple of detectors on the shelf. We were able to order them right away, and they were in our hands within a week instead of the eight weeks we had feared. The spectrometer company we were partnering with, Ocean Optics, was also able to jump on the change right away. Although some of the modifications would delay us, and the new detectors were much more costly—$5,000 instead of $100 each—ChemCam no longer seemed in mortal danger, at least not from the detectors!
But our problems were not over. At this point a completely different component began giving us a devil of a time. ChemCam was to use two types of optical fibers to transfer light to our spectrometers. One was the long 20-foot cable that JPL was to supply, which was coming along nicely at this time. The other was a set of fiber bundles within the instrument proper. These units were called “bundles” because they each consisted of between twelve and nineteen very small fibers about the width of a human hair. We used them to shape the light. At one extremity, the fiber ends were arranged in a circle, and at the other side the ends were all lined up in a row, the shape we needed the light to have when entering the spectrometers. The bundles, which were designed to improve the instrument’s performance, were a bit of an afterthought, so they had not been ordered at the beginning of the project. We were now trying to play catch-up.
The fiber company was slow to respond initially, and so I booked a flight to their plant in Massachusetts to explain how important their little units were to our project. As soon as I sat down in the plane, I knew something was wrong—the crew was taking its time even though the departure was imminent. Eventually a flight attendant came on the PA system and announced a mechanical problem. Two and a half hours later we left. By the time I got to Minneapolis, my connecting flight had already departed. Trying to squeeze the trip into as little time as possible, I had planned to catch a flight back just after my morning meeting the next day. But there were no more flights east, so now everything was messed up.
The airline wanted to book me on a flight that would arrive in Massachusetts after my return flight had already taken off to go back. I explained my predicament, and the airline found a way to get
me to my meeting sooner, but it would require an extra stop in Chicago. It was the middle of February and the coldest night of the year in the windy city. I arrived very late and had to walk a quarter mile to my assigned hotel, where I would get a few hours of sleep before catching a 6 A.M. flight. Ferocious gusts of below-zero wind whipped at me and blew dirty snow in my face as I trudged to the deserted hotel lobby.
Fortunately, the morning flight went well. The company officials gave me a warm welcome, and we were able to talk through the contract. They shortened the delivery time to about twelve weeks, which seemed tolerable. These fibers, being so tiny, were incredibly difficult to work with, requiring gluing and polishing under a microscope. We were definitely pushing the capabilities. As they explained their processes I became much more sympathetic about the time they said it would take to make the units. I decided we could handle the delay.
Unfortunately, when we received the fibers, they only partially worked. Some light came through them, so it looked like they were working, but measurements with a photon meter showed that much of the light was being lost somewhere inside the fibers. It was going to take at least another round to get it right.