There were other ascent engine development problems, primarily with welding and fabricating the injector, and with localized gouging of the ablative throat and nozzle in long duration firings. There were overpressure “spikes” during startup at altitude, when the engine exhausted onto a plate in close proximity, simulating the descent stage from which it would launch on the Moon. These restricted the options available to the designers for stopping instability. The Bell engine’s ablative nozzle was made of thermoplastic material similar to the command module’s heat shield. It resisted the extremely high temperatures (three thousand degrees Fahrenheit) inside the engine by its excellent thermal-insulation properties and by charring when overheated, leaving in place a firm, charred residual material that largely preserved the nozzle’s original shape. Irregularities in the injector spray pattern could cause local hot streaks within the chamber that produced cuts and gouges in the nozzle throat, the location within the engine where the rate of heat transfer to the walls was the greatest. Baffles on the injector face improved the engine’s stability, but usually made nozzle gouging and erosion worse, so the designers found themselves between the proverbial “rock and a hard place.”
Two failures in the fall of 1966 gave added impetus to the quest for a solution. The first spontaneous instability, without bomb detonation in the chamber, occurred during an altitude test of the engine at White Sands; this was with a flat-faced injector. Soon afterward a baffled injector configuration failed a bomb test at Bell. More engineers, design variations, and tests were added to the schedule, but results were elusive. At one point a configuration successfully passed ten bomb tests but failed the eleventh. We had set a criteria of twelve successful tests before declaring a design stable; after that, we increased the hurdle to twenty tests.
In the midst of this activity Dandridge and Bob Thompson of Grumman Propulsion were flying from La Guardia Airport to Buffalo on one of their many visits to Bell. Seated together, they discussed engine tests and bombs intently for most of the flight. As they disembarked, four burly men in dark suits confronted them, flashed FBI shields, and hustled them into an anteroom. For the next two hours the agents questioned them severely, demanding to know what was behind all this talk of bombs that the flight attendant had overheard. It took much explaining by Dandridge and Thompson, and telephone calls to Grumman security and Bob Carbee, before the FBI was convinced that their implausible story was true.
The cut-and-try fixes dragged on without conclusive results, and by mid-1967 NASA was very concerned; they saw this problem as a potential Apollo program “show stopper.” NASA hired Rocketdyne to develop an alternative injector that could be installed into the Bell engine. As soon as Rocketdyne had some injector hardware and began to obtain test results, there was a continuous series of high level reviews and visits between Rocketdyne at Canoga Park, California, and Bell at Niagara Falls, New York. General Phillips, Apollo program director, George Low, Apollo spacecraft director, and NASA propulsion expert Guy Thibideaux were involved, as was I, Joe Gavin, and Dan-dridge and his group for Grumman. Bill Wilson of NASA at Rocketdyne, Steve Domokos, Rocketdyne ascent engine program manager, and Dave Feld of Bell maintained close communications and a cooperative attitude, transforming what could easily have been a very sticky contractual situation into close teamwork across companies and organizations.
Rocketdyne succeeded in producing a stable injector, but the engine they designed had other problems, including hard starts, rough running and manufacturing and assembly difficulties. Bell was unable to show a positive fix on their injector despite repeated tries. At one point, in a combination of jest and frustration, Dave Feld declared, “Perhaps combustion instability is an East Coast phenomenon. Let’s send our engine to Rocketdyne and have them test it out there.”
To Feld’s surprise, we quickly accepted his suggestion. The tests were repeated in the Rocketdyne facility at Santa Susanna, and the bomb-induced instability was still present. With the Bell engine in hand, Rocketdyne was then directed to install their injector and combustion chamber into it, retaining the Bell nozzle, valves, and mounting hardware. After some adjustments to the injector spray pattern to minimize throat erosion, this combination of the two companies’ engines performed very well. Bomb-induced oscillations damped out in less that four hundred milliseconds and thrust and specific impulse5 were within specifications.
In May 1968 Phillips and Gavin visited Bell and Rocketdyne and reviewed the latest design options and the encouraging test data. They presented three design and contracting options to Low: (1) Bell engine and Bell injector (still not positively stable), (2) Rocketdyne injectors installed in Bell engines at Rocketdyne, or (3) Rocketdyne injectors installed in Bell engines at Bell. Low chose option two, with the entire assembly put together and furnished by Rocketdyne. By June 1968 this design had passed fifty-three bomb tests without instability, and it was completely qualified in August.6 The long ordeal of ascent engine development was over and a major item was removed from the show stoppers list. Not a moment too soon, because the heavy LM flight schedule of 1969 was almost upon us. LM had already lost its berth on Apollo 8, which was to have been the first manned LM flight with the CSM in Earth orbit. (LM-3 wasn’t ready when needed to fly in late 1968.) Instead, Apollo 8 became a manned lunar orbit mission with CSM only at Christmas 1968. The original Apollo 8 mission was slipped to Apollo 9, flown in March 1969.
Stress Corrosion
The Super Weight Improvement Program resulted in the widespread use of chemical milling for structural parts on LM that rendered aluminum alloy parts more susceptible to stress corrosion, because it left a relatively rough and open-pored surface finish. Even without chem milling, certain aluminum alloys were vulnerable to this phenomenon, which in the 1960s was just being encountered and understood. Stress corrosion is intergranular corrosion occurring at the metal grain boundaries, usually visible only through a microscope looking at an etched and polished surface. A combination of steady stress and moisture or humidity is required to produce this corrosion. The stress levels can be relatively low—well below the recommended working stress of the material. Cracks and failures from stress corrosion typically do not develop until months or years after the original stress is applied—and this became a hidden time bomb for the LM structure, surfacing widely in the schedule-critical years of 1967 and 1968 and finding its way to the show stoppers list.
The most common applied stress causing this problem was “fit-up” stress, which resulted when parts did not fit or nest together exactly. When the fasteners were tightened, the parts were deflected until they made complete contact and the stress resulting from the deflection was locked into the material. Some joints, particularly the rod ends inserted into LM structural tubes, required a press fit to provide a solid, immovable connection. In such cases a predetermined stress level was purposely locked in upon assembly.
Minimizing fit-up stress required parts that fit together well or were carefully shimmed upon assembly to eliminate deflections when the fasteners were tightened. Training of assembly mechanics was revised to emphasize the importance of proper parts fit up and techniques for accomplishing it. We also reviewed the engineering tolerances on parts assemblies, particularly press fit joints.
Some stress corrosion of thin tabs was noted on LTA-1 as early as 1964, and a stress corrosion inspection and survey was performed on all LMs then under construction. The problem continued at a low level, with an occasional crack found, until a rash of cracked parts was discovered in mid-1967, beginning with LM-1. The cracks were mostly in the press fit ends of structural tubes, although some thin tabs were found cracked also. Mueller was furious. At that late date the Apollo schedule was threatened by this insidious problem—it could exist on almost any part, anywhere within the LM. The day after an inspection, a new crack could develop, because the nature of the phenomenon was chronic and progressive. LM stress corrosion was branded a show stopper.
Heavily pressured from above, I went to gen
eral quarters. Led by Bob Carbee, our best structural design and materials troubleshooters, Len Paulsrud, Will Bischoff, and Frank Drum assembled a team from Engineering, Manufacturing, and Quality Control to inspect all the accessible structure on the LMs under construction. Visual inspection was conducted with flashlights and magnifying glasses. Suspect areas were brushed with Zy-glo dye penetrant, which glowed under ultraviolet light and enhanced the visibility of tiny cracks.
By mid-February 1968 we had thoroughly inspected six LMs (numbers 3 through 8) and over fourteen hundred accessible components. No major cracks were found. We also switched from 7075-T6 aluminum alloy to the more stress corrosion resistant—T73 temper, effective on LM-4 and subsequent vehicles. This was accomplished by retrofitting the structural tubes on these vehicles. By changing out the tubes that had been the major source of trouble and showing by inspection that fit up parts with thin tabs were not cracked, I felt we had the problem under control, and NASA agreed. By the end of the month, Mueller told NASA Administrator Webb that he was no longer worried about stress corrosion.7
Inspections continued throughout the LM program, and occasionally a cracked part was found and replaced. LM never suffered a structural failure of any kind, so the effect of stress corrosion on the flight missions was nil. But stress corrosion was a nagging problem that never entirely went away, as there was always a small chance that the next inspection would turn up a newly cracked part.
Battery Problems
The LM batteries were constructed with silver and zinc electrodes and liquid potassium hydroxide electrolyte and were not designed to be repeatedly charged and discharged. A limited number of recharges was allowed during ground tests, but in flight the batteries were launched fully charged and discharged until depleted. The EPS had four batteries in the descent stage, producing 3 ampere-hours per pound of battery weight, and two in the ascent stage, which yielded 2.5 ampere-hours per pound. The batteries had alternating plates of silver and zinc connected to positive and negative terminal bus bars by metallic jumpers and separated from each other by paper insulation. The plates were mounted in a row inside a vented plastic case that was filled with electrolyte. A straightforward design with seemingly not much that could go wrong. But looks can be deceiving. Before the LM program ended, in the course of resolving a long list of development problems, Grumman was forced to learn more about these batteries than even the manufacturer knew.
My confidence in the batteries waned when I visited the Eagle Picher factory near Joplin, Missouri, in 1966 as part of a tour of several LM subcontractors. Eagle Picher was primarily a manufacturer of paints and industrial chemicals based on paint pigments. They got into the battery business because some of the electrode materials, like lead and zinc, were used in their paints, and they saw a way of diversifying their product line with new applications of materials that they understood. Knowing this background did not prepare me for the sight of their factory complex—it was a sprawling industrial wasteland with many acres of low sheds and two-story brick buildings, dominated by dozens of tall smokestacks spewing great clouds of gray and white smoke. White dust covered everything—the roofs and walls of the buildings, the ground, the cars in the parking lots, everything.
The LM battery assembly area was located in one of the low corrugated metal sheds. Inside the windows were open to the swirling particle-laden atmosphere and there were layers of white dust on the windowsills and floors. At rows of worktables, strapping farm boys were doing delicate battery assembly, folding sheets of insulation over the plates, mounting the plates on the bus bars and connecting the jumpers, and installing the plate assemblies into the battery case. One muscular fellow caught my eye particularly; he was wearing a soiled undershirt and had a cigarette dangling from his lips. As I watched incredulously, the ash on his cigarette grew longer as he peered over the open case into which he had just placed a plate assembly, until it finally broke off and fell inside.
That was the last straw for me. I motioned for Eagle Picher’s LM battery program manager to follow me into a glass partitioned cubicle off the assembly floor and exploded. “Don’t you people have any concept of quality?” I shouted. “For God’s sake, look at this place! Clean up the building, close the windows, install air conditioning and filters, prohibit smoking, make the workers wear clean clothes and smocks, maybe even replace the farm hands with nimble-fingered women.”
After that visit I insisted that Grumman hold a monthly review with Eagle Picher, alternating between Joplin and Bethpage. If I could not attend the Joplin meeting, which was frequently the case, Carbee filled in for me if he could. I also prevailed upon Joe Kingfield, LM Quality Control manager, to station a resident Grumman quality inspector at Eagle Picher. It seemed like cruel and unusual punishment, to assign a Grumman man to such a place, but I considered it necessary. Our inspector filed a weekly report on Eagle Picher’s activities and progress in “cleaning up their act.”
All this was in the nature of a preemptive strike, since as yet no battery problems had emerged. Our concern was soon justified as erratic battery performance began to appear in tests, traced to assembly errors and contamination. As battery development progressed, many other problems surfaced. Case cracking and jumper failures during vibration tests were recurring problems, and required several modifications to design details before the batteries could pass the vibration portion of qualification testing. Underperformance was a problem that required redesign to maximize the active area of the plates and to determine the optimum electrolyte concentration. Venting of hydrogen from the case during battery operation was provided by a small plastic relief valve in the sealed cover. These valves sometimes stuck shut during cold-temperature testing, causing hydrogen gas pressure to build up and the cover to pop free of the case. In space this would cause complete battery failure, as all the electrolyte would leak out. The durability of the paper separator also gave concern. There were some short circuits between plates when pinholes developed in the separators after several ground test recharges. More durable separator material and further limitation of recharges solved this problem.
This list of miscellaneous problems continued through 1967 into 1968, giving me and Grumman a background level of concern. The batteries passed their qualification tests, although several months behind schedule, and the problems that remained did not seriously threaten the Apollo program schedule. I remained wary of the batteries until the completion of the program, preconditioned by my original view of Eagle Picher as a dirty gray scene of desolation from Dante’s Inferno.
Tank Failures
Of the thousands of possible failures in the Apollo Mission and the LM, one of the most terrifying was rupture or explosion of a tank. Dozens of tanks on LM held the vital consumables of space exploration: rocket propellants, helium pressurant, oxygen, and water. Most of these tanks contained high levels of pressure energy and could explode in event of failure. Since all aerospace contractors used the same group of subcontractors to make their tanks and the same materials and similar processes and procedures, a tank failure anywhere in the Apollo program sent a shock wave of concern to everyone.
In mid-1965 there were two series of tank failures, both on the CSM program but directly affecting LM. CSM reaction control system fuel and oxidizer tanks each failed at their manufacturer, Bell Aerosystems, which also made very similar tanks for the LM RCS. Then a failure occurred at Beech Aircraft, which was making hydrogen and oxygen tanks for the CSM fuel cell assembly. I sent John Strakosch from LM Structural Design and Frank Drum from Materials to both companies to meet the engineers leading the failure investigations and establish an information channel to Grumman on the findings. They were also able to suggest lines of inquiry to the investigation teams.
Strakosch and Drum returned very concerned because the origin of the failures was unknown and they applied directly to LM. A list of possible causes had been prepared and was being vigorously explored by NASA, North American, Bell, and Beech. Both tanks were of highly str
essed titanium and the failures seemed to originate at or near the welded circumferential seam.
Thanks to careful sleuthing, these cases were solved within a few months. The RCS tank failures were caused by the propellant manufacturer improving his process to increase the purity of the nitrogen tetroxide oxidizer that he produced. The revised production process reduced the amount of trace contaminants, including nitrous oxide, which had played a beneficial, if unsuspected, role in protecting the titanium from attack by the nitrogen tetroxide. By specifying a minimum allowable amount of nitrous oxide in the product, the problem was resolved. To assure control over the formulation of such commercial products, NASA began buying propellants for the whole program under a government specification.
The cause of the Beech problem was quite different. It occurred because a weld rod of lower strength titanium alloy had been inadvertently used to weld the oxygen tank. This was determined by metallurgical examination and analysis. A torrent of procedures and regulations followed, aimed at tightening control and accountability for weld rod and certifying that the proper alloy has been used in each weld.
In October 1966 one of the large service module propellant tanks ruptured while under pressure test at NAA in Downey. After intensive investigation the cause was determined to be incompatibility between titanium and the methanol (methyl alcohol) that was used as the pressure test liquid.8 This oversight stimulated NASA to conduct a comprehensive survey of all the fluids to which the Apollo tanks were exposed in their lifetimes and to perform laboratory tests to establish whether the tank material and the fluid were compatible. It seemed fairly basic, but until then it had not been done. This case reminded me of an experience I had years earlier while developing a small liquid rocket at Lockheed. My test rocket failed because I had not known that the nitric acid oxidizer would attack the nickel in the high strength stainless steel alloy we were using.
Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight) Page 20