Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight)

by Kelly, Thomas J.

  Radcliffe was tall, handsome, and articulate, a big man with the lithe grace of a trained athlete. A highly competitive long-distance runner at Syracuse University, he continued to train while at Grumman, frequently running ten miles from work to his home in Lloyd Harbor. This was long before jogging was a popular activity: one night a suspicious Bethpage policeman brought him to the station house for questioning, sure that anyone running along the road must be running away from a crime scene. He also competed in cross-country ski races in the White Mountains of New Hampshire, holding his own against not only other Americans but also athletes from Finland and other Scandinavian countries. He loved sailing on Long Island Sound, challenging the elements in a thirty-five-foot 1906 Herreshof-designed sloop with grossly excessive sail area that was always teetering on the edge of capsizing in strong winds. An athlete, outdoorsman, and engineer, he threw himself wholeheartedly into anything he undertook.

  Radcliffe was a flight test engineer before joining the LM program. He worked with the design engineers and the test pilots to develop the flight test programs to validate and certify Grumman’s experimental airplanes. He flew aboard the test flights, recording gauge readings from the instrument panel as the test pilot maneuvered the new airplane through its paces. He had developed a healthy skepticism regarding engineers’ claims for their designs: “Well, we’ll see how good it is when we test it.” It was the perfect attitude for one charged with proving that LM’s rocket propulsion systems were safe and reliable.

  His flight test work had shown Radcliffe how good Grumman’s products were—the best carrier-based military aircraft in the world. He was fiercely proud of the quality and excellence of Grumman airplanes; to him the “Grumman ironworks” and “as sterling on silver” were articles of faith not to be profaned.2 A minister’s son, he saw the LM program heading for an apocalyptic end unless it reformed and returned to the high standards set by the company’s founders.

  Radcliffe started with Joe Gavin and Bob Mullaney and worked his way down through the LM management hierarchy, preaching against the evil of leaks and the need to reform our designs without delay. When he reached me I was shocked by what I heard. I had not realized that the rigs were still leaking badly despite the improved seal designs we had provided. Carbee and his fluid systems design group leaders joined the meeting at my request, and after Radcliffe repeated his message of warning we discussed what to do next. Radcliffe’s opinion was unequivocal: “Eliminate all mechanical joints, and learn how to make brazed joints that don’t leak.” We agreed to work toward this goal.

  In the ensuing weeks we eliminated the AN (army-navy standard), Gamah, and other threaded fittings that were used with replaceable components. Instead we brazed these components directly into the system with high-temperature nickel-silver brazing alloy, using a tubing stub provided on the component. For replacement, the Manufacturing Engineering group worked out a technique of cutting out the component and using the induction brazing tool to heat the stubs on the component and the plumbing side of the joint to remove the two halves of the cut braze sleeve (“de-plucking” was the descriptive if inelegant term the shop gave to this operation). The stub ends were cleaned and polished and the new component was brazed into the system using the standard techniques. This made maintenance of these fluid systems more difficult and time-consuming, but if we made the brazes properly, they did not leak. It was necessary to X-ray every brazed joint to assure that full penetration of the brazing alloy had been achieved with minimum void area. If it had not, the joint had to be reheated with the electrical induction heating tool using slightly longer time and higher temperature than the previous attempt. Up to four reheats were allowed in trying to get a good joint—beyond that, the joint had to be cut out and resized, a new braze sleeve had to be installed, and the brazing process was repeated. Portable X-ray equipment was used to inspect the joints in place on the shop floor, but no one could work in the immediate area while X-raying was in progress.

  There were still some areas where we retained mechanical connections, mostly on the propellant and pressurant tanks. Large openings in the tanks were necessary for cleaning, inspection, and installation of quantity gauging sensors. I retained bolted flange designs in these areas, and at the tank manufacturers’ plants we found such designs to be leak-tight.

  Even Radcliffe found the improved system design to be acceptable, except that he occasionally had to reheat brazed joints or tighten bolted flanges that had developed leaks. Once he was conducting a high-level NASA delegation, headed by Wernher von Braun, through a tour of Grumman’s test facility at White Sands. It was the Huntsville team’s first visit, and they were very interested and asked many questions. When they visited the descent-stage altitude chamber to observe a flight-weight test rig loaded with live propellants, the test conductor briefed them in front of a portable projection screen, standing with his back to the test rig. As he spoke, Radcliffe, who was in the back of the chamber facing the test article, suddenly saw an ominous white cloud rising from the rig. He immediately cut the speaker off and shouted, “Gentlemen, there’s a fuel leak. Get out now—follow me!”

  The Huntsville group moved with alacrity, following Radcliffe into the adjacent control room. The altitude chamber door was actuated shut, safely containing the fumes. It was embarrassing, but thanks to Radcliffe’s quick action nobody was injured or even exposed to the deadly propellant fumes.

  Leaks continued as an occasional nuisance item in cold flow and at White Sands until the LM-1 fiasco in June 1967. LM-1 was delivered in the midst of shakedown problems with the spacecraft assembly and test operation in Plant 5, Bethpage. When we delivered LM-1, Grumman and the local NASA inspectors thought we had a leak-tight spacecraft, based upon passing helium sniffer tests while pressurized with helium at a low pressure of 50 pounds per square inch (psi). (Operating propellant tank pressures were between 175 and 235 psi, different for each of the three systems.) Soon after it was received at KSC, LM-1 was found to have widespread leakage in the propulsion and RCS systems. The people at Cape Kennedy quickly characterized Grumman’s first flight-worthy spacecraft that we had proudly, if tardily, shipped as a “piece of junk that leaked like a sieve.” It was a hard blow to the morale of the people at S/CAT, who had put in long hours and seven day weeks to deliver LM-1 and were convinced that it was a quality product.

  We initially thought that the cape’s findings were due to differences in leak detection procedures and equipment. To test this hypothesis a QC crew from Cape Kennedy came to Bethpage and performed leak tests on systems in LM-2 and LM-3 that had been found leak-tight at Bethpage. Discouragingly, they found some leaks that had escaped detection by the home team. Moreover, the leaks were real—on both LM-1 at the cape and LM-2 and LM-3 some of the leaks detected by the sniffers could also be seen in the bubble test. Although I was unsure why this happened, I declared that we would adopt the cape leak test regimen to the letter and have experienced cape inspectors train our people in its use.

  It took three weeks to standardize our equipment and complete the Cape Kennedy procedures training at Bethpage. The cape used a different model helium sniffer than we, although made by the same manufacturer, Veeco. Embarrassed and responding to pressure from NASA, Joe Gavin became directly involved in the leak problem. At my recommendation he put Will Bischoff, deputy Structural Design Section head, in charge of an intensive leak fix effort. The bolted flanges on the tanks were the worst problem, followed by the few other smaller mechanical component connections that had survived my earlier purge.

  Bischoff consulted with the tank manufacturers, Aerojet (ascent propulsion), Allison (descent propulsion) and Bell Aerosystems (RCS), and with O-ring and sealing experts around the country, and developed a new design for the tank flanges. It had dual O-rings, revised groove dimensions and tolerances, and a test port between the two O-rings to detect leakage. Test samples performed very well, as did the first tanks with the new flange design. After an all-out effort by the tank m
anufacturers, we replaced the tank flanges in LM-1 with an interim improved design which was compatible with the existing tank side of the joint.

  Bischoff’s team also developed an improved dual O-ring flange design for pipe-mounted components. We used this on components that were themselves problems and required frequent replacement, such as the pressure regulators. The better-behaved components were directly brazed into the system.

  Working with Manufacturing Engineering and Grumman fluid systems engineers from Cape Kennedy, the Bischoff team went over our brazing processes thoroughly and upgraded them regarding cleanliness, dimensional tolerances, and prebrazing sizing of the tubing and braze sleeves to assure dimensional accuracy. Brazing and X-ray crews were given additional training, delivered in part by the manufacturers of the induction brazing and portable X-ray equipment. With time, the percentage of first-time acceptable and leak-tight brazes increased and the number of reheats went down.

  LM-1 was finally made leak-tight at the cape after three months of intensive effort, aided by six of Radcliffe’s best “leak fixers” on loan from White Sands. The damage to Grumman’s reputation was severe. The next spacecraft we delivered, LM-3, was pounced upon savagely when it arrived in June 1968 and immediately checked for leaks. This time we had invited the Cape Kennedy receiving inspection team up to Bethpage to join our people in the predelivery inspections and tests, so the cape people agreed that LM-3 was leak-tight when shipped. Two minor leaks were found in the cape receiving inspection and were quickly repaired. (However, LM-3 had more than one hundred other deficiencies, some of them major, including stress corrosion and wire and splice problems. Nineteen areas were selected for consideration by George Mueller’s Certification Review Board before LM-3 could be cleared for flight. The unavailability of LM-3 for flight in 1968 prompted George Low to devise an alternative lunar-orbit mission with CSM alone—Apollo 8, flown in December 1968.)3

  Propulsion and RCS leakage remained a concern throughout the duration of the LM program. Constant vigilance and retraining were required to attain leak-tight systems—any minor slip would soon be shown up by a squealing sniffer in S/CAT or at Cape Kennedy. The frequency of leaks was greatly reduced from the mortifying debacle of LM-1 or the constant problems that had bedeviled Radcliffe at White Sands, but the occasional leakage that did occur reminded us constantly of the difficult and unforgiving nature of pressurized fluid systems in space. If it leaked in space or on the Moon there would be no way to stop it or replenish the precious lost propellant.

  Ascent Engine Instability

  Compared with the familiar internal combustion engine with its rotating crankshaft, camshafts, pumps, injectors and reciprocating pistons, valves and pushrods, a rocket engine seems relatively simple. The LM engines were simpler yet, especially the ascent engine, which sought extremely high reliability by straightforward design and rugged construction. Designed to perform a single continuous burn for about seven minutes at constant thirty-five hundred pounds of thrust, the ascent engine had neither pumps, igniters, gimbals, nor a fuel-cooled nozzle bell. It was simpler than the common oil burners used for home heating. Standing four feet high and thirty-one inches in diameter at the nozzle exit, it did not even look formidable, despite the typical rocket engine profile of cylindrical combustion chamber and bell shaped nozzle. Yet this innocuous-looking device proved to be one of the greatest threats to the Apollo program schedule. Frequently the LM ascent engine made the notorious “show stoppers” list as a problem that could stop the enormous, nationwide Apollo program dead in its tracks. When that happened, Mueller, Gen. Sam Phillips, and other NASA leaders applied pressure to Grumman that made life miserable for Lew Evans, George Titterton, and Joe Gavin, and they shared their unhappiness down the chain of command. As hard as we all pressed, ascent engine combustion instability was a chronic problem that yielded only slowly and grudgingly to trial-and-error solutions.

  Combustion instability is the most fundamental technical problem in a liquid-propellant rocket engine. Rocket engine combustion requires that the fuel and oxidizer propellants flow into the combustion chamber at a constant rate; be thoroughly mixed in a fixed ratio of fuel to oxidizer, depending upon the particular propellant combination, and be maintained in a confined volume upstream of the nozzle throat (smallest cross-sectional area) for a sufficient time for the chemical reactions of combustion to occur. This is typically done by forcing the propellants under pressure through an injector, a metal plate with many precisely dimensioned holes (orifices) that meter the flow to achieve the desired rate, and direct the streams of fuel and oxidizer to impinge upon one another at a fixed point away from the surface of the injector. The injector resembles a high-flow showerhead—water is typically used in the first checks of the injector’s flow rate and stream impingement patterns.

  Combustion instability can occur because the rate of energy release in the combustion in the chamber is extremely high, and many physical and chemical variables interact there, each capable of influencing the others. Combustion chemistry, chamber pressure and temperature, injector flow patterns, propellant flow rates, chemical and acoustic energy can all perturb one another in the chamber’s roaring inferno. Geometrical tolerances in the injector, bubbles in the propellant, variations in propellant supply pressure, and acoustic and thermal shocks at startup can further act to initiate or sustain instability. Acoustic pressure pulses bouncing back from the chamber walls can bend some of the liquid streaming from the injector orifices, causing more or less rapid combustion and generating new waves of pressure. Instability results in large amplitude oscillations in chamber pressure and heat transfer into the chamber and nozzle walls, which can increase uncontrollably until the engine explodes or ruptures. Typically instability does not occur every time a rocket engine is fired—it’s a statistical phenomenon, with instability failures occurring on average once every X number of starts.

  Combustion instability has been a problem in rocketry since von Braun and his people encountered it at Peenemünde during World War II. It was usually fixed by making ad hoc changes to the injector and the combustion-chamber geometry, changes arrived at by “cut and try” on each new engine design, an approach that did not provide any reliable rules for the next design. When it was encountered on the mighty F-1 engine for the Saturn S-1C first stage, which stood fifteen feet tall and produced 1.5 million pounds of thrust, it received unprecedented attention. An F-1 engine destroyed itself due to combustion instability in a test stand at Edwards Air Force Base on 28 June 1962, setting off a feverish round of design changes and tests led by Jerry Thomson of NASA Marshall and Paul Castenholtz of Rocketdyne, the F-1’s builder.

  At first the effort was frustratingly slow, partly because the random nature of the instability did not produce the problem often enough or in any predictable manner. When it did occur, the instability destroyed a huge F-1 engine—an expensive way to learn you still have the problem. After losing two more engines in early 1963, Thomson and Castenholtz devised a technique of exploding a small bomb (like a blasting cap) inside the chamber of a firing engine and observing how quickly the pressure oscillations triggered by the bomb damped out. Arbitrarily they decided the engine would be considered stable if the oscillations damped out within 400 milliseconds, that is, .4 seconds. The bomb test technique allowed the engineers, not nature and statistics, to initiate instability on the test stand. They were ready to shut the engine down quickly if the oscillations diverged rather than damping out, thus avoiding the loss of an engine in their tests.

  The heroic efforts of Thomson and Castenholtz in slaying the dragon of instability on the F-1 is another story well worth reading.4 They did not succeed until early 1965, when the F-1 engine was qualified after more than two and a half years of struggling with this intractable technical problem. In their wake they left the Apollo program management extremely sensitive to the gravity of rocket engine instability, and they also left a well-developed bomb test technique.

  Although we and our
ascent engine subcontractor, Bell Aerosystems, were aware of instability problems on the F-1 and other contemporary engines, we thought our engine’s small size and simple injector pattern might render it immune to that affliction. The initial test firings at Bell in 1963 and early 1964 went well, until NASA realized that Grumman had not imposed bomb stability test requirements on Bell. The ascent engine was derived from the successful Agena engine, designed for the air force’s unmanned spy satellite program, for which there were no bomb stability requirements. NASA said this was not good enough for a manned spacecraft engine and we somewhat shamefacedly agreed—it had been an oversight on Grumman’s part.

  The first bomb stability tests in mid–1964 showed a problem—the combustion chamber pressure oscillations triggered by the bomb did not damp out. They did not diverge either, just continued at constant amplitude for the duration of the firing. The bomb did not always cause undamped oscillations, and even when it did the engine seemed to be no worse for the wear after the test. Manning Dandridge, our LM Propulsion Section head, and Dave Feld, ascent engine program manager at Bell, were puzzled by this uncommon engine behavior, and they consulted widely with combustion instability experts in NASA, the air force, and industry. No one had seen exactly this instability signature before (steady, undamped oscillations), but all agreed it was unacceptable and must be eliminated.

  For the next two years Bell tried every instability cure that they, we, or NASA could think of, to no avail. Dandridge’s usual cheerful optimism wore thin; he grew increasingly anxious and frustrated. At first supremely confident that the next baffle design or injector spray pattern modification would solve the problem, he spent long hours with his engineers working up additional variations to test. Dandridge trusted in objective analyses to solve engineering problems, yet the complexity and number of variables associated with combustion instability thwarted all efforts at mathematical modeling of the phenomenon.