All too often we in S/CAT would goof or blunder all by ourselves, and no amount of capable LM program leadership could save us from ridicule and upper management or NASA intervention. One of the worst foul-ups occurred on LTA-8, that devilishly complex thermal test article. At one point in the testing we were required to drain and flush the water-glycol coolant that circulated throughout both LM stages, cooling electrical equipment and components and the astronauts’ suits and oxygen supply. The test team mounted a fifty-five-gallon drum on the highest level of the workstand, connected by clear plastic tubing to various points of the LM’s cooling circuit. Unthinkingly they used an old drum with a rusted bottom that dripped water-glycol solution all over the spacecraft, requiring several days of painstaking swabbing and cleaning with an acid neutralizing agent to set right. At Titterton’s meeting I hung my head in apology and endured a bitter, sarcastic tongue-lashing.
The cockpit instruments in the LM had hermetically sealed cases to protect them from humidity and dirt. After we had installed the instruments in three cockpits, a clever NASA inspector at Bethpage devised a new technique that was simple but effective in disclosing dirt particles trapped within the sealed case. He simply held the instrument on the workbench, glass face down, and shook and tapped it gently. Then, retaining the glass-down orientation, he lifted it up over his head and looked for dirt particles on the inside surface of the glass. If any at all were visible, the instrument had to be returned to its supplier to have the case opened, cleaned, and resealed.
This improved inspection technique had never been thought of before, either by the instrument manufacturers or ourselves, and almost none of the instruments that had been delivered could pass it. For the manned flight spacecraft the new test was mandatory, so for LM-3, the cockpit of which was almost complete, we had to remove and replace all the instruments. We faulted ourselves for not devising this obvious quality check long beforehand.
My Chickens Come Home to Roost
My assignment to run S/CAT was a unique learning experience and an appropriate form of retribution. On the assembly floor I came face to face with the troublesome design features I had approved, and in some cases demanded, when I was project engineer, which caused untold hours of toil for the manufacturing technicians who had to make them work in the real world.
Foremost among these were the extremely thin 26-gauge kapton insulated wire and miniature electrical connectors used throughout the LM. Adopted as a weight reduction measure during SWIP in 1966, these fragile wires and tiny connectors were an endless source of problems with wire breakage and improperly mated connectors. Breakage was common in the wires, which were used wherever signal voltages were applied with essentially no current, making them more abundant than the larger-gauge wires.
On the assembly floor I often watched sympathetically as a frustrated technician demonstrated the difficulty of mating and demating a miniature connector containing dozens of fine wires, doing it by feel with gloved hands in a cramped and all-but-inaccessible space. One smiling tech with small but powerful fingers was in great demand because he could handle the most difficult locations.
We ran a special vehicle-level vibration test on LM-1 using electrically driven vibration generators. This test gave us confidence that these wires would not break when the LM was being shaken in flight during launch or from its own rocket engines. Still the wire breakage problem due to installation and handling was a constant drag on assembly and test operations, although it gradually lessened as the technicians improved their techniques for gentle handling of wire bundles. For LM-4 and subsequent vehicles we switched to a special high-strength copper alloy in the wires, which alleviated the breakage problems.
The fire-retarding and moisture-sealing requirements of potting and covering with Beta cloth booties added to the difficulty of handling electrical connectors of all sizes. Much of the potting and bootie installation could be done on the workbenches before the wire harness assemblies were installed into the vehicle, but there were some areas where the wire bundle was threaded through structure and a connector had to be placed on one end inside the spacecraft. The potting material took several hours to cure under heat lamps, after which portable X-ray equipment was used to verify that the wires inside were all properly routed to the connector pins or sockets. A klaxon horn brayed a warning on the floor before the X-ray was turned on—all personnel had to leave the immediate area. Some nights as I lay in bed that klaxon was still reverberating in my brain. If wires had to be replaced in a connector for any reason, they were physically cut out of the potting, the new wires, pins, or sockets were slipped into place, and the connector was repotted and X-rayed again, a cycle that took at least one whole shift. With many thousands of wires and hundreds of connectors in the LM, this was not an uncommon occurrence.
Even the basic aluminum alloy structure of the LM imposed exacting demands upon the manufacturing process. In our relentless quest for weight reduction, we engineers had made widespread use of the high-strength alloy 7075 and of chemical milling, which enhanced susceptibility to stress corrosion. Controlling stress corrosion required carefully fitting each part upon assembly to avoid clamp-up stresses when the fasteners were tightened. This involved educating our mechanical technicians and inspectors to the causes of stress corrosion and the most common fit-up problems of LM parts, and then enforcing rigorous compliance with the approved fit-up procedures. Even so we had numerous cases in which cracks were discovered in thin aluminum flanges that had retained excessive clamp-up stresses when installed. The astronauts were especially vocal in urging us to stamp out this problem, as they no doubt pictured their return vehicle crumbling beneath them on the Moon, victim of its own locked-in stresses.
Delivering LM-1
By early June 1967, despite all these problems, we were approaching the delivery of LM-1 to KSC. As the first flight LM with many special provisions for unmanned flight, the birth pangs of LM-1 were long and painful. The LM mission programmer (LMP), a combination of computer, electrical relays, and switching racks, was the programmable robotic brain that would perform the mission automatically under command of the MIT-designed Apollo computer (primary mode) or the LMP itself (backup mode). This complex, special-purpose unit required extensive qualification testing and design modifications before we began to have confidence in it. Even more troublesome was the development flight instrumentation, a secondary instrumentation system with pressure, temperature, stress, and vibration sensors and its own separate telemetry and recorders. DFI was intended to provide additional engineering data on the early LM flights that could be used to correct and refine the design prior to the first lunar landing; it was only installed on LM-1 to LM-3. Since it was not essential to mission operations, DFI was not qualified to the same high parts selection and reliability standards as the Instrumentation system used for mission critical measurements, but was more akin to experimental aircraft test instrumentation. In mid-February 1967 I noted that 56 out of 320 total DFI measurements were down, with little improvement over recent weeks despite extensive trouble shooting, transducer (sensor) replacement, and wire repair.1 Many of the DFI measurements were considered mandatory for LM-1’s mission of verifying LM systems performance in space.
To cure the DFI problems I got Gene Goltz, head of the Instrumentation Department, to lead a “tiger team” of instrumentation specialists from both the LM program and Flight Test in a complete item-by-item review of the DFI system. They considered component and system design, test results, supplier quality performance, and environmental requirements of the system. Goltz’s team recommended strengthening some DFI requirements, particularly the use of vibration testing as a quality acceptance test screening of transducers, modems, and other critical components. They recommended that we drop some suppliers whose components were not performing reliably. Although it took some time, the tiger team’s output resulted in acceptable levels of DFI reliability.
Aside from all the problems with LM-1 itself, everything e
lse in the test and checkout operation was simultaneously being developed and tried out for the first time. The ground-support equipment—the cable sets, connectors, adapters, fluid servicing carts, and propellant-loading equipment that connected LM-1 to the computer-controlled automated checkout equipment—also was being designed, built, tested, and evaluated by spacecraft checkout. Hundreds of GSE end items were required, each of which had to be qualified as a deliverable item to the government. Howard Peck, an assistant project engineer, led a team of several hundred engineers and technicians in the development of LM GSE. Even the ACE computers and consoles, developed and maintained by General Electric, were regularly modified and improved as experience was gained in actual test operations.
The test procedures specified each step of test operations, no matter how trivial or routine, and required constant rewriting and improvement to be usable under actual test conditions. It was not unusual at this stage for a test procedure to be rewritten three or four times before a document emerged that could actually be used to conduct the test. With continual interruptions for spacecraft and GSE troubleshooting, repairs, and modifications, test procedure rewrites, and ACE hardware and software upgrades, a nominal six-hour test could take weeks to complete.
That LM-1 advanced through its scheduled sequence of testing at all under such conditions was a tribute to the persistence and drive of the LM-1 test team, and particularly its leader, spacecraft team manager Jim Harrington. Jim was a short, freckle-faced Irishman with a brown cowlick and a bouncy, confident manner. He endeared himself to me during a high-level meeting with NASA’s top leadership. George Mueller had asked to meet Harrington, as he was concerned about LM-1’s inability to hold schedule. While questioning him in front of the assembled NASA and Grumman top brass, including our president, Lew Evans, Mueller ventured an opinion that LM-1 was having so much trouble getting through its test and checkout phase that it could never be relied upon in flight, and that perhaps it should be converted to a “hangar queen” while LM-2 became the first flight vehicle.
“That’s a fine theory, Dr. Mueller,” said Jim, staring unblinkingly at him while flashing a beatific smile, “but you’re absolutely wrong.”
Harrington then outlined the various causes for LM-1 delays, many of which did not even involve the spacecraft itself but would have affected whatever LM was the first to undergo rigorous test and checkout. He ended up convincing Mueller and the other NASA officials that when it was delivered, LM-1 would be a reliable, high-quality product.
Jim’s capable leadership and unfailing good humor carried LM-1 forward on the assembly floor too, and by late May it had completed the final and comprehensive all-systems test, the redoubtable OCP 61018. The ascent and descent stages were then demated and moved for pressurized fluid systems testing in the reinforced concrete bunker cold flow facility, located across the parking lot behind the Plant 5 LM assembly hangar, and we scheduled the formal customer acceptance readiness review with NASA for 21 June.
The CARR was such a tragicomic circus that I recounted it at length in chapter 1. Suffice to say here that we did, after a long, difficult day of briefings and “chit” dispositions, receive approval from NASA’s Apollo Spacecraft Program Office to ship LM-1 to KSC, subject to cleaning up a long “crab list” of questions, documentation, and minor retests. On 22 June 1967, I stood on the Grumman Bethpage runway in a stiff wind and bright sunshine and watched as the specially designed shipping containers holding LM-1’s ascent and descent stages were loaded into the bloated belly of the “Guppy” aircraft, a modified Boeing Stratocruiser used to transport the outsized Apollo spacecraft. With great relief I watched the huge airplane slowly lift off and climb skyward, using most of the six-thousand-foot runway. It was bound for KSC—our first flight spacecraft delivery.
Our relief was short lived. LM-1, scornfully derided as “junk, garbage” by NASA’s Petrone, was promptly rejected by receiving inspection at KSC due to plumbing leaks, triggering Skurla’s demands for action. A concentrated leak-fix effort led by Will Bischoff, our subsystem engineer for Structures and Mechanical Systems, was successful, but the damage was done. LM-1 had given Grumman a bad name for quality at KSC, a bad name from which we would be slow to recover.
A week after the LM-1 CARR, NASA and Grumman held a CARR on LTA-8. This was much smaller than the LM-1 extravaganza, and it was held in the austere but acoustically adequate conference room in Plant 25. Titterton did not attend, nor did Low; the NASA and Grumman delegations were chaired by Gilruth and Joe Gavin, respectively. This CARR did not result in approval to ship LTA-8 to the Manned Spacecraft Center but generated a long series of action items and discrepancy reports to be worked off. Some of these required lengthy rework in the shop, such as an action to replace all the cockpit instrument panels with another set in which the wiring had the latest fire retardant potting, booties and overcoating. The all-systems test, OCP 61018, had to be repeated after the new panels were installed. The manned LTA-8, in addition to requiring all the changes resulting from the Apollo 1 fire, also had an extensive DFI installation, which had at least as many problems as LM-1’s version of DFI.
Tom Kelly (left) and Dick McLaughlin at Grumman-Bethpage for delivery of the LM-1. (Courtesy Northrop/Grumman Corporation) (Illustration credit 12.3)
An Uphill Climb
Gradual improvement in S/CAT’s performance took place over the summer, as we worked simultaneously on LTA-8, LM-2 to LM-5, and M-6. The latter was a steel boiler-plate mockup of the LM crew compartment completely outfitted inside with flight-type plumbing, wiring, and equipment, in which ignition tests would be conducted in 5 psia pure oxygen to verify that our cabin was safely fire resistant. In September we completed the action items and DR closeouts for LTA-8 and delivered that unique, nonstandard vehicle to MSC for installation in the large thermal vacuum chamber, where it would perform an extensive, manned test program that was rife with uncommon test conditions and potential safety issues. At least it was out of Bethpage and we could concentrate on building and testing flight LMs!
Another step forward was the official designation of Spacecraft Assembly and Test as a major operational organization within the LM program at Grumman, replacing the temporary and rather vague command-post structure. President Lew Evans kicked off the new S/CAT organization with a rousing speech to the program and the corporate department heads, during which he also formalized the position of the spacecraft directors as leaders of the assembly and test teams for each LM. The spacecraft directors were all highly experienced test engineers, mostly from the Flight Test Department, and one of them, LM-3 director Tom Attridge, was a senior experimental test pilot. Evans challenged the new organization to rapidly improve its ability to increase quality and maintain schedules. I was given carte blanche to add up to fifty test engineers to S/CAT, either by recruiting internally from Engineering and Flight Test or by external hiring. I followed through quickly on this, since I already had a list of internal prospects whom I had intended to pursue as soon as I received budget authority.
The spacecraft directors were a feisty group, accustomed to command positions and pumped up by Lew Evans’s charge to straighten out S/CAT, getting it to meet schedules and maintain quality. They worked hard to make their LMs the best of all, and developed a dedicated engineering and manufacturing team devoted to that particular vehicle.
An upturn in our fortunes was the addition to S/CAT management of Lynn Radcliffe, who three years earlier had established a lunar module rocket propulsion test facility at a new NASA site on the White Sands Missile Range in New Mexico. Radcliffe built it into an effective test facility where the LM ascent and descent propulsion systems and reaction control system were test fired under simulated high-altitude conditions, using both boiler-plate and flight-weight test rigs. Joe Gavin wisely observed that Wright and I were wearing down from the unremitting strain of endless deadlines and problems and around-the-clock operations. We needed help at our level, real help. The scale of S/CAT operations
had grown to involve hundreds of people, all being pushed hard to do quality work but also meet schedules, and personnel and human relations problems were escalating. Radcliffe’s greatest strengths lay in human relations, administration and organization, areas that sorely needed strengthening in S/CAT.
Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight) Page 26