4. Strive for simplicity and ample design safety margins.
This guideline became the principal line of defense for systems, such as propulsion, and elements, such as structure and landing gear, that either could not be made redundant or would gain no reliability benefit from redundancy. NASA imposed a program-wide set of structural design safety factors: 1.1 times maximum predicted applied stress before yield of the material, and 1.5 times before failure. These provided adequate margins while recognizing the need for “just good enough” designs to achieve spacecraft weight goals.
5. Test extensively and exhaustively in various environments and stress levels, including stress to failure. Document all failures and investigate until the specific cause is found and design, manufacturing, or operational corrections have been made.
A particularly useful test was acceptance-vibration testing of systems and components, which tended to disclose both design and manufacturing defects that could be corrected. Joe Gavin led a crusade to refine the design and improve reliability by relentlessly tracking down and correcting the cause of test failures. Gavin proclaimed throughout the program, “There are no random failures; every test failure has a specific cause that must be found and corrected.”6
We developed these reliability approaches and applied them to LM with NASA’s advice and approval at every step. One obvious result was that the number of LM components grew dramatically, accompanied by a major increase in weight. In January 1964 NASA approved increasing the LM control weight to 29,500 pounds (fully loaded, without crew).7 We agreed to try to achieve a target weight of 25,000 pounds, but the propellant tanks were resized for the control weight. Weight control became more important as the design moved from sketches to drawings to hardware, until in 1965 it became my primary concentration.
An important program-wide NASA decision in mid-1963 greatly simplified the spacecraft’s design and lunar mission planning. The competitions for the Apollo spacecraft and for LM both specified in-flight maintenance (IFM) and repair. Built-in test circuits would detect failed replaceable assemblies or components, which would be carried as spares inside the CM and LM crew compartments, and be manually replaced by the flight crews as needed. Although we dutifully complied with this approach in our LM proposal and delegated much of its analysis and implementation to RCA, I never liked it, and once we won the LM contract I tried to change it.
I was convinced that in-flight maintenance would degrade reliability instead of improving it, for many reasons. For one thing, the built-in test circuitry itself was complex and required adding sensors or test ports at critical system locations, which themselves became additional potential failure points. The connectors or mechanical attachments that were required to make the components removable in flight were less reliable than the alternate designs of fixed attachments of hard-mounted components that were only replaced by skilled technicians in a factory clean-room environment. In-flight maintenance made the wiring harness and electrical connectors more susceptible to short circuits and corrosion from humidity and liquid spills in the cabin because protective techniques such as hermetic sealing and connector “potting” (sealing with waterproof hardening putty) probably could not be used. If broadly applied, IFM would require most of the electronic equipment to be located in accessible areas within the crew compartment, increasing its size and internal heat load. Because the spare components and assemblies would have to be stored within the crew compartments, they would be exposed to the humid internal environment. The number of spares of each type would have to be estimated by failure rate analyses that would not be perfect, resulting in payload wasted carrying unused spares that could be more effectively applied to providing redundancy in the basic design. My list of objections was long and, I thought, convincing. Owen Maynard and his NASA LM engineers became as determined as I to eliminate IFM.
Other powerful voices within NASA also began attacking IFM. Houston Flight Operations director Christopher Kraft argued that the crew simply would not have time to repair faulty hardware during LM operations. When George Mueller took over as Manned Space Flight chief in Washington in September 1963, he also had reservations about it. Shortly thereafter IFM was deleted from the entire Apollo spacecraft. Instead the crew would rely on operational displays, the caution and warning system, and ground-based support from the Mission Operations Center in Houston to detect malfunctions. Switchable redundancy would be “wired in,” and all electronics inside the cabin would be hermetically sealed or potted to protect against moisture and contaminants.8 This encouraged us as designers to locate as much electronics as possible outside of the crew compartment, making it smaller and more flexible in accommodating lunar surface mission requirements. I believe this sound NASA decision contributed to Apollo’s success.
Project Christmas Present
In the fall of 1963 North American invited Grumman and MIT to join a task force at Downey devoted to establishing an integrated set of Apollo program schedules. The schedules they had prepared in October 1962 had been rendered meaningless by subsequent delays, and as spacecraft integrator they could not properly function without detailed schedule goals. The task force generated the Apollo spacecraft development test plan (ASDTP), the first comprehensive set of subsystem, ground-test, and flight-test schedules linking the CSM, LM, and GNC system with one another and with the Saturn booster. The initial draft of the plan was submitted by the contractors to Houston just before Christmas and was dubbed Project Christmas Present by the task force.
The Grumman contingent at Downey was led by Reynold “Ren” Witte and Theodore “Ted” Moorman. Both were experienced test engineers; Witte’s background was in ground testing from the Structural Test Group in Engineering and Moorman’s in aircraft flight testing with the Flight Test Department. Both were well-organized and effective leaders. They tapped into the test engineering corporate memory at Bethpage and directed the ten to twenty Grumman engineers on site who were temporarily assigned to support them. Witte and Moorman had led the LM development test negotiations with NASA after contract award and were thoroughly familiar with the intricate interrelations and prerequisites between critical test milestones on the LM subsystems and in LM’s flight-development program. In the ASDTP exercise they explored and established the constraints that LM development milestones imposed on the CSM, the GNC system and the Saturn, and vice versa. The ASDTP task force was a most beneficial activity for all the Apollo contractors.
The LM program in Project Christmas Present included ten flight LMs, the first two of which were unmanned, and six LM test articles for ground tests, as follows: LTA-2, launch vibration tests at Huntsville; LTA-10, SLA fit and mass model tests at North American-Tulsa; LTA-1, “house spacecraft” at Bethpage for electronics tests and support of fabrication, assembly, and checkout; LTA-8, thermal vacuum tests at Houston; and LTA-3 and LTA-5, structural and vibration tests at Bethpage. The ground-test program also contained boiler-plate and flight-weight propulsion test rigs for fluids tests in the cold flow facility at Bethpage and hot rocket firings at White Sands. The first unmanned LM flight was scheduled for late 1967, and the ground test articles were planned for use during 1966 and 1967.
Grumman Leads Mission Planning
As our LM designs took shape we encountered questions on LM requirements that could only be answered by a better understanding of how the lunar landing mission would be conducted. The design of the crew compartment, for example, was greatly influenced by the crew’s activities on the lunar surface. When would the spacesuits be worn? How many times would they be doffed and donned in the LM cabin, and how much volume would this require? What about the return of lunar samples; size, weight, contents? One set of questions generated others. Determining the required capacity and duty cycle of the EPS called for detailed knowledge of the mission time line; accounting hour-by-hour for what mission activities were in progress and what equipment was turned on. The same was true of thermal analysis, determining the heat loads used to size the ECS. Comm
unications system requirements, duty cycles, and antenna positioning and usage could not be finalized without a detailed mission plan, nor could any of the LM systems. I realized that we must have a mission baseline for our design.
Rathke and I proclaimed the need for a design reference mission (DRM) to give all Apollo contractors a basis for finalizing their system design requirements. We asked Tom Barnes to take our existing lunar-orbit rendezvous studies and expand their mission definition. We also began talking up the idea informally at Apollo program meetings with NASA, NAA, and MIT.
In September 1963, shortly after the approval of a definitive contract with North American Aviation, Joseph F. Shea replaced Charles Frick as Apollo spacecraft program manager in Houston.9 Joe Shea was brought into NASA’s newly formed Office of Manned Space Flight in November 1961 from Space Technology Laboratories by NASA’s administrator, James Webb, and D. Brainerd Holmes, associate administrator for Manned Space Flight.10 Known from the Titan and Minuteman ballistic missile programs as a brilliant aerospace systems engineer, he was made Holmes’s deputy and given the challenge of evaluating the competing lunar mission modes. Shea was an articulate man of overpowering intellect, a skilled debater, persuasive in argument and a powerful program leader. He was tall and handsome, with an athletic build and dark Irish good looks—jutting jaw, fair complexion, prominent black eyebrows, and black crew-cut hair. He was dangerously dynamic, so likely to prevail in most arguments that the entire Apollo program depended upon his wisdom and good judgment. Although he could witheringly destroy an opponent’s arguments as capably as any trial lawyer, he also had a wonderful sense of humor and enlivened many a meeting with his witty, inventive puns.
When I mentioned to Shea our need for better definition of the lunar mission to pin down LM design requirements, he threw the ball back to me by recommending that Grumman lead a mission study with participation by the other Apollo contractors. Thus was born the Apollo Mission Planning Task Force (AMPTF) in January 1964. With Barnes in charge, the AMPTF set up shop in one of the large Apollo conference rooms in Plant 25 and was joined by team members from NAA, MIT, and NASA-Houston. Tom Barnes was a great team leader. Friendly, constructive, and totally without ego or institutional bias, he inspired confidence and cooperation from the entire task force. Barnes was a talented systems engineer who explored problems relentlessly, asking key “what if” questions that sometimes led to new ways of defining or resolving things. He did it in such an easygoing but provocative manner that others were stimulated to new insights and contributions.
The task force started by defining the basic mission objectives: “Land two astronauts and scientific equipment on the near-Earth-side surface of the Moon and return them safely to Earth.” A second objective was to carry at least 250 pounds of scientific equipment to be set up on the Moon and to bring back 100 pounds of lunar soil and rocks.11 The AMPTF created a detailed description and analysis of all flight mission activities from liftoff to splashdown and recovery—the DRM. They also investigated possible failure modes and contingencies to determine their effect on mission planning and on spacecraft design requirements.
Four months of intensive mission planning and analysis took place with dozens of engineers from NASA and the contractors participating. To make possible precise launch and trajectory calculations using the exact relative positions of Earth, Moon, and spacecraft and figuring the rocket firings necessary to execute mission maneuvers and flight path corrections, it was necessary to choose a specific date for the DRM. The team selected 6 May 1968 for liftoff. Using the same minute-by-minute crew time line planning technique that had been developed on Projects Mercury and Gemini, the AMPTF extended the detailed flight plan to cover three astronauts and two spacecraft (CSM and LM) that for part of the mission functioned independently. Actions required of the crew, the spacecraft, and the ground network to perform the mission were documented in the DRM time line, and trajectory calculations and error analyses were performed to establish system performance and accuracy requirements. The result was the most complete prelaunch mission planning yet attempted for Apollo, providing a good basis for further development of design requirements, operational ground rules, and mission plans.
The DRM clarified the docking requirements for both the CSM and LM. Initial docking and extraction of the LM from the SLA would be carried out by the crew from their couches in the CM. The resulting connection would provide a rigid pressurized tunnel permitting crew access between both spacecraft. Upon return from the Moon, the rendezvous maneuver would be performed by the two-man LM crew, with CM-active backup available, but the docking would be done by the lone crewman in the CM in the same manner as the initial docking. Only the upper LM hatch was required for docking, whereas the forward LM hatch was needed for lunar surface egress.
The DRM became a treasure trove of information for contractor engineers seeking firm requirements to which to design their systems and components. At Grumman we set up a formal process to tabulate the requirements specified or implied by the DRM and compare them with the design specifications for the LM spacecraft and its systems and components; correcting the design specs where necessary to bring them into conformance. This assured that the spacecraft we were designing and subcontracting would be able to perform its overall intended function of lunar landing and return.
The planners looked for failure modes at each step of the mission and sketched out recovery plans where possible. They also determined the accuracy required in critical mission phases. For example, the midcourse trajectory corrections on the way to the Moon had to be accurate within three or four feet per second or else the spacecraft would crash into the surface. Upon reentry the CM must hit the outer edge of the Earth’s atmosphere within a flight path angle “window” of two degrees. Too steep an angle would result in a rapid plunge into the atmosphere, burning up the spacecraft like a meteor, while too shallow an approach angle would skip the CM off the top of the atmosphere and send it on an eternal orbit of the Sun.
One major result of the AMPTF contingency planning was the identification of the “LM Lifeboat” mission. While postulating the effect of various CSM failures on the outbound leg of the mission, the planners realized that a number of them could be countered by using the LM as a lifeboat and utilizing its propulsion, guidance and control, life-support, and other systems to return the crew to the vicinity of the Earth’s atmosphere for reentry in the CSM. To provide this rescue capability, some of the LM consumables, such as oxygen, water, and electrical power, would have to be increased by 10 to 15 percent above that needed to perform the basic mission. Because LM then existed only on paper, we decided to make the tanks that much larger. At a later date it could be decided whether to actually load the additional consumables into them. Six years after it first appeared in the AMPTF’s report, this vital crew rescue mode was dramatically utilized on Apollo 13.
Loose Cannon
As we performed comparative analyses on the LM systems we came to doubt the reliability estimates for the MIT guidance, navigation, and control system that was provided to both the LM and the CM. We arrived at this conclusion while preparing our own estimates of the reliability of the backup abort guidance system, for which Grumman was responsible. This led us to challenge MIT’s GNC reliability estimates, with disastrous results for Grumman.
Our GNC experts came to believe, and they convinced me, that MIT’s GNC had a factor of one hundred lower reliability than MIT claimed, an opinion based mainly upon Grumman’s interpretation of summary mean-time-between-failure (MTBF) data for guidance system components on the Polaris, Titan, and Minuteman ballistic missile programs, as published in a GE report. It was instigated by a former Honeywell reliability engineer working for Grumman who may have had a hidden agenda. (MIT had beaten Honeywell in the competition for the Apollo GNC and said engineer was on the Honeywell proposal team.) After Grumman published its conclusions in a report to the Apollo program office, MIT and NASA hit the roof.
Joe Shea
convened a meeting of all interested parties in early January 1964 to find the truth and punish the guilty. We gathered in the well-appointed Apollo program conference room in Houston. About thirty NASA, MIT, air force, and Bellcom GNC experts attended, led by Jim Elms, representing Bob Gilruth, MSC director, and Joe Shea, Apollo spacecraft program manager. Gavin, Rathke, Whitaker, and I sat opposite this stone-faced group. Shea warned everyone that the meeting was being tape-recorded.
Elms welcomed everyone and said the meeting would be a technical discussion to resolve questions concerning Apollo guidance system reliability. He turned the meeting over to Shea. Shea glowered under his black eyebrows and hunched forward toward his microphone, his hands knotted tightly together on the desk, his gold MIT ring visible:
Gentlemen, we have a serious problem. The problem is that Grumman believes the MIT guidance system is two orders of magnitude inferior to other available systems, and that the Apollo program is being jeopardized by this choice. The issue centers upon the evaluation of the data used in drawing this conclusion and establishing its validity.
I intend to force a black-and-white conclusion as a result of this meeting. Either there is or there is not a significant basic difference in the inherent reliability of the MIT system and other comparative data. Someone, Grumman or MIT, will have to leave this meeting admitting he was wrong—mea culpa, mea maxima culpa.
Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight) Page 11