Almost a year later another tank failure occurred, this time an LM descent propulsion supercritical helium tank. The inner pressure vessel ruptured while being pressure tested at the manufacturer, Airesearch. The failure originated at the weld, so at first we thought it was another case of a mislabeled weld rod. Henry Graf, supercritical helium system manager at Airesearch, impounded the remainder of the spool of weld rod used in the failed tank, and metallurgical examination showed it to be the proper alloy. Moreover, microscopic inspection of the failure showed some tiny intergranular cracks, typical of stress corrosion. Yet as far as anyone knew the tank had never been exposed to any fluids not approved for compatibility with titanium.
Henry Graf became obsessed with finding the cause of this failure. He led his engineering and quality staff through a minute examination of every step in the manufacturing process, starting with the receipt of the titanium forgings and the quality pedigree that accompanied them. At each step of the process, they looked at what had been done on the failed tank, and asked whether anything in this step was different from their process on previous tanks. Graf’s careful detective work paid off, discovering a cause so trivial that a less observant investigator would surely have overlooked it. Graf noticed one minor difference in the process for this tank and those that had preceded it: instead of using new cloth pads to wipe the tank surfaces prior to welding, washed, reused cloths were employed. Examination of the washed cloths showed traces of detergent, and test samples that were wiped with them failed under combined stress and humidity testing. The trace detergent attacked titanium! There could be no more gripping example of the extreme sensitivity of highly stressed tank material and welds to contamination.
In September 1968 a descent propellant tank destined for LM-9 (Apollo 15) was found to have a cracked weld after proof test at the manufacturer, Airite Division of Sargent Industries. The test had been conducted with the tank full of distilled water at room temperature. Airite was able to verify that the proper weld rod had been used and that all fabrication processes and procedures were complied with. However, some impurities were found in the water used for the test. Frank Drum spent time with Airite in California and compared Airite’s welding procedures to those of our other titanium tank suppliers, Aerojet and Airesearch. He considered Airite’s procedures to be less conservative than the others in some key process steps, such as tack welding. Aerojet always did tack welding by machine inside a vacuum welding chamber, while Airite did the tack welds by hand in the factory environment before putting the tank into the chamber to complete the full continuous weld by machine.
At our insistence Airite upgraded their procedures to include the best practices of our other suppliers, and consulted with them on the “tricks of the trade” involved in making them work well. With guidance from NASA and Grumman, they also redesigned the butt weld joint to a “J-groove” design, based upon improved strength of this design in coupon samples. Airite further tightened their already strict manufacturing quality controls and improved cleanliness and housekeeping in their shops.
As part of the SWIP weight reduction effort, Grumman had prepared a lightweight descent propellant tank design and put it out for competitive bids. Airite won the competition, replacing the original descent tank supplier, Allison Division of General Motors. Allison tanks were in LMs up to the LM-5 (Apollo 11) and Airite tanks were installed in LM-6 and LM-7 when the failure occurred. LM-8’s tanks had been delivered but not installed. At Grant Hedrick’s suggestion, we began a cryoproof test program for the Airite descent tanks, to see if we could verify that sufficient weld safety margin existed beyond the proof pressure level to let us clear the LM-6 and LM-7 tanks without removing them. Cryoproofing consisted of testing the tank to proof pressure (one and a half times normal operating pressure) while filled with liquid nitrogen at minus three hundred degrees Fahrenheit. Welds were more brittle at this low temperature, so this was a more severe test of the fracture toughness of the weld.
An extensive fracture toughness test program was conducted on coupon samples cut from the failed tank, to show that tanks which passed cryoproofing had ample margin for all mission stresses. Samples were tested at room temperature and in liquid nitrogen, with water and with freon (which was being considered as an alternate proof test fluid), and with and without machined slots, simulating preexisting cracks or flaws.
As a result of this thorough investigation, in which materials, welding, and fracture mechanics experts from NASA-Houston and Marshall, North American, and all the tank suppliers in the program consulted or participated, we developed an acceptable lightweight tank manufactured at Airite. However, we replaced the tanks in LM-6 and LM-7 with heavier Allison tanks and installed them also into LM-8. The Airite SWIP tanks, which were also made longer to contain the added propellant required to accommodate the LM weight increase for the extended duration missions, went into LM-10 to LM-13 (Apollos 15 to 18).9 These tanks were cryoproofed before delivery.10
I consulted closely with Grumman’s chief technical engineer, Grant Hedrick, on these tank problems, and found him a wellspring of good advice and wise judgment. In the Airite tank problem in particular, Hedrick called all the shots. Frank Drum and his Materials Section, with their expertise in materials properties, inspection techniques, and metallurgy, were also very helpful. Throughout our tank problems, we worked closely with Joseph Kotanchik, branch chief of Structures and Mechanics at NASA-Houston, and his very capable group. They made the full facilities and expertise of NASA available to us and provided valuable suggestions. At one point during the Airite tank weld fracture toughness test program, all ten of NASA-Houston’s fatigue test machines were loaded with our coupon test specimens.
We were nervous enough about the LM tanks that we sought advanced test and inspection techniques to further verify their quality, as in our cryoproofing program to recover from the Airite tank failure. The routine inspections and tests that we required on all tanks included weld X-ray and dye penetrant (Zy-glo) inspections and proof pressure test with water at ambient temperature. The supercritical helium tank was also cryoproof pressure tested at its extremely low (cryogenic) operating temperature of –450 degrees Fahrenheit.
We experimented with installing a network of acoustic sensors on the ascent propellant tanks during proof pressure tests. Pinging sounds were emitted as the tank material stretched under pressure, and the system could triangulate with multiple sensors to locate the point from which the sound originated. Dye penetrant inspections were performed at these points of origination to see if incipient flaws could be detected. Results were inconclusive and no flaws were detected, so we abandoned this approach.
Like most LM components, the tanks were extremely sensitive to the slightest mistake at any point in their fabrication, assembly and test process. This meant that they were totally dependant upon the skill, craftsmanship, and integrity of their builders, as was the whole LM and all its parts. A prime example of a tank builder’s integrity and concern was shown by Bruce Baird, ascent propellant tank manager for Aerojet General at Downey, California. Baird, a short man with a serious but youthful face, came into my office one day and declared he had a problem. He showed me color photos of a lunar module ascent tank that had recently completed acceptance tests at Downey and was ready for delivery. The problem was that, in the final heat treat, this tank had turned a dark purplish color, whereas all other ascent tanks were colored light straw after heat treat. Something different had been present in the atmosphere of the heat treat furnace for this tank, and Baird did not know what it was. Although the tank had passed its proof pressure test, he feared that it might have been damaged in some way that would reveal itself later, perhaps in flight.
Hedrick and our other experts went out to Downey to see the suspect tank, and examined its records and the whole Aerojet tank fabrication and test process. They concluded that the tank was probably all right, but since the discoloration could not be explained and Baird felt so strongly about it, they
agreed not to accept this tank for flight but to use it for the ultimate test-to-failure demonstration in the tank qualification test program. In the test to failure the purple tank exceeded its design ultimate pressure, so there had been nothing wrong with it after all. Nonetheless, we admired Baird for his dedication to quality and the success of the Apollo mission. LM never had a tank failure or leak in flight, thanks to Graf, Baird, and many others like them at Grumman and our suppliers.
10
Schedule and Cost Pressures
In those heady days at the beginning of Project Apollo, cost was never mentioned, except when NASA questioned whether we had enough money to do what was necessary. We were free to plan, design, expand staff and facilities, and do whatever it took to get the program moving. Congress had almost unanimously approved President Kennedy’s goal of landing men on the Moon by the end of the 1960s, despite its uncertain price tag.1 The initial program estimate was $8 billion, but it was commonly expected to cost $20 billion. Congress funded all federal expenditures yearly, so even a long-duration program such as Apollo got its money one year at a time, with no guarantee of how much money, if any, would be approved the following year. Before long NASA, just like all other agencies and programs, had to fight for money in Congress. Congress began questioning and trimming Apollo’s costs in 1965, when NASA’s budget exceeded $5 billion. The rapid growth of Apollo expenditures brought charges of waste and lack of control. NASA spent $5.1 billion in fiscal year 1965 and $5.2 billion in fiscal year 1966, of which $2.5 billion in fiscal year 1965 and more than $3 billion in fiscal year 1966 was for Apollo. The NASA fiscal year 1967 budget request of $5.58 billion was cut to $5,012 billion by President Johnson and further trimmed to $4,968 billion by Congress. Apollo’s budget escaped unscathed, but the follow-on Apollo applications program was all but deleted.2
The LM program at Grumman directly felt this budgetary belt tightening. Grumman’s expenditures had risen dramatically, from $135 million in fiscal year 1964 to $350 million in fiscal year 1966, attracting the attention of NASA management. NASA’s first move against Grumman’s increased spending fit well with a desire that had been building in Headquarters to convert the Apollo contractors to incentive contracts from cost plus fixed fee. With an incentive contract the contractor’s fee was determined by program accomplishments as well as cost and schedule performance. Both Administrator James Webb and George Mueller believed this to be necessary, not only to motivate their contractors financially to improve performance but also to show Congress that they were being hard-nosed in managing them. The Department of Defense under Secretary Robert McNamara was also moving to incentive contracts.
Incentive Contract Negotiation
In March 1965 Joe Shea kicked off a major exercise aimed at renegotiating Grumman into an incentive contract. The effort began with NASA examining the updated material we had prepared, which gave the detailed work statements, schedules, and estimated cost to completion for each contract line item. The top level LM system specification, the overall document describing the LM and its systems, and the performance and interface (P&I) specification, the top-level summary of LM’s technical performance and characteristics and its technical interactions with any other element of the Apollo-Saturn system, were also prepared for negotiation.
A large, high-ranking NASA contingent arrived in Bethpage to conduct the review and negotiations, led by Joe Shea, R. Wayne Young (who had replaced Bill Rector as the LM project officer), and Tom Markley from the Apollo Spacecraft Program Office. Joe Gavin and Bob Mullaney headed the Grumman effort. Rathke and I led the large Engineering supporting effort, which included assistant project engineers Carbee, Whitaker, and Coursen, the System and Subsystem engineers and section heads, the cognizant engineers, and any other support they required. As in the original contract negotiations in Houston, our people paired off with their counterparts into fact-finding and negotiation teams, meeting in the many conference rooms in Plants 25 and 5.
Rathke and I worked all day on the many other engineering efforts in progress: finishing the design, getting the drawings out, trimming LM weight growth, and resolving technical problems. At 5:00 P.M. we assembled the team leaders and reviewed the progress and status on each line item and work package. Often these meetings lasted until 9:00 or 10:00 at night. After two weeks of fact finding, NASA started making their recommendations.
NASA was looking for 25 to 40 percent cuts in cost and manpower. We were shocked; this certainly was nothing like the original negotiations. None of us believed we could do the job for the cost NASA wanted, and I set the team leaders off to prepare rebuttals. For the next several days we worked with the team leaders to refine their rebuttal evidence and presentations. I thought we had strong positions in every area.
After the rebuttals were made to the NASA teams, our team leaders reported that most of their NASA counterparts agreed with them, with usually minor exceptions. The total estimated cost was going up, even beyond our initial position, because of oversights and underestimates that became apparent during the detailed negotiations. Joe Shea became increasingly testy; he was gruff with us in our management interface meetings, and there were reports that he had been dressing down his own troops for failing to reduce LM costs.
After a month’s effort, Shea abruptly canceled the whole exercise. Most of the NASA delegation returned to Houston, but Young, Bill Lee, and a few others remained to tie up loose ends. NASA agreed to use the unofficial positions agreed by the team leaders in each line item and work package as the planning baseline for the LM program.
In June 1965 Bob Gilruth met with Grumman president Clint Towl to review the outcome of the Shea exercise. They agreed that they could not baseline an incentive contract yet. Gilruth then informed Towl that NASA was imposing a lunar module management plan, limiting Grumman to spend no more than $78 million for the last quarter of fiscal year 1965, an amount considerably less than Grumman’s estimate.
We worked with our subcontractors to negotiate incentive contracts with them. Our subsystem engineers, cognizant engineers, and the subcontracts managers led this effort, which continued throughout the summer. We also reviewed and reestimated the Grumman in-house work packages. In September Grumman submitted an incentive contract conversion proposal to NASA that became the basis for continuing negotiations during the fall and winter. Engineering support to LM program management was continually provided as required, but it was a “low-key” activity carried out by small teams, often over the phone. NASA’s emphasis had changed from cutting the estimates to trying to understand the most likely cost. In this improved environment, agreements came quickly at the team level and were supported by program management.
In February 1966 agreement was announced on an updated contract for the Grumman LM program containing incentives on schedule, cost, technical and mission performance. It carried the LM program through 1969 at an estimated cost of $1.42 billion. The contract also invoked a revised set of LM specifications and statement of work, which defined Grumman’s responsibilities for the remainder of the program. Gavin and Mullaney held internal meetings to publicize the incentive provisions of the contract, so that everyone working on LM would know what the specific short term goals and priorities were. Highest priority was meeting the scheduled ship date for LM-1, the first flight LM, of 15 November 1966.
Hjornevik Review
Unfortunately, reaching agreement with NASA and our subcontractors on schedule and cost goals was no guarantee of achieving them. The ink on the new contract was barely dry when our forecasts at Bob Mullaney’s weekly program meeting showed further cost growths and schedule slips. This continued into the spring, despite vigorous efforts by the subsystem engineers and subcontract managers to contain it. The deteriorating situation attracted the attention of Grumman’s newly appointed president, Lew Evans.
Evans was short and stocky, with a full face, black hair, twinkling eyes, and mischievous smile. He radiated energy and charisma, the natural leadership ability tha
t sweeps others willingly into its train. When talking to Evans you felt that his whole attention was focused on you and your concerns as his clear blue eyes bore into you with total concentration. Even his many facial tics (eye blinking, mouth stretching, neck rotation) enhanced the impression of a human dynamo bursting with energy. He genuinely liked people and was interested in them as human beings, not just cogs in his enterprise. Above all he conveyed a spirit of optimism and unlimited opportunities for those who followed him, and a belief that the future for Grumman and its people was boundless.
Evans was born in what is now North Korea, the son of a globe-trotting mining engineer. He lived in many places, including several years in Mexico, where he became fluent in Spanish and developed a taste for spicy food. After graduating from the University of California in 1942 he served with distinction in the U.S. Army Air Corps. He graduated from Harvard Law School in 1947, was admitted to the bar, and served four years as assistant counsel of the Bureau of Aeronautics in Washington, D.C. While at the bureau he observed the key roles that politics and personal relationships played in obtaining contracts from the navy, or any government agency.
Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight) Page 21