These meetings were long and painful. Lanzkron would not accept waffled answers to his questions. If the person he was cross-examining implicated someone else as part of his excuses, Lanzkron insisted that the third party be brought into the meeting immediately to defend himself or rebut the charges. If information was lacking, he demanded that it be located and presented before the meeting adjourned. The meetings frequently continued until 8:00 or 9:00 P.M. with no break for lunch. When the meeting began Lanzkron prominently displayed on the table in front of him an apple and a glass of water, which everyone knew was his lunch. The Grumman people either brought brown-bag lunches or slipped out in ones and twos for a quick bite in the Plant 5 cafeteria.
After holding his first few LM GSE meetings, Lanzkron requested a management meeting with Gavin, Mullaney, Rathke, and me. He expressed shock and dismay at what he had found at Grumman. Calmly and methodically he presented the data our own people had given him, which showed that we were making no progress against the GSE schedules but steadily falling further behind. His devastating assessment was that Grumman’s GSE management was not up to the job and Grumman’s corporate GSE capability was inadequate. He demanded strong corrective action from management. Gavin and Mullaney admitted that we were deficient in the GSE part of the program and asked for time to prepare a corrective action plan. Smiling triumphantly, Lanzkron gave them two weeks. He confirmed the time of the follow-up meeting with Gavin before he left the room.
Unknown to me at the time, this pressure on GSE from NASA reinforced a similar thrust from the navy, which required more GSE to support fleet operations and maintenance of Grumman’s aircraft and their ever more complex systems. This resulted in a major change in Grumman’s corporate organization—the creation of a new operating department for Integrated Logistic Support (ILS). On a level with the traditional departments of Engineering, Flight Test, and Manufacturing, ILS was responsible for every activity and product required to support Grumman’s aircraft and spacecraft in their operating environment. This included GSE design and manufacture, spare-parts provisioning, logistic-support analyses, and publications and manuals. The former Materials and Processes (M&P) Engineering and E&P Manufacturing groups were included in the new department, and there was a reallocation of Engineering and Manufacturing floor space to support ILS expansion. To lead this new organization management selected Edward Dalva, a veteran Grummanite and proven project engineer and manager who had directed the development of the W2F and E2A airborne early-warning aircraft systems. Dalva proved effective in infusing energy and a “can-do” spirit into what had been a neglected backwater activity. He greatly expanded Grumman’s sales in the ILS area, making it a major segment of the company’s business base and increasing our customers’ satisfaction with the total performance of Grumman’s products.
But LM could not wait for corporate GSE to get its house in order; we needed help now. Gavin and Mullaney moved to strengthen the LM GSE management. Our LM GSE project engineer, Dick Spinner, and the Manufacturing manager, Bob Wagenseil, were simply no match for Lanzkron, often withering pathetically under his onslaughts. John Coursen, a savvy project engineer with a record of accomplishment on the F10F and Fin programs, replaced Spinner, and Tony Oddo, assistant director of E&P Manufacturing, replaced Wagenseil. They recruited other capable people to bolster LM GSE management. Gavin and Mullaney also set in motion a plan to expand and integrate LM GSE manufacturing and test facilities, which were scattered throughout various tooling and E&P shops. In the brief follow-up meeting with Lanzkron, he nodded approvingly when these changes were announced and said he looked forward to seeing what these new people could do.
I spent more time on GSE, often attending at least part of their weekly meeting with Lanzkron as well as continuing to hold my own biweekly technical staff meeting with GSE Engineering. As I became familiar with the GSE program, I soon found myself doing what the GSE “natives” did: thinking of and referring to the GSE end items by their five- or four-digit numbers. In some cases the dialogue at the GSE meetings sounded like the legendary comedians’ convention where someone calls out the number of a joke and everyone laughs.
In late 1966 LM GSE Manufacturing was consolidated in a refurbished building in Syosset, Long Island, known as the Davega building for its former occupant, a sporting-goods chain. With the GSE Engineering and Manufacturing staffs greatly expanded and intense concentration and effective leadership by Coursen and his team, plus increased support from the corporate ILS Department, the LM GSE program gradually turned the corner. The end-item list stabilized, and our ability to make schedules and hold to them gradually improved. At its peak in early 1967, LM GSE Engineering and Manufacturing employed more than fifteen hundred people. The total cost of LM GSE was greater than our 1962 negotiated estimate for the entire LM program, an indication of how GSE was underestimated by all at the start of the program.
Lanzkron was reassigned by Shea to key trouble spots elsewhere on the program, but he returned to Grumman for a time in 1967 to help us resolve pervasive manufacturing assembly and quality-control problems. Although his brand of assistance was painful to endure, it was effective and contributed substantially to both Grumman’s success and that of the Apollo program.
Configuration Management and Scheduling
The management of the Apollo program was a staggering task, more complex in its scope, number of participants, and interrelations between program elements than anything yet attempted by the aerospace industry. The activities and outputs of more than 175,000 people in thousands of organizations across the United States had to be coordinated, scheduled, and provided with technical interface data wherever their products interacted with someone else’s. Interlocking schedules for each program participant had to be prepared and scheduled against actual progress. The whole array of schedules had to be capable of rapid revision whenever new developments, such as unforeseen test failures or delivery slippages, caused a change in technical approach or plans.
As a model of how to manage such complexity, NASA used the ballistic missile programs then under development by the air force and the navy. They adopted major techniques from each of them: from the air force, the configuration management system, and from the navy, the program evaluation and review technique (PERT).
Configuration management, developed and used on the Atlas, Titan, and Minuteman programs, was a formal, structured method for defining and controlling the detailed configuration of aerospace flight vehicles and GSE. Rigorous control over the millions of parts in a missile or spacecraft was essential to prevent unexpected failures due to the actual vehicle differing from what the engineering drawings showed it to be. The leaders of NASA’s manned spaceflight program had learned this lesson early, through a terrifying incident that occurred as they all watched the first launch of an unmanned Mercury capsule on a Redstone booster, the last flight prerequisite to a manned suborbital mission. MR-1, as it was called, failed ignominiously and dangerously on the launch pad on 20 November 1960. The Redstone rocket fired briefly and lifted a few inches off the pad, then abruptly shut down. The Redstone settled back on the launch pad, but the Mercury spacecraft continued to go through its preprogrammed sequence of events for liftoff and landing; firing the escape tower rocket, then the radar chaff dispenser, the drogue parachute, and the main parachute. The Redstone was left standing on the pad, unsecured, with its propellant tanks pressurized and no connection to ground control from the blockhouse. Fortunately there was no fire or explosion, and NASA technicians eventually went out to the pad and successfully disarmed the live Redstone rocket.
The cause of this mishap was determined to be a failure of configuration management: the “as built” system was not “as designed” in the engineering drawings. Subsequent to the prior launch from the pad, a technician filed about one-quarter inch from one of the two prongs in the tail plug, an electrical connector at the base of the launch umbilical tower that plugged into the base of the Redstone, and did not tell anyone w
hat he did. When the Redstone lifted off about an inch, the tail plug was supposed to pull out of the base of the rocket, switching the Redstone over to internal power. However, if one of the two prongs was to disconnect before the other, the circuitry was designed to shut the Redstone’s engine down immediately. This is what happened when one connector prong had been made shorter than the other. NASA was extremely lucky that MR-1 did not explode on the pad in a spectacular failure that might have caused cancellation of the whole manned spaceflight program. This narrow escape made all those who witnessed it, including Gilruth and von Braun, determined to have a disciplined configuration management program throughout NASA.1
The air force’s configuration management system was thorough and rigorous and was documented in five manuals known as the “375 series.” With minor revisions NASA adopted it completely and educated its contractors on how to apply it. I listened to an air force lecture on configuration management, after which Bill Rector and I discussed how we would apply it to LM. We agreed that the M-5 mockup review would also be the preliminary design review (PDR) as defined by the configuration management manuals, marking the completion of the definition phase of the program. After PDR no major configuration changes could be made without prior approval by the LM Change Control Board (CCB), staffed by both NASA and Grumman. The design and development phase for unmanned LMs would end shortly before LM-1 delivery with a critical design review (CDR), and for manned LMs before LM-3 delivery2 After CDR no changes that affected form, fit, or function of any part or component could be made without prior CCB approval. The nomenclature implies the rigor and discipline that this system imposed. Making it happen would require continuous education and repetitive indoctrination of all those on the program, a major task of training and communications. The LM program was Grumman’s first exposure to rigorous air force 375 configuration management, as the navy had not yet imposed it on our aircraft programs.
The PERT scheduling and control system was first put into major program usage by Admiral Rayburn on the Polaris Fleet Ballistic Missile program and improved by other users, including the Minuteman program. It was adopted by the Department of Defense as a standard tool of program management, and numerous software programs were developed to provide computer-assisted generation of the PERT schedules and networks. NASA adopted a version of PERT as its standard across all programs, including Apollo. As in configuration management, the LM program was Grumman’s first use of PERT, so we had to train our people in the system and its application to LM. During 1964 we conducted many training sessions for our people in the use of CM and PERT.
Many of PERT’s features were well suited to the Apollo program. It provided flexible, computer-aided scheduling of program activities and allowed for uncertainties by inputs of “best, worst, and most likely” completion dates for each event. It showed the relationships between events through graphic network diagrams in which the activities or events were positioned and coded to show their relationship to other events, whether as prerequisite constraints or as parallel nonconstraining activities. Using computer software, PERT allowed the program to be scheduled at a low level of detail that could directly be related to measurable actions and outputs in Engineering, Manufacturing, or test. It used the computer to “roll up” the highly detailed network into summary schedules sorted by spacecraft, by test program, or whatever views were meaningful at the overall program level.
Rathke and Mullaney established the LM program Scheduling Group to learn the PERT system and help us apply it. The group was headed by Larry Moran from the Scheduling Group in corporate engineering. Larry was tall, thin, and energetic, with a tanned and freckled face, black hair brushed back in a low pompadour, and a ready laugh and smile. A heavy smoker, he looked unhealthy hunched over the networks and schedules on his drafting board, wreathed in smoke with a cigarette dangling from his lips. He was a natural leader and had a wonderful sense of humor, which helped his hardworking group endure the long hours they put in.
Larry Moran built a group of program schedulers who were far more than just PERT technicians. They understood interactions between program events better than anyone else on LM, and as scheduling problems developed they were very constructive in helping program and engineering management devise “workarounds” to recover lost time or accommodate unforeseen events and requirements. The schedulers roamed freely throughout the LM program, determining the actual status of events scheduled on the PERT networks and posting them on sliding schedule wallboards displayed in the Plant 25 main conference room. Gavin and Mullaney conducted the weekly LM program meeting using Moran and his charts to show the schedule problem areas and discuss recovery options. Summary schedules and actual status were rolled up from our PERT networks and fed into the computers at NASA-Houston for integration into the total Apollo spacecraft schedules and networks that NASA maintained.
Moran’s group and their PERT diagrams became the principal tool that Rathke, Carbee, Whitaker, Coursen, and I used to plan and schedule Engineering’s activities. The networks included all the drawings and the analyses required from Whitaker’s Systems Engineering groups that were constraints on drawings. The flow of major test activities was added as the networks matured. The drawing estimates and scheduled delivery promises came from the leaders of the individual Engineering design sections based upon their past experience, knowledge of the current design, and their workload and staffing. Bob Carbee for the flight and ground test spacecraft and John Coursen for GSE, probed, challenged, and demanded performance against schedule from their people—day after day, week after week. It took time, but the results of their persistent efforts gradually became visible as the growth in our estimated number of drawings required and slippage against the established delivery schedules decreased throughout 1965 and 1966.
Mountains of Work
We faced a mountain of required drawing outputs that built up rapidly throughout 1965 and plateaued in 1966 at more than four hundred drawings a week. The pressure of schedule was relentless. The expansion of our Engineering groups continued in 1965 and 1966, completely filling Plant 25. Engineers worked in large bullpen areas on the open floor with drawing boards and desks jammed five abreast. (To reach the aisle, the person working at the center board had to pass behind and disturb two people on either side.) Group leaders went from board to board several times a day, collecting nearly finished drawings to be turned over to Drafting or the Release Group and updating the status inputs for PERT. Problems involving lack of information from other Engineering groups, or anything that held up completion of a drawing were taken to the group leaders for action or resolution. There was a corner of the first floor of Plant 25 where a large plywood table had been constructed for preparing oversized drawings and “white masters.” The table was three feet high and about ten feet square. Several engineers or draftsmen worked on their hands and knees on the table, drawing directly on large sheets of vellum paper or on the white painted aluminum sheets on which the “white master” tooling templates were drawn and later cut out to fit into the tool that defined LM skin contours or structure. The three large Engineering floors in Plant 25 were like throbbing frames in a giant beehive, bustling with purposeful activity, visible output, and the hum of accomplishment.
Plant 35, a two-story building with almost the same floor area as Plant 25, was completed in 1965. It provided some relief to the overcrowding in Plant 25 and space for the continuing growth in Engineering staff. By 1966 LM Engineering also occupied substantial office areas in Plants 5 and 30 as the number of engineers neared its peak of almost three thousand. Art Gross and the Facilities Management Group were busy with office moves, expansions, and reconfigurations, which had to be carried out with a minimum of disruption to the ongoing work.
The Drafting Group was a veteran operation in corporate Engineering, dating back to Vice President of Engineering Dick Hutton, a draftsman-engineer who was the first person hired by Chief Engineer Bill Schwendler when Grumman was founded. Draftsmen took
drawings after the engineer had drawn and specified the basic design and added the finishing touches, such as dimensions, notes, and additional views, that were needed to make them intelligible and useful to manufacturing. They created detailed parts drawings from assembly drawings and engineering instructions. They did preliminary work to create a drawing by preparing cross sections or views upon which the engineer could work out his design additions or modifications. We often had more than four hundred draftsmen on LM, a number that could be readily be changed by bringing in “job shoppers,” contract draftsmen from manpower supply companies. Extensive use of overtime in Engineering and Drafting evened out the peaks and valleys in manpower demand.
Corporate Drafting was run by Howard Krier and the LM Drafting Group by Ross Chandler. They were a hard-bitten pair of aircraft design veterans and not about to take any nonsense from a young project engineer like me, or from an upstart government agency like NASA for that matter. Krier was a fatherly looking fellow with salt-and-pepper hair and clear-rimmed glasses who considered himself the final authority on anything pertaining to drafting. He had a high-pitched voice that became increasingly annoying to me in debates when he would stubbornly refuse to give ground. Chandler, black-haired and usually toting a large cigar in his mouth, was somewhat younger and had the patient but exasperated look of one who believes he has seen it all.
I had one meeting with them in which, in response to pressure from Shea in Houston, I challenged the schedule status and manpower estimates of all LM Engineering, including Drafting. They unrolled dozens of detailed estimating sheets broken down to individual drawings and went through them page by page while Krier kept up a high-pitched skirl of justification and Chandler blew cigar smoke over my head. By the time they were finished I was glad to escape with an estimate no higher than their original position. After that I relied on Rathke to deal with them—he understood them better and, if necessary, could blow cigar smoke back at Chandler.
Moon Lander: How We Developed the Apollo Lunar Module (Smithsonian History of Aviation and Spaceflight) Page 15