The Mystery of Flight 427

by Bill Adair

  The result was a spiral-bound booklet called the “Boeing Contribution to the USAir Flight 427 Accident Investigation Board.” It was the classic Boeing approach—slick color renderings of what Germano saw from the cockpit and a view from behind the plane, matched with Boeing’s analysis. The renderings were the same ones that Haueter had seen on the posters in the conference room a year and a half earlier, but this time they had Boeing’s comments on why the plane was innocent:

  The rudder system on the Boeing 737 airplane has been operated successfully for 73 million flight hours. In all of these hours of service history, there has been no known occasion when there was a full uncommanded rudder input. Nor has there been any known occasion when a rudder malfunction produced an event that was not controllable by the flight crew. Most important, the extensive investigation conducted to date into the rudder system used on USAir Flight 427 confirms that this system was fully operational during the upset.

  Boeing was careful not to criticize the pilots too much. The booklet said it was understandable that they had made a mistake after they were startled by the wake turbulence: “It is known that pilots respond to roll upsets by using rudder. It is also well documented that rudder inputs once made can be forgotten or ignored and maintained for the remainder of the flight.” The booklet said Emmett and Germano were surprised when they were jostled by the wake turbulence. They tried to respond to the roll, but overcorrected with the left rudder pedal.

  “In all likelihood, the crew became absorbed in making other control inputs as the upset sequence developed, and simply failed to perceive that a full rudder input had been made.” The booklet quoted from a USAir document that said when pilots respond to upsets, “our biggest problem has been stepping on the wrong rudder!!” The booklet said there was no evidence of any failure of the rudder PCU and that Flight 427 would have been recoverable if the pilots had not pulled back on the stick. It ended with this:

  Boeing believes there are persuasive reasons to support a conclusion that the USAir Flight 427 accident was caused by an unexpected encounter with wake turbulence, rudder commands by the crew and a failure to apply correct recovery techniques.

  Haueter believed that the “Boeing Contribution” was designed to kill Phillips’s safety recommendations. The booklet didn’t mention them, but it arrived the week before the board was scheduled to discuss them and it clearly tried to deflect attention away from the plane.

  But the booklet ended up having no effect on the recommendations. The safety board delayed them by two weeks so that Bob Francis, the board member working on the TWA 800 crash, could have more time to review them, but the safety fixes were approved unanimously with no significant changes.

  The recommendations were sent to David Hinson, the FAA administrator, in an exhaustive twenty-six-page letter that cited the Eastwind incident and the crashes of USAir 427 in Pittsburgh and United 585 in Colorado Springs. The letter said the investigation of the USAir crash had not been completed, but that the NTSB had found 737 safety problems “that need to be addressed.”

  McSweeny, the FAA aircraft certification chief, was his usual defensive self in responding to the recommendations. He downplayed their importance, saying that the FAA had been addressing many of the same issues with previous airworthiness directives that grew out of the Critical Design Review. He said they would consider the NTSB’s recommendations but that “you want to be very careful when you change [the 737]. You need to make sure that your tinkering does not cause a problem.” Once again, McSweeny sounded as though he was protecting Boeing.

  21. THERMAL SHOCK

  1966

  Bendix Electrodynamics

  North Hollywood, California

  A hydraulic valve had to pass a battery of tests to get accepted by Boeing. One test shook it violently, like a can of house paint in a mixer. Another test moved the valve back and forth 5 million times. The most brutal test froze the valve to minus 40 degrees Fahrenheit and injected it with hot hydraulic fluid. That represented the worst imaginable condition—an overheated hydraulic pump when the plane was in frigid air at 35,000 feet. Hot fluid would shoot into the frozen valve, causing it suddenly to expand. The test was called thermal shock.

  Boeing did not manufacture its own valves, just as it didn’t build most of the parts for its planes. Instead, it relied on hundreds of suppliers such as Bendix. The company did not make the unique valve for the 737’s rudder, but it was bidding to make a similar one for Boeing’s giant new plane, the 747. Bendix engineers built a prototype of the valve to undergo the standard battery of tests—the paint shaker, the marathon, and thermal shock.

  The tests were held in a gray stucco building in an industrial section of North Hollywood, not far from the Burbank airport. The lab, which took up most of the first floor, was filled with a thick, oily smell from all the hydraulic fluid. The room was a veritable torture chamber for a hydraulic valve. The lab even had special steel containers called crash boxes that were used the first time a valve was pressurized, in case it exploded.

  Upstairs was a man named Ralph Vick, an engineer who worked on some of the company’s most important projects. Vick was not directly involved in the bid for the 747 valve, but he kept close tabs on the tests because he—like everyone else in the company—desperately wanted to win the big Boeing contract.

  The torture tests on the 747 valve were no different from hundreds of others performed in the Bendix lab that year. The technicians placed the valve in a tiny freezer and hooked up the hydraulic lines. Once the valve had cooled to sub-zero, they flipped a switch and heard the steady whine of the hydraulic pumps. They moved the valve back and forth, as if a pilot were stepping on the pedals. Then someone flipped another switch, and piping hot fluid shot inside. Usually the valve kept moving. But this one strained and then stuck for a few seconds.

  It had failed the test.

  When Vick heard about the results, he knew it was a setback but not a catastrophe. The valve was an amazingly tight device, with only a few millionths of an inch between each slide, so a very tiny design error could cause a jam. The Bendix engineers went back to their drawing boards and redesigned the tolerances. The new valve passed without problems.

  May 1996

  L’Enfant Plaza Hotel

  Washington, D.C.

  Vick unpacked his suitcase in his hotel room and sat down at the desk with a legal pad. He had come to Washington for the first meeting of the “Greatest Minds in Hydraulics” to review Haueter and Phillips’s work. The goal of the panel was to look for new tests that the safety board should try. At sixty-seven, Vick was a quiet, serious man, a good choice for the group because he had designed dozens of valves and had been awarded twenty-five patents. He was quite familiar with the unique valve-within-a-valve used for the 737 rudder.

  Sitting in his hotel room, he recalled the Bendix test thirty years earlier when hot fluid hit cold metal and the prototype valve stuck for a few seconds. That jam turned out to be no big deal—a redesign took care of the problem. But he wondered if the rudder valve on the USAir plane had stuck the same way. He sketched a brief outline of the test on a piece of paper and gave it to Phillips the next day.

  “I think we should look at this,” Vick said. “It may be something.”

  The NTSB had not done a thermal shock test on the 427 valve because there had been no comments on the cockpit tape about a hydraulic problem. If one of the pumps had broken, it would have triggered a warning light in the cockpit and the pilots would likely have mentioned it. But Phillips agreed to try the test. He was open to any suggestion.

  The power control unit from the USAir crash would be frozen to 40 degrees below zero, similar to the outside air temperatures at 30,000 feet, and then it would be pumped with hot hydraulic fluid.

  No one expected a breakthrough. The 737 valve had passed its own thermal shock test when it was certified in the 1960s. Besides, the temperature range was far more extreme than anything the PCU encountered in real life. Boeing o
fficials viewed the test as a waste of time. McGrew said a 737 would encounter thermal shock conditions only if it flew to the moon.

  On August 26, the Greatest Minds in Hydraulics and Phillips’s systems group gathered at Canyon Engineering, a tiny hydraulics company in an industrial park in Valencia, California. Cox thought the place looked more like a garage than a modern test facility. They had chosen Canyon because the chairman of the expert panel worked there, but the company did not have the sophisticated test equipment that Boeing and Parker did. Phillips brought the PCU in a sturdy navy blue chest, like a violinist carrying his prized Stradivarius. He took the 60-pound case to his hotel room each night to make sure that no one could tamper with the device.

  At Canyon the PCU was placed in a big white Coleman cooler, the same kind you would take on a picnic. It looked like an amateur setup. Holes were cut in the cooler for pipes and tubes and then sealed with gray duct tape. When someone opened the cooler, frost formed on the PCU, making it look like a giant Popsicle. Cox and several others in the room said they were concerned that the temperatures were not controlled closely enough to produce legitimate results. But they forged ahead with the tests to see what would happen.

  The group tested two PCUs—a new one straight from the factory and the one from the crash. To make sure that the hydraulic fluid was similar to Flight 427’s, they used fluid drained from other 737s. They used a pneumatic cylinder to act like the pilot’s feet, pushing the valve back and forth. The room filled with a steady rhythm of clicks and hisses as the cylinder moved the valve left and right.

  Click, hiss, click, hiss. They put the factory PCU through its calisthenics at room temperature, testing it fifty times. It responded normally. They let gaseous nitrogen into the cooler and watched the temperature gauges plummet to minus 30 or 40 degrees, to simulate the cold air at 30,000 feet. Click, hiss, click, hiss. No problems. Finally, they tried two tests to simulate an overheated hydraulic pump, heating the fluid to 170 degrees. Click, hiss, click, hiss. The hot fluid hit the cold valve, but there were no problems. The factory PCU worked great.

  They removed it from the Coleman cooler and installed the PCU from Flight 427. Click, hiss, click, hiss. No problems at room temperature. Click, hiss, click, hiss. The frigid unit was blasted with hot fluid, but it still worked fine. It was their last day in Valencia, and the tests were going so smoothly that several people started to pack up and say good-bye. Another theory had been ruled out. It was time to move on.

  They had reached the most extreme condition. The PCU was depressurized, frozen with the nitrogen gas, and then injected with piping-hot fluid. They made the test especially severe by performing it with the A and B hydraulic systems separately, so the hot fluid would not be diluted by cooler fluid from the other system.

  The hot fluid hit the cold valve. Click, hiss, click, hiss, click, hiss, click, hisssssssssssssssssss.

  The hissing changed pitch. The valve had jammed.

  “It didn’t come back,” said someone in the room.

  “That’s interesting,” said someone else. “Reeeeaaalllll interesting.”

  A second later, the arm went back to neutral and began cycling again. Click, hiss, click, hiss.

  They stopped the test and talked about what had happened. Did they have a breakthrough? Nobody could be sure. The test conditions were so poorly controlled that any result was questionable. A computer operator who had been collecting test data had mistakenly deleted everything, so they had little evidence of what they had seen. Everyone agreed to try it again.

  Click, hiss, click, hisssssssssss. Click, hisssssssssss. The valve was moving slower than it was supposed to. Click, hisssssssssssssssssssssssss. It stuck again.

  They agreed that the test should be done again in a more controlled setting. The Boeing team criticized the tests, saying they were too extreme and that the valve could have been damaged. So the next morning, Phillips woke up at 4 A.M. and drove to Parker-Hannifin so they could perform a test to make sure the valve was okay.

  The test was crucial. When they had first examined the valve after the crash, they had not found any scratches inside it. If they found scratches now, it would prove that a jam left a scratch, which would indicate there had not been a jam on Flight 427. Also, a scratch would mean that the valve had been altered since the crash, which would rule out any further tests. The whole theory about a valve malfunction would go down the drain.

  The Parker technicians took the valve apart, measuring and documenting each piece. They put them under a microscope, examining each surface for scratches or scrapes. They found none and no evidence of a jam. Phillips breathed a sigh of relief.

  They had proved that the valve could jam—and leave no evidence behind.

  Six weeks later, Phillips’s group reconvened in a Boeing laboratory in Seattle. This time, instead of testing the PCU in the Coleman cooler, they used a specially designed foam box with a window on top. The box’s cooling system was more powerful and precise, with temperatures closely monitored by a computer.

  They ran through the same tests they had done in Valencia, starting with the factory PCU at room temperature and then trying a variety of thermal shocks. There were no clicks and hisses this time because the pneumatic system had been replaced by a hydraulic one. In some tests, the technicians just pulled on a lever to move the valve. Once again, the factory PCU passed every test.

  The technicians removed it and replaced it with the PCU from Flight 427. It passed the first tests with no trouble. Then came Condition G, a repeat of the most extreme test in the Coleman cooler. They removed hydraulic pressure from the PCU and let it soak in the cold air until it reached minus 40 degrees. The system A hydraulic fluid was heated to 170 and shot directly into the PCU. The technician moving the valve back and forth felt it slow down. He didn’t notice it bind, but a computer showed it had jammed momentarily. They repeated the action, and the technician felt the lever kick back when he tried to move it to the right. He tried again, felt it stick to the left and then jam.

  Once again they had shown that the 427 valve was unique. It jammed when the factory unit did not.

  Yet Boeing was right. The extreme temperature range necessary for a thermal shock just didn’t happen in real life. And there was no proof that it had happened on the USAir plane.

  Despite their skepticism, Boeing engineers said they would examine the charts from the tests for anything unusual. They might learn how the valve and rudder reacted to a jam.

  A few days later, in a building overlooking Paine Field in Everett, a young Boeing engineer named Ed Kikta sat at his desk, reviewing the charts. He could see the test data on his computer screen, but he liked to print the results so he could study them more closely. The charts showed the flow of hydraulic fluid during each test, higher when it was pushing the rudder and down to zero when it was not. Kikta expected that when the outer valve jammed during the thermal shock, the inner valve would compensate and send an equal amount of fluid in the opposite direction, which would keep the rudder at neutral. That was the great safety feature of the 737 valve. It could compensate for a jam.

  But as Kikta studied the squiggly lines for the return flow, he saw dips that were not supposed to be there. When he matched them to another graph showing the force on the levers inside the PCU, he made an alarming discovery. When the outer valve had jammed, the inner valve had moved too far to compensate. That meant the rudder would not have returned to neutral, the way it was supposed to.

  The rudder would have reversed.

  That could be catastrophic. A pilot would push on the left pedal, expecting the rudder to go left, but it would go right.

  To make sure he hadn’t made a mistake, Kikta showed the results to the other engineers in the room. They agreed with his interpretation. It appeared that the valve had reversed. Kikta looked up and saw that his boss, Jim Draxler, was putting his coat on, getting ready to leave. Kikta stopped him.

  “I think I’ve found something in the data,” Kikta sai
d. “We might have a problem here.” Draxler took his coat off, set down his briefcase, and listened to what Kikta had to say. The consequences of his discovery were enormous. If he was right, it meant the PCU was not performing the way Boeing had promised. The valve-within-a-valve was supposed to provide redundancy if one slide jammed. But this meant a single jam could cripple a plane.

  The next morning Draxler convened a group that he called his grizzled veterans, engineers who had lots of experience with flight controls. Kikta explained his findings and showed them the charts. Draxler went around the room, asking each one about the significance of Kikta’s discovery. They were unanimous: It was a serious problem that needed to be fixed quickly.

  Boeing sprang into action. The company ordered Parker Hannifin to run its own tests to check Kikta’s conclusions. Parker engineers confirmed the results and discovered that when they jammed the outer valve, the levers in the PCU appeared to flex slightly, which allowed the inner valve to line up with the wrong holes.

  Boeing was notorious for being the slow-moving “Lazy B,” but not this time. Fear was a powerful motivator. Engineers usually needed weeks to get an airplane for a test, but now they got one off the assembly line in just twenty-four hours. The plane landed at Boeing Field and was pulled into the B-52 hangar where the fat guy tests had been held. As a cold downpour fell outside on the night of October 29, 1996, the 737 was rigged with the special device that Parker had built to simulate the jam. Hewett, the Boeing test pilot, climbed into the cockpit while Kikta stood on a platform on the tail of the plane, watching the rudder and the PCU. Hewett pushed on the pedals, moving the rudder from side to side. The first two tests went smoothly, and the rudder operated as intended.