by Bill Adair
Then came a more rigorous test. Hewett slowly stepped on the left pedal and the rudder moved properly. He then jammed his foot on the right pedal as hard as he could. It kicked back with tremendous force.
The rudder swung the wrong direction.
Further tests showed that the likelihood of the rudder reversing depended on where the outer slide jammed. If it jammed closer to its neutral position, the rudder was less likely to reverse. But if it jammed when it was farther from neutral, a reversal was more certain. It was about midnight now and everyone was exhausted. They all drove home worrying about what they should do to fix the plane. “Everyone was concerned,” Draxler recalled later. “We didn’t know what it meant, how it all fit together.”
The next day, Boeing notified the FAA that the company had found a problem with the rudder PCU but wanted twenty-four hours to figure out how to deal with the problem. The FAA agreed.
Intense meetings went on all day in Renton and Everett as the Boeing engineers discussed how to respond. They broke into two teams, one to come up with a plan for how pilots could detect and respond to a jam and another to look at long-term design changes to the PCU. They worked into the night. By 11 P.M., they got approval from senior management for a pilot test and some short- and long-term changes to the PCU.
The next day, Halloween, about ten Boeing officials drove to the FAA office in Renton, a big mirrored cube of a building beside Interstate 405. They weren’t sure what the FAA would do. Would the agency want to ground the airplane? The PCU no longer protected against jams the way it was supposed to, which could mean that the plane no longer met certification standards.
About twenty-five Boeing and FAA officials gathered in a conference room. McGrew took a seat just beneath a smiling photo of FAA administrator David Hinson. In the corner were a TV and a VCR, with the clock display flashing “12:00” throughout the meeting. Draxler began by explaining what they had found in the tests, with Hewett frequently interrupting to give his perspective. It took a unique kind of windup to trigger the phenomenon, they said. You had to press on one pedal and then stomp hard on the other to make the primary slide line up with the wrong holes and cause the reversal.
Did this match what had happened to the USAir plane? an FAA official asked.
The Boeing engineers said all they knew from the test was that if you jam the outer slide, you could get a reversal. Jams were extremely unlikely because of the many filters in the hydraulic system, which removed particles before they caused problems. In thirty years and more than 50 million flights, there had been only seven confirmed jams. None had resulted in an accident or injury. And there was no evidence that one had occurred on the USAir plane.
Another FAA official pointed out that the new evidence seemed to counter Boeing’s claims that the pilots had caused the crash.
McGrew spoke up. “We’ve received a lot of public criticism about hiding things and not wanting to spend a lot of money,” he said. “But I frankly don’t care [what it costs]. If there is something wrong with the airplane, I want to fix it.”
Steve O’Neal, the FAA flight-test engineer, was impressed by McGrew’s comments. Until that point, O’Neal had felt McGrew was overprotective of the airplane. But McGrew now seemed sincere in saying that he was open to anything.
The meeting ended. Boeing said it would issue an alert service bulletin to warn airlines about the condition. The bulletin would require mechanics to perform a test every 250 hours, stomping on the pedals to check for jams. The FAA planned to issue an emergency airworthiness directive that mandated the tests. Boeing also said it would develop a long-term plan to redesign the valve to prevent a reversal. That fix was likely to take several years.
These directives were more symbolism than real action, designed to reassure the public that the FAA was taking action. The engineers knew the tests would not be very effective. They would catch a jam if it occurred at the precise moment of the test, but a jam could still occur at any time.
Despite the seriousness of Boeing’s discovery, FAA officials say they did not give serious thought to grounding the 737 fleet. The plane had a good safety record, they said, and a jam was still considered highly unlikely. Even O’Neal, who had wanted to ground the fleet two years earlier, agreed with his bosses this time.
While the Boeing-FAA meeting was going on in Renton, Haueter and Phillips were 2,000 miles away in Pittsburgh, unaware of the dramatic developments. They had returned to the Holiday Inn near the Pittsburgh airport to meet with all the parties. Haueter and each of his group leaders gave updates on the investigation. Phillips reviewed the results of the thermal shock tests (without knowing of Boeing’s finding) and discussed what work still needed to be done. Rick Howes, the Boeing coordinator for the investigation, sat through the all-day meeting without saying a word about the company’s big discovery.
There was the usual sparring between ALPA and Boeing—this time over Boeing’s latest estimates of the rudder movement—and then the meeting ended uneventfully. Haueter put a slide on the overhead projector with a comment that an English scientist once made to Charles Lindbergh. Haueter said it had become the slogan of the 427 investigation: “Everything that happens was once infinitely improbable. Therefore, nothing that happens should be surprising.”
People chuckled and the meeting broke up. Haueter, Phillips, and Tom Jacky, the NTSB performance chairman, took a flight back to Washington. As they got off the plane at National Airport and walked toward the subway station, Haueter’s beeper went off. The NTSB had a new pager system that could transmit words as well as phone numbers.
Haueter glanced down at it. MAJOR FINDING REL TO PIT / DEFECT FOUND ON SERVO VALVE, the pager said.
“This is a joke,” he said. “This isn’t real. Some jerk has figured out our paging system.” They went their separate ways and headed home.
The message had come from Schleede, Haueter’s boss, who had been working late in the NTSB office when McGrew and John Purvis, the head of Boeing’s accident investigations, called to tell him about the finding. Schleede transmitted the message to Haueter and then walked downstairs to the bar at the L’Enfant Plaza Hotel, where Hall was having a drink.
“Jim,” Schleede said, “I think we’ve got it.”
The next day, the FAA briefed Haueter, Phillips, and other NTSB officials about the finding. Haueter realized that it was a major piece of his puzzle.
“This isn’t the way I thought it would end,” he told Phillips as they walked back after the meeting. “I expected it was going to be a fight all the way to the end, putting all these little pieces together, with people saying we wouldn’t have enough evidence. And all of a sudden here is something no one expected.”
That day, Boeing sent a telex message to every airline in the world that flew 737s:
Alert Alert Alert Alert Alert Alert Alert Alert
Boeing Alert Service Bulletin 737-27A1202 dated November 1, 1996
The dual servo valve is designed to overcome the effects of a jammed primary or secondary slide. Although there has never been a report of a secondary slide jam, tests just completed at Boeing have shown that, under certain conditions, some jams of the secondary slide can result in anomalous rudder motion.
Anomalous rudder motion. It was a Boeing euphemism for a catastrophic situation—a rudder jam and a reversal.
Cox heard rumblings about the discovery on Halloween night but didn’t hear the news until the morning of November 1, when the alert was issued. He had spent an extra day in Pittsburgh and was summoned to the office of USAir vice president/flight operations William Barr. A group of pilots and safety officials were meeting to discuss the service bulletin and how USAir should respond. There were major implications for USAir because it had the third-largest fleet of 737s in the world.
Barr asked Cox point-blank, “Is the airplane safe?”
“Yes,” Cox said. He was convinced that a jam was still highly unlikely and that, even if one occurred, pilots could recover. USAir had
been the first airline to raise its minimum speed above the crossover point, so USAir pilots had an extra cushion of safety. And the airline’s pilots were already doing a rigorous rudder check, which meant they were effectively conducting the test before every flight.
That afternoon Boeing issued a carefully worded press release that downplayed the seriousness of the discovery:
Boeing recently discovered that, under certain conditions, a jam of the PCU’s secondary slide could possibly interfere with the intended operation of the unit. The discovery was made by several Boeing engineers during a careful review of data generated by a National Transportation Safety Board test for the effects of thermal shocks on the PCU….
“This is the nature of our business,” said Charlie Higgins, Boeing vice president/airplane safety and performance. “We identify very unlikely possibilities and take steps to eliminate them, or at least to further reduce their likelihood of ever happening. That’s one of the ways we keep enhancing the safety of the aviation system.”
It was too late for Cox to fly back to St. Petersburg, so he ended up at a Motel 6 near the Pittsburgh airport, watching reports on the rudder discovery on CNN. “This could wrap up 427 quickly,” he said in a phone conversation between reports. “This could be the ‘Aha!’” He said the lack of markings inside the valve was evidence that the gremlin in the rudder system “is a thief in the night. It leaves no trail.”
He praised Boeing for being so forthcoming about the discovery. “I’m extremely pleased. They stepped forward.”
Just before Thanksgiving, Phillips went back to the Parker plant in Irvine to compare the valve from the USAir plane with the ones from the Eastwind and the factory PCUs. He wanted to find out if there was something that made the 427 valve jam when the others would not.
Every rudder valve was slightly different. All valves had to meet certain Boeing and FAA standards, but the holes for hydraulic fluid on each one were never exactly the same. The tests so far suggested that some valves could be more prone to reverse than others.
At Parker, the three valves were each disassembled and examined and then hooked into a test rig to see how far off neutral they had to be moved before the rudder would reverse. The factory valve performed the best. It would not reverse until the outer slide was 38 percent extended. But the USAir and Eastwind valves would reverse more easily, when the slides were 12 and 17 percent extended, respectively. Also, a measurement of the distance between the valve slides found that the USAir unit was considerably tighter than the other two.
That was the final piece of evidence that Haueter had waited for. At last he had proved that the USAir valve was unique. After three years and hundreds of tests, he now had a scenario for what had happened to Flight 427.
It went like this:
It was a smooth flight from Chicago to Pittsburgh, so there was not much movement by the yaw damper. That lack of movement might have allowed particles to build up in the hydraulic fluid. There could have been a modest thermal shock to the PCU because of a problem with a hydraulic pump—not enough to set off a warning to the pilots but enough to send hot fluid rushing into the cold valve, making it suddenly expand.
The PCUs on other 737s might have tolerated that without trouble. But the valve on this particular USAir plane was especially tight. The thermal shock and the contaminants caused a jam. And the jam happened when one slide inside the valve was slightly off center and more likely to reverse. The pilot or the yaw damper was commanding the rudder to go right, but it went hardover to the left.
All of this occurred at the most vulnerable speed for a 737, when the plane was flying at 190 knots—the crossover point when a rudder hardover could not be countered by turning the wheel. The pilots compounded the problem when they pulled back on the control column, which made the plane lose speed and stall. The plane spiraled down and crashed into the hill in Hopewell.
It always takes a chain of events to cause a crash. In this case, it took the wake turbulence, the startling of the pilots, the fact that the plane was flying at the crossover point, the uniqueness of the valve, the jam and reversal, and the mistake of pulling back on the stick. If any one of those things had been different, the crash would not have occurred.
What the hell is this? Haueter thought he finally knew the answer.
He had moles in Boeing who gave him inside information about the company’s strategy, alerting him when Boeing was preparing a full-court press to lobby the NTSB or when the company was softening its approach. The moles disagreed with the Boeing position of blaming the pilots, which Haueter found reassuring. He was glad that some Boeing employees had made their own decisions and believed the rudder had reversed. Haueter suspected there also were NTSB moles who told Boeing what he and Phillips were thinking. So he wasn’t surprised when Howes, the Boeing coordinator for the investigation, called shortly before Christmas 1996.
“We understand there are people at the NTSB who, if they wrote the report today, would find fault with the airplane,” Howes said. “We don’t know how you can possibly say that. We’d like to know who these people are so we can help straighten them out.”
“Rick,” Haueter said, “I don’t know how the report is going to go until it goes to the five board members. But many people have quite frankly told me that they think the airplane rudder system caused the accident—and several of those work for Boeing.”
The Boeing team realized it was facing long odds. Haueter was still holding weekly conference calls with the parties, but they had less to discuss each week. After one call in early 1997, McGrew, Howes, and aerodynamic expert Jim Kerrigan sat in the Boeing air safety conference room looking glum.
The “Boeing Contribution” had been a flop. M-Cab had not been convincing. Haueter and Phillips wanted to blame the plane. It looked like Boeing was about to lose.
“It’s a feeling of banging our heads against the wall,” sighed Kerrigan.
They could not understand why Haueter and Phillips had become so narrowly focused on the plane. There were no conclusive data that showed there had been a jam—no scrapes, no marks. Yes, they had discovered a new failure in the rudder system, but there was no proof it had occurred on 427. Haueter and Phillips had a purely circumstantial case and had not seriously considered the circumstances on the other side, which suggested the pilots had caused the crash.
“You would hope the management of the NTSB doesn’t form opinions ahead of time,” Howes said, but everyone in the room knew that was wishful thinking. All three engineers thought the NTSB was obsessed with finding a cause for the crash and making sure the board did not end up with another unsolved case.
McGrew acknowledged that Boeing had made a few strategic mistakes in the investigation. The whole episode had been an education for him about the politics of aviation safety. He had always believed that all you needed to prove a point was solid data, but this investigation showed that the NTSB was getting more and more political under Hall and that Boeing had to play the political game if it wanted to succeed. McGrew said he had not done a good job of communicating with Hall and other NTSB officials along the way.
The realization that Boeing was about to lose was especially painful because so many employees had invested so much time in the investigation over the past two years. “People have burned a lot of hours on this,” said Howes. “To have it come down to something we don’t think is right is very discouraging.”
Deep down, McGrew was convinced that the pilots caused the crash, but he didn’t think there was enough evidence to list that as the probable cause. “We still cannot say we know what made the rudder go in. We think we know, but we’ll never know for sure.”
A friend called Brett the day after Boeing’s announcement about the thermal shock discovery. “Hey, did you hear? They found the cause of Joan’s crash.”
Brett rushed out and bought the November 2 Chicago Tribune, but the paper did not have anything about the discovery. He found an item in the Wall Street Journal two days late
r, but it was a short story that did not capture the magnitude of the finding. He was losing hope that they would ever be able to prove what had happened. It was clear the rudder was to blame. Why couldn’t the NTSB finally pin it on Boeing and USAir?
“I feel like Nicole Brown Simpson’s relatives,” Brett said. “There’s this mass of information, but they can’t get a conviction.”
He had gotten a mixed response from entertainment companies when he made his pitch about T. Rex’s Dino World Cafe. Everyone seemed to like the concept, but nobody would commit any money for a prototype. After several rejections from big companies, Brett decided to raise the money himself. He would give it a year. If he didn’t succeed, he’d find a new job.
The possibility of getting rich from the restaurant was one of the driving forces for him, but he did not think of his lawsuit that way. It was a tool of vengeance. “The money doesn’t mean that much. I got a fair amount from life insurance and that hasn’t brought me any happiness.”
He said he had finally realized that Boeing and USAir were both so huge that even a multimillion-dollar payment would barely cause a ripple in their finances. “There isn’t enough money in the world to make them suffer,” he said. In his view, Boeing and USAir were ducking responsibility in their court filings and public statements about the crash. Brett told Demetrio that in order to settle the suit he wanted an admission of guilt from the companies.
“You and I will be a fossil on another planet before that happens,” Demetrio said.
On January 15, 1997, Brett happened to flip on the TV as he sat down to eat his lunch. A network was covering Vice President Al Gore’s speech to a conference on aviation safety and security.
“As you know,” Gore said, “the investigations into the crashes of Boeing 737s in Colorado Springs and Pittsburgh have not yet been closed. But those investigations have identified improvements that could help eliminate the chance of rudders playing a role in future accidents. These changes can and should be made without delay.”