by Ashlee Vance
So, I think generally you do want to have a timeline where, based on everything you know about, the schedule should be X, and you execute towards that, but with the understanding that there will be all sorts of things that you don’t know about that you will encounter that will push the date beyond that. It doesn’t mean that you shouldn’t have tried to aim for that date from the beginning because aiming for something else would have been an arbitrary time increase.
It’s different to say, “Well, what do you promise people?” Because you want to try to promise people something that includes schedule margin. But in order to achieve the external promised schedule, you’ve got to have an internal schedule that’s more aggressive than that. Sometimes you still miss the external schedule.
SpaceX, by the way, is not alone here. Being late is par for the course in the aerospace industry. It’s not a question of if it’s late, it’s how late will the program be. I don’t think an aerospace program has been completed on time since bloody World War II.
Dealing with the epically aggressive schedules and Musk’s expectations has required SpaceX’s engineers to develop a variety of survival techniques. Musk often asks for highly detailed proposals for how projects will be accomplished. The employees have learned never to break the time needed to accomplish something down into months or weeks. Musk wants day-by-day and hour-by-hour forecasts and sometimes even minute-by-minute countdowns, and the fallout from missed schedules is severe. “You had to put in when you would go to the bathroom,” Brogan said. “I’m like, ‘Elon, sometimes people need to take a long dump.’” SpaceX’s top managers work together to, in essence, create fake schedules that they know will please Musk but that are basically impossible to achieve. This would not be such a horrible situation if the targets were kept internal. Musk, however, tends to quote these fake schedules to customers, unintentionally giving them false hope. Typically, it falls to Gwynne Shotwell, SpaceX’s president, to clean up the resulting mess. She will either need to ring up a customer to give them a more realistic timeline or concoct a litany of excuses to explain away the inevitable delays. “Poor Gwynne,” Brogan said. “Just to hear her on the phone with the customers is agonizing.”
There can be no question that Musk has mastered the art of getting the most out of his employees. Interview three dozen SpaceX engineers and each one of them will have picked up on a managerial nuance that Musk has used to get people to meet his deadlines. One example from Brogan: Where a typical manager may set the deadline for the employee, Musk guides his engineers into taking ownership of their own delivery dates. “He doesn’t say, ‘You have to do this by Friday at two P.M.,’” Brogan said. “He says, ‘I need the impossible done by Friday at two P.M. Can you do it?’ Then, when you say yes, you are not working hard because he told you to. You’re working hard for yourself. It’s a distinction you can feel. You have signed up to do your own work.” And by recruiting hundreds of bright, self-motivated people, SpaceX has maximized the power of the individual. One person putting in a sixteen-hour day ends up being much more effective than two people working eight-hour days together. The individual doesn’t have to hold meetings, reach a consensus, or bring other people up to speed on a project. He just keeps working and working and working. The ideal SpaceX employee is someone like Steve Davis, the director of advanced projects at SpaceX. “He’s been working sixteen hours a day every day for years,” Brogan said. “He gets more done than eleven people working together.”
To find Davis, Musk called a teaching assistant* in Stanford’s aeronautics department and asked him if there were any hardworking, bright master’s and doctoral candidates who didn’t have families. The TA pointed Musk to Davis, who was pursuing a master’s degree in aerospace engineering to add to degrees in finance, mechanical engineering, and particle physics. Musk called Davis on a Wednesday and offered him a job the following Friday. Davis was the twenty-second SpaceX hire and has ended up the twelfth most senior person still at the company. He turned thirty-five in 2014.
Davis did his tour of duty on Kwaj and considered it the greatest time of his life. “Every night, you could either sleep by the rocket in this tent shelter where the geckos crawled all over you or take this one-hour boat ride that made you seasick back to the main island,” he said. “Every night, you had to pick the pain that you remembered least. You got so hot and exhausted. It was just amazing.” After working on the Falcon 1, Davis moved to the Falcon 9 and then Dragon.
The Dragon capsule took SpaceX four years to design. It’s likely the fastest project of its ilk done in the history of the aerospace industry. The project started with Musk and a handful of engineers, most of them under thirty years old, and peaked at one hundred people.* They cribbed from past capsule work and read over every paper published by NASA and other aeronautics bodies around projects like Gemini and Apollo. “If you go search for something like Apollo’s reentry guidance algorithm, there are these great databases that will just spit out the answer,” Davis said. The engineers at SpaceX then had to figure out how to advance these past efforts and bring the capsule into the modern age. Some of the areas of improvement were obvious and easily accomplished, while others required more ingenuity. Saturn 5 and Apollo had colossal computing bays that produced only a fraction of the computer horsepower that can be achieved today on, say, an iPad. The SpaceX engineers knew they could save a lot of room by cutting out some of the computers while also adding capabilities with their more powerful equipment. The engineers decided that while Dragon would look a lot like Apollo, it would have steeper wall angles, to clear space for gear and for the astronauts that the company hoped to fly. SpaceX also got the recipe for its heat shield material, called PICA, through a deal with NASA. The SpaceX engineers found out how to make the PICA material less expensively and improved the underlying recipe so that Dragon—from day one—could withstand the heat of a reentry coming back from Mars.* The total cost for Dragon came in at $300 million, which would be on the order of 10 to 30 times less than capsule projects built by other companies. “The metal comes in, we roll it out, weld it, and make things,” Davis said. “We build almost everything in-house. That is why the costs have come down.”
Davis, like Brogan and plenty of other SpaceX engineers, has had Musk ask for the seemingly impossible. His favorite request dates back to 2004. SpaceX needed an actuator that would trigger the gimbal action used to steer the upper stage of Falcon 1. Davis had never built a piece of hardware before in his life and naturally went out to find some suppliers who could make an electromechanical actuator for him. He got a quote back for $120,000. “Elon laughed,” Davis said. “He said, ‘That part is no more complicated than a garage door opener. Your budget is five thousand dollars. Go make it work.’” Davis spent nine months building the actuator. At the end of the process, he toiled for three hours writing an e-mail to Musk covering the pros and cons of the device. The e-mail went into gory detail about how Davis had designed the part, why he had made various choices, and what its cost would be. As he pressed send, Davis felt anxiety surge through his body knowing that he’d given his all for almost a year to do something an engineer at another aerospace company would not even attempt. Musk rewarded all of this toil and angst with one of his standard responses. He wrote back, “Ok.” The actuator Davis designed ended up costing $3,900 and flew with Falcon 1 into space. “I put every ounce of intellectual capital I had into that e-mail and one minute later got that simple response,” Davis said. “Everyone in the company was having that same experience. One of my favorite things about Elon is his ability to make enormous decisions very quickly. That is still how it works today.”
Kevin Watson can attest to that. He arrived at SpaceX in 2008 after spending twenty-four years at NASA’s Jet Propulsion Laboratory. Watson worked on a wide variety of projects at JPL, including building and testing computing systems that could withstand the harsh conditions of space. JPL would typically buy expensive, specially toughened computers, and this frustrated Watson.
He daydreamed about ways to handcraft much cheaper, equally effective computers. While having his job interview with Musk, Watson learned that SpaceX needed just this type of thinking. Musk wanted the bulk of a rocket’s computing systems to cost no more than $10,000. It was an insane figure by aerospace industry standards, where the avionics systems for a rocket typically cost well over $10 million. “In traditional aerospace, it would cost you more than ten thousand dollars just for the food at a meeting to discuss the cost of the avionics,” Watson said.
During the job interview, Watson promised Musk that he could do the improbable and deliver the $10,000 avionics system. He began working on making the computers for Dragon right after being hired. The first system was called CUCU, pronounced “cuckoo.” This communications box would go inside the International Space Station and communicate back with Dragon. A number of people at NASA referred to the SpaceX engineers as “the guys in the garage” and were cynical about the start-up’s ability to do much of anything, including building this type of machine. But SpaceX produced the communication computer in record time, and it ended up as the first system of its kind to pass NASA’s protocol tests on the first try. NASA officials were forced to say “cuckoo” over and over again during meetings—a small act of defiance SpaceX had planned all along to torture NASA. As the months went on, Watson and other engineers built out the complete computing systems for Dragon and then adapted the technology for Falcon 9. The result was a fully redundant avionics platform that used a mix of off-the-shelf computing gear and products built in-house by SpaceX. It cost a bit more than $10,000 but came close to meeting Musk’s goal.
SpaceX reinvigorated Watson, who had become disenchanted with JPL’s acceptance of wasteful spending and bureaucracy. Musk had to sign off on every expenditure over $10,000. “It was his money that we were spending, and he was keeping an eye on it, as he damn well should,” Watson said. “He made sure nothing stupid was happening.” Decisions were made quickly during weekly meetings, and the entire company bought into them. “It was amazing how fast people would adapt to what came out of those meetings,” Watson said. “The entire ship could turn ninety degrees instantly. Lockheed Martin could never do anything like that.” Watson continued:
Elon is brilliant. He’s involved in just about everything. He understands everything. If he asks you a question, you learn very quickly not to go give him a gut reaction. He wants answers that get down to the fundamental laws of physics. One thing he understands really well is the physics of the rockets. He understands that like nobody else. The stuff I have seen him do in his head is crazy. He can get in discussions about flying a satellite and whether we can make the right orbit and deliver Dragon at the same time and solve all these equations in real time. It’s amazing to watch the amount of knowledge he has accumulated over the years. I don’t want to be the person who ever has to compete with Elon. You might as well leave the business and find something else fun to do. He will outmaneuver you, outthink you, and out-execute you.
One of Watson’s top discoveries at SpaceX was the test bed on the third floor of the Hawthorne factory. SpaceX has test versions of all the hardware and electronics that go into a rocket laid out on metal tables. It has in effect replicated the innards of a rocket end to end in order to run thousands of flight simulations. Someone “launches” the rocket from a computer and then every piece of mechanical and computing hardware is monitored with sensors. An engineer can tell a valve to open, then check to see if it opened, how quickly it opened, and the level of current running to it. This testing apparatus lets SpaceX engineers practice ahead of launches and figure out how they would deal with all manner of anomalies. During the actual flights, SpaceX has people in the test facility who can replicate errors seen on Falcon or Dragon and make adjustments accordingly. SpaceX has made numerous changes on the fly with this system. In one case someone spotted an error in a software file in the hours right before a launch. SpaceX’s engineers changed the file, checked how it affected the test hardware, and, when no problems were detected, sent the file to the Falcon 9, waiting on the launchpad, all in less than thirty minutes. “NASA wasn’t used to this,” Watson said. “If something went wrong with the shuttle, everyone was just resigned to waiting three weeks before they could try and launch again.”12
From time to time, Musk will send out an e-mail to the entire company to enforce a new policy or let them know about something that’s bothering him. One of the more famous e-mails arrived in May 2010 with the subject line: Acronyms Seriously Suck:
There is a creeping tendency to use made up acronyms at SpaceX. Excessive use of made up acronyms is a significant impediment to communication and keeping communication good as we grow is incredibly important. Individually, a few acronyms here and there may not seem so bad, but if a thousand people are making these up, over time the result will be a huge glossary that we have to issue to new employees. No one can actually remember all these acronyms and people don’t want to seem dumb in a meeting, so they just sit there in ignorance. This is particularly tough on new employees.
That needs to stop immediately or I will take drastic action—I have given enough warnings over the years. Unless an acronym is approved by me, it should not enter the SpaceX glossary. If there is an existing acronym that cannot reasonably be justified, it should be eliminated, as I have requested in the past.
For example, there should be no “HTS” [horizontal test stand] or “VTS” [vertical test stand] designations for test stands. Those are particularly dumb, as they contain unnecessary words. A “stand” at our test site is obviously a *test* stand. VTS-3 is four syllables compared with “Tripod,” which is two, so the bloody acronym version actually takes longer to say than the name!
The key test for an acronym is to ask whether it helps or hurts communication. An acronym that most engineers outside of SpaceX already know, such as GUI, is fine to use. It is also ok to make up a few acronyms/contractions every now and again, assuming I have approved them, eg MVac and M9 instead of Merlin 1C-Vacuum or Merlin 1C-Sea Level, but those need to be kept to a minimum.
This was classic Musk. The e-mail is rough in its tone and yet not really unwarranted for a guy who just wants things done as efficiently as possible. It obsesses over something that other people might find trivial and yet he has a definite point. It’s comical in that Musk wants all acronym approvals to run directly through him, but that’s entirely in keeping with the hands-on management style that has, mainly, worked well at both SpaceX and Tesla. Employees have since dubbed the acronym policy the ASS Rule.
The guiding principle at SpaceX is to embrace your work and get stuff done. People who await guidance or detailed instructions languish. The same goes for workers who crave feedback. And the absolute worst thing that someone can do is inform Musk that what he’s asking is impossible. An employee could be telling Musk that there’s no way to get the cost on something like that actuator down to where he wants it or that there is simply not enough time to build a part by Musk’s deadline. “Elon will say, ‘Fine. You’re off the project, and I am now the CEO of the project. I will do your job and be CEO of two companies at the same time. I will deliver it,’” Brogan said. “What’s crazy is that Elon actually does it. Every time he’s fired someone and taken their job, he’s delivered on whatever the project was.”
It is jarring for both parties when the SpaceX culture rubs against more bureaucratic bodies like NASA, the U.S. Air Force, and the Federal Aviation Administration. The first inklings of these difficulties appeared on Kwaj, where government officials sometimes questioned what they saw as SpaceX’s cavalier approach to the launch process. There were times when SpaceX would want to make a change to its launch procedures and any such change would require a pile of paperwork. SpaceX, for example, would have written down all the steps needed to replace a filter—put on gloves, wear safety goggles, remove a nut—and then want to alter this procedure or use a different type of filter. The FAA would need a week to review the new pro
cess before SpaceX could actually go about changing the filter on the rocket, a lag that both the engineers and Musk found ridiculous. On one occasion after this type of thing happened, Musk laid into an FAA official while on a conference call with members of the SpaceX team and NASA. “It got hot and heated, and he berated this guy on a personal level for like ten minutes,” Brogan said.
Musk did not recall this incident but did remember other confrontations with the FAA. One time he compiled a list of things an FAA subordinate had said during a meeting that Musk found silly and sent the list along to the guy’s boss. “And then his dingbat manager sent me this long e-mail about how he had been in the shuttle program and in charge of twenty launches or something like that and how dare I say that the other guy was wrong,” Musk said. “I told him, ‘Not only is he wrong, and let me rearticulate the reasons, but you’re wrong, and let me articulate the reasons.’ I don’t think he sent me another e-mail after that. We’re trying to have a really big impact on the space industry. If the rules are such that you can’t make progress, then you have to fight the rules.
“There is a fundamental problem with regulators. If a regulator agrees to change a rule and something bad happens, they could easily lose their career. Whereas if they change a rule and something good happens, they don’t even get a reward. So, it’s very asymmetric. It’s then very easy to understand why regulators resist changing the rules. It’s because there’s a big punishment on one side and no reward on the other. How would any rational person behave in such a scenario?”
In the middle of 2009, SpaceX hired Ken Bowersox, a former astronaut, as its vice president of astronaut safety and mission assurance. Bowersox fit the mold of recruit prized by a classic big aerospace company. He had a degree in aerospace engineering from the U.S. Naval Academy, had been a test pilot in the air force, and flew on the space shuttle a handful of times. Many people within SpaceX saw his arrival at the company as a good thing. He was considered a diligent, dignified sort who would provide a second set of eyes to many of SpaceX’s procedures, checking to make sure the company went about things in a safe, standardized manner. Bowersox ended up smack in the middle of the constant pull and push at SpaceX between doing things efficiently and agonizing over traditional procedures. He and Musk were increasingly at odds as the months passed, and Bowersox started to feel as if his opinions were being ignored. During one incident in particular, a part made it all the way to the test stand with a major flaw—described by one engineer as the equivalent of a coffee cup not having a bottom—instead of being caught at the factory. According to observers, Bowersox argued that SpaceX should go back and investigate the process that led to the mistake and fix its root cause. Musk had already decided that he knew the basis of the problem and dismissed Bowersox after a couple of years on the job. (Bowersox declined to speak on the record about his time at SpaceX.) A number of people inside SpaceX saw the Bowersox incident as an example of Musk’s hard-charging manner undermining some much-needed process. Musk had a totally different take on the situation, casting Bowersox as not being up to the engineering demands at SpaceX.