In the Beginning...Was the Command Line

by Neal Stephenson

  THE HOLE HAWG OF OPERATING SYSTEMS

  Unix has always lurked provocatively in the background of the operating system wars, like the Russian Army. Most people know it only by reputation, and its reputation, as the Dilbert cartoon suggests, is mixed. But everyone seems to agree that if it could only get its act together and stop surrendering vast tracts of rich agricultural land and hundreds of thousands of prisoners of war to the onrushing invaders, it could stomp them (and all other opposition) flat.

  It is difficult to explain how Unix has earned this respect without going into mind-smashing technical detail. Perhaps the gist of it can be explained by telling a story about drills.

  The Hole Hawg is a drill made by the Milwaukee Tool Company. If you look in a typical hardware store you may find smaller Milwaukee drills, but not the Hole Hawg, which is too powerful and too expensive for homeowners. The Hole Hawg does not have the pistol-like design of a cheap homeowner’s drill. It is a cube of solid metal with a handle sticking out of one face and a chuck mounted in another. The cube contains a disconcertingly potent electric motor. You can hold the handle and operate the trigger with your index finger, but unless you are exceptionally strong, you cannot control the weight of the Hole Hawg with one hand; it is a two-hander all the way. In order to fight off the counter-torque of the Hole Hawg, you use a separate handle (provided), which you screw into one side of the iron cube or the other depending on whether you are using your left or right hand to operate the trigger. This handle is not a sleek, ergonomically designed item as it would be in a homeowner’s drill. It is simply a foot-long chunk of regular galvanized pipe, threaded on one end, with a black rubber handle on the other. If you lose it, you just go to the local plumbing supply store and buy another chunk of pipe.

  During the eighties I did some construction work. One day, another worker leaned a ladder against the outside of the building that we were putting up, climbed up to the second-story level, and used the Hole Hawg to drill a hole through the exterior wall. At some point, the drill bit caught in the wall. The Hole Hawg, following its one and only imperative, kept going. It spun the worker’s body around like a rag doll, causing him to knock his own ladder down. Fortunately he kept his grip on the Hole Hawg, which remained lodged in the wall, and he simply dangled from it and shouted for help until someone came along and reinstated the ladder.

  I myself used a Hole Hawg to drill many holes through studs, which it did as a blender chops cabbage. I also used it to cut a few six-inch-diameter holes through an old lath-and-plaster ceiling. I chucked in a new hole saw, went up to the second story, reached down between the newly installed floor joists, and began to cut through the first-floor ceiling below. Where my homeowner’s drill had labored and whined to spin the huge bit around, and had stalled at the slightest obstruction, the Hole Hawg rotated with the stupid consistency of a spinning planet. When the hole saw seized up, the Hole Hawg spun itself and me around, and crushed one of my hands between the steel pipe handle and a joist, producing a few lacerations, each surrounded by a wide corona of deeply bruised flesh. It also bent the hole saw itself, though not so badly that I couldn’t use it. After a few such run-ins, when I got ready to use the Hole Hawg, my heart actually began to pound with atavistic terror.

  But I never blamed the Hole Hawg; I blamed myself. The Hole Hawg is dangerous because it does exactly what you tell it to. It is not bound by the physical limitations that are inherent in a cheap drill, and neither is it limited by safety interlocks that might be built into a homeowner’s product by a liability-conscious manufacturer. The danger lies not in the machine itself but in the user’s failure to envision the full consequences of the instructions he gives to it.

  A smaller tool is dangerous too, but for a completely different reason: it tries to do what you tell it to, and fails in some way that is unpredictable and almost always undesirable. But the Hole Hawg is like the genie of the ancient fairy tales, who carries out his master’s instructions literally and precisely and with unlimited power, often with disastrous, unforeseen consequences.

  Pre-Hole Hawg, I used to examine the drill selection in hardware stores with what I thought was a judicious eye, scorning the smaller low-end models and hefting the big expensive ones appreciatively, wishing I could afford one of them babies. Now I view them all with such contempt that I do not even consider them to be real drills—merely scaled-up toys designed to exploit the self-delusional tendencies of soft-handed homeowners who want to believe that they have purchased an actual tool. Their plastic casings, carefully designed and focus-group tested to convey a feeling of solidity and power, seem disgustingly flimsy and cheap to me, and I am ashamed that I was ever bamboozled into buying such knicknacks.

  It is not hard to imagine what the world would look like to someone who had been raised by contractors and who had never used any drill other than a Hole Hawg. Such a person, presented with the best and most expensive hardware-store drill, would not even recognize it as such. He might instead misidentify it as a child’s toy, or some kind of motorized screwdriver. If a salesperson or a deluded homeowner referred to it as a drill, he would laugh and tell them that they were mistaken—they simply had their terminology wrong. His interlocutor would go away irritated, probably feeling rather defensive about his basement full of cheap, dangerous, flashy, colorful tools.

  Unix is the Hole Hawg of operating systems,* and Unix hackers—like Doug Barnes and the guy in the Dilbert cartoon and many of the other people who populate Silicon Valley—are like contractors’ sons who grew up using only Hole Hawgs. They might use Apple/Microsoft OSes to write letters, play video games, or balance their checkbooks, but they cannot really bring themselves to take these operating systems seriously.

  THE ORAL TRADITION

  Unix is hard to learn. The process of learning it is one of multiple small epiphanies. Typically you are just on the verge of inventing some necessary tool or utility when you realize that someone else has already invented it, and built it in, and this explains some odd file or directory or command that you have noticed but never really understood before.

  For example, there is a command (a small program, part of the OS) called “whoami,” which enables you to ask the computer who it thinks you are. On a Unix machine, you are always logged in under some name—possibly even your own! What files you may work with, and what software you may use, depends on your identity. When I started out using Linux, I was on a nonnetworked machine in my basement, with only one user account, and so when I became aware of the whoami command it struck me as ludicrous. But once you are logged in as one person, you can temporarily switch over to a pseudonym in order to access different files. If your machine is on the Internet, you can log on to other computers, provided you have a user name and a password. At that point the distant machine becomes no different in practice from the one right in front of you. These changes in identity and location can easily become nested inside each other, many layers deep, even if you aren’t doing anything nefarious. Once you have forgotten who and where you are, the whoami command is indispensible. I use it all the time.

  The file systems of Unix machines all have the same general structure. On your flimsy operating systems, you can create directories (folders) and give them names like Frodo or My Stuff and put them pretty much anywhere you like. But under Unix the highest level—the root—of the filesystem is always designated with the single character “/” and it always contains the same set of top-level directories:

  /usr /etc /var /bin /proc /boot /home /root /sbin /dev /lib /tmp

  Each of these directories typically has its own distinct structure of subdirectories. Note the obsessive use of abbreviations and avoidance of capital letters; this is a system invented by people to whom repetitive stress disorder is what black lung is to miners. Long names get worn down to three- or four-letter nubbins, like stones smoothed by a river.

  This is not the place to try to explain why each of the above directories exists and what is contained in them. A
t first it all seems obscure; worse, it seems deliberately obscure. When I started using Linux, I was accustomed to being able to create directories wherever I wanted and to give them whatever names struck my fancy. Under Unix you are free to do that, of course (you are free to do anything), but as you gain experience with the system you come to understand that the directories listed above were created for the best of reasons and that your life will be much easier if you follow along. (Within /home, by the way, you have pretty much unlimited freedom.)

  After this kind of thing has happened several hundred or thousand times, the hacker understands why Unix is the way it is, and agrees that it wouldn’t be the same any other way. It is this sort of acculturation that gives Unix hackers their confidence in the system and the attitude of calm, unshakable, annoying superiority captured in the Dilbert cartoon. Windows 95 and MacOS are products, contrived by engineers in the service of specific companies. Unix, by contrast, is not so much a product’ as it is a painstakingly compiled oral history of the hacker subculture. It is our Gilgamesh epic.

  What made old epics like Gilgamesh so powerful and so long-lived was that they were living bodies of narrative that many people knew by heart, and told over and over again—making their own personal embellishments whenever it struck their fancy. The bad embellishments were shouted down, the good ones picked up by others, polished, improved and, over time, incorporated into the story. Likewise, Unix is known, loved, and understood by so many hackers that it can be re-created from scratch whenever someone needs it. This is very difficult to understand for people who are accustomed to thinking of OSes as things that absolutely have to be created by a company and bought.

  Many hackers have launched more or less successful reimplementations of the Unix ideal. Each one brings in new embellishments. Some of them die out quickly, some are merged with similar, parallel innovations created by different hackers attacking the same problem, others still are embraced and adopted into the epic. Thus Unix has slowly accreted around a simple kernel and acquired a kind of complexity and asymmetry that is organic, like the roots of a tree, or the branchings of a coronary artery. Understanding it is more like anatomy than physics.

  For at least a year, prior to my adoption of Linux, I had been hearing about it. Credible, well-informed people kept telling me that a bunch of hackers had got together an implentation of Unix that could be downloaded, free of charge, from the Internet. For a long time I could not bring myself to take the notion seriously. It was like hearing rumors that a group of model-rocket enthusiasts had created a completely functional Saturn V by exchanging blueprints on the Net and mailing valves and flanges to each other.

  But it’s true. Credit for Linux generally goes to its human namesake, one Linus Torvalds, a Finn who got the whole thing rolling in 1991 when he used some of the GNU tools to write the beginnings of a Unix kernel that could run on PC-compatible hardware. And indeed Torvalds deserves all the credit he has ever gotten, and a whole lot more. But he could not have made it happen by himself, any more than Richard Stallman could have. To write code at all, Torvalds had to have cheap but powerful development tools, and these he got from Stallman’s GNU project.

  And he had to have cheap hardware on which to write that code. Cheap hardware is a much harder thing to arrange than cheap software. A single person (Stallman) can write software and put it up on the Net for free, but in order to make hardware it’s necessary to have a whole industrial infrastructure, which is not cheap by any stretch of the imagination. Really the only way to make hardware cheap is to punch out an incredible number of copies of it, so that the unit cost eventually drops. For reasons already explained, Apple had no desire to see the cost of hardware drop. The only reason Torvalds had cheap hardware was Microsoft.

  Microsoft refused to go into the hardware business, insisted on making its software run on hardware that anyone could build, and thereby created the market conditions that allowed hardware prices to plummet. In trying to understand the Linux phenomenon, then, we have to look not to a single innovator but to a sort of bizarre Trinity: Linus Torvalds, Richard Stallman, and Bill Gates. Take away any of these three and Linux would not exist.

  OS SHOCK

  Young Americans who leave their great big homogeneous country and visit some other part of the world typically go through several stages of culture shock: first, dumb wide-eyed astonishment. Then, a tentative engagement with the new country’s manners, cuisine, public transit systems, and toilets, leading to a brief period of fatuous confidence that they are instant experts on the new country. As the visit wears on, homesickness begins to set in, and the traveler begins to appreciate, for the first time, how much he or she took for granted at home. At the same time it begins to seem obvious that many of one’s own cultures and traditions are essentially arbitrary and could have been different; driving on the right side of the road, for example. When the traveler returns home and takes stock of the experience, he or she may have learned a good deal more about America than about the country they went to visit.

  For the same reasons, Linux is worth trying. It is a strange country indeed, but you don’t have to live there; a brief sojourn suffices to give some flavor of the place and—more importantly—to lay bare everything that is taken for granted, and all that could have been done differently, under Windows or MacOS.

  You can’t try it unless you install it. With any other OS, installing it would be a straightforward transaction: in exchange for money, some company would give you a CD-ROM, and you would be on your way. But a lot is subsumed in that kind of transaction, and it has to be gone through and picked apart.

  We like plain dealings and straightforward transactions in America. If you go to Egypt and, say, take a taxi somewhere, you become a part of the taxi driver’s life; he refuses to take your money because it would demean your friendship, he follows you around town, and weeps hot tears when you get in some other guy’s taxi. You end up meeting his kids at some point and have to devote all sorts of ingenuity to finding some way to compensate him without insulting his honor. It is exhausting. Sometimes you just want a simple Manhattan-style taxi ride.

  But in order to have an American-style setup, where you can just go out and hail a taxi and be on your way, there must exist a whole hidden apparatus of medallions, inspectors, commissions, and so forth—which is fine as long as taxis are cheap and you can always get one. When the system fails to work in some way, it is mysterious and infuriating and turns otherwise reasonable people into conspiracy theorists. But when the Egyptian system breaks down, it breaks down transparently. You can’t get a taxi, but your driver’s nephew will show up, on foot, to explain the problem and apologize.

  Microsoft and Apple do things the Manhattan way, with vast complexity hidden behind a wall of interface. Linux does things the Egypt way, with vast complexity strewn about all over the landscape. If you’ve just flown in from Manhattan, your first impulse will be to throw up your hands and say, “For crying out loud! Will you people get a grip on yourselves!?” But this does not make friends in Linux-land any better than it would in Egypt.

  You can suck Linux right out of the air, as it were, by downloading the right files and putting them in the right places, but there probably are not more than a few hundred people in the world who could create a functioning Linux system in that way. What you really need is a distribution of Linux, which means a prepackaged set of files. But distributions are a separate thing from Linux per se.

  Linux per se is not a specific set of ones and zeroes, but a self-organizing Net subculture. The end result of its collective lucubrations is a vast body of source code, almost all written in C (the dominant computer programming language). “Source code” just means a computer program as typed in and edited by some hacker. If it’s in C, the file name will probably have .c or .cpp on the end of it, depending on which dialect was used; if it’s in some other language, it will have some other suffix. Frequently these sorts of files can be found in a directory with the name /src
, which is the hacker’s Hebraic abbreviation of “source.”

  Source files are useless to your computer, and of little interest to most users, but they are of gigantic cultural and political significance, because Microsoft and Apple keep them secret while Linux makes them public. They are the family jewels. They are the sort of thing that in Hollywood thrillers is used as a McGuffin: the plutonium bomb core, the top-secret blueprints, the suitcase of bearer bonds, the reel of microfilm. If the source files for Windows or MacOS were made public on the Net, then those OSes would become free, like Linux—only not as good, because no one would be around to fix bugs and answer questions. Linux is “open source” software, meaning simply, anyone can get copies of its source code files.

  Your computer doesn’t want source code any more than you do; it wants object code. Object code files typically have the suffix .o and are unreadable to all but a few, highly strange humans, because they consist of ones and zeroes. Accordingly, this sort of file commonly shows up in a directory with the name /bin, for “binary.”

  Source files are simply ASCII text files. ASCII denotes a particular way of encoding letters into bit patterns. In an ASCII file, each character has eight bits all to itself. This creates a potential “alphabet” of 256 distinct characters, in that eight binary digits can form that many unique patterns. In practice, of course, we tend to limit ourselves to the familiar letters and digits. The bit-patterns used to represent those letters and digits are the same ones that were physically punched into the paper tape by my high school teletype, which in turn were the same ones used by the telegraph industry for decades previously. ASCII text files, in other words, are telegrams, and as such they have no typographical frills. But for the same reason they are eternal, because the code never changes, and universal, because every text-editing and word-processing software ever written knows about this code.