The Telescope in the Ice

by Mark Bowen

  Dave was reintroduced to neutrino astronomy by a charming and highly regarded, somewhat off-beat particle physicist by the name of Kwan-wu Lai. The Km3 concept, which was the focus of the meeting Dave eventually convened in Arcadia, California, was originally Kwan’s idea. The plan was to build a kilometer-scale neutrino telescope in a deep trench off the California coast, near Santa Barbara.

  During a sabbatical at the Jet Propulsion Laboratory in 1993 or so, Kwan gathered a group to look into the Km3 concept and invited Dave Nygren to join. For a while there, Dave was flying down from Berkeley once a week. His interest was in developing technologies basic to the entire concept of large Cherenkov telescopes, rather than any specific instrument. At one of the group’s weekly meetings, he suggested taking a “phase neutral” approach, that is, developing technologies applicable to both water and ice. His desire to acquaint himself with all the technologies then being employed in DUMAND and AMANDA was part of his motive for convening the Arcadia meeting in the spring of 1994.

  A few months after the meeting, he experienced a moment of serendipity as he sat in on what he calls a show-and-tell session for a master’s thesis by an electrical engineering student at Berkeley named Stuart Kleinfelder. Kleinfelder had “designed a chip that could swallow a waveform with high frequency components and capture it at high speed, but then read it out more slowly,” Dave explains. “This is not the kind of chip that you would use for video, because something’s happening all the time. But in something like neutrino astronomy, not much of anything is happening most of the time. So if something interesting happens you can capture it and then process it at low speeds.… This saves a lot of power.” Saving power is an important consideration when you’re running thousands of modules in a remote location, a mile and more out of reach.

  Since Kleinfelder’s chip would turn the pulse from a phototube into a string of numbers that could be sent to the surface digitally, it would eliminate the dispersion problem altogether. And it would turn out that digital signal processing had manifold other advantages that even Dave Nygren didn’t appreciate at the time.

  “So the whole thing started with the random occurrence of seeing a chip that Stuart Kleinfelder had designed for no purpose other than getting a master’s thesis,” he says. “I saw this as a truly disruptive technology that could and should be explored.… It was the hope that I saw to really get into this business, so to speak, and do it right.”

  At Dave’s urging, the Km3 group at the Jet Propulsion Laboratory managed to swing down $100,000 for the design and construction of two “digital optical modules,” or DOMs, incorporating a phototube, Kleinfelder’s chip, and an on-board “computer-on-a-chip.”

  By the summer of 1996, Shannon Jackson, an undergraduate on a summer research fellowship at JPL, was assembling the DOMs. That same summer, Dave asked Jerry Przybylski, a crack engineer at his own employer, Lawrence Berkeley Laboratory, if he would consider a trip to the South Pole the following winter in order to install them. Jerry managed to persuade his wife to let him go by promising her a vacation in New Zealand after his deployment, and hopped on board. Thus the DOM became at least the fifth creature in the increasingly complex zoology of the AMANDA optical module.

  The Km3 group petered out the following year without spawning a working collaboration.

  * * *

  Another area in urgent need of improvement was deployment: the physicists were suffering out there. It was far too stressful and taking far too long a time to drop their strings of pearls into the ice.

  Albrecht Karle left his first deployment season, AMANDA-B4, in the firm conviction that something had to be done about the so-called breakout scheme. Each string consisted of two cables, essentially, one for the electronics and the other a strong metal cable that acted as a sort of backbone to prevent the string from stretching. The problem was that over the length of a string the electrical connectors and the mechanical attachments to the metal cable inevitably “walked away” from with each other. As the deployment team attached each successive module, the mechanical and electrical connections would get farther and farther apart, and picking up the slack became a time-consuming, seat-of-the-pants affair that led to cold fingers, since some of the connections required bare hands, and long deployments, which threatened the viability of a string. It was impossible to account for the differential stretching of the two cables ahead of time. There had to be some way of adjusting for the mismatch in the heat of battle.

  During his first off-season, Albrecht devised a breakout scheme that solved the match-up problem altogether. Bob Morse refers to it as “the German whips and chains.” The basic idea is to attach chains to the metal cable at the desired distance between modules and pick up the slack with the electrical cable by attaching each module to whichever link on the chain happens to match up with the electrical connections. The key to this arrangement is an item known as a chain clutch sling, a handy little gadget that Albrecht ran across in a cable and rigging store that happened to be located next door to his favorite movie theater in Berlin.

  Albrecht was willing to take on almost anything in those days: “As you’re younger, you have this unlimited amount of confidence,” he says. Thus, when Pat Mock announced at the spring collaboration meeting that he would be stepping down as deployment manager, Albrecht agreed to take his place. One of his conditions was to ask the principal investigators to sign off on an organizational chart that gave him, a lowly post-doc, the authority to make tactical decisions on the Ice. He wanted to ensure “that there was no risk that more senior people would put pressure on me at the pole, possibly by e-mail.… To me it symbolized the transition towards organization.”

  He got his mandate, but any organization it may have instilled was symbolic at best.

  * * *

  It’s not clear what set Steve Barwick off that year. Perhaps Wisconsin finally stood up to his annual attempt to wring extra money out of them or NSF didn’t give him as much as he wanted either. In any event, sometime that summer, he went on strike. He stopped answering phone calls and e-mails and refused to do any work on the project.

  This may have been the worst instance of it, but in fact Steve went on strike about once a year. The previous year at Pole, for example, when the drill kept shorting out and Albrecht helped fix it, Steve had retreated to his room to read books for a week. It wasn’t anger exactly, it was a sort of panic that set in whenever the pressure built.

  This strike presented more than the usual challenge, since he’d placed himself in the strategic position of integrating the string cables and transporting them to Port Hueneme for their sea voyage to Antarctica. Cable integration was usually carried out in the warehouses of a company on the outskirts of San Diego, and Steve usually acted as straw boss and sent a few of his students down to help. This year, he held his students back, too. Bob Morse had to put together a group of replacements to fly to San Diego and get the job done.

  That was the first forest fire of the 1996–97 campaign.

  * * *

  PICO’s problematic head driller of the previous year had been replaced by a member of his crew named Tim Macovicka, who got on well with Bruce Koci and Albrecht. So the drilling went well, but that was about it.

  Albrecht remembers that “everything that could possibly go wrong did go wrong.” His own season started out badly the very day he reached Pole, when he had trouble adjusting to the altitude. The station doctor decided (incorrectly, in Albrecht’s view) that he was showing signs of high-altitude pulmonary edema—a potentially fatal condition that can be resolved only by dropping in altitude—and sent him back to McMurdo for a few days.

  Despite Macovicka’s improved leadership, it took a while to wake the drill up from its winter hibernation, so the first hole wasn’t completed until mid-December. And so, the first deployment began. The German whips and chains notwithstanding, this was still a madcap affair. There was only one shift, made up of whatever students and post-docs happened to be on the Ice; ther
e were no engineers or trained technical staff; it was carried out in the wide open; and it was all hands on deck until the job was done.

  They had attached all the optical modules to the first string, one by tedious one, and were lowering it to its final resting place at the bottom of the hole, when about a thousand meters down, the pressure sensors began telling them that the string might be stuck: the readings weren’t changing even though the winch was clearly lowering it into the hole. They pulled it up a few meters and lowered it back down, and it stuck again. After a second try, Albrecht consulted Bruce and they decided to pull it out altogether. For whatever reason, it seemed that the hole had necked down at that depth.

  The deployment team had no choice but to keep working, even though they’d been awake for about thirty hours. Several were falling asleep in place, and one or two were so exhausted that they had to be sent back to their berths. They figured out a way to “un-deploy” on the fly, pulling the dripping, ice-covered modules and cables out of the hole, somehow scraping the ice from the electrical components, removing the modules with bare hands, sticking them back into their cushioned boxes, and stowing them in a Jamesway where they could thaw and dry out.

  The deployers got some sleep while the drillers reamed out the hole, and they finally deployed the first string on Christmas Day, thus missing the traditional South Pole Christmas banquet. They ate their holiday meal sitting in the Jamesway with their optical modules.

  “So that was my very profound initial experience for how string installation can go way different from what you expect,” Albrecht remembers. “And how … this whole operation … it’s tough.”

  John Jacobsen, the artist/physicist who had helped demonstrate AMANDA’s sensitivity as a supernova detector, had recently completed his doctoral thesis and was now employed as a post-doc in Madison. He made his first trip to Pole that season and remembers his initial impressions vividly.

  When their LC-130 taxied to a stop, John and his fellow passengers stepped down onto the snow to find no one to greet them or tell them where to go. He wandered around in the whiteness for a while, eventually found his way down the snowy ramp into the dome, looked up at the crapsicles hanging from the ceiling, and said to himself, “You’ve got to be fucking kidding me. I cannot believe I have to spend the next month and a half here!” Then he entered the dingy galley to find Albrecht sitting alone at a table, looking as though “he’d just survived the bubonic plague.” The second deployment had ended a few hours earlier.

  John was also hit hard by the altitude. After a briefing by the station manager that he hardly took in, he struggled across the Ice to his cubicle in Summer Camp, lay down on his bunk, and at the first pangs from his bladder realized that his room was missing an essential appointment: the piss can. This is no joke. It’s quite an ordeal when you’re suffering from the listlessness, headaches, and nausea of acute altitude sickness to “get into what basically amounts to a space suit sans oxygen supply every time nature calls,” as he e-mailed to some friends back in the real world, and trek a hundred yards across the Ice to the men’s room. To compound the problem, the best thing you can do to combat altitude sickness is drink lots of water. On one of his early forays to the plywood shack that housed the toilets, he had the good fortune to meet a guy who told him he could get his very own piss can in the galley. Standard issue was an empty industrial-sized tomato can.

  Being an AMANDA “beaker,” aka scientist, he was pulled in on deployments whenever they came around, but his main job was to write software, which he did in a small, cluttered, oblong room filled with computers, known affectionately as Back of Science. It was located in one of the inverse refrigerators inside the dome.

  John approached the writing of computer code in the same way he approached painting: as an art form. In fact, he was beginning to realize that he was more interested in that aspect of his work than the physics. His thesis had consisted mainly of Monte Carlo simulations, and he also liked to work “close to the metal,” as he put it, at the lowest level of hardware: the chips and circuit boards that actually ran the AMANDA machine. He’d been writing code for about ten years and was becoming an adept.

  He ended up going back to Pole about fourteen times. The following year, he spent many happy hours in Back of Science, listening to music (especially David Bowie’s Outside) and writing a program he named polechomper, which took the information directly from AMANDA’s data acquisition system, stored it on a tape drive at Pole, and also sent it by way of a NASA satellite to a bank of computers at the White Sands Missile Range in southern New Mexico, where a second of his programs, sableblanc, forwarded it to a computer at Lawrence Berkeley Laboratory. There it was made available to the full collaboration on the Web. Prior to polechomper and sableblanc, AMANDA’s data was stored only on tapes, which had to be hand-carried north once a year.

  This seems like a good moment to consider the contrast between working at the South Pole and working at an accelerator laboratory in the real world, like Fermilab or CERN. Not only would John have been living in a comfortable apartment, which he could have stayed in year-round, a bank of supercomputers would have been attached directly to his detector. The primary data from this experiment, on the other hand, needs to be sent through an extremely narrow pipeline, low-bandwidth satellites that are above the horizon for only a few hours a day, in order to reach adequate computers. And this pipeline isn’t even capable of transmitting all the raw data. The AMANDA and IceCube scientists have had to invent clever ways of filtering and compressing their data at Pole—hopefully without missing anything—since there is a hard limit on how much they can accumulate in any given time period. This severe challenge, which usually goes unmentioned, is probably unique in high-energy physics.

  John’s code was not only effective, it was beautiful. About a year after he wrote polechomper, he received the following e-mail from Madison grad student Tyce “Ty” DeYoung, who was working to extend the code and must have been at Pole at the time.

  Subject: You are a god

  Hi John,

  I’ve been looking over polechomper—mere words cannot express how grateful I am that you write such clear code. You are like a god to me.

  Slightly punchy,

  Ty

  They were an inventive and humorous bunch. The password for one of the early AMANDA Web sites was “cheapass”: Center for High-Energy AstroPhysics at Southpole Station.

  * * *

  Jerry Przybylski arrived at the station a week or two after John did, having stopped in Pasadena on his way south from Berkeley in order to pick up the two digital optical modules at the Jet Propulsion Laboratory. He packed the DOMs, a couple of desktop computers, and some troubleshooting equipment into plastic cargo boxes, about three hundred pounds altogether, and carried them south as checked baggage.

  Jerry remembers his first few minutes at Pole as being “very dramatic, sort of like checking into the planet Hoth from Star Wars V.” Once he had walked clear of the wings and whirling propellers of his plane, he became disoriented, because all he could see was white. He had seen the dome from the air, but from the ground it was hidden by recently plowed snow and the drifts that had been slowly burying the structure in the twenty-plus years since it had been built.

  Albrecht gave him a corner in one of the Jamesways out by the AMANDA drill shack and arranged for a 10,000-pound spool of cable, one of the strings, to be parked on the Ice beside it. Jerry found a way to attach the DOMs to the cable, and in such crude conditions tried to get them working between interruptions for deployments and “cable drags”: once a string was deployed, the top end was attached to a surface cable that had to be dragged across the Ice to a building on the surface known as the Martin A. Pomerantz Observatory, or MAPO, where it was attached to the computers and other electronics that ran the instrument.

  Jerry describes Pole as being a “very unfriendly environment” for working with electronics. The dry conditions and constant wind generate an enormous amount
of static electricity, so you can destroy chips and other electrical components if you don’t discharge the static on your body now and again by touching a grounded piece of metal. One day, he was simply walking past one of the DOMs when a spark jumped through the air to it from one of his hands. By a small miracle it wasn’t destroyed.

  When he attached the DOMs to the cable, he found that they didn’t work and he had some sophisticated troubleshooting to do—a task that was additionally complicated by the difficulty of communicating with Shannon Jackson and his software expert back in Pasadena: e-mail only works when the communications satellites are above the horizon. Jerry had to rewrite the so-called firmware in the computer-on-a-chip that ran the DOMs, and that meant “burning” the new firmware into new chips. He had to depressurize and unseal the glass Benthospheres that housed the electronics, replace the offending chips, reseal the spheres, and pump them back up to pressure. Then he discovered a fatal bug in the software that ran the DOMs from the surface. He solved that problem, hung around to deploy the DOMs and run tests from the surface, and escaped Pole just in time to rendezvous with his wife in New Zealand. It was a frantic experience all around—as the DOM would be from beginning to end.

  He must have felt strange leaving his babies behind, since he wouldn’t have any idea how they performed for about ten months. Over the winter, their data was stored on optical disks that didn’t reach his hands until someone found the time to put them on a northbound Herc near the start of the following season. Although one of the DOMs died a couple of months after it was deployed (Jerry never knew if it was the one he’d zapped), they produced just enough data to keep the project alive.