The Telescope in the Ice

by Mark Bowen

  Jim worked very hard to partner with the foundation and Raytheon. In their regular conference calls, he says, he noticed right off that the manager of science support at Raytheon seemed to think it made him look good to make Wisconsin look bad by pointing out their mistakes. Meanwhile, the NSF folks liked to play the potentate, sitting in the middle, adjudicating disputes. Jim called the Raytheon guy separately and said, “If you ever bring up an issue on that call where you’re pointing at us, then I’m gonna beat the shit outa ya. I’m not gonna let it lie, you know, cuz we gotta solve this stuff before we talk to NSF.” (Unlike Francis, he is not afraid to be the bad guy.) He brought a manager from Raytheon into the IceCube organizational chart, at the level of some of the physicists, and this man participated in IceCube’s weekly meetings: “So, here’s our dirty laundry, you can look at it, and, you know, be part of the team.” It worked.

  In the spring of 2004, the NSF board that is responsible for such decisions recommended that IceCube proceed to construction, in the recognition that the total cost to the foundation over the next seven years would be slightly more than $240 million. This released about $50 million for the current fiscal year. In the memorandum formalizing their recommendation, the board indicated that the recent hiring of “permanent Project Director (PD), James Yeck,” was the most important factor driving the decision, noting that he was “widely recognized as one of the leading project managers in the world.”

  * * *

  Meanwhile, they were sprinting toward their first IceCube season. The optimistic plan was for four strings. Jim hedged his bet and committed to only one.

  20. Failure and Success

  The IceCube physicists may have had something to learn about project management, but they were good at picking people. Francis and Bob Morse had brought two superb additions into the team the year before Jim Yeck entered the picture: Jeff Cherwinka and Jim Haugen.

  Cherwinka had worked at Madison’s Physical Sciences Laboratory for almost ten years in the eighties and nineties, followed that with two years at Cornell as chief engineer on a particle physics detector, and then returned to Madison to work in private industry. When the company he worked for was put up for sale, he ran across a posting at PSL and applied, thinking it would be a return to high-energy physics. The last thing he expected was to be working on a hot water drill. He was hired as principal engineer on the drill in July 2002.

  Not long after he arrived, Morse, Halzen, and Paulos began looking for someone to supervise the making of the digital optical modules, and Jeff recommended Jim Haugen, who had been his boss at his industrial job. The other three resisted at first, since they thought they needed an engineering type, but Jeff convinced them that what they really needed was a production manager.

  Haugen is an astoundingly upbeat and energetic individual who refers to himself as “a lifelong Wisconsin guy.” He graduated from Madison with “kind of like a double major in math/physics with a minor in electrical engineering,” worked in Silicon Valley for ten years, and then lived in Thailand for about five, managing contract manufacturing facilities that could produce millions of electronic chips a day “with twenty different packaging types, to forty different customers, seven days a week, twenty-four hours a day—three million units of stuff, out the door, with the right label, with the right quantity. So, yeah, that was a whole different thing.” He ended up taking over responsibility for everything that would go into the Ice, all the way from two and a half miles down to the connections at the computer farm in the new “counting house” that would be built on the surface—not just the DOMs: the cables and pressure sensors, numerous one-off devices, and so on. This would add up to about one-third of IceCube’s $270 million budget. He may have been overqualified for the task, and that was a good thing.

  * * *

  One of the many enhancements that came out of the years of research and development that had gone into AMANDA was the inclusion of a two-dimensional air shower array named IceTop, which would sit on the surface of the Ice directly above the submerged IceCube array, a mile and more below. IceTop, in turn, was an improvement upon SPASE, the South Pole Air Shower Experiment, which had been installed a couple of years before AMANDA, you will remember, and proved to be unexpectedly useful in the AMANDA effort. Since SPASE could not only detect showers of muons raining down from above but also tell their direction, and since some of those showers were headed toward AMANDA, it provided all kinds of ways of testing and calibrating the neutrino telescope below. It could be used to check AMANDA’s pointing accuracy and angular resolution, for example. Even though SPASE was not optimally placed, because it was offset horizontally, it was used to “veto” a small portion of the down-going air shower background. If SPASE detected an air shower headed in AMANDA’s direction, AMANDA could be programmed not to be fooled by the flood of light it would perceive shortly thereafter.

  IceTop would be significantly more helpful in the way of vetoing, since it would be situated directly above IceCube and cover the entire instrument. (As with SPASE, it would also prove more helpful than the physicists ever imagined.) It would consist of eighty-one “stations,” each consisting of two large tanks of clear ice and each tank containing two submerged DOMs. The tanks would be sealed tightly against incoming light, so that the DOMs would perceive only the Cherenkov light produced by air shower particles inside the tanks (see photographs 20, 21, and 30). This added up to an extra 324 DOMs for Jim Haugen to build.

  * * *

  When he joined the project in March 2003, Haugen discovered that the production of the hardware was about as out of control as Yeck would discover the management situation to be when he arrived six months later. The DOM was the most sophisticated component, and Haugen knew he could not possibly allow it to evolve into the hodgepodge of zoological curiosities that had populated AMANDA. They would need to build 5,500 of them all told, accounting for extras, and the last one installed, seven or eight years hence, would need to be functionally identical to the first. That meant they had to verify and fix the design and build about 350 of them in time for their first drilling season, which was only eighteen months away. Meanwhile, the physicists didn’t even have a parts list! Some guy at Lawrence Berkeley Laboratory kept the parts for the main circuit board on the shelves in his office.

  As a veteran of the semiconductor industry, where there is such an obsession with making products absolutely identical that companies will go to the length of building $2 billion factories in three or four different countries that are exactly the same down to the bolts in the walls and carpets on the floors, Haugen was amazed to learn that the agreement between the different IceCube institutions called for the DOMs to be assembled in three separate locations: the Physical Sciences Laboratory in Wisconsin; Christian Spiering’s institute in Zeuthen, Germany; and Stockholm University. Not only did this present a challenge to making them identical, it would add a lot of cost.

  He walked into Jim Yeck’s office one day and explained that he could easily set up a small plant in Thailand that could produce 5,500 identical DOMs for a labor cost of five to ten dollars apiece. He recalls that Yeck “was like, ‘’Kay, Haugen. Calm down. That’s not how we’re doing it.… Forget that crazy idea.… You’re not in industry anymore. This is a big science project.’” Yeck had just come from the Large Hadron Collider, where on one experiment they were building identical detectors in twelve different countries. It was common, especially in Europe, for different nations and institutions to contribute in kind as well as in cash, and it had the added advantage of keeping them all engaged. Aside from the cultural value, Yeck also didn’t mind the idea of moving the center of gravity ever so slightly away from Madison, even though it would cost in the end about $350 per module or an extra $2 million all told.

  Haugen worked successfully to bridge the cultural gap—and not just with the Europeans; this was an industry versus academia kind of thing. “I’m a bit, ‘’Kay, let’s roll up our sleeves, let’s get ’er goin’, let’s�
�’ And, you know, we had to just kinda take ’er a little bit slowly and back off here, and do some design verification and things like that.” He flew to Zeuthen and held a meeting with Christian Spiering and Rolf Nahnhauer, the “tough German guy” who was in charge of building the DOMs there, and Per Olof Hulth from Stockholm and Allan Hallgren from Uppsala, a longtime AMANDA veteran.

  He explained that together they were going to build 5,500 high-quality, identical DOMs, ship them on time every year, and meet their budget. He also introduced such radical industrial notions as bills of materials and process travelers. He deployed “the force of charm and trustworthiness, and drinking beer and being, you know, mostly right—you’re never right all the time … and eventually people started trusting you.” It didn’t take long for the physicists to realize that the inherently team-based ethic of manufacturing sometimes made it more fun and even more exciting than doing science.

  Haugen held the final design review for the DOM in May 2004, about fourteen months after he arrived, and they managed to get 360 of them onto a ship at Port Hueneme, headed south, in September. This was pretty much the last time the DOM raised the blood pressure of anyone involved.

  Six years later, as construction was nearing its end, he asked Madison physicist Mark Krasberg to do some analytical tests to see if there were any differences between the first DOMs and the last, and Krasberg couldn’t find a thing.

  * * *

  Pressures remained high on the drill team, however. Jeff Cherwinka and colleagues had put together a testing station at the Physical Sciences Laboratory that duplicated the tower and sheave that would be used to lower the hose and electrical cables into the Ice, and they’d dug a deep hole in the ground to approximate the drill hole.

  In the early spring of 2004, when they put the whole thing together and tested it for the first time, they discovered a potentially show-stopping problem with the usual suspect, the hose. With the weight of the drill head and other assemblies that would be attached at the bottom end and the fact that the hose itself was not neutrally buoyant in water, the force of gravity acting to stretch it as it wound off the pulley would come to about 10,000 pounds by the time the full two and a half miles of it had been lowered into the drill hole.

  They had searched the globe for a manufacturer for the hose and finally identified one not far from Padua, Italy. Although they had specified it for longitudinal strength, the engineer from Space Science and Engineering who had been responsible for testing this spec had dropped the ball. When they put the hose under load for the first time in Wisconsin, they could see it necking down and hear it tearing as it rolled off the pulley. It was far too late to redesign it (at a cost of about $1 million a pop), so they had to find a work-around. They ended up transferring some of the load to the metal cable that supported the electrical wires that controlled the drill head by taping the hose to the cable at intervals, using IceCube’s version of duct tape, a fiberglass tape with a silicon adhesive that was affectionately known as driller’s tape. They would strengthen the hose twice in subsequent years, but they’d always keep using the tape for good measure.

  In the end, as I’ve mentioned, they barely got the drill to the pole a year after they had originally planned.

  * * *

  The final technology that was absolutely critical to the building of IceCube was plain old South Pole knowhow, and there was no one in the world better equipped with that than the people who had worked on AMANDA.

  One huge concern was deployment. To the people who actually did that work, says Gary Hill, the prospect of a cubic-kilometer detector “was just absurd, you know, ’cuz just look at us out here on the Ice.… It’s cold!… It just seemed hard to believe that things would come together to do an IceCube.”

  As he had with his “German whips and chains,” however, Albrecht Karle came up with a relatively civilized solution. Gary remembers, sometime in 2000 or 2001, Albrecht taking him out to the large courtyard by the physics building in Madison and outlining a prospective “deployment building” in chalk.

  “He says, ‘Yeah, this is where the hole will be. This is where the DOMs will be stored.’ DOMs being stored? You know, DOMs are in boxes, aren’t they? I mean you take them out and you deploy them. ‘No, no. We’re gonna have the DOMs inside, ready to go, on shelves and things. And this is where the control room will be.’ Control room?… And, you know, essentially that’s basically how it ended up. We have this building, it’s heated, you work inside.… You can basically work in a fairly easy environment in there. You don’t have to dress up too much. I mean sometimes it’s a bit colder near the hole and things, but, yeah. You know, these things all came through.”

  The deployment building was integrated into the drill tower, and the entire unit, which could be sledded around on the Ice, became known as the TOS (rhymes with the goss in gossamer), short for Tower Operations Structure.

  * * *

  Albrecht went south as on-ice lead for the 2004–05 season and, as he puts it, just “someone who could help.” Gary led one of the two deployment shifts; Kurt Woschnagg and Darryn Schneider also worked on deployment; and they all helped in innumerable ways, simply by knowing people at the station, knowing where things were, and knowing how the station worked. There were also several experienced AMANDA drillers, including a good percentage of Swedes. Per Olof and company had been supplying drillers every year since they had joined AMANDA and would continue to do so all through IceCube.

  Two critical people were missing, however: Bob Morse and Bruce Koci. Bob underwent quadruple bypass heart surgery in November, and Bruce, very sadly, had been diagnosed with non-Hodgkin’s lymphoma. These two gave what help they could by telephone or e-mail from Wisconsin, but this went only so far. Their physical presence—and especially Bruce’s—might have made all the difference.

  * * *

  When the drilling team and their many tons of equipment arrived at Pole in the fall of 2004, they came face-to-face with the fact that this drill was an entirely different beast from AMANDA’s. For one thing, it was extremely complicated; it took weeks to put it together. Christmas came and went, New Year’s came and went. And then, when they were finally ready to drill, they learned that NSF had failed to post an environmental assessment on the Web, so they had to wait through a comment period before it would be legal.

  Since South Pole Station is a very small place, this had a negative effect on the drillers’ morale. Everyone was aware of the problems. Jim Yeck remembers all this “macho/ego stuff runnin’ around.” (He’d gone down early to help set up the TOS and get the lay of the land and then returned north to work things from the real world.) The scuttlebutt around the station was that they wouldn’t be able to drill at all that season, and the prospect of complete failure coupled with the fact that the drillers found themselves without a whole lot to do in a place that didn’t offer much in the way of entertainment made for a corrosive mixture. So Jim and Jack Lightbody at NSF came up with the idea of a “pilot hole.” It wouldn’t be official construction, just some testing, but they could use it for deployment if it turned out to be acceptable.

  The water finally began to circulate in their gigantic contraption on roughly the tenth of January, only about a month before the station would close for winter. Albrecht, whose job was to keep an eye on the entire situation, compares the experience to building a large airplane and finally starting it up, rumbling down the runway, and lifting off for the first time. You’re in the air; you can’t just pull over and park. “It was like, ‘Oh my god! We’re drilling! I guess we’re drilling, yes…’”

  It was clear from the start that something was wrong.

  There is a truism in engineering that when something goes wrong, it’s usually more than one thing. One problem was that the water pumps kept shutting off. On the Ice, they thought it was the computerized motor drives on the pumps, but afterward they realized it was actually the pumps themselves, which weren’t spec’d for the 10,000-foot altitude. They would
overheat and send a signal to the drives to shut themselves off. This had a ripple effect: when the water stopped flowing, the water boilers would overheat and shut themselves off. By drilling more slowly than they had planned, the drillers could limp along with this handicap. But it wasn’t the only thing.

  This is where one can’t help but think that if Bruce Koci had been there, things would have gone very differently. There is no doubt that the man on the ground that season, Jeff Cherwinka, is an excellent engineer. He eventually solved every problem and became a master of the art. But at that point Bruce was far and away the best field engineer and troubleshooter in the game. He had an uncanny ability to sense what was happening with a drill and in the ice a mile beneath his feet, while Jeff had never drilled ice before.

  Everyone seems to use the word oscillation to describe what was happening down where the action was, at the drill head. It would surge down into the ice, melting itself a pathway, and then rise back up by as much as forty feet. This chaotic motion made for a very poor-quality hole. What one shoots for in hot water drilling is for the water spraying out of the nozzle at the bottom end of the drill head to melt the ice at just the right speed to match the rate at which the head is being lowered. This allows it to fall in a plumb line under the force of gravity and melt out a straight, dead-vertical hole of constant diameter. It seemed that they were lowering the head too rapidly even at their handicapped rate, but it was hard to tell exactly how rapidly, because the many sensors they had put in place to measure the rate were sending back different readings. They struggled along for about a day, only reaching a depth of about a thousand meters—slower than they had managed with AMANDA—and finally put their heads together and decided to abort.