by Mark Bowen
As I have mentioned, there was still no working DAQ or experiment control software four years later, in January 2005, when the first IceCube string was installed. Albrecht Karle had seen this coming and asked John Jacobsen, who was consulting with LBL but living in Chicago, to lead a group that would try to get something working on the Ice, and John had succeeded.
LBL is an extremely expensive place to do business with. By 2005, more than a dozen engineers had been working on the IceCube software for several years, at a cost in the neighborhood of $3 million per year, yet, amazingly, Jacobsen didn’t even use LBL software as the basis for his seat-of-the-pants code for the first string. He modified some code that had been developed at Madison and the University of Pennsylvania for the purpose of testing the DOMs before they went into the Ice.
The main architect of that software was a research scientist at Madison by the name of Kael Hanson. Kael had done his graduate work at the University of Michigan on MACRO, a neutrino detector in a tunnel under Gran Sasso, the highest mountain in Italy’s Apennine Alps. He had moved to Penn in 2000, to do a post-doc under professor Doug Cowan, who had gotten involved in AMANDA in the late nineties. So Kael was originally an AMANDA guy, and he has the adventurous approach to physics—and life, apparently—that goes with the territory. He relates that when he was down on the Ice in 2002, Bob Morse and Albrecht Karle offered him a job in Madison. He got back to Philadelphia, packed up his family (he and his vibrant wife, Gloria, have five children), and drove to Wisconsin—only to find that Bob and Albrecht had forgotten about the offer!
“It was at Pole we made all of the arrangements,” he writes. “Lack of oxygen, easy to see how things got messed up.”
They found him a job in a couple of weeks.
Kael had a knack for electronics and software. When IceCube came into being, he was put in charge of testing the DOMs, and this led to his becoming manager of all the various gadgets that went into the Ice.
By the end of the second, successful season, when they deployed eight strings, LBL had managed to produce some semi-functional DAQ software, but none for experiment control. Again John Jacobsen had to hack something together, and this brought the situation to a head. Francis, Albrecht, and Jim Yeck got together and decided to promote Kael to co-manager of software development, on a par with the existing manager at LBL, who soon decided to leave the project in order to work on a neutrino experiment at Daya Bay, China.
In typical Jim Yeck fashion, Kael was dropped in at the deep end of the pool. He now “owned” a project that had burned through almost $20 million with little to show for it, employing about fifteen people dispersed around the country who didn’t get along very well. Although he was guaranteed all the material support he would need and Jim and Francis ran political cover for him, he found that no matter what he tried, he could not get the group to team up and produce working software. As the Antarctic season approached, his nerves began to fray. He was having a hard time sleeping, and his health was deteriorating.
In September, scant weeks before the season would begin, he had what he calls “a vision.” He realized that no matter how much money he spent and no matter how many people he hired, the situation would not improve. “The inspiration hit almost overnight,” he writes. “I can understand the Muse concept—strange how it felt like it wasn’t coming from my brain, but I was just the channel.” He decided to lay off all but four people: himself, John Jacobsen, a programmer in Madison named Dave Glowacki, and another at LBL named Keith Beattie. Some still remember this as Kael’s “DAQ coup.” The cuts were everywhere, even in Madison, but LBL felt the most pain, since it lost the most people, and, needless to say, Kael incurred a lot of resentment. Collaboration-wide, however, most people realized the deed had to be done.
Kael, John Jacobsen, and Dave Glowacki traveled south in January 2007 with what Kael describes as “something really rough” to try on their new twenty-two-string array. Dave was the DAQ guru, John worked on experiment control. The software was so rough that as the countdown to station closing began, the instrument was crashing every few minutes and it was looking as though it might go through its entire first year of being large enough to take data without taking any at all. Things got so desperate that they considered buying a few days by missing the last LC-130 to McMurdo and catching a Twin Otter north instead. This is a smaller plane that can land in colder conditions. There are usually one or two Twin Otter flights after the last 130. They are operated by a contractor out of Calgary, Alberta, named Ken Borek Air, so they tend to route north toward Canada through Punta Arenas, Chile, rather than New Zealand.
At the very last minute, John figured out a way to make the instrument restart automatically every time it crashed. He remembers that they looked at each other and said, “Don’t look at the screen again. Let’s go!” and ran out to catch the last 130.
He had managed to get the skeleton of the experiment control system that they still use today working pretty well. Back home, they continued to improve the software, uploading updates via satellite. By April or May the instrument was capable of surviving for several hours at a time. It got steadily better from there.
To Francis, this debacle demonstrates that if NSF had decided to run IceCube out of LBL rather than Madison, the instrument probably wouldn’t have been built. They would have run out of money when it was about the size of AMANDA. “Because remember how the game is played, right?… Your budget is basically the size of your detector.”
In contrast, once IceCube recovered from its first season and had the wind at its back, Jim Yeck began to work the other side of the coin, searching for ways to save money in order to buy back scope. Remember that in the early budget negotiations with NSF he had backed off to seventy strings from their original goal of eighty. Now he wanted them back.
“I’ve had enough project experience to know that one way you save money is you spend money, depending on what you spend it on, right?… Like you spend it on the drill. You put more holes in the Ice per year, you have one less season, you save millions of dollars. Some people think that the way you save money is you don’t spend any. That’s not true in a project, because schedule will most likely be the biggest decider in what the cost is. If you drag it out, you’re spending more money. So think schedule, think spending money to make schedule. That’s your starting point.”
Over the 2009–10 Antarctic summer, which would have been their last according to the baseline schedule that Jim had laid out in 2004, they deployed an astounding twenty strings against a target of twelve and brought the total in the Ice to seventy-nine. And they still had money left for one more season.
This allowed them to dream. Not only could they add more strings, they could expand their scientific reach.
* * *
Even in the 1990s, before Super-K discovered neutrino oscillation, the AMANDA physicists had been imagining ways to study oscillation themselves. As with Super-K, they would employ the atmospheric neutrino beam, and the idea would be to watch neutrinos change from one flavor to another as they traversed the Earth. Since IceCube views the entire northern sky, the neutrinos it detects travel different distances through the planet in order to reach it. Those arriving from the direction of the north pole traverse the longest distance, and those from nearest the horizon the least. Since the chance that a neutrino will oscillate increases the farther it travels in dense matter, different flavor mixtures will arrive at IceCube from the different directions.
This is pure particle physics, as opposed to astrophysics. The Super-K discovery has spawned a global, multi-billion-dollar effort to examine every detail of oscillation, mainly because, as I’ve said, it’s the only physics so far discovered that violates the standard model. And the IceCube physicists have carved out an exclusive niche for themselves, since they can study neutrinos at higher energies than any other experiment—orders of magnitude higher than any neutrino produced by an accelerator, or detected by Super-K, since that instrument’s Greisen-type des
ign limits it to lower energies. And higher energies hold particular interest, because the study of oscillation is basically the study of the weak nuclear force, and anomalous behavior in weak interactions, if there is any, is more likely to occur at higher energies.
I’ve also pointed out that Per Olof Hulth had a playful approach to physics. He was also in a good position to take advantage of that inclination, since he wasn’t as bogged down in the project management as, say, Francis Halzen was. When it began to look as though they’d have some room to play with IceCube, he came up with a proposal for making it better at measuring oscillation and several other things.
Enough was known about oscillation that they could run some well-informed Monte Carlos in order to design an optimum configuration, and what they came up with was a more densely packed sub-array, in which the strings would be closer together and the DOMs on each string closer to each other than in the standard IceCube array. This would open up an intermediate energy range, between those of Super-K and IceCube. It would also make the instrument more sensitive to low-mass WIMPs in the Sun and galactic center and to supernovae (areas of study that were actually more interesting to Per Olof—and Francis—than oscillation).
Buford Price, who has a similarly creative approach to science, had been leading an increasingly sophisticated study of the ice at the South Pole for more than a decade by then. One of the features his group had identified is a relatively dense layer of dust at depths of 2,000 to 2,100 meters, which was laid down during a particularly cold, dry spell at about the mid-point of the most recent ice age, roughly 65,000 years ago. To avoid the scattering and absorption of light in the dust layer, the collaboration decided to locate Per Olof’s sub-array below it, at the deepest levels of IceCube, where the ice also happens to be the clearest. This sub-array was an in-fill, in the center of the IceCube array: it incorporated several standard IceCube strings as well. Per Olof obtained a grant of $9 million from the Knut and Alice Wallenberg Foundation in Sweden, a longtime supporter of AMANDA and IceCube, in order to fabricate the six specialized strings for the sub-array. They decided to name it DeepCore.
A schematic of IceCube at its completion in December 2010.
IceCube Collaboration
The first DeepCore string was deployed in 2009–10, the year they brought the total to seventy-nine. The following year, 2010–11, their last year of deployment, they dropped the last five DeepCore strings into the Ice along with the last two standard IceCube strings and brought the total to eighty-six, eighty of which were pure IceCube, their original goal. Jim Yeck had gotten his scope back and more.
* * *
The final season was a victory lap—easy after the previous year’s twenty strings. The seventh string of the season and the very last for IceCube was deployed on December 18, 2010, exactly a week before Christmas—quite a contrast from my millennial season, when we didn’t deploy the first string until December 19.
Per Olof had come up with another playful idea, and that was to attach a video camera to the bottom of the string—they called it the Swedish Camera—in order to see what the ice looked like going down and watch the water refreeze after drilling. The second purpose was of special interest: they wanted to get an idea of how many bubbles formed as the ice grew back and find out what other strange things might be happening. (They discovered that a column of bubbles forms in the center of the hole and that the optical modules are pushed to the side of the hole as the ice refreezes. The modeling of the instrument has become so detailed that these features are now incorporated into their Monte Carlo and data analysis software.)
Kurt Woschnagg remembers “a Christmas-y feeling” during that last deployment. He just happened to be around when people started to assemble and decide who should take the various roles. Kurt usually did something in the background like monitoring the depth of the string, but for the first time ever, he was assigned to “position one.” He’d be one of the people actually attaching the DOMs to the cable, and he was very happy to be where the action was on that special day. Gary Hill also made sure to be there the entire time. Since he led one of the deployment teams, he kept himself free of specific tasks in order to keep an eye on the big picture.
It took awhile to hook up the camera, and it wasn’t that fine-tuned a deployment team to begin with, plus they wanted to savor the experience. They had set up a video link so that their colleagues in the real world could tune in, and Gary would wander into the room where that camera was every once in a while to commentate. People from all over the station were sticking their noses in just to watch. “Everyone’s in a good mood and it’s progressing nicely,” remembers Kurt. “Every now and then we would stop, and people [would] sign DOMs and … take pictures.” Robert Schwarz dropped by. So did an old friend named Paul Sullivan, who had wintered with Robert and Gary in 1997.
A Swedish crew was operating the Swedish Camera, so Kurt, who had grown up in Sweden, joined them after he finished his shift at position one. Ryan Bay, a protégé of Buford Price who had now converted more-or-less completely from particle astrophysics to climatology, was raptly monitoring the camera’s display for the layers of dust and volcanic ash that were signatures of specific “time horizons,” known geologic events of the distant past, as the camera dropped past them. Since the drillers had already finished their work, they were drinking champagne.
As the end approached, the TOS became very crowded and even more festive. A silence fell as they brought the last DOM into the room and attached it in a slow, ceremonial way. They all chanted, “DOMs away!” in unison and started the drop.
Gary remembers feeling that he’d blinked and IceCube was over. It seemed that he’d been in the same room deploying the first string just the other day. While AMANDA brought unexpected adventures every season, on IceCube, he says, “Every year down there was almost as if you’d turn up on the plane, you hop off the plane, you walk into the station, and then you’d see the same people, and by that time it felt like you’d never left. You’d get into your room; even if it was a different room to other seasons, you know, in a few minutes you felt like you’d never left the place.… [You’d] set up the drill camp, see all the drillers, a lot of them the same from year to year, go through a few disasters here and there with the drill where every year, you know, things would go wrong. And so my actual deployment experience at South Pole is sort of one continuous time down there, right? It all gets compressed into, ‘Well, we did the first string, and, Oh!, now it’s the last one.’ …
“And then,” he concluded in low storytelling voice, “as quickly as it all began, it was all over.”
“There were speeches,” remembers Kurt. “It was just a—that was a fantastic feeling actually, when that was going on. It was just fun. We didn’t really want to—hah!—want to see the last one disappear. We really wanted to extend that moment.” (See photograph 31.)
A big celebration had gotten under way outside, but Kurt declined the champagne and stayed inside to help monitor the drop: “That I really wanted to do. I wanted to be part of those who positioned the last string at its final depth.”
When it was all done, they sent a message to Francis: “We made the impossible look easy.” He posted it on the door to his office. “That,” he says, “is my summary of the construction of IceCube.”
* * *
The strange thing about the end of construction was that it made the instrument less of a tourist attraction at Pole. When you could show people the drill and the drilling and the hole and the cabling and the DOMs lined up on their shelves, all the distinguished visitors, DVs, wanted to stop by for a tour. When it was done, there was just this funny-looking building filled with computers, sitting on the Ice. Jim Haugen, who was usually the one who showed the DVs around, had seen this coming and secretly shipped eighty-six flags to the station, three feet wide by eight feet tall. Eighty were bright orange to mark the IceCube strings; six were red for DeepCore. There was already an array of small black flags out there markin
g the IceTop tanks.
The eighteenth was a Saturday. At about six p.m., when the last string was positioned and tied off for freeze-in, most people walked back to the station to continue celebrating and enjoy their Sunday off. Jim had been awake for about twenty-four hours at that point, and so had Gary, who had found himself a spot on an empty DOM shelf and taken a nap. Sven Lidström and a couple of other drillers also stayed out on the Ice in order to keep an eye on the quiescent drill, which hadn’t been put to bed yet. There was still water in the hoses that had to keep circulating.
Jim pulled the flags from their hiding spot and revealed his plan. Then he, Gary, Sven, and one or two other drillers “went out under the dead of a blizzard,” says Gary, “and went around—hah!—this whole bloody array for several hours, zhzhzhzhzh, with a drill and sticking these damn flags in.” They couldn’t see each other for the blowing snow, so they wandered around randomly in the huge hexagon, somehow managing to fill the whole thing in.
“So when people woke up for Sunday breakfast? They looked out and now what you saw was this beautiful, giant field of eighty-six flags,” says Haugen. “Eighty orange flags and six red flags for the DeepCore flapping in the breeze where twelve hours ago there was nothing. So it was cool.”
John Jacobsen’s pattern in recent years had been to fly to the Ice sometime in January, after the hardware was hooked up, get the software up and running with the new year’s crop of strings, and teach the winterovers how to run the instrument. Each year’s instrument was named for its size, so the previous year’s, which comprised seventy-nine strings, had been known as IC79. On the eighteenth of January 2011, John broadcast an e-mail under the title “IC79 est morte, vive IC86.”