Argonne supercomputer goes green

PORTLAND, Ore. — Operators of the supercomputing center at Argonne National Laboratory used innovations in both computer architecture and cooling methods to achieve over $1 million in annual energy savings for its IBM Blue Gene/P supercomputer.

Supercomputers typically consume multiple megawatts of electricity. At 557 teraflops, Argonne's Blue Gene/P is one of the fastest supercomputers in the world, but uses as little as one-third the power consumed by other supercomputers.

"The most important part of how we got 'green' was to work with [IBM] to design a system from the ground up specifically for energy efficiency, but without sacrificing performance," said Peter Beckman, director of Argonne's Leadership Computing Facility. "Our platform uses two to three times less power than other installed systems of similar computing size."

The biggest reason the IBM Blue Gene/P runs cooler is its clock speed, which was reduced to just 850 MHz, about three times slower than typical high-end cores. Power savings were greater than three-fold, according to officials at the Illinois lab. The reason is that the relationship between speed and power consumed is not linear, but exponential. To compensate for slower speed, the system uses many low-voltage parallel cores, in particular 163, 840 cores with 80 terabytes of semiconductor memory and eight petabytes of mass storage.

"By scaling back the frequency and voltage of each of the cores, and then aggregating very large number together into a single machine, we get a supercomputer that is both green and fast," said Beckman.

Each Blue Gene/P board saves energy by housing four cores below a copper heat sink in a system-on-chip that includes all suppport circuitry except its semiconductor memory chips (shown soldered to the board).

The architecture of each supercomputer processor board was designed to reduce energy consumption, Beckman said, by incorporating four cores, networking controllers, memory managers and other logic onto a single SoC, thereby eliminating power-hungry interface circuitry that must be glued together on supercomputer cores.

"By putting everything on a single custom chip, all of the extra power required to buffer and drive links to other chips is greatly reduced," said Beckman.

Argonne Labs also used an innovative cooling system to reduce power consumption. Air conditioners were replaced with fans to move 300,000 cubic feet of water-chilled air per minute to maintain a room temperature of 64 degrees F. The technique uses only 60 percent more energy for cooling than the supercomputer itself draws, compared to over twice the power for typical supercomputers.

"We worked to model the airflow required to efficiently cool the supercomputer, and then built a structure to enclose the hardware and use high-efficiency air handlers to reduce our power," said Beckman. "We also tied into the Argonne chilled water plant, which uses cooling towers to chill the water when the weather is cold. So when it is about 35 degrees outside, we can easily chill water essentially for free, saving about $20,000 to $25,000 a month in electrical costs."

Argonne Labs is currently constructing a new building which will tap into the chilled water plant to circulate water through pipes inside the racks holding the supercomputers. Electrical components will then use mini-heat exchangers to cool the supercomputers chips directly with the cold water rather than air.

The lab also is seeking to reduce power usage in electronic components through smart power management functions that turn off chips and storage systems when they are not being used. Power-intensive computing jobs are also being scheduled to run at night, when temperatures are lower and the power grid has excess capacity.