SDSC’s ‘Comet’ Supercomputer Helps Confirm Gravitational Wave Discovery

Published February 24, 2016

This month’s announcement by the National Science Foundation (NSF) that scientists for the first time detected gravitational waves in the universe as hypothesized by Albert Einstein 100 years ago has opened up a new era of exploration for astronomers and astrophysicists.

The NSF-funded Comet supercomputer at the San Diego Supercomputer Center (SDSC) at the University of California, San Diego, which went into operation in May 2015, was one of several high-performance computers used by researchers to help confirm that landmark discovery before a formal announcement was made.

On September 14, 2015 at 4:50:45 a.m. eastern standard time, scientists detected gravitational waves by both of the NSF-funded Laser Interferometer Gravitational-Wave Observatory (LIGO) detectors, according to the agency’s February 11 announcement. The waves reached Earth from the southern hemisphere, passed through the Earth, and emerged at the Earth's surface, first at the LIGO interferometer near Livingston, Louisiana, and then, seven milliseconds later and 1,890 miles away at the second LIGO interferometer, in Hanford, Washington.

Scientists soon realized they had the tell-tale “chirp” signature of two black holes merging. At the moment of the collision, about three times the mass of the sun was converted into gravitational waves with a peak power outlet of about 50 times that of the whole visible universe, according to the NSF/LIGO announcement.

The collision of the two black holes happened 1.3 billion years ago, generating waves that have been traveling through the universe ever since. The discovery, however, immediately rippled with full force throughout and beyond the astrophysics research community, confirming Einstein’s belief that gravitational waves existed. That belief was an outgrowth of his groundbreaking general theory of relativity, which depicted gravity as a distortion of space and time triggered by the presence of matter.

Until now, scientists had found only indirect evidence of their existence. Einstein correctly thought that by the time the gravitational waves reached us they would be much weaker, so weak no one would ever be able to measure them. That’s only because he likely could not have imagined today’s technology that enables LIGO’s two detectors to detect movements that are a tiny fraction of the width of an atomic nucleus.

The collision of two black holes – an event detected for the first time ever by the Laser Interferometer Gravitational-Wave Observatory (LIGO) – is seen in this computer simulation. LIGO detected gravitational waves, or ripples in space and time, generated as the black holes merged. Time has been slowed by a factor of 100. The stars appear warped due to the strong gravity of the black holes. Image credit: Simulating eXtreme Spacetimes (SXS) project, http://www.black-holes.org

While the production discovery analyses for the actual gravity wave detection last September were run on clusters at Syracuse University and at the Albert Einstein Institute -Hannover, Germany, SDSC’s Comet supercomputer was one of several systems which contributed processing to the analysis and review of the result. LIGO researchers have been using high-performance computers provided through XSEDE, and before that, TeraGrid and the National Partnership for Advanced Computational Infrastructure (NPACI) programs, to study gravitational waves for many years.

“Investments by the National Science Foundation in basic research ‘moonshot’ programs such as LIGO, as well as innovative and computational resources like Comet, are critical in providing researchers with a national cyberinfrastructure comprised of the most advanced tools and expertise needed to solve the grand challenges of our time,” said Sandra A. Brown, Vice Chancellor for Research at UC San Diego. 

“The detection of gravitational waves represents the pinnacle of human achievement,” said SDSC Director Michael Norman, who is also the Principal Investigator for the Comet program. “As an astrophysicist, it is doubly rewarding for me because it underscores the coupled nature of experimental and computational science in solving some the deepest mysteries in the universe.”

“My understanding is that as LIGO prepared to listen to a cosmic conversation, they did not know how loud the first signal would be,” said Frank Würthwein, a UC San Diego physics professor who leads SDSC’s distributed high-throughput computing activities. “But they prepared to have sensitivity to stuff that is a lot quieter.” Würthwein is also executive director of the Open Science Grid, a multi-disciplinary research partnership specializing in high-throughput computational services funded by the U.S. Department of Energy and the NSF. OSG provided access to numerous high-performance computing systems, including Comet, for LIGO researchers to verify the finding.

LIGO researchers consumed almost 630,000 hours of computational time on Comet, using the system’s new Virtual Cluster interface for the analysis of the data. This ‘bare metal’ approach – a term that describes an environment in a computer system or network in which a virtual machine is managed like a physical computer, and performs as well as one – means that a virtual computing cluster looks, feels, and performs almost exactly like the physical hardware, thereby reducing the entry barrier for researchers by letting them project an environment they already know onto Comet.  

“LIGO’s discovery of gravitational waves from the binary black hole required large-scale data analysis to validate the discovery claim,” said Duncan Brown, The Charles Brightman Professor of Physics at Syracuse University’s Department of Physics who studies gravitational waveforms for black hole and neutron star binaries. “This includes measuring how significant the signal is compared to noise in the detector, and re-analyzing the data with simulated signals to ensure that we understand the astrophysical sensitivity of the search.

“Comet’s virtual clusters allowed us to rapidly spin up an Open Science Grid environment that we used to run analysis on Comet,” added Brown. “These compute cycles were extremely important for us to complete large-scale simulations and fast validation of the search.”

“Normally the computing workflows for LIGO are run on its own clusters and its partners,” said Edgar M. Fajardo Hernandez, a programmer analyst in SDSC’s high-throughput computing group. “OSG enabled scientists from the LIGO collaboration to access elastically – meaning increasing and decreasing on demand – computer resources.”

Saul Teukolsky, the Hans A. Bethe Professor of Physics and Astrophysics at Cornell University, has been using Comet since early 2015 for a project named “Gravitational Waves from Compact Binaries: Computational Contributions to LIGO.” Teukolsky’s most recent allocation on Comet is for about 7 million hours of computing time on the system – significantly more than a typical allocation.

“We have been calculating gravitational waveforms for the LIGO project by solving Einstein's equations numerically,” said Teukolsky. “We start with two black holes that may be spinning and put them in orbit around each other. We then evolve the system in time and read off the wave signal that is produced. In the discovery paper, the LIGO team showed a figure with their measured data. Superimposed on top was a theoretical waveform produced by our collaboration, which agreed very well. This is how we really know the waves were produced by two black holes and not something else.”

Each simulation can take from about a week to one month to complete, depending on how complicated the orbit is, according to Teukolsky. “The numerical method we use (spectral methods) is extremely efficient, but doesn't scale well with more than about 100 processors. However, we have to run cases with many different values of the black hole masses and spins in order to model all the possible waveforms, and Comet is ideal for this situation.”

“Nearly all of our work has been LIGO related,” said Teukolsky. “I believe that the theoretical work that I and other researchers do supports the experimental work done by other LIGO researchers in that these wave form simulations produce the kind of templates that they need.”

In addition to black hole simulations, Teukolsky’s research team also run binary neutron star cases. “LIGO hopes to detect those soon,” he said.

“We are proud that Comet is supporting LIGO researchers, and we look forward to making both this innovative system and SDSC’s high-performance computing experts available in the future as we enter this exciting new era of gravity wave astronomy,” said SDSC’s Norman.