SDSC and Sylabs Spread the Word on Singularity

Published March 27, 2019

The San Diego Supercomputer Center (SDSC) at UC San Diego, and Sylabs.io recently hosted the first-ever Singularity User Group meeting, attracting users and developers from around the nation and beyond who wanted to learn more about the latest developments in an open source project known as Singularity. Now in use on SDSC’s Comet supercomputer, Singularity has quickly become an essential tool in improving the productivity of researchers by simplifying the development and portability challenges of working with complex scientific software.

Released in 2015, Singularity is an open source-based container platform designed for scientific and high-performance computing (HPC) environments where reproducibility is essential for scientific research. Singularity containers provide the means to package an application along with all of its dependencies and environment – the binaries and libraries in its toolchain, configurations, and data – everything except the operating system kernel. By explicitly excluding the kernel, the remaining runtime can be encapsulated within a single file container known as the Singularity Image Format (SIF).

Because Singularity containers are simple, fast, secure, and emphasize integration over isolation, they are a good fit for high-performance computing and enterprise performance computing use cases that routinely involve special-purpose devices such as GPUs, InfiniBand interconnects, and parallel file systems.

The March 12-13 event’s keynote speaker was Sylabs Founder and CEO Gregory Kurtzer, who pioneered the computer program that performs containerization while at the Lawrence Berkeley National Laboratory.

“SDSC has been a staunch supporter of Singularity since the project started in 2015,” said Kurtzer, noting that its adoption “has progressed to the point that literally thousands of compute-intensive applications and workflows are routinely containerized via Singularity.”

The Singularity User Group (SUG) event, sponsored by Lenovo, attracted an extended user community that included advanced computing staff from Indiana University, the Pittsburgh Supercomputing Center (PSC), as well as national laboratories including Argonne, Fermilab, Lawrence Livermore, Oak Ridge, and Sandia.

Industry attendees came from AMD, Ellexus, Intel, Lenovo, Mellanox, NVIDIA, and VMware, along with other government and academic institutions including the Federal Reserve Bank of Kansas City, Globus, Harvard University, University of Michigan, and Australia’s Monash University.

Hands-on sessions were conducted in addition to a series of talks describing specific use cases, including an overview of Singularity being employed on SDSC’s Comet supercomputer.

“SDSC was an early adopter of Singularity,” said SDSC Computational Scientist Mahidhar Tatineni, who also manages SDSC User Services Group. “The scientific community is developing software at an unprecedented rate. As the computational workflows and diversity of the underlying systems that these are being run on grows, we have found Singularity to be a powerful tool for improving the efficiency and portability of these workflows.”

Researchers on Comet are using Singularity for projects including those in neuroscience, high-energy physics, molecular dynamics, and many others. At UC San Diego, researchers on the Adolescent Brain Cognitive Development (ABCD) initiative are using Singularity as part of a 10-year project aimed at understanding how the adolescent brain develops.

“The scope of ABCD and the need to ensure portability and reproducibility make Singularity an easy choice for the project,” said Tatineni. “It has the side benefit of reducing the support workload on my team since many of the software build issues have been addressed via containerization. It’s a win-win for us and the project.”

The recent Singularity User Group event also covered recent offerings from Sylabs including:

• Singularity Desktop, which now allows users to design, build, test, and sign/verify Linux-based Singularity containers on macOS.
• A demonstration of Singularity’s plug-in architecture, scheduled to appear with the next release of the software (Singularity 3.2.0) in the April 2019 timeframe.
• Singularity’s integration with Slurm, demonstrated via the Singularity Container Runtime Interface (CRI) for Kubernetes and identified as key to enabling new types of compute workflows that include Artificial Intelligence (AI), Deep Learning, and Machine Learning.

“Singularity has progressed significantly in the past four years, and by engaging directly with users and developers, we can collectively ensure that the software continues to evolve in directions relevant to the needs of its community,” said Kurtzer.

About SDSC

As an Organized Research Unit of UC San Diego, SDSC is considered a leader in data-intensive computing and cyberinfrastructure, providing resources, services, and expertise to the national research community, including industry and academia. Cyberinfrastructure refers to an accessible, integrated network of computer-based resources and expertise, focused on accelerating scientific inquiry and discovery. SDSC supports hundreds of multidisciplinary programs spanning a wide variety of domains, from earth sciences and biology to astrophysics, bioinformatics, and health IT. SDSC’s petascale Comet supercomputer is a key resource within the National Science Foundation’s XSEDE (eXtreme Science and Engineering Discovery Environment) program.

About Sylabs.io

Sylabs was founded with the mission to advocate for open source technologies while providing high-performance computing container solutions as well as professional support and services. Sylabs’ capabilities for machine learning, deep learning, and compute-driven analytics satisfy a broad range of customer needs that take enterprise and HPC commercial markets to the next level. The company, headquartered in the San Francisco Bay Area, is backed by RStor, the only multi-cloud platform that enables organizations to compute, connect, and operate above the cloud.