System Architecture

Expanse's standard compute nodes are each powered by two 64-core AMD EPYC 7742 processors and contain 256 GB of DDR4 memory, while each GPU node contains four NVIDIA V100s (32 GB SMX2), connected via NVLINK, and dual 20-core Intel Xeon 6248 CPUs. Expanse also has four 2 TB large memory nodes.

The entire system, to be integrated by Dell, is organized into 13 SDSC Scalable Compute Units (SSCUs), comprising 56 standard nodes and four GPU nodes, and connected with 100 GB/s HDR InfiniBand. Every Expanse node has access to a 12 PB Lustre parallel file system (provided by Aeon Computing) and a 7 PB Ceph Object Store system.

The Expanse cluster is managed using the Bright Computing HPC Cluster management system, and uses the SLURM workload manager for job scheduling. While the system is suited for modest-scale jobs as few as tens of cores to several hundred cores, Expanse also handles high-throughput computing jobs via integration with the Open Science Grid, which can have tens of thousands of single-core jobs, and provides connectivity to commercial clouds via the job queuing system. A low-latency interconnect based on Mellanox High Data Rate (HDR) InfiniBand supports a fabric topology optimized for jobs of one to a few thousand cores that require medium-scale parallelism.

One of the key innovations of Expanse is its ability to support composable systems, allowing researchers to create a virtual 'tool set' of resources, such as Kubernetes, for a specific project and then re-compose it as needed. Expanse will also feature direct scheduler-integration with the major cloud providers, leveraging high-speed networks to ease data movement to/from the cloud.