System Architecture

Expanse's standard compute nodes are each powered by two 64-core AMD EPYC 7742 processors and contain 256 GB of DDR4 memory, while each GPU node contains four NVIDIA V100s (32 GB SMX2), connected via NVLINK, and dual 20-core Intel Xeon 6248 CPUs. Expanse also has four 2 TB large memory nodes.

The entire system, to be integrated by Dell, is organized into 13 SDSC Scalable Compute Units (SSCUs), comprising 56 standard nodes and four GPU nodes, and connected with 100 GB/s HDR InfiniBand. Every Expanse node has access to a 12 PB Lustre parallel file system (provided by Aeon Computing) and a 7 PB Ceph Object Store system.

The Expanse cluster is managed using the Bright Computing HPC Cluster management system, and uses the SLURM workload manager for job scheduling. While the system is suited for modest-scale jobs as few as tens of cores to several hundred cores, Expanse also handles high-throughput computing jobs via integration with the Open Science Grid, which can have tens of thousands of single-core jobs, and provides connectivity to commercial clouds via the job queuing system. A low-latency interconnect based on Mellanox High Data Rate (HDR) InfiniBand supports a fabric topology optimized for jobs of one to a few thousand cores that require medium-scale parallelism.

One of the key innovations of Expanse is its ability to support composable systems, allowing researchers to create a virtual 'tool set' of resources, such as Kubernetes, for a specific project and then re-compose it as needed. Expanse will also feature direct scheduler-integration with the major cloud providers, leveraging high-speed networks to ease data movement to/from the cloud.

Expanse Hardware and Component Specifications

System Component Configuration
AMD Rome Standard Compute Nodes
Node count 728
Clock speed 2.25 GHz
Cores/node 128
DRAM/node 256 GB
NVMe/node 1 TB
Node count 52
CPU cores/node 40
CPU Type 6248 Xeon
CPU Clock speed 2.5 GHz
CPU DRAM/node 384 GB
GPUs/node 4
GPU Type V100 SMX2
Memory/GPU 32 GB
NVMe/node 1.6 TB
Large-memory AMD Rome Nodes
Node count 4
Clock speed 2.25 GHz
Cores/node 128
DRAM/node 2 TB
SSD memory/node 3.2 TB
Storage Systems
File systems Lustre, Ceph
Lustre Storage 12 PB
Ceph Storage 7 PB