Comet to Expanse Transition Tutorial

SDSC’s new supercomputer, Expanse, goes into full production on November 1, 2020 and replaces Comet, which will be retired on March 31, 2021. Although the transition should be straightforward for most users, there are some important differences in both the hardware and software between these two systems. We will start with an overview of the Expanse architecture, which is based on AMD’s 64-core EPYC Rome processor and NVIDIA’s V100 GPU. Moving from Intel to AMD processors introduces a new set of compilers, math libraries and tools. We’ll explain how to use the AMD software stack to build and run applications for optimal performance. GPU applications will port easily to Expanse and the upgrade from Comet’s P100s to the newer V100s generally results in significantly reduced run times. With regard to job submission, we continue to use the Slurm resource manager. Partition names are left unchanged, but job scripts will have to be updated to reflect the larger core counts and, for highly scalable applications, maximum job size. We also introduce a new charging model that fairly takes into account the usage of all resources (memory, CPU, GPU). The tutorial concludes with a discussion of interactive computing using the Expanse Portal and efficiently moving data from Comet to Expanse.

• Overview/User Environment
  • 11:00 - 11:30 AM Overview of System and Allocations
• Running Jobs
  • 11:30 AM - 12:15 PM Slurm differences, Running Jobs
  • 12:15 PM - 12:25 PM 10-minute break
  • 12:25 PM - 12:55 PM Modules, compiling and basic optimizaton - CPU
  • 12:55 PM - 1:25 PM Modules, compiling and basic optimizaton - GPU 
  • 1:25 PM - 1:35 PM 10-minute break
  • 1:35 PM - 1:45 PM Job Charging, TRES 
  • 1:45 PM - 2:00 PM Using the Expanse portal 
  • 2:00 PM - 2:15 PM Interactive computing and running Jupyter Notebooks 
• Data Management
  • 2:15 PM - 2:30 PM Data transfer mechanisms