User Guide

TSCC harnesses the power of the Simple Linux Utility for Resource Management (SLURM) to effectively manage resources and schedule job executions. To operate in batch mode, users employ the sbatch command to dispatch tasks to the compute nodes. Please note: it's imperative that heavy computational tasks are delegated exclusively to the compute nodes, avoiding the login nodes.

Before delving deeper into job operations, it's crucial for users to grasp foundational concepts such as Allocations, Partitions, Credit Provisioning, and the billing mechanisms for both Hotel and Condo models within TSCC. This segment of the guide offers a comprehensive introduction to these ideas, followed by a detailed exploration of job submission and processing.

This can be useful for reviewing the members of a specific account and may help provide insights when troubleshooting an issue.

Note: The “account” in sacctmgr and other Slurm commands is Slurm accounts (rather than user account), which is used for allocation purposes.

Partitions

 $ scontrol show partition

This will show something like:

The default walltime for all queues is now one hour. Max walltimes are still in force per the below list.

If you want to obtain information about all the partitions in the system, you can alternatively use the following command:

$ scontrol show partitions

Note the ‘s’ at the end of the command.

For additional information, some of the limits for certain partitions are provided for each partition in the table below. The allowed QOS must be specified for each partition.

We'll be diving deeper into partitions in the Job Submission section later in this guide.

Quality of Service (QOS):

QOS for each job submitted to Slurm affects job scheduling priority, job preemption and job limits. QOS available are:

• hotel
• hotel-gpu
• Condo
• hcg- <project-name>
• hcp- <project-name>
• condo-gpu
• hca -<project-name>

In the previous list, <project-name> refers to the allocation id for the project. For TSCC (Slurm), that is the Account, or simply put, the group name of the user. 

More on QOSs will be discussed in the Job submission section of this guide.

How to Specify Partitions and QOS in your Job Script

You are required to specify which partition and QOS you'd like to use in your SLURM script (*.sb file) using the #SBATCH directives. Keep in mind the specificactions of each QOS for the different parftitions as shown in tables 2 and 3. Here's an example for a job script that requests one node from the hotel partition:

#!/bin/bash

#SBATCH --partition=hotel

#SBATCH --QOS=hotel

#SBATCH --nodes=1

# ... Other SLURM options ...

# Your job commands go here

CPU nodes 

Table 2: CPU Partitions information. hcg = [H]PC [C]ondo [G]old, hcp = [H]PC [C]ondo [P]latinum

GPU nodes

Table 3: GPU Partitions information. hca = [H]PC [C]ondo [A]ccelerator

Job Charging in Condo

For condo allocations, the charging of the jobs is based on the memory and the number of cores used by the job. The charging also varies based on the type of GPU used.

Job Charge:

( (Total # cores/job + (.2 * Total memory requested/job) + (Total # A100/job * 60) + (Total # A40/job * 10) + (Total # RTX6000s/job * 30) + (Total # RTX3090s/job * 20) ) * Job runtime (in seconds)/60

Example using Job Charge:

Let's assume a researcher wants to run a job that requires:

• 16 cores
• 32GB of requested memory
• 1 A100 GPU
• The job has a runtime of 120 minutes (or 2 hours), or 7200 seconds.

Calculation

Core Charge: 16 cores

Memory Charge: 0.2 * 32GB = 6.4, but we'll use 6 in this case given that SLURM will only take integers for this calculations.

A100 GPU Charge: 1 A100 GPU  * 60 = 60

Sum these charges: 16 + 6 + 60 = 82

Now, multiply by the job runtime:  82 * 7200 seconds / 60 = 9,840 SUs

Result

The total cost for running the job would be 9,888 Service Units. Charging is always based on the resources that were used by the job.  The more resources you use and the longer your job runs, the more you'll be charged from your allocation. 

Job Charging in Hotel

(Total # cores/job + (.2 * Total memory requested/job) + (Total #GPU/job * 30)) * Job runtime (in seconds)/60

Calculation

Running Jobs in TSCC

Before diving into how to submit jobs interactively or non-interactively in TSCC, we need to clarify a few important concepts that are crucial for job performance and scheduling.

Slurm Job Partition

As discussed previously, in SLURM, a partition is a set of nodes where a job can be scheduled. On the TSCC system, partitions vary based on factors such as hardware type, maximum job wall time, and job charging weights, which determine the cost of running a job (as explained further in the job charging section). Each job submitted to SLURM must conform to the specifications and limits of the partition it is assigned to.

Please, refer to the table below to check for the different partitions in the Hotel and Condo programs.

To get information about the partitions, please use:

$ sinfo

a100          up 7-00:00:00      1    mix tscc-gpu-14-27

a100          up 7-00:00:00      2  alloc tscc-gpu-14-[25,29]

a100          up 7-00:00:00      1   idle tscc-gpu-14-31

a40           up 7-00:00:00      1    mix tscc-gpu-10-6

condo         up 14-00:00:0      7  down* tscc-1-27,tscc-4-[17-18,41],tscc-11-58,tscc-13-26,tscc-14-39

condo         up 14-00:00:0      8   comp tscc-1-[11,17,41],tscc-4-[15-16],tscc-11-50,tscc-13-[8,12]

condo         up 14-00:00:0     30    mix tscc-1-[2-10,18,25-26,28-32,39-40],tscc-4-19,tscc-6-37,tscc-8-[35-36],tscc-9-[14,17],tscc-10-13,tscc-11-[49,54],tscc-13-2,tscc-14-24

condo         up 14-00:00:0      4  alloc tscc-1-[1,35],tscc-13-11,tscc-14-1

condo         up 14-00:00:0     80   idle tscc-1-[12-16,33-34,36-38,42-45],tscc-8-37,tscc-9-[15-16],tscc-11-[44,47-48,51-53,55-57,59,61-68,70-75,77],tscc-13-[1,9-10,13-14,27-29,32-37],tscc-14-[2-16,20-23,40-44]

...

To get specific information about one partition, please use:

$ sinfo -p <partition_name>

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST

hotel*       up 7-00:00:00      1  maint tscc-11-0

hotel*       up 7-00:00:00      1  down* tscc-11-26

hotel*       up 7-00:00:00      2   comp tscc-11-[27-28]

hotel*       up 7-00:00:00      6    mix tscc-11-[1,4,29-32]

hotel*       up 7-00:00:00     25   idle tscc-11-[2-3,5-8,10-25,33-35]

hotel-gpu    up 2-00:00:00      1  drain tscc-gpu-m10-02

hotel-gpu    up 2-00:00:00      1   resv tscc-gpu-1-01

hotel-gpu    up 2-00:00:00     16   idle tscc-gpu-10-9,tscc-gpu-m10-[01,03-16]

SLURM Job Quality of Service (QOS)

In SLURM, Quality of Service (QOS) is a critical factor that influences various aspects of a job's execution, such as the maximum job wall time, scheduling priority, job preemption, and job limits. Each job submitted to SLURM is assigned a QOS, which determines how resources are allocated and how the job competes for scheduling. On TSCC, groups can have multiple QOS options. For example, the "Hotel" program includes hotel and hotel-gpu QOS, while the "Condo" program includes several QOS types, such as condo, condo-gpu, and specific QOS allocations like hcq, hcp, and hca, which are restricted to condo groups based on their ownership of specific node types, including gold, platinum, and GPU nodes. These QOS categories help manage and optimize resource use across different projects and hardware configurations.

The table below shows a summary of all available QOSs in the Hotel and Condo programs.

To keep in mind:

hcg-<allocation>: [H]PC [C]ondo [G]old, only for condo groups who owns gold node(s)

hcp-<allocation>: [H]PC [C]ondo [P]latinum, only for condo groups who owns platinum node(s)

hca-<allocation>: [H]PC [C]ondo [A]ccellerator, only for condo groups who owns GPU node(s)

Checking available Quality of Service

To check your available Quality of Service (QOS) in SLURM, you can use the $ sacctmgr show assoc command:

$ sacctmgr show assoc user=$USER format=account,user,qos%50

   Account       User                                                QOS

---------- ---------- --------------------------------------------------

    account_1    user_1                             hotel,hotel-gpu,normal

The information above illustrates the accessible or available QOSs for the allocation (account) you are part of. This means that, when submitting a job, you need to keep this in mind to ensure that SLURM can successfully schedule and run your job in a timely manner.

Also, you can check further characteristics of any QOS by running:

$ sacctmgr show qos format=Name%20,priority,gracetime,PreemptExemptTime,maxwall,MaxTRES%30,GrpTRES%30 where qos=hotel

 Name   Priority  GraceTime   PreemptExemptTime     MaxWall                        MaxTRES                        GrpTRES

-------------------- ---------- ---------- ------------------- ----------- ------------------------------ ------------------------------

hotel          0   00:00:00                      7-00:00:00                 cpu=196,node=4

Here is a brief description of the columns in the table of the response above:

1. Name: This is the name of the QOS. In this example, it's hotel.
2. Priority: The priority level assigned to jobs submitted under this QOS. A higher value means jobs in this QOS will have higher scheduling priority.
3. GraceTime: This represents the time allowed before a job is subject to preemption (if applicable). A value of 00:00:00 means no grace time is provided.
4. PreemptExemptTime: The amount of time a job can run before becoming eligible for preemption.
5. MaxWall: This is the maximum wall time (duration) that a job can run under this QOS. In this case, it is 7-00:00:00, meaning the job can run for a maximum of seven days.
6. MaxTRES: The maximum resources a single job can request, specified here as cpu=196, node=4. This means a maximum of 196 CPUs and 4 nodes can be used by a job under this QOS.
7. GrpTRES: The maximum resources that a group or user can use collectively across all jobs under this QOS.

In SLURM, the Quality of Service (QOS) determines which partition a job can be submitted to. This mapping ensures that the job runs within the appropriate resource limits and scheduling policies defined for that QOS. When submitting a job, it is crucial to specify both the partition and the QOS to ensure that SLURM can properly allocate resources and schedule the job accordingly. Without matching the correct QOS and partition, the job may not run as expected or could be rejected.

Hotel QOS configuration

Condo QOS configuration

Note: Jobs submitted with condo or condo-gpu QOS become preemptable after 8 hours (plus 30 min grace time).

hcq, hcp and hca QOS configuration

The hcq, hcp, and hca QOS configurations are used for jobs submitted to the gold, platinum, or GPU partitions, as there are no home queues or dedicated partitions for these QOS types. These configurations limit the total resources available to your group, based on the nodes you've purchased. Jobs submitted with these QOS options are not preemptible, have a high priority to start, and a maximum wall time limit of 14 days. Additionally, nodes are labeled with the appropriate QOS.

For example, if a PI purchases eight 36-core, 256 GB RAM nodes, the nodes are added to the gold partition, contributing 288 CPU cores and 2 TB of RAM. The QOS hcq-<allocation_name> would be available for the group’s jobs, with a total resource limit of either 288 CPUs, 2 TB of memory, or 8 nodes.

Multi-Category Security (MCS) labelling

In Multi-Category Security (MCS) labeling, jobs submitted to the platinum, gold, or GPU partitions with the hcq, hcp, or hca QOS are labeled based on the group or allocation name. This labeling ensures that if a job from a PI's group is running on a node with their QOS, the entire node is labeled for that group, preventing jobs from other groups from being scheduled on the same node. As a result, only jobs from the same group and QOS can utilize the remaining resources on that node.

MCS labeling is applied to make job assignment easier for SLURM, as it clearly defines which jobs can run on which nodes. Additionally, it enhances security by ensuring that resources are only accessible to the group that owns the node for that particular job.

Using srun for interactive Job submission

Note: It is absolutely crucial not to run computational programs, applications, or codes on the login nodes, even for testing purposes. The login nodes are specifically designed for basic tasks such as logging in, file editing, simple data analysis, and other low-resource operations. Running computationally intensive tasks on the login nodes can negatively impact other users' activities, as they share the same login node. By overloading the login node, your tests or programs can slow down or interfere with the login and basic tasks of other users on the system. To maintain system performance and ensure fair resource use, it is crucial to restrict computational work to the appropriate compute nodes and use login nodes only for light tasks.

Now that we have a grasp on the basic concepts explained above, we can start going deep on how to submit and monitor jobs for effecitve use of computational resource.

The first type of job we're going to discuss is the Interactive job. An interactive job allows users to set up an interactive environment on compute nodes. The key advantage of this setup is that it enables users to run programs interactively. However, the main drawback is that users must be present when the job starts. Interactive jobs are typically used for purposes such as testing, debugging, or using a graphical user interface (GUI). It is recommended not to run interactive jobs with a large core count, as this can be a waste of computational resources.

You can use the srun command to request an interactive session. Here's how to tailor your interactive session based on different requirements:

srun -t short for --time hh:mm:ss  \

     -N short for --nodes, number of nodes \

     -n short for --ntasks, total number of tasks to run job on \

     --ntasks-per-node optional to --ntasks \

     -c short for --cpus-per-task, number of threads per process* \

     -A* short for --account <Allocation> \

     -p short for --partition <partition name> \

     -q short for --qos \

     -G short for --gpus number of GPU card \

     --mem , memory (details later) \

     --x11 , enabling X11 forwarding \

     --pty , in pseudo terminal \

     bash , executable to run 

NOTE

The “account” in Slurm commands like srun is Slurm accounts (rather than user account), which is used for allocation purposes.

Example: Requesting a Compute Node

To request one regular compute node with 1 core in the hotel partition for 30 minutes, use the following command:

$ srun --partition=hotel --pty --nodes=1 --ntasks-per-node=1 -t 00:30:00 -A xyz123 --qos=hotel --wait=0 --export=ALL /bin/bash

In this example:

• --partition=hotel: Specifies the debug partition.
• --pty: Allocates a pseudo-terminal.
• --nodes=1: Requests one node.
• --ntasks-per-node=1: Requests 1 tasks per node.
• -t 00:30:00: Sets the time limit to 30 minutes.
• -A xyz123: Specifies the account.
• --wait=0: No waiting.
• --export=ALL: Exports all environment variables.
• --qos=hotel: Quality of Service.
• /bin/bash: Opens a Bash shell upon successful allocation.

A more advanced version of interactive jobs may include using MPI:

$ srun --overlap -n 8 <mpi_executable>

where the --overlap flag is required for interactive + MPI, otherwise srun will hang.

For properly setting your job so it runs in a timely manner and doesn't hang, one key aspecto to consider is the allocated or requested CPU to the job. The table below shows the maximum CPU that can be requested out of a node:

For example, if you consider the submission of the two interactive jobs below, you will see that both will be able to start because each requests at most the maximum number of CPUs allowed by the partition:

$ srun -N 2 -n 128 -c 1 -p condo -q condo -A account_1 -t 8:00:00 --pty bash

$ srun -N 2 -n 2 -c 60  -p condo -q condo -A account_1 -t 8:00:00 --pty bash

Both jobs request 2 nodes (-N 2), and each node is from a condo partition, which allows a maximum of 64 CPUs per node.

• For the first job, the total number of requested CPUs is calculated as 1 (-c parameter) multiplied by 128 (-n parameter), resulting in 128 CPUs across the 2 requested nodes. Since each node can provide 64 CPUs, the total requested (128 CPUs) exactly matches the available CPUs across the two nodes (64 CPUs per node x 2 nodes = 128 CPUs).

• For the second job, the total number of requested CPUs is 60 (-c parameter) multiplied by 2 (-n parameter), which equals 120 CPUs across the two condo nodes. Together, the two nodes provide a total of 128 CPUs, which is sufficient to accommodate the 120 CPUs requested by the job.

By the same logic, the next job submission should fail right away given that the amount of requested CPUs is larger than the one that can be provided by the requested nodes:

$ srun -N 2 -n 129 -c 1 -p condo -q condo -A account_1 -t 8:00:00 --pty bash

Requesting more nodes (-N > 1) can increase the maximum number of CPUs available to a job; however, you should ensure that your program can efficiently run across multiple nodes. It is generally recommended to use CPUs from the same node whenever possible. For example, if you need to use a total of 8 CPUs, it is not recommended to request -N 2 -n 4, which would distribute the tasks across two nodes. Instead, you should request -N 1 -n 8 to use all 8 CPUs on a single node, optimizing resource usage and reducing unnecessary overhead.

Also, consider how much memory you will allocate to your job. The default memory allocated to a job in SLURM is 1GB per CPU core. The default unit of memory is MB, but users can specify memory in different units such as "G", "GB", "g", or "gb". Users are allowed to specify the memory requirements for their job, but it is recommended to choose an appropriate amount to ensure the program runs efficiently without wasting computational resources. The charging factor for memory usage is 0.2 per GB. The maximum amount of memory a user can request is specified on the following table. It's advisable to run small trial tests before large production runs to better estimate the memory needs.

The --mem flag is used to specify the amount of memory on one node. Requesting more nodes can allocate more memory to the job. However, you need first to ensure that your job can run on multiple nodes.

GPU Jobs

GPU jobs should be submitted by specifying the --gpus (or -G) option. For example, using --gpus 2 will allocate 2 GPU cards to the job. If requesting multiple GPUs, ensure that your program can utilize all of them efficiently. Different nodes offer varying GPU configurations; for instance, all hotel GPU nodes and Condo A100/A40 nodes have 4 GPUs per node, while Condo RTX A6000 and RTX 3090 nodes have 8 GPUs per node. Allocated GPUs are referenced through the CUDA_VISIBLE_DEVICES environment variable, and applications using CUDA libraries will automatically discover the assigned GPUs through this variable. You should never manually set the CUDA_VISIBLE_DEVICES variable.

Submitting Batch Jobs Using sbatch

To submit batch jobs, you will use the sbatch command followed by your SLURM script. Here is how to submit a batch job:

$ sbatch mycode-slurm.sb

> Submitted batch job 8718049

In this example, mycode-slurm.sb is your SLURM script, and 8718049 is the job ID assigned to your submitted job.

In this section, we will delve into the required parameters for job scheduling in TSCC. Understanding these parameters is crucial for specifying job requirements and ensuring that your job utilizes the cluster resources efficiently. Required Scheduler Parameters

• --partition (-p): Specifies which partition your job will be submitted to. For example, --partition=hotel would send your job to the hotel partition.
• --qos (-q): Specifies which QOS your job will be used.
• --nodes (-N): Defines the number of nodes you need for your job.
• --ntasks-per-node OR --ntasks (-n): Indicates the number of tasks you wish to run per node or in total, respectively. If both are specified, SLURM will choose the value set for --ntasks.
• --time (-t): Sets the maximum time your job is allowed to run, in the format of [hours]:[minutes]:[seconds].
• --account (-A): Specifies the account to which the job should be charged.
• --gpus: Indicates the total number of GPUs needed by your job.

Examples

Example 1: Submitting to 'hotel' Partition

Let's say you have a CPU-intensive job that you'd like to run on one node in the ‘hotel’ partition, and it will take no more than 2 hours. Your SLURM script may look like this:

Example 2: Submitting to ‘hotel-gpu’ Partition

Suppose you have a job requiring 1 GPUs, shared access, and expected to run for 30 minutes. Here's how you could specify these requirements:

Where tensorflowtest.py can be a simple Hello world script such as:

Example 3: OpenMP Job

OpenMP (Open Multi-Processing) is an API that supports multi-platform shared-memory parallel programming in C, C++, and Fortran. It allows you to write programs that can run efficiently on multi-core processors. By using OpenMP directives, you can parallelize loops and other computational blocks to run simultaneously across multiple CPU cores, thereby improving performance and reducing execution time.

To run an OpenMP job in TSCC, you can use the following batch script as a template. This example is for a job that uses 16 CPU cores on a single node in the shared partition. The test script at the end of the template called ‘pi_openmp’ is located here: /cm/shared/examples/sdsc/tscc2/openmp. To run the bash script with the test script, copy the entire openmp directory to your own space, and from there execute the batch job: 

Breakdown of the Script

• --job-name: Specifies the job name.
• --output: Sets the output file where stdout and stderr are saved. SLURM merges stdout and stderr by default. See the above OpenMP job script example for separating stderr.
• --partition: Chooses the partition.
• --qos: Chooses the QOS.
• --nodes, --ntasks, --cpus-per-task: Define the hardware resources needed.
• --export: Exports all environment variables to the job's environment.
• --time: Sets the maximum runtime for the job in hh:mm:ss format.
• module load: Loads necessary modules for the environment.
• export OMP_NUM_THREADS: Sets the number of threads that the OpenMP job will use.

Example 4: GPU Job

The script in the previous example can be found in here:

/cm/shared/examples/sdsc/tscc2/amber/gpu/amber-slurm.sb

Note the #SBATCH -G 1 parameter used to request 1 GPU

Example Application Scripts:

Navigate to:

$ ls /cm/shared/examples/sdsc/tscc2/

abinit  amber  cp2k  cpmd  gamess  gaussian  gromacs  lammps  namd  nwchem  qchem  quantum-espresso  siesta  vasp openmp mpi

This directory holds test scripts for applications like abinit, vasp, and more, optimized for both GPU and CPU nodes.

Look for the testsh scripts or *.sb file within each folder. Although you can't execute any script directly from here, you can copy them—and the entire application directory—to your own space to run it. This will help you avoid dependency issues.

Important Note on submitting jobs

You always must specify the allocation name in your job request, whether it is Condo or Hotel partition, interactive or batch job. Please, add --account=<allocation_name> (-A=<allocation_name>) to your srun command, or #SBATCH --account=<allocation_name> (#SBATCH -A=<allocation_name>) to your job scripts. You can run the command " sacctmgr show assoc user=$USER format=account,user " to find out your allocation name.

$ sacctmgr show assoc user=$USER format=account,user

Account User
---------- ----------
account_1 user_1

Checking Job Status

You can monitor the status of your job using the squeue command:

$ squeue -u $USER

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

8718049 hotel mycode user PD 0:00 1 (Priority)

• JOBID: This is the unique identification number assigned to each job when it is submitted. You will use this JOBID when you wish to cancel or check the status of a specific job.
• PARTITION: This indicates which partition (or queue) the job is submitted to. Different partitions have different resources and policies, so choose one that fits the job's requirements.
• NAME: This is the name of the job as specified when you submitted it. Naming your jobs meaningfully can help you keep track of them more easily.
• USER: This field shows the username of the person who submitted the job. When you filter by $USER, it will display only the jobs that you have submitted.
• ST: This stands for the status of the job. For example, "PD" means 'Pending,' "R" means 'Running,' and "C" means 'Completed.'
• TIME: This shows the elapsed time since the job started running. For pending jobs, this will generally show as "0:00".
• NODES: This indicates the number of hotel nodes allocated or to be allocated for the job.
• NODELIST(REASON): This provides a list of nodes assigned to the job if it's running or the reason why the job is not currently running if it's in a pending state.

$ nvidia-smi

Tue Sep 24 00:06:17 2024

+---------------------------------------------------------------------------------------+

| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |

|-----------------------------------------+----------------------+----------------------+

| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|                                         |                      |               MIG M. |

|=========================================+======================+======================|

|   0  Tesla P100-PCIE-16GB            On | 00000000:04:00.0 Off |                    0 |

| N/A   36C    P0               53W / 250W|   2632MiB / 16384MiB |     86%      Default |

|                                         |                      |                  N/A |

+-----------------------------------------+----------------------+----------------------+

|   1  Tesla P100-PCIE-16GB            On | 00000000:05:00.0 Off |                    0 |

| N/A   35C    P0               26W / 250W|      0MiB / 16384MiB |      0%      Default |

|                                         |                      |                  N/A |

+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+

| Processes:                                                                            |

|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |

|        ID   ID                                                             Usage      |

|=======================================================================================|

|    0   N/A  N/A     67028      C   ...iqo2tw4hqbb4mluy/bin/pmemd.cuda.MPI      370MiB |

|    0   N/A  N/A     67029      C   ...iqo2tw4hqbb4mluy/bin/pmemd.cuda.MPI      370MiB |

+---------------------------------------------------------------------------------------+

Find account(s) by description substring match

Let's say you want to filter results by partial substrings of account names. You can get useful information by running:

$ tscc_client -d account_1

Important Note:

Do not try to run for loops or include the client into a script that could result in multiple database invokations, such as:

Doing this might bog down the system.

Users who try to run these kind of scripts or commands will have their account locked.

TSCC employs SBATCH and SLURM. Therefore, it's crucial to understand how to monitor usage. Here are the new commands to use on TSCC:

1. Account Balances: On TSCC (SLURM):

$ sreport user top usage User=<user_name>

Please note that SLURM relies on a combination of sacctmgr and sreport for comprehensive usage details.

2. Activity Summary:

$ sacct -u <user_name> --starttime=YYYY-MM-DD --endtime=YYYY-MM-DD

For group-level summaries on TSCC:

sreport cluster AccountUtilizationByUser start=YYYY-MM-DD end=YYYY-MM-DD format=Account,User,TotalCPU

(Note: Adjust the YYYY-MM-DD placeholders with the desired date range. This command will display account utilization by users within the specified period. Filter the output to view specific group or account details.)