CommandsforJobMonitoring

rtracejob

rtracejob is used to compare resource requests to resource usage for an individual job. Currently it can be used to display a single job information or the summary of array job information. By typing in rtracejob -h, the help information will be displayed to explain the arguments and available functions for rtracejob:

    usage: rtracejob [-h] [-l] [--dump_failed_joblist] jobID

    positional arguments:
     jobID                 the slurm job ID for displaying job information. 
                           For array jobs if root job ID is given (for 
                           example, in job ID 1234567_12 the root job ID 
                           is 1234567) the summary of array jobs will be 
                           displayed.

    optional arguments:
     -h, --help            show this help message and exit
     -l, --list_subjobs    with this option rtracejob will print out all 
                           of sub jobs information automatically, in 
                           default it's off
     --dump_failed_joblist
                           with this option rtracejob will dump the failed 
                           sub jobs ID to a file in name of 
                           "failed_joblist_(your jobid).txt", 
                           in default it's off

rtracejob is useful for troubleshooting when something goes wrong with your job. For example, rtracejob jobID can be used to display information for a single job:

    [bob@vmps12 ~]$ rtracejob 1234567
    +------------------+--------------------------+
    |  User: bob       |      JobID: 1234567      |
    +------------------+--------------------------+
    | Account          | chemistry                |
    | Job Name         | python.slurm             |
    | State            | Completed                |
    | Exit Code        | 0:0                      |
    | Wall Time        | 00:10:00                 |
    | Requested Memory | 1000Mc                   |
    | Memory Used      | 13712K                   |
    | CPUs Requested   | 1                        |
    | CPUs Used        | 1                        |
    | Nodes            | 1                        |
    | Node List        | vmp505                   |
    | Wait Time        | 0.4 minutes              |
    | Run Time         | 0.4 minutes              |
    | Submit Time      | Thu Jun 18 09:23:32 2015 |
    | Start Time       | Thu Jun 18 09:23:57 2015 |
    | End Time         | Thu Jun 18 09:24:23 2015 |
    +------------------+--------------------------+
    | Today's Date     | Thu Jun 18 09:25:08 2015 |
    +------------------+--------------------------+

A user might want to check how much memory a job used compared to how much was requested, or how long it took a job to execute relative to how much wall time was requested. In this example, note the Requested Memory reported is 1000Mc, meaning 1000 megabytes per core (the “c” stands for “core”). This is the default for jobs that specify no memory requirement. If you see a lowercase “n” on the `Requested Memory` line, this stands for “node” and occurs when a --mem= line is included in a SLURM script, which allocates the amount of memory listed per node in the allocation.

Additionally, rtracejob can also used to display the summary of an array job. For example,

    [lf8@gw342 ~]$ rtracejob 5270444

    +---------------------------+-------------------------------------------+
    +                   SUMMARY of ARRAY JOBS                               +
    + User name: liuf8          | job ID: 5270444                            
    +---------------------------+-------------------------------------------+
    + Account                   | accre                                  
    + Job Name                  | array.slurm                            
    + No. of Submitted SubJobs  | 5                                      
    + No. of Finished SubJobs   | 5                                      
    + No. of Successful SubJobs | 5                                      
    + No. of Failed SubJobs     | 0                                      
    + Requested Memory          | 500mn                                  
    + Max Memory Used by SubJobs| 154472k                                
    + Original Requested Time   | 00:10:00                                
    + Max Running Time          | 00:00:42                                
    + Min Running Time          | 00:00:26                                
    + Max Waiting Time          | 00:00:00                                
    + Min Waiting Time          | 00:00:00                                
    +---------------------------+-------------------------------------------+

The job ID 5270444 is the root job ID, and 5270444_1, 5270444_2 are the sub jobs that belong to this array job. You can also pass either sub job ID directly to the rtracejob command, or get information about all sub jobs by passing the --list_subjobs flag (e.g. rtracejob 5270444 --list_subjobs). By providing the root job ID, rtracejob will scan all of the sub jobs’ information and finally print a summary of the given array job.

qSummary

qSummary provides an alternate summary of jobs and cores running across all groups in the cluster. It is possible to filter the results by selecting a specific account through the -g option.

[jill@vmps12 ~]$ qSummary
GROUP      USER        ACTIVE_JOBS  ACTIVE_CORES  PENDING_JOBS  PENDING_CORES
-----------------------------------------------------------------------------
science                    18            34             5             7
          jack             5             5             4             4
          jill             13            29             1             3
-----------------------------------------------------------------------------
economics                  88           200           100           100
           emily           88           200           100           100
-----------------------------------------------------------------------------
Totals:                   106           234           105           107

As shown, the output from qSummary provides a basic view of the active and pending jobs and cores across groups and users within a group. qSummary also supports a -g argument followed by the name of a group, a -p argument followed by the partition name, and a -gpu switch if you like to see GPU rather than CPU info. For example:

     
[jill@vmps12 ~]$ qSummary -p batch_gpu -gpu
GROUP      USER        ACTIVE_JOBS  ACTIVE_GPUS  PENDING_JOBS  PENDING_GPUS
-----------------------------------------------------------------------------
science                     4             8             1             2
           jack             0             0             1             2
           jill             4             8             0             0
-----------------------------------------------------------------------------
economics                   4            16             0             0
           emily            4            16             0             0
-----------------------------------------------------------------------------
Totals:                     8            24             1             2

slurm_resources

To list your current SLURM group membership, type slurm_resources.

This command will list all your slurm accounts and special resources such as GPUs they have access to organized by partition (batch, interactive, batch_gpu, etc..). If you do have GPU access, it will return an example for your SLURM script.

  [bob@gw01 ~]$ slurm_resources

Accounts to use for accessing the batch (default) partition:

Account
-------------------
fe_accre_lab
accre_guests


Accounts and GPU types for accessing the batch_gpu partition:

Accounts           GPU Type                      GPU Limit
------------------ ---------------------------- ----------
fe_accre_lab_acc   nvidia_geforce_rtx_2080_ti            2
fe_accre_lab_acc   nvidia_rtx_a6000                      2
fe_accre_lab_acc   nvidia_titan_x                        4
accre_guests_acc   nvidia_geforce_rtx_2080_ti            2
accre_guests_acc   nvidia_rtx_a6000                      6

You have access to accelerated GPU resources in the batch_gpu partition.
As a usage example, if you wanted to request 2 GPUs of type "nvidia_geforce_rtx_2080_ti"
for a job with account "fe_accre_lab_acc" on the partition "batch_gpu",
then you would add the following lines to your SLURM script:

#SBATCH --account=p_dsi_acc
#SBATCH --partition=batch_gpu
#SBATCH --gres=gpu:nvidia_geforce_rtx_2080_ti:2

Note that the "GPU Limit" column on the table above lists is the burst
limit for that GPU type for the specified account. For example, the
scheduler will not allow more than 2 GPUs of type
"nvidia_geforce_rtx_2080_ti" to be used simulatenously by account "fe_accre_lab_acc".
If more GPUs of that type are requested by that account, the scheduler
will not allow them to run until other running jobs have completed.
You may not request more than the GPU limit for a single job.


Accounts and QOSs for accessing the interactive partition:

Account            QOS                   CPU limit  Memory Limit (GiB)
------------------ -------------------- ---------- -------------------
fe_accre_lab_int   debug_int                    80               960.0
accre_guests_int   debug_int                    80               960.0

Use the "qosstate [QOS_NAME]" command to get current usage for
the QOS specified by QOS_NAME with a breakdown of usage by user
and slurm account.

Note that the "debug_int" QOS is a special QOS available to all
cluster users for quick debugging of slurm scripts. On this special
QOS there are additional restrictions of a maximum wall clock time
of 00:30:00 and each user can use a maximum of cpu=16,mem=192G at
one time.

As an example, to submit a job to the interactive partition using
the account "accre_int" and QOS "debug_int" you would add the following
lines to your slurm script:

#SBATCH --account=accre_int
#SBATCH --partition=interactive
#SBATCH --qos=debug_int

showLimits

As the name suggests, showLimits will display the resource limits imposed on accounts and groups on the cluster for the batch partition. Running the command without any arguments will list all parent accounts and group accounts on the cluster. Optionally, showLimits also accepts a -g argument followed by the name of a group or account. For example, to see a list of resource limits imposed on a parent account named science_account (this account does not actually exist on the cluster):

[jill@vmps12 ~]$ showLimits -g science_account
ACCOUNT         GROUP       FAIRSHARE   MAXCPUS   MAXMEM(GB)  MAXCPUTIME(HRS)
-----------------------------------------------------------------------------
science_account                12        3600       2400           23040
              biology          1        2400       1800               -
              chemistry        1         800        600               -
              physics          1         600        600            8640
              science          1           -       2200           20000
-----------------------------------------------------------------------------

Limits are always imposed on the parent account level, and occasionally on the group level when multiple group accounts fall under a single parent account. If a particular limit is not defined on the group level, the group is allowed access to the entire limit under its parent account. For example, the science group does not have a MAXCPUS limit defined, and therefore can run across a maximum of 3600 cores so long as no other groups under `science_account` are running and no other limits (MAXMEM or MAXCPUTIME) are exceeded.

We leave FAIRSHARE defined on the parent account level only, so groups within the same account do not receive elevated priority relative to one another. The value 1 for FAIRSHARE defined at the group level means that all groups under the parent account receive equal relative priority.

qosstate

The qosstate command will display the current usage of all Quality of Services (QoSs) available to your user in the interactive and interactive_gpu partitions:

[bob@cn1287 ~]$ qosstate

Live interactive QOS usage report for QOS available to user: bob

QOS                     CPUs (used/limit)   Memory GiB (used/limit)
===================================================================
debug_int                          1 / 80               1.0 / 960.0
bob_lab_2_int                      0 / 4                 0.0 / 30.0
bob_lab_int                        4 / 8                16.0 / 60.0

When run with a specified QoS as an argument it will display the users and accounts which are using the QoS and how many resources they are using:

[bob@cn1287 ~]$ qosstate bob_lab_int

Live interactive QOS usage report for QOS available to user: appelte1

QOS                     CPUs (used/limit)   Memory GiB (used/limit)
===================================================================
bob_lab_int                         4 / 8               16.0 / 60.0


Breakdown of bob_lab_int usage by slurm account and user

Account            User             CPUs      Memory GiB
========================================================
bob_lab_int
                   bob                 4            16.0

SlurmActive

This command is under re-development.