Slurm report
Automatic usage report
You can generate accounting statistics from Slurm using the slurmreport script, located in /ceph/hpc/bin/.
Usage:
/ceph/hpc/bin/slurmreport [-m] [-c | -w] [-a accountlist] [-s startdate -e enddate] [-r report-directory]
where:
-m: Send report by E-mail to root@localhost
-a: Select an account (or list of accounts)
-s -e: Select Start and End dates of the report
-c: Select current month from the 1st until today
-w: Select the last week
-r: Copy the report to a specified directory as well
./slurmreport -s 2021-08-18 -e 2021-08-01
Date format could be either YYYY-MM-DD or MMDD
If you run the script without specifying additional features, the report will be generated for the last month and placed into your home directory.
Manual usage report
Create an accounting report manually using sreport command:
Usage report in CPU Hours
Report cluster account utilization by filing specified fields for the desired period of time start/end:
sreport cluster AccountUtilizationByUser -T billing user=$USER -t Hours start=2021-01-01 end=2021-05-25 format=Proper%20,Login%20,Accounts,Used
Example:
sreport cluster AccountUtilizationByUser -T billing user=user -t Hours start=2021-01-01 end=2021-10-25 format=Proper%20,Login%20,Accounts,Used
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2021-01-01T00:00:00 - 2021-10-07T14:59:59 (24156000 secs)
Usage reported in CPU Hours
--------------------------------------------------------------------------------
Proper Name Login Account Used
-------------------- -------------------- --------------- -----------
User User user vega-users 5009241
Used=5009241 is in Thread Hours.
To get result in Core Hours divide the number by 2.
Usage report in TRES Hours
Fill specified fields for the desired period of time start/end:
Create report for your project account:
ACC=$(sacctmgr show assoc cluster=vega user=$USER format=account%40 -n) ; ACC=$(tr -d ' ' <<< "$ACC") ; sreport --tres billing cluster AccountUtilizationByAccount account=$ACC -t Hours start=2021-01-01 end=2022-01-01 format=Proper%20,Login%20,Accounts%30,Used
Create report for your user account only
sreport -T billing cluster UserUtilizationByAccount -t Hours start=2021-01-01 end=2022-01-01 user=$USER format=Proper%20,Login%15,Account%40,Used%20
Usage report in CPU Hours for account
Fill specified fields for the desired period of time start/end:
Create report for your project account:
sreport -t hourper --tres=billing cluster AccountUtilizationByUser Account=eurohpc-2010paXXX-users Start=2021-07-01 End=`date -d tomorrow +%Y-%m-%d` format=Accounts%25,Login%20,Used
Example:
$ sreport -t hourper --tres=billing cluster AccountUtilizationByUser Account=eurohpc-2010paXXXX-users Start=2021-07-01 End=`date -d tomorrow +%Y-%m-%d` format=Accounts%25,Login%20,Used
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2021-07-01T00:00:00 - 2022-03-24T13:59:59 (23036400 secs)
Usage reported in TRES Hours/Percentage of Total
--------------------------------------------------------------------------------
Account Login Used
------------------------- -------------------- -------------------
eurohpc-2010paXXXX-users 1919017(0.07%)
eurohpc-2010paXXXX-users user1 5316(0.00%)
eurohpc-2010paXXXX-users user2 791721(0.03%)
eurohpc-2010paXXXX-users user3 4064(0.00%)
eurohpc-2010paXXXX-users user4 851589(0.03%)
eurohpc-2010paXXXX-users user5 266326(0.01%)
Used=1919017 is in Thread Hours.
To get result in Core Hours divide the number by 2.
This example shows the total usage of all resources on (cpu and gpu partition) for the project (1919017) and is divided by users.
User1, user2, user3, user4, and user5 can view their own usage by partition. For usage on cpu partition is used the following command:
sreport job SizesByAccount -T billing User=user1 -t Hours Start=2021-07-01 End=`date -d tomorrow +%Y-%m-%d` account=eurohpc-2010pa5872-users grouping=9999999 Partition=cpu format=Account%25
--------------------------------------------------------------------------------
Job Sizes 2021-07-01T00:00:00 - 2022-03-24T13:59:59 (23036400 secs)
TRES type is billing
Time reported in Hours
--------------------------------------------------------------------------------
Account 0-9999998 TRE >= 9999999 TR % of cluster
------------------------- ------------- ------------- ------------
eurohpc-2010pa5872-users 1 0 100.00%
Output is shows that user1 had used 1 Thread Hours, or 0.5 Core Hours.
For usage on gpu partition is used following command:
$ sreport job SizesByAccount -T billing User=user1 -t Hours Start=2021-07-01 End=`date -d tomorrow +%Y-%m-%d` account=eurohpc-2010pa5872-users grouping=9999999 Partition=gpu format=Account%25
--------------------------------------------------------------------------------
Job Sizes 2021-07-01T00:00:00 - 2022-03-24T13:59:59 (23036400 secs)
TRES type is billing
Time reported in Hours
--------------------------------------------------------------------------------
Account 0-9999998 TRE >= 9999999 TR % of cluster
------------------------- ------------- ------------- ------------
eurohpc-2010pa5872-users 5315 0 100.00%
Output is shows that user1 had used 5315 Thread Hours, or 2657.5 Core Hours.
Summarizing: user1 had usage of 1 Thread Hours on cpu partition and 5135 Thread Hours on gpu partition, whats gives 5316 total Thread Hours.