Monitor your jobs
Once your job has been submitted to Slurm via the sbatch command, multiple tools are available to monitor and control your job.
1 – Getting general job information
Slurm native tools
The squeue command gives information about the current jobs queued or running on MCMeSU (more information with man squeue).
Without arguments, squeue will display the current jobs for all users :
[user1@mc-login1]$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
28 std ****** ****** R 1:21:05 10 mc-n[009-018]
38 std ****** ****** R 20:05 4 mc-n[002-005]
You can use myqueue or squeue -u $USER to show your jobs only :
[user1@mc-login1]$ myqueue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
28 std ****** ****** R 1:21:05 10 mc-n[009-018]
If you know youd job identifier (output of the squeue --user=$USER or myqueue command), you can get information about your specific job with the scontrol show jobid command.
[user1@mc-login1]$ scontrol show jobid 28
JobId=28 JobName=*****
UserId=*****(11111) GroupId=users(100) MCS_label=N/A
Priority=21 Nice=0 Account=***** QOS=std WCKey=*
JobState=RUNNING Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
RunTime=01:34:06 TimeLimit=1-00:00:00 TimeMin=N/A
SubmitTime=2024-04-25T13:56:56 EligibleTime=2024-04-25T13:56:56
AccrueTime=2024-04-25T13:56:56
StartTime=2024-04-25T13:56:56 EndTime=2024-04-26T13:56:56 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-04-25T13:56:56 Scheduler=Backfill
Partition=std AllocNode:Sid=mc-login1:11111
ReqNodeList=(null) ExcNodeList=(null)
NodeList=mc-n[009-018]
BatchHost=mc-n009
NumNodes=10 NumCPUs=320 NumTasks=320 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=320,mem=2500G,node=10,billing=320
AllocTRES=cpu=320,mem=2500G,node=10,billing=320
Socks/Node=* NtasksPerN:B:S:C=32:0:*:1 CoreSpec=*
MinCPUsNode=32 MinMemoryCPU=8000M MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
Command=/scratch/*****/ZOOMImage/job.sh
WorkDir=/scratch/*****/ZOOMImage
StdErr=/scratch/*****/ZOOMImage/slurm-28.out
StdIn=/dev/null
StdOut=/scratch/*****/ZOOMImage/slurm-28.out
Power=
If you want to display detailed status of job arrays, use squeue -r or myqueue -r.
2 – Deleting or stopping a job
At any time, you can delete (or stop) a queued (or running) job by using the scancel <jobID> command :
[user1@mc-login1]$ scancel 123456
If you’re job is running, Slurm will handle the killing of its processes.
3 – Connecting to a computing node for more details
Once your job is running (status R in qstat or qqueue), you can connect to one of the used nodes with ssh in order to inspect the status of the node for instance, or to dynamically interact with your job. Identify the nodes your job has been dispatched to with or squeue -j <job_id> and run one of the following commands depending on the server your job has been dispatched to :
# To connect to computing node mc-n001
ssh mc-n001
Once connected to a node, you can run the same “classical” system monitoring tools (top, pstree, watch, ...) that you would on a desktop computer.