Job submission
MCMeSU Supercomputer uses a queuing system (aka batch scheduler) to match users jobs with available computing resources.
Users submit their programs to the job scheduler (Slurm), which maintains a queue of jobs and distributes them on the compute nodes according to the servers status, scheduling policies and jobs parameters (number of compute nodes / cores, estimated execution time, required memory, etc.).
The interface with Slurm is done via a text file – the job script or Slurm script – created by the user, which will define your job requirements and execution steps. This file is mainly comprised of two sections :
- the header in which you specify the job requirements (execution time, number of CPU cores to use, memory requirements…) in the form of Slurm directives.
- the body in which you will write the commands to load specific software, define environment variables, and run your job.
You can use the script generator to effortlessly create a script.
Job script header
The header is a succession of Slurm directives which syntax is :
#SBATCH [option]
The most relevant options on MCMeSU Supercomputer are :
- #SBATCH -p std Partition (or type of resources) to use, possible values are thin, std, bigm and gpu (default is std)
- #SBATCH --time=4-03:20:10 Maximal execution time of the job, for example here : 4 days, 3 hours, 20 minutes and 10 seconds (default is 24 hours)
- #SBATCH -J jobName Name of the job
- #SBATCH -o stdout.txt Standard output file during job script execution
- #SBATCH --mail-user=user@example.com Email address for email notifications
- #SBATCH --mail-type=ALL Enables email notification for all job scheduling evens
Then, according to the kind of parallelism your application is using, you should set the number of CPUs or parallel tasks required.
If you do not know the kind of parallelism your application supports, please contact us for guidance.
Shared-Memory/Multithreaded/OpenMP Execution
For shared-memory computing (i.e. multithreaded or OpenMP-based execution), you should specify :
- #SBATCH -N 1 Number of servers that will be used (multithreaded execution can only work with a single server)
- #SBATCH -n 1 Number of parallel tasks that will be launched (multithreaded execution needs only one task)
- #SBATCH -c 32 Number of CPUs that will be allocated to each task (set this value to the number of threads you need)
Distributed/MPI Execution
For distributed computing (i.e. multiprocessing or MPI-based execution), you should specify :
- #SBATCH -N 1 Number of servers that will be used (this value should be based on the number of tasks you want to launch)
- #SBATCH -n 32 Number of parallel MPI tasks that will be launched
- #SBATCH -c 1 Number of CPUs that will be allocated to each MPI task (use only one CPU per task, unless your application supports hybrid MPI-OpenMP execution)
Here is, for instance, a typical header for a MPI job spawning 32 processes on a single server with a maximum execution time of 30 minutes :
#SBATCH -N 1
#SBATCH -n 32
#SBATCH -c 1
#SBATCH --time=00:30:00
We advise you to look into sbatch manual page (man sbatch) to learn more about job configuration.
Job script body
After having defined your resources needs in the header, you will have to define the execution steps of your job.
This includes (but is not limited to) :
- Setting-up environment variables
- Copying data files, program executables
- Specifying the output directories
- Specifying the execution command
- Cleaning-up
A job script is a shell script, thus you can use the same constructs and commands as in a regular Linux shell script.
Job Environment Variables
Different environment variables are accessible within a Slurm job script.
They will allow you to interact with Slurm resource allocation, temporary directories and various job information :
- $SLURM_SUBMIT_DIR Directory where the sbatch command was executed
- $SLURM_JOB_ID Job ID number given to this job. This number is used by many of the job management tools such as
squeue,myqueueorscancel. - $SLURM_JOB_NAME Name of the job. This can be set using the -J option in the slurm job script (or from the command line).
- $SLURM_JOB_NODELIST List of all nodes allocated to the job.
- $SLURM_NUM_NODES Total numver of nodes in the job's resource allocation.
- $SLURM_ARRAY_TASK_ID Job array ID (index) number.
- $SLURM_NTASKS Number of tasks in the job (by default, one CPU is allocated per task).
- $SCRATCH Your base directory in the scratch storage.
- $STORE Your base directory in the store storage.
- $TMPDIR Local temporary disk storage unique to each node and each job
- $USER Your user name. Useful if you would like to dynamically generate a directory on some scratch space.
- $HOSTNAME Name of the computer currently running the script.
Submit your script/job
Submitting your job to Slurm is easy once you have authored the corresponding job script file, using the sbatch command :
sbatch [path/to/your/script]
For instance, if you saved your script file as myScript.sh, just run the following command in a terminal:
sbatch myScript.sh
Note that you can also specify SBATCH directives in the command line as sbatch options, which will override the ones defined in the script.
For instance, to override the CPU requirements and target partition of a Slurm script, run the command:
sbatch -p bigm -n 64 myScript.sh