Job Arrays
Job arrays in Slurm allow users to submit and manage a collection of similar jobs easily. Instead of submitting individual jobs separately, a job array allows you to submit multiple similar jobs using a single job script.
Job arrays are particularly useful for identical tasks that operate on different data sets or parameters. For example, you may have a simulation that you need to run many times, with a different set of parameters each run. Or you may have a processing task that you have tested successfully on a single sample, and you need to apply it to 100 samples.
Compared to submitting a single job that performs all the similar tasks you want to achieve, using job arrays will allow your computation to be split in multiple individual jobs that will be processed concurrently.
Set up the job array directive in your script
To create a job array, you must use the --array parameter in your job script or sbatch command.
For example, if you have 10 samples to process, #SBATCH --array=1-10 will tell Slurm to run the script 10 times, meaning create 10 independent array tasks numbered from 1 to 10.
#SBATCH --array=0-15 # 16 jobs (indices from 0 to 15)
#SBATCH --array=10-16:2 # 4 jobs (indices: 10, 12, 14, 16)
#SBATCH --array=2,3,5,7,11,13 # 6 jobs with given indices
#SBATCH --array=1-10000%32 # 10,000 jobs, maximum 32 jobs running in parallel
Any Slurm directives that you specify for resources at the top of your script, for example --cpus-per-task or --nodes etc will be applied to each array task.
You only need to specify the resources required for a single sample/task, not the entire array.
The $SLURM_ARRAY_TASK_ID variable
When working with Slurm job arrays, a key variable to be aware of is $SLURM_ARRAY_TASK_ID. Slurm assigns this variable to each task in the array based on the --array directive specified at the beginning of your script. For example, if you use #SBATCH --array=1-10, the variable $SLURM_ARRAY_TASK_ID will be set to 1 for the first task, 2 for the second task, 3 for the third, and so on, up to 10.
In the simplest scenario, you can directly use the $SLURM_ARRAY_TASK_ID variable within your script to manage your job array.
Suppose you have input files named like this: sample1.data, sample2.data, sample3.data, ..., sample10.data.
Each of your 10 Slurm array tasks can handle a different sample file by using the line sample $\{SLURM_ARRAY_TASK_ID}.data.
This way, task 1 will process sample1.data, task 2 will process sample2.data, and so on.

#!/bin/bash
#SBATCH -J example_array_job # Job name
#SBATCH --array=1-10 # Array range
# Construct the filename based on the task ID
input_file="sample${SLURM_ARRAY_TASK_ID}.data"
# Print the input file for debugging purposes
echo "Processing file: ${input_file}"
# Run your command or script using the input file
your_command ${input_file}
Passing the $SLURM_ARRAY_TASK_ID Variable to Python or R Script
If your Slurm script calls a custom Python or R script, you can access the $SLURM_ARRAY_TASK_ID variable just like any other environment variable. In Python, for example, you can use os.getenv("SLURM_ARRAY_TASK_ID") to retrieve its value. This approach allows you to customize the behavior of each array task dynamically based on the task ID.