Scratch storage

A scratch space is a high-performance storage that must be used for your data and jobs files.

On MeSU supercomputer, the scratch storage is named scratchbeta.
It is a Lustre filesystem, served through Infiniband network, and it has much better performances than the home filesystems.
It is available on the login nodes, on beta nodes and on gamma nodes.

Quotas and backups

Back-up : please keep in mind that the scratch directory doesn’t have any backup system.
It is therefore important to check that you do not store any important data on /scratchbeta, without having a backup of them elsewhere.

Quota : scratch space does not have any quota, we monitor however the age of the files and may remove files unused for three months.

Access to scratch directory

The scratch directory is created on your first login, and is located at /scratchbeta/$USER
If you do not tell PBS to use those spaces explicitly, your job will be executed from your current working directory (most often your home directory).

Steps to use scratch directory :

  • Compile your code if needed (you can do it in your /home directory, but also in your scratch spaces).
  • Move any input data files to your scratch directory : /scratchbeta/$USER
  • If you wish to run your job directly from /scratchbeta :
    • Copy your executable and your PBS script in a dedicated directory in your scratch space.
    • Change your working directory to /scratchbeta/$USER with cd
    • Launch your script with qsub from /scratchbeta/$USER
  • If you wish to run your job from your /home directory:
    • edit your PBS script to use the appropriate environment variables and read/write from/to the correct directories
    • Launch your script from your home directory

For instance, here is a script you could launch from your /home directory, which will write in your /scratchbeta directory :

#!/bin/bash
#PBS -q beta
#PBS -l select=1:ncpus=24:mpiprocs=24
#PBS -l walltime=60:00:00
#PBS -N myTestJob

# load appropriate modules
module purge
module load intel/intel-compilers-18.0/18.0

#move to PBS_O_WORKDIR
cd $PBS_O_WORKDIR

# Define scratch space scratch
SCRATCH=/scratchbeta/$USER/myprogram_scratch_space
PROJECT=’my project name’
mkdir $SCRATCH
mkdir $SCRATCH/$PROJECT

# copy some input files to  $SCRATCH directory
cp some_input_files  $SCRATCH/$PROJECT

#execute your program
cd $SCRATCH/$PROJECT || exit 1
myprogram 1> myprogram.out 2> myprogram.err

# copy some output files to submittion directory and delete temporary work files
cp -p some_output_files $PBS_O_WORKDIR || exit 1

#clean the temporary directory
rm -rf ${SCRATCH}/${PROJECT}

Just keep in mind the quotas on homes and the absence of backups on scratch spaces. in your home must be only the results and the codes you need and your personal software (conda environnement, your own compilation of a program with the options you specify …)