Skip to main content

Data Management

Understanding the different types of storage available on MCMeSU is crucial for efficient data management and job execution. Here’s an overview of the three storage : HOME, SCRATCH, and STORE.

HOME Storage​

HOME storage is the landing space of a user connection to MCMeSU, each user owns a dedicated directory /home/$USER or $HOME.

Characteristics​

  • Fair Performance : Designed to allow a large number of simultaneous lightweight tasks like preparing scripts, editing files, and submitting jobs.
  • Small capacity : Users are expected to store less than 100 GB in their HOME.
  • Security : This storage is secured and made redundant.

Use-case Examples​

  • Job submission scripts ;
  • Personal configuration files (e.g., .bashrc, .profile) ;
  • Source code for applications.

SCRATCH Storage​

SCRATCH storage is designed to be a high-performance temporary workspace, each user owns a dedicated directory /scratch/$USER or $SCRATCH

Characteristics​

  • High Performance : Designed for fast read/write operations, crucial for intensive computational jobs.
  • Large Capacity : Typically more spacious than other storage types to accommodate large-scale operations.
  • Temporary : Old files (>3 months old) on scratch storage can be automatically purged without notice to free up space, do not store important unused data here.

Use-case Examples​

  • Intermediate data from simulations or analyses.
  • Temporary files and job outputs.
  • Results not yet saved elsewhere.

STORE Storage​

STORE storage is intended for long-term storage of large amounts of data, each user owns a dedicated directory /store/$USER or $STORE.

Characteristics​

  • Archiving : Store data that needs to be preserved after processing, such as final results from simulations or research projects.
  • Large Capacity : Optimized for massive storage capacity at the expense of read/write performance.
  • Security : This storage is secured and made redundant.

Use-case Examples​

  • Final project results.
  • Raw data collected for future analysis.
  • Backups of code and configurations.

Moving data to STORE​

For moving large datasets, we recommend using rsync. It is a powerful tool used to copy and synchronize files between different locations, either locally or over a network.

To move your data safely, here's a sequence of commands you may follow. We'll assume that you're backing up a sub-directory in your SCRATCH named "mydata".

First, we set up a destination directory :

cd "$STORE"
mkdir backup-scratch
cd backup-scratch

Then, we use rsync to copy the sub-directory from the SCRATCH to the STORE :

rsync -arv --progress "$SCRATCH/mydata/" .

If you interrupt the transfer or want to ensure that data has been copied, you can repeat the same command (only the missing data will be transferred).

Finally, when rsync is done, we can delete the copied data :

cd "$SCRATCH"
rm -rf mydata

Note that the rsync command can be aborted at any time if required.
You can carry out the transfer several times, as long as you use the same command each time.

Summary​

To help you visualize these concepts, here’s a summary table:

Type of StoragePrimary Use-caseCharacteristicsExamples of Content
HOMEConfiguration and preparationSmall capacity, fair performance, securedJob scripts, config files
SCRATCHIntensive and temporary computationsHigh performance, purged, no redundancyIntermediate data, job outputs
STORELong-term storage and archivingLarge capacity, low performance, securedFinal results, backups, raw data

Process of using your data on MCMeSU​

1. Sending your data and script in your HOME storage, STORE storage or directly to your SCRATCH storage using scp or sftp. Computation process

2. After preparing your job, transfer all the files you need for your job on the SCRATCH storage (you can organise or compile your code from your HOME, SCRATCH or STORE directory). Computation process

3. Start your script with sbatch from SCRATCH, it is optimized for fast transfer and computation. Computation process

4. Once your job is done, you can find your results in SCRATCH. Transfer some results and scripts files in your HOME storage if need adjustements to start another job, or to the STORE storage if you want to save your results. Computation process

Best Practices​

  • Appropriate Use: Utilize each type of storage for its intended purpose. Avoid keeping important files on SCRATCH storage.
  • Monitoring: Regularly monitor the space you use in SCRATCH storage to avoid data loss due to purges (for example: du -hs $SCRATCH).
  • Regular Backups: Backup important data on STORE storage or other external means to safeguard against accidental losses.

These different types of storage are designed to offer you flexibility and performance. Each has a specific role that, when properly understood and utilized, can greatly enhance your efficiency and data management.

Data Usage Report​

Storage spaces of MCMeSU are continually monitored.

Using the mydata command, you can get a report of your storage occupation :

[user1@mc-login1]$ mydata

Data storage report of user

HOME : 8.3 GB [0.0% of total capacity]
SCRATCH : 2.1 TB [0.5% of total capacity]
STORE : 8.6 TB [1.2% of total capacity]

Last refresh : 2026-02-25 15:01:58

In case we notice an unusual storage usage, we will get in touch with you.