Introduction
Introduction to Parallel Programming
What is MPI?
MPI stands for Message Passing Interface. It is a standardized and portable way to program parallel applications that run on distributed memory systems, such as clusters or supercomputers.
Unlike multithreading (which uses shared memory), MPI assumes that each process has its own private memory. These processes communicate by sending and receiving messages.
MPI is the most common standard used in High Performance Computing (HPC) environments.
Core MPI Concepts
Communicator
A communicator in MPI defines a group of processes that can exchange messages. It provides a communication context, meaning that messages sent within one communicator are isolated from others.
The most commonly used communicator is:
MPI.COMM_WORLD: the default communicator that includes all processes launched by your MPI job.
Each communicator manages its own set of process IDs (called ranks). Two communicators can have the same ranks referring to different processes.
Rank and Size
rank: the unique ID of a process within a communicator. Ranks start at 0 and go up tosize - 1.size: the total number of processes in the communicator.
Example: Rank, Size, and Communicator (Single Node)
We'll use Python and C for the examples, but any programming language can be used with MPI.
- Python
- C language
Python file: mpi_beginner_one_node.py
from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
print(f"Hello! I am process {rank} out of {size} processes in the communicator.")
Slurm script : mpi_beginner_one_node.sh
#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_one_node
#SBATCH --output=mpi_for_beginner_one_node.txt
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=std
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module load python/3.12.5
srun python mpi_beginner_one_node.py
This script will start 4 MPI processes on a single node. To run it:
sbatch mpi_beginner_one_node.sh
C file: mpi_beginner_one_node.c
#include <stdio.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
printf("Hello! I am process %d out of %d processes in the communicator.\n", rank, size);
MPI_Finalize();
return 0;
}
We compile the program with mpicc :
module load mpi/openmpi/4.1.8/gcc
mpicc -o mpi_beginner_one_node mpi_beginner_one_node.c
Slurm script : mpi_beginner_one_node.sh
#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_one_node
#SBATCH --output=mpi_for_beginner_one_node.txt
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=std
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module load mpi/openmpi/4.1.8/gcc
srun ./mpi_beginner_one_node
This script will start 4 MPI processes on a single node. To run it:
sbatch mpi_beginner_one_node.sh
Output :

Each of the 4 MPI processes printed its rank and the total size. All processes ran in parallel on the same node.
Example: Multi-node Execution
To see MPI working across multiple nodes, let's slightly modify the program.
- Python
- C language
Python file : mpi_beginner_multi_nodes.py
from mpi4py import MPI
import socket
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
print(f"Hello! I am process {rank} out of size {size}. Running on host: {socket.gethostname()}")
We added socket.gethostname to display the host (node) where each process is running.
Slurm script : mpi_beginner_multi_nodes.sh
#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_multi_nodes
#SBATCH --output=mpi_for_beginner_multi_nodes.txt
#SBATCH --ntasks=8
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --partition=std
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module load python/3.12.5
srun python mpi_beginner_multi_nodes.py
This script requests 8 tasks, distributed across 2 nodes (4 processes per node).
To run it:
sbatch mpi_beginner_one_node.sh
C file : mpi_beginner_multi_nodes.c
#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
#include <unistd.h> // for gethostname()
int main(int argc, char *argv[]) {
int rank, size;
char hostname[256];
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
gethostname(hostname, sizeof(hostname));
printf("Hello! I am process %d out of %d. Running on host: %s\n", rank, size, hostname);
MPI_Finalize();
return 0;
}
We added gethostname to display the host (node) where each process is running.
We compile the program with mpicc :
module load mpi/openmpi/4.1.8/gcc
mpicc -o mpi_beginner_multi_nodes mpi_beginner_multi_nodes.c
Slurm script : mpi_beginner_multi_nodes.sh
#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_multi_nodes
#SBATCH --output=mpi_for_beginner_multi_nodes.txt
#SBATCH --ntasks=8
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --partition=std
export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}
module load mpi/openmpi/4.1.8/gcc
srun ./mpi_beginner_multi_nodes
This script requests 8 tasks, distributed across 2 nodes (4 processes per node).
To run it:
sbatch mpi_beginner_one_node.sh
Output

As we can see, the job ran 8 MPI processes across two different nodes (mc-n005.mesu and mc-n015.mesu). This illustrates distributed parallelism: the same code is executed on different physical machines.
The order of the output lines may vary each time you run the program. This is because all MPI processes run in parallel and print at different times, independently of their rank.
Summary
In this tutorial, we introduced:
-
The concepts of communicator, rank, and size.
-
How to run an MPI program using SLURM on:
-
A single node with multiple MPI processes
-
Multiple nodes in a distributed HPC setup
-