Skip to main content

Introduction

Introduction to Parallel Programming​

What is MPI?​

MPI stands for Message Passing Interface. It is a standardized and portable way to program parallel applications that run on distributed memory systems, such as clusters or supercomputers.

Unlike multithreading (which uses shared memory), MPI assumes that each process has its own private memory. These processes communicate by sending and receiving messages.

MPI is the most common standard used in High Performance Computing (HPC) environments.


Core MPI Concepts​

Communicator​

A communicator in MPI defines a group of processes that can exchange messages. It provides a communication context, meaning that messages sent within one communicator are isolated from others.

The most commonly used communicator is:

  • MPI.COMM_WORLD: the default communicator that includes all processes launched by your MPI job.

Each communicator manages its own set of process IDs (called ranks). Two communicators can have the same ranks referring to different processes.


Rank and Size​

  • rank: the unique ID of a process within a communicator. Ranks start at 0 and go up to size - 1.
  • size: the total number of processes in the communicator.

Example: Rank, Size, and Communicator (Single Node)​

We'll use Python and C for the examples, but any programming language can be used with MPI.

Python file: mpi_beginner_one_node.py​

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print(f"Hello! I am process {rank} out of {size} processes in the communicator.")

Slurm script : mpi_beginner_one_node.sh​

#!/bin/bash

#SBATCH --job-name=mpi_for_beginner_one_node
#SBATCH --output=mpi_for_beginner_one_node.txt
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --cpus-per-task=1
#SBATCH --partition=std

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load python/3.12.5

srun python mpi_beginner_one_node.py

This script will start 4 MPI processes on a single node. To run it:

sbatch mpi_beginner_one_node.sh

Output :​

mpi_result_01

Each of the 4 MPI processes printed its rank and the total size. All processes ran in parallel on the same node.

Example: Multi-node Execution​

To see MPI working across multiple nodes, let's slightly modify the program.

Python file : mpi_beginner_multi_nodes.py​

from mpi4py import MPI
import socket

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

print(f"Hello! I am process {rank} out of size {size}. Running on host: {socket.gethostname()}")

We added socket.gethostname to display the host (node) where each process is running.

Slurm script : mpi_beginner_multi_nodes.sh​

#!/bin/bash

#SBATCH --job-name=mpi_for_beginner_multi_nodes
#SBATCH --output=mpi_for_beginner_multi_nodes.txt
#SBATCH --ntasks=8
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1
#SBATCH --partition=std

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

module load python/3.12.5

srun python mpi_beginner_multi_nodes.py

This script requests 8 tasks, distributed across 2 nodes (4 processes per node).

To run it:

sbatch mpi_beginner_one_node.sh

Output​

mpi_result_02

As we can see, the job ran 8 MPI processes across two different nodes (mc-n005.mesu and mc-n015.mesu). This illustrates distributed parallelism: the same code is executed on different physical machines.

note

The order of the output lines may vary each time you run the program. This is because all MPI processes run in parallel and print at different times, independently of their rank.

Summary​

In this tutorial, we introduced:

  • The concepts of communicator, rank, and size.

  • How to run an MPI program using SLURM on:

    • A single node with multiple MPI processes

    • Multiple nodes in a distributed HPC setup