Skip to main content

Split and combine data

Scatter, Gather, and Reduce​

In real parallel programs, it's common to split data across processes, perform computations independently, then combine the results. MPI provides collective operations to do exactly this:

  • scatter: distributes chunks of data to each process

  • gather: collects data from each process to one

  • reduce: aggregates data using operations like sum, max, min, etc.

Scatter and Gather Example​

We will send a list of numbers from process 0, give each process a slice, let them double it, and collect the results back.

Python file: mpi_scatter_gather.py​

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()

# Only rank 0 creates the full data to scatter
if rank == 0:
data = [i for i in range(size)]
print(f"[Rank 0] Original data: {data}")
else:
data = None

# Create a variable to receive one element per process
recv_data = comm.scatter(data, root=0)
print(f"[Rank {rank}] Received value: {recv_data}")

# Each process doubles its value
processed = recv_data * 2

# Gather results at rank 0
gathered_data = comm.gather(processed, root=0)

if rank == 0:
print(f"[Rank 0] Gathered result: {gathered_data}")

SLURM script: mpi_scatter_gather.sh​

#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_scatter_gather
#SBATCH --output=mpi_for_beginner_scatter_gather.txt
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --partition=std

module load python/3.12.5

srun python mpi_scatter_gather.py

Output:​

mpi_result_05

note

The order of the output lines may vary each time you run the program. This is because all MPI processes run in parallel and print at different times, independently of their rank.

Each process gets one element, processes it, and sends it back.

reduce: Aggregating Values​

We compute the sum of values distributed across processes.

Python file: mpi_reduce.py​

from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

local_value = rank + 1
print(f"[Rank {rank}] Local value: {local_value}")

total = comm.reduce(local_value, op=MPI.SUM, root=0)

if rank == 0:
print(f"[Rank 0] Total sum: {total}")

SLURM script: mpi_reduce.sh​

#!/bin/bash
#SBATCH --job-name=mpi_for_beginner_reduce
#SBATCH --output=mpi_for_beginner_reduce.txt
#SBATCH --ntasks=4
#SBATCH --nodes=1
#SBATCH --partition=std

module load python/3.12.5

srun python mpi_reduce.py

Output:​

mpi_result_06

Summary​

  • Use scatter to divide data among processes

  • Use gather to collect data back to one process

  • Use reduce to aggregate values (sum, min, max, etc.)