← Home

Scaling Behavior of Gromacs

gromacs molecular dynamics benchmarking cuda gpu

Old work. Smuggling this into the open. Original work done October 2025.

Introduction #

GROMACS is a widely used molecular dynamics simulation package known for its high performance and efficiency, particularly for biological systems, and using GPU-accelerated compute. The release of GROMACS 2025.3 introduced several enhancements to GPU usage, including long range electrostatic interactions via particle-mesh Ewald (PME).

A static scaling test was performed on both GROMACS 2024.2 and GROMACS 2025.3 to compare and evaluate the performance of the two versions using a fixed problem size and fixed computational resources, but varying the specific MPI rank, OpenMP thread, and PME offload configurations. Then, a more intensive exploration of 2025.3 configurations was conducted to identify performance differences between GPU card models and multiple PME ranks (-npme > 1).

The goal was to not only showcase the improvements in GROMACS 2025.3 but also to provide insights into how different configurations can impact performance.

Compilation Notes #

The newer GROMACS 2025.3 was compiled using the NVHPC SDK 25.1, which includes the NVIDIA compilers and libraries optimized for NVIDIA GPUs. In particular:

  • cuFFTmp was enabled for shared GPU FFT for PME tasks, allowing for multiple PME ranks (-npme > 1) to be used effectively with GPU offloading
  • new 'cutting edge' GPU direct communication via HPCX remote-direct-memory-access (RDMA) was enabled for GPU-GPU communication
  • full CUDA-aware build via HPC SDK's OpenMPI/IB (NVHPC builds possible as of 2025.2 see commit)
  • BLAS/FFTW3 enabled for CPU-only tasks

Of note, this build has hidden environment variables which effect performance:

# Specify that GPU RDMA communication should be used
export GMX_ENABLE_DIRECT_GPU_COMM=1
# Specify that GPU PME decomposition should be used
export GMX_GPU_PME_DECOMPOSITION=1

These settings are recommended for optimal performance with GROMACS 2025.3 on GPU-accelerated systems.

The older GROMACS 2024.2 was compiled using GCC with OpenMPI and CUDA 11.8, which is a more traditional setup for GROMACS installations.

Test System #

The test system used was the Satellite Tobacco Mosaic Virus (STMV) system, which consists of 1,066,628 atoms. The system was prepared using the CHARMM36m force field and solvated in a water box with appropriate ions to neutralize the charge. The input files were taken from the Heterogeneous Parallelization Benchmark Info and Systems repository, specifically the stmv system, which was provided as supplmental material in Páll et. al's "Heterogeneous parallelization and acceleration of molecular dynamics simulations in GROMACS." J. Chem. Phys. 7 October 2020; 153 (13): 134110. https://doi.org/10.1063/5.0018516.

Simulation Runs #

Test One: GROMACS 2024.2 vs 2025.3 #

Test Environment #

The test was conducted on the following hardware:

  • Node: HPE ProLiant Gen10 Plus
    • CPU: AMD EPYC 7662 (64 cores, 128 threads, 2.0 GHz)
    • GPUs: 4x NVIDIA A10 (24GB VRAM, 448 Tensor Cores, 10752 CUDA Cores)
    • RAM: 64/512 GB DDR4
  • Software:
    • GROMACS 2024.2: compiled with GCC + OpenMPI + CUDA 11.8
    • GROMACS 2025.3: compiled with NVHPC SDK 25.1 + OpenMPI + CUDA 12.6

Setup #

Each simulation was run for 50000 steps with the last 10000 steps used for performance measurement. The following command was used to run the simulations, with variations in the number of MPI ranks, OpenMP threads, and PME offload configurations:

mpirun -np <MPI_RANKS> gmx mdrun -v \
-nsteps 50000 -resetstep 40000 \
-pin on -ntomp $OMP_NUM_THREADS \
-pme gpu -npme <NPME> \ 
-bonded gpu -update gpu -nstlist 300

Where <MPI_RANKS>, $OMP_NUM_THREADS, and <NPME> were varied according to the test configurations outlined below. All simulations were run on the same node, with 4x NVIDIA A10 GPUs, and the same input files were used for both GROMACS versions to ensure a fair comparison.

Furthermore, the following environment variables were set to optimize performance for version 2025.3:

export GMX_ENABLE_DIRECT_GPU_COMM=1 
export GMX_GPU_PME_DECOMPOSITION=1

Test Configurations and Results #

The tests were performed using a fixed problem size (1,066,628 atoms) and varying the number of MPI ranks and OpenMP threads between versions. For 2025.3 alone, the npme variable was also varied across the 8x8 configuration. The following configurations were tested:

GROMACS VersionMPI RanksOpenMP ThreadsnpmePerformance (ns/day)
2024.2416127.50
2024.288124.73
2024.21641MPI failure
2025.3416129.86
2025.388138.27
2025.388225.20
2025.38839.888
2025.3164126.26

Notes #

2025.3 outperformed 2024.2 across all tested configurations. The most significant performance improvement was observed in the configuration with 8 MPI ranks and 8 OpenMP threads, where GROMACS 2025.3 achieved a performance of 38.27 ns/day compared to 24.73 ns/day for GROMACS 2024.2, representing a 54.7% increase in performance on the same hardware and problem size. The configuration with 16 MPI ranks and 4 OpenMP threads was not successful in GROMACS 2024.2 due to MPI failures, while GROMACS 2025.3 managed to run this configuration, achieving a performance of 26.26 ns/day. Multiple PME ranks (-npme > 1) in GROMACS 2025.3 did not yield better performance for this specific test case; however, this may vary with different systems and configurations and is worth further exploration for systems with larger PME (Coulombic dominant) workloads.

Test Two: GROMACS 2025.3 GPU Model and npme Exploration #

A more intensive exploration of GROMACS 2025.3 configurations was conducted to identify performance differences between GPU card models and multiple PME ranks (-npme > 1). The tests were performed on the various GPU models available on the cluster, using the same STMV system and input files as in Test One. In addition, the nstlist parameter, which controls the frequency of neighbor list updates, was varied to assess its impact on performance. In this test, a fixed configuration of 8 MPI ranks and 4 OpenMP threads was used. The following command was used to run the simulations:

export GMX_ENABLE_DIRECT_GPU_COMM=1 
export GMX_GPU_PME_DECOMPOSITION=1 
export OMP_PLACES=cores 
export OMP_PROC_BIND=close 

mpirun -np 8 gmx mdrun -v \
-nsteps 50000 -resetstep 40000 \
-pin on -ntomp 4 \
-pme gpu -npme $4 \ 
-bonded gpu -update gpu \
-nb gpu -nstlist $3 \ 
-noconfout

Where $1 is the number of GPUs, $2 is the GPU model (e.g., A10, A100, L4, etc.), $3 is the nstlist value (e.g., 60, 120, or 300), and $4 is the number of PME ranks (e.g., 1, 2, 3). Each simulation was run for 50000 steps with the last 10000 steps used for performance measurement.

Test Configurations and Results #

glistngpusnstnpmens/day
a1023003killed
a10023003killed
a4023003killed
h10023003killed
l423003killed
l40s23003killed
a1021203killed
a10021203killed
a4021203killed
h10021203killed
l421203killed
l40s21203killed
a102603killed
a1002603killed
a402603killed
h1002603killed
l42603killed
l40s2603killed
l40s4300123.557
l40s4120120.733
l40s2300117.745
a104300117.358
l44300117.312
l40s460116.704
l40s2120116.035
l44120115.731
a104120115.314
h1002300114.34
l40s260113.368
a402300113.308
l4460113.192
a10460113.106
h1002120113.027
a402120112.797
a1002300112.452
a102300111.802
a1002120111.452
l42300111.315
h100260111.061
a102120110.936
l42120110.766
a40260110.203
a10026019.609
l426019.475
a1026019.283
h100230026.066
h100212025.631
h10026025.53
l40s430024.241
a10430024.127
a10412024.1
l4430024.029
a40212024.02
a1046024.007
l40s46023.999
l40s26023.98
l40s412023.794
l446023.73
a40230023.716
l4412023.698
l40s230023.605
a100212023.569
l426023.566
a100230023.555
l4230023.536
a10230023.498
l40s212023.47
l4212023.43
a10212023.412
a1026023.308
a10026023.293
a4026023.169
l4430032.999
l4412032.79
l40s412032.725
l40s46032.577
l40s430032.571
a10412032.505
a10430032.445
a1046032.395
l446032.379

Analysis #

The results indicate that the NVIDIA L40S GPU model provided the best performance for the STMV system with 1,066,628 atoms when using 4 GPUs, achieving a maximum performance of 23.557 ns/day with nstlist=300 and npme=1. The NVIDIA A10 and L4 models also performed well, with maximum performances of 17.358 ns/day and 17.312 ns/day, respectively, under similar configurations. The NVIDIA H100, A100, and A40 models showed lower performance in this specific test case, with maximum performances of 14.34 ns/day, 12.452 ns/day, and 13.308 ns/day, respectively.

This points to FP32 performance, rather than memory bandwidth or tensor core count, as the limiting factor for GPU offload performance with this test system, and this is probably indicative of how GROMACS utilizes GPU resources for this type of workload.

The tests with multiple PME ranks (-npme > 1) generally resulted in lower performance compared to using a single PME rank (-npme=1). This suggests that for the STMV system, the overhead associated with managing multiple PME ranks outweighs the benefits of parallelizing the PME calculations across multiple GPUs, even with >1M atoms.

Finally, varying the nstlist parameter showed that a higher value (e.g., nstlist=300) generally led to better performance, likely due to reduced frequency of neighbor list updates, which can be computationally expensive and cannot be vectorized efficiently, as the number of neighbor interactions varies per atom.

Conclusion #

The static scaling test demonstrates the significant performance improvements in GROMACS 2025.3 over GROMACS 2024.2, particularly when utilizing a balanced configuration of MPI ranks and OpenMP threads. The enhancements in GPU offloading, direct GPU communication, and overall optimization in the newer version contribute to these performance gains.

The exploration of GROMACS 2025.3 configurations across different GPU models and PME rank settings highlights the importance of selecting the appropriate hardware and configuration for optimal performance. The NVIDIA L40S GPU model demonstrated superior performance for the STMV system, and using a single PME rank with a higher nstlist value was found to be the most effective configuration.

While it is now possible to use multiple PME ranks with GPU offloading in GROMACS 2025.3, it may not always yield better performance, and careful benchmarking is recommended.

References #