abstract:  The introduction of accelerator devices such as graphics processing units (GPUs) has had profound impact on molecular dynamics simulations and has enabled order-of-magnitude performance advances using commodity hardware. To fully reap these benefits, it has been necessary to reformulate some of the most fundamental algorithms, including the Verlet list, pair searching and cut-offs. Here, we present the heterogeneous parallelization and acceleration design of molecular dynamics implemented in the GROMACS codebase over the last decade. The setup involves a general cluster-based approach to pair lists and non-bonded pair interactions that utilizes both GPUs and CPU SIMD acceleration efficiently, including the ability to load-balance tasks between CPUs and GPUs. The algorithm work efficiency is tuned for each type of hardware, and to use accelerators more efficiently we introduce dual pair lists with rolling pruning updates. Combined with new direct GPU-GPU communication as well as GPU integration, this enables excellent performance from single GPU simulations through strong scaling across multiple GPUs and efficient multi-node parallelization.
author: Szilárd Páll; Artem Zhmurov; Paul Bauer; Mark Abraham; Magnus Lundborg; Alan Gray; Berk Hess; Erik Lindahl
bibliography: merged_stripped.bib
date: 2024-10-02
title: Heterogeneous Parallelization and Acceleration of Molecular Dynamics Simulations in GROMACS

[^1]

[^2]

# Introduction

Molecular dynamics (MD) simulation has had tremendous success in a number of application areas the last two decades, in part due to hardware improvements that have enabled studies of systems and timescales that were previously not feasible. These advances have also made it possible to introduce better algorithms and longer simulations have enabled more accurate calibration of force fields against experimental data, all of which have contributed to increasing trust in computational studies. However, the high computational cost of evaluating forces between all particles combined with integrating over short time steps (${\sim} \SI{2}{\femto \second}$) has led to fundamental challenges for the field as the speed of individual processor cores is no longer increasing. Without algorithms that can better exploit new parallel hardware, the timescales accessible in simulations will hit a brick wall. Unlike some other fields, improving resolution by increasing the model detail, e.g. with quantum effects or increasing the size of the system, cannot replace reaching longer timescales. In most cases molecular dynamics targeting a specific application depends critically on achieving faster simulations by reducing the time each MD step takes.

One successful recent approach has been the introduction of enhanced sampling based on ensembles of simulations. When combined with parallelization of individual runs, this makes it possible to use the largest high performance computing (HPC) resources in the world to study even small systems. However, even for HPC systems a high rate of producing trajectories is imperative to sample dynamics covering adequate timescales, which means cost-efficiency and throughput are of paramount importance.

The design choices in GROMACS are guided by a bottom-up approach to parallelization and optimization, partly due to the code's roots of high performance on cost-efficient hardware. This is not without challenges; good arguments can be made for focusing either top-down on scaling or just sticking to single-GPU simulations. However, by employing state-of-the-art algorithms and efficient parallel implementations the code is able to target hardware and efficiently parallelize from the lowest level of SIMD (single instruction, multiple data) vector units to multiple cores and caches, accelerators, and distributed-memory HPC resources.

We believe this approach makes great use of limited compute resources to improve research productivity, and it is increasingly enabling higher absolute performance on any given resource. Exploiting low-level parallelism can be tedious and has often been avoided in favor of using more hardware to achieve the desired time-to-solution. However, the evolution of hardware is making this trade-off increasingly difficult. The end of microprocessor frequency scaling and the consequent increase in hardware parallelism means that targeting all levels of parallelism is a necessity rather than option. The MD community has been at the forefront of investing in this direction, and our early work on scalable algorithms, fine-grained parallelism and low-level portable parallelization abstractions have been previous steps on this path.

Accelerators such as GPUs are expected to provide the majority of raw FLOPS in upcoming Exascale machines. However, the impact of GPUs can also be seen in low- to mid-range capacity computing, especially in fields like MD that have been able to utilize the high instruction throughput as well as single precision capabilities; this has had particularly high impact in making consumer GPU hardware available for scientific computing.

While algorithms with large amounts of fine-grained parallelism are well-suited to GPUs, tasks with little parallelism or irregular data-access are better suited to CPU architectures. Accelerators have become increasingly flexible, but still require host systems equipped with a CPU. While there has been some convergence of architectures, the difference between latency- and throughput-optimized functional units is fundamental, and utilizing each of them for the tasks at which they are best suited requires *heterogeneous parallelization*. This typically employs the CPU also for scheduling work, transferring data and launching computation on the accelerator, as well as inter- and intra-node communication. Accelerator tasks are launched asynchronously using APIs such as CUDA, OpenCL or SYCL to allow concurrent CPU–GPU execution. The heterogeneous parallelization model adds complexity which comes at a cost, both in terms of hardware (latency, complex topology) and programmability, but it provides flexibility (every single algorithm does not need to be implemented on the accelerator) and opportunities for higher performance. Heterogeneous systems are evolving fast with very tightly coupled compute units, but the heterogeneity in HPC will remain and is likely best addressed explicitly.

Our first GPU support was introduced in GROMACS 4.5 and relied on a homogeneous acceleration by implementing the entire MD calculation on the GPU. The same approach has been used by several codes (e.g. ACEMD, AMBER, HOOMD-blue, FENZI, DESMOND-GPU) and has the benefit that it keeps the GPU busy, avoiding communication as long as scaling is not a concern. However, this first approach also had shortcomings: only algorithms ported to the GPU can be used in simulations, which limits applicability in large community codes. Implementing the full set of MD algorithms on all accelerator frameworks is not practical from porting and maintenance concerns. In addition, our experience showed that outperforming the highly optimized CPU code in GROMACS by only relying on GPUs was difficult, especially in parallel runs where the CPU-accelerated code excels. To make use of GPUs without giving up feature support, while providing speedup to as many simulation use-cases as possible, utilizing both CPU and GPU in heterogeneous parallelization was necessary.

Heterogeneous offload is employed by several MD codes (NAMD , LAMMPS , CHARMM , or GENESIS ). However, here too the GROMACS CPU performance provided a challenge: since the tuned CPU SIMD kernels are already capable of achieving iteration rates around 1 ms per step without GPU acceleration, the *relative* speedup of adding an accelerator was less impressive at the time.

To address this, we started from scratch by recasting algorithms into a future-proof form to exploit both GPUs and CPUs (including multiple devices) to provide very high absolute performance while supporting virtually all features no matter what hardware is available. The pair-interaction calculation was redesigned with a cluster pair algorithm to fit modern architectures, which replaces the traditional Verlet list based on particles. Clusters are optimized to fit the hardware, and the classical cut-off setup has evolved into accuracy-based approaches for simulation settings to allow multi-level load balancing and on-the-fly tuning based on system properties. Together with CPU SIMD parallelization and multi-threading, this has allowed efficient offloading of short-range non-bonded calculations to GPUs and brought major gains in performance.

New algorithms and the heterogeneous acceleration framework have made it possible to track track the shift towards dense heterogeneous machines and balancing CPU/GPU utilization by offloading more work to powerful accelerators. The most recent release has almost come full circle to allow offloading full MD steps, but this version also supports most features of the MD engine by utilizing both CPUs and GPUs, it targets multiple accelerator architectures, and provides scaling both across multiple accelerator devices and multiple nodes. This bottom-up heterogeneous acceleration approach provides flexibility, portability and performance for a wide range of target architectures, ranging from laptops to supercomputers and from CPU-only machines to dense multi-GPU clusters. Below we present the algorithms and implementations that have enabled it.

These are relatively complex concepts, so to aid the reader we will first discuss the general ideas in sections II-IV, after which we dedicate section V to details of core algorithms, returning to performance benchmarks and discussions in the final two sections.

# Computational challenges in MD simulations

The core of classical MD is the time-evolution of particle systems by numerically integrating Newton's equations of motion. This requires calculating forces for every time step, which is the main computational cost. While this can be parallelized, the integration step is inherently iterative.

The total force on each particle involves multiple terms: non-bonded pair interactions (typically Lennard-Jones and electrostatics), bonded interactions (e.g. bonds, angles, torsions), and possibly terms like restraints or external forces. Given particle coordinates, each term can be computed independently (Fig. ).

While there are good examples of MD applications with gigantic systems, the most common approach is to keep the simulation size fixed and small to improve absolute performance. Improving the time-to-solution thus requires strong scaling. Historically, virtually all time was spent computing forces, which made it straightforward to parallelize, but well-optimized MD engines now routinely achieve step iteration rates at or below a millisecond . Thus, the MD problem is increasingly becoming latency-sensitive where synchronization, bandwidth and latency both within nodes and over the network are becoming major challenges for homogeneous as well as heterogeneous parallelization due to Amdahl's law. This has partly been compensated for by increasing density of HPC resources where jobs rely more on intra-node communication than the inter-node network. This in turn has enabled a shift from coarse parallelism using MPI and domain decomposition to finer-grained concurrency within force calculation tasks, which is better suited for multicore and accelerator architectures.

Dense multi-GPU servers often make it possible for simulations to remain on a single node, but as the performance has improved the previous fast intra-node communication has become the new bottleneck compared to the fast synchronization within a single accelerator or CPU. As a side-effect, simulations that a decade ago required extreme HPC resources are now routinely performed on single nodes (often with amazingly cost-efficient consumer hardware). This has commoditized MD simulations, but to explore biological events we depend on advancing absolute performance such that individual simulations cover dynamics in the hundreds of microseconds, which can then be combined with ensemble-level parallelism to sample multi-millisecond processes.

In the mid 2000s, processors hit a frequency wall and the increases in transistor count were instead used for more functional units, leading to multi- and manycore designs. Specialization has enabled improvements such as wider and more feature-rich SIMD units as well as application-specific instructions, and recent GPUs have also brought compute-oriented application-specific acceleration features. Compute units however need to be fed data, but memory subsystems and interconnects have not showed similar improvement. Instead, there has been an increasing discrepancy between the speed of computation and data movement. The arithmetic intensity per memory bandwidth required to fully utilize compute units has increased threefold for CPUs in the last decade, and nearly tenfold for GPUs. In particular, most accelerators still rely on communication over the slowly evolving PCIe bus while their peak FLOP rate has increased five times and GROMACS compute kernel performance grew by up to an order of magnitude. This imbalance has made heterogeneous acceleration and overlapping compute and communication increasingly difficult. This is partly being addressed through tighter host–accelerator integration with NVIDIA NVLink among the first (other technologies include Intel CXL or AMD Infinity Fabric), and this trend is likely to continue.

MD as a field was established in an era where the all-important goal was to save arithmetic operations, which is even reflected in functional forms such as the Lennard-Jones potential (the power-12 repulsion is used as the square of the power-6 dispersion instead of an expensive exponential). However, algorithm design and parallelization is shifting from saving FLOPS to efficient data layout, reducing and optimizing data movement, overlapping communication and computation, or simply recomputing data instead of storing and reloading. This is shifting burden of extracting performance to the software, including authors of compilers, libraries and applications, but it pays off with significant performance improvements, and the resulting surplus of FLOPS suddenly available will enable the introduction of more accurate functional forms without excessive cost.

# Parallelization of MD in GROMACS

## The structure of the MD algorithm

The force terms computed in MD are independent and expose task parallelism within each MD step. Force tasks typically only depend on positions from the previous step and other constant data, although domain decomposition (DD) introduces an additional dependency on the communication of particle coordinates from other nodes. The per-step concurrency in computing forces (Fig. ) is a central aspect in offload-based parallel implementations. The reduction to sum forces for integration acts as a barrier; only when new coordinates become available can the next iteration start. The domain decomposition algorithm on the other hand exposes coarse-grained data parallelism through a spatial decomposition of the particle system. Within each domain, finer-grained data-parallelism is also available (in particular in non-bonded pair interactions), but to improve absolute performance it is the total step iteration rate in this high-level flowchart that has to be reduced to the order of 100 μs.

## Multi-level parallelism

Modern hardware exposes multiple levels of parallelism characterized by the type and speed of data access and communication between compute units. Hierarchical memory as well as intra- and inter-node interconnects facilitate handling data close to compute units. Targeting each level of parallelism has been increasingly important on petascale architectures, and GROMACS does so to improve performance.

On the lowest level, SIMD units of CPUs offer fine-grained data-parallel execution. Similarly, modern GPUs rely on SIMD-like execution called SIMT (single instruction, multiple thread) where a group of threads execute in lockstep (width typically 32–64). CPUs have multiple cores, frequently with multiple hardware threads per core. Similarly, GPUs contain groups of execution units (multiprocessors/compute units), but unlike on CPUs distributing work across these cannot be controlled directly, which poses load balancing challenges. On the node level, multiple CPUs communicate through the system bus. Accelerators are attached to the host CPU or other GPUs using a dedicated bus. These CPU–GPU and GPU–GPU buses add complexity in heterogeneous systems and, together with the separate global memory, represent some of the main challenges in a heterogeneous setup. Finally, the network is essentially a third-level bus for inter-node communication. An important concern is not only fast access on each level, but also the non-uniform memory access (NUMA): moving data between compute units has non-uniform cost. This also applies to intra-node buses as each accelerator is typically connected only to one NUMA domain, not to mention inter-node interconnects where the topology can have large impact on communication latency.

The original GROMACS approach was largely focused on efficient use of low- to medium-scale resources, in particular commodity hardware, through highly tuned assembly (and later SIMD) kernels. The original MPI- (and PVM) based scaling was less impressive, but in version 4.0 this was replaced with a state-of-the-art neutral-territory domain-decomposition combined with fully flexible 3D dynamic load balancing (DLB) of triclinic domains. This is combined with a high-level task decomposition that dedicates a subset of MPI ranks to long-range PME electrostatics to reduce the cost of collective communication required by the 3D FFTs, which means multiple-program, multiple-data (MPMD) parallelization. Domain decomposition was initially also used for intra-node parallelism using MPI as well as our own thread-based MPI library implementation. Since the DD algorithm ensures data locality, this has been a surprisingly good fit to NUMA architectures, but it comes with challenges related to exposing finer-grain parallelism across cores and limits the ability to make use of efficient data-exchange with shared caches. Algorithmic limitations (minimum domain size) also restrict the amount of parallelism that can be exposed in this manner. While the design had served well, significant extensions were required in order to target manycore and heterogeneous GPU-accelerated architectures.

The multilevel heterogeneous parallelization was born from a redesign that extended the parallelization to separately target each level of hardware parallelism, first introduced in version 4.6. New algorithms and programming models have been adopted to expose parallelism with finer granularity. Our first major change was to redesign the pair-interaction calculation to provide a flexible and future-proof algorithm with accuracy-based settings and load balancing capabilities, which can target either wide SIMD or GPU architectures. On the CPU front, SIMD parallelism is used for most major time-consuming parts of the code. This was necessitated by Amdahl's law: as the performance of non-bonded kernels and PME improved, previously insignificant components such as integration turned into new bottlenecks. This was made fully portable by the introduction of the GROMACS SIMD abstraction layer, which started as the replacement of raw assembly with intrinsics and now supports a range of CPU architectures using 14 different SIMD instruction sets, with additional ones in development.

To utilize both CPUs and GPUs, intra-node parallelization was extended with an accelerator offload layer and multithreading. The offload layer schedules GPU tasks and data movement to ensure concurrent CPU–GPU execution and it has evolved as more offload abilities were added. OpenMP multithreading was first introduced to improve PME scaling, and later extended to the entire MD engine. To allow assembling larger units of computation for GPUs, we increased the size of MPI tasks and have them run across multiple cores instead of dispatching work from a large number of MPI ranks per node. This avoids bottlenecks in scheduling and execution of small GPU tasks. Multithreading algorithms have also gone through several generations with a focus on data-locality, cache-optimizations, and load balancing, improving scalability to larger thread counts. Hardware topology detection based on the hwloc library is used to guide automated thread affinity setting, and NUMA considerations are taken into account when placing threads.

For single-node CPU-only parallelism, execution is still done with sequentially dependent tasks (Fig. A), which allows relying on implicit dependencies and sharing output across force calculations. Expressing concurrency (Fig. ) to allow parallel execution over multiple cores and GPU has required explicitly expressing dependencies. The new design uses multi-threading and heterogeneous extensions for handling force accumulation and reduction. On the CPU, per-thread force accumulation buffers are used with cache-efficient sparse reduction instead of atomic operations. This is important e.g. for bonded interactions where a thread typically only contributes forces to a small fraction of particles in a domain. When combined with accelerators, force tasks can be assigned to either CPU or GPU, with additional remote force contributions received over MPI. With forces distributed in CPU and GPU memories, we use a new reduction tree to combine all contributions. Explicit dependencies of this reduction for the single GPU case are indicated by black arrows in Fig. . Fulfilling dependencies may require CPU–GPU transfers to the compute unit that does the reduction, and the heterogeneous schedule is optimized to ensure these overlap with computation (Fig.  panels C and D).

# Heterogeneous parallelization

Asynchronous offloading in GROMACS is implemented using either CUDA or OpenCL APIs, and has two main functionalities: explicit control of CPU–GPU data movement as well as asynchronous scheduling of concurrent task execution and synchronization. This design aims to maximize CPU–GPU execution overlap, reduce the number of transfers by moving data early, keeping data on the accelerator as long as possible, ensuring transfer is overlapping with computation, and optimize task scheduling for the critical path to reduce the time per step.

## Offloading force computation

GROMACS initially chose to offload the non-bonded pair interactions to the GPU, while overlapping with PME and bonded interactions being evaluated on the CPU (Fig. B). While this approach requires CPU resources, it has the advantage of supporting domain decomposition as well as all functionality, since any special algorithm can execute on the CPU. When combined with DD, interactions with particles not local to the domain depend on halo exchange. This is handled by splitting non-bonded work into two kernels: one for local-only interactions and one for interactions that involve non-local particles. Based on co-design with NVIDIA, stream priority bits were introduced in the GPU hardware and exposed in CUDA. This made it possible to launch non-local work in a high priority stream to preempt the local kernel and return remote forces early, while the local kernel execution can overlap with communication. Currently only a single priority bit is available, but increasing this should facilitate additional offloading; this is a less complex solution than e.g. a persistent kernel with dynamic workload-switching. The intra-node load balancing, together with control and data flow of the heterogeneous setup with with short-range force offload (non-bonded and bonded) is illustrated in Fig. .

The gradual shift in CPU–GPU performance balance in heterogeneous systems brought the need for offloading further force tasks to avoid the CPU becoming a bottleneck or, from a cost perspective, not needing expensive CPUs. Consequently, we added offload of PME long-range electrostatics both in CUDA and OpenCL. PME is offloaded to a separate stream (Fig. C) to allow overlap with short-range interactions. The main challenge arises from the two 3D FFTs required. Simulations rely on small grids (typical dimensions $32$–$256$), which has not been an optimization target in GPU FFT libraries, so scaling FFT to multiple GPUs would often not provide meaningful benefits. However, we can still allow multi-GPU scaling by reusing our MPMD approach and placing the entire PME execution on a specific GPU. We have also developed a hybrid PME offload to allow back-offloading the FFTs to the CPU, while the rest of the work is done on the GPU. This is particularly useful for legacy GPU architectures where FFT performance can be low. Additionally, it could be beneficial for strong scaling on next-generation machines with high-bandwidth CPU–GPU interconnects to allow grid transfer overlap and exploiting well-optimized parallel CPU 3D FFT implementations. The last force task to be offloaded was the bonded interactions. Without DD, this is executed on the same stream as the short-range non-bonded task (Fig. D). Both tasks consume the same non-bonded layout-optimized coordinates and share force output buffer. With DD, the bonded task is scheduled on the nonlocal stream (Fig. ) as it is often small and not split by locality.

The force offload design inherently requires data transfer to/from the GPU every step, copying coordinates to, and forces from, the GPU prior to force reduction (black boxes in Fig. B–E), followed by integration on the CPU. With accelerator-heavy systems this can render a offload-based setup CPU-limited. In addition, GPU compute to PCIe bandwidth is also imbalanced. High performance interconnects are not common, and typically used only for GPU–GPU communication. This disadvantages the offload design as it leaves the GPU idle for part of each step, although this can partly be compensated for with pair list pruning described in the algorithms section.

Pipelining

force computation, transfer and integration or using intra-domain force decomposition can reduce the CPU bottlenecks. However, slow CPU–GPU transfers are harder to address by overlapping since computation is faster than data movement. For this reason our recent efforts have aimed to increasingly eliminate CPU–GPU data movement and rely on direct GPU communication for scaling.

## Offloading complete MD iterations

To avoid the data transfer overhead , GROMACS now supports executing the entire innermost iteration, including integration, on accelerators (Fig. E). This can fully remove the CPU from the critical path, and reduces the number of synchronization events. At the same time, the CPU is employed for pair search and domain decomposition (done infrequently), and special algorithms can use the now free CPU resources during the GPU step. In addition to the force tasks performed on the GPU, the data ownership for the particle coordinates, velocities and forces is now also moved to the GPU. This allows shifting the previous parallelization trade-off and minimize GPU idle time. The inner MD loop however still supports forces that are computed on the CPU, and often the cycle of copying the data from/to the GPU and evaluating these forces on the CPU takes less time than the force tasks assigned to the GPU (Fig. E). The CPU can now be considered a supporting device to evaluate forces not implemented on GPU (e.g. CMAP corrections), or those not well-suited for GPU evaluation (e.g. pulling forces acting on a single atom). This keeps the GPU code base to a minimum and balances the load by assigning a different set of tasks to the GPU depending on the simulation setup and hardware configuration. This is highly beneficial both for high-throughput and multi-simulation experiments on GPU-dense compute resources or upgrading old systems with a high-end GPU. At the same time, it also allows making efficient use of communication directly between GPUs, including dedicated high-bandwidth/low-latency interconnects where available. In our most recent implementation, data movement can be automatically routed directly between GPUs instead of staging communication through CPU memory. When a CUDA-aware MPI library is used, communication operates directly on GPU memory spaces. Our own thread-MPI implementation relies on direct CUDA copies. Additionally, by exchanging CUDA events, it can use stream synchronization across devices which allows fully asynchronous communication offload leaving the CPU free from both compute and coordination/wait. The external MPI implementation requires additional CPU–GPU synchronization prior to communication, but allows the new functionality to be used across multiple nodes. Much of the GPU–GPU communication, either between short-range tasks or between short-range and PME tasks, is of a halo exchange nature where non-contiguous coordinates and forces are exchanged, which requires GPU buffer packing and unpacking operations. In particular for this, keeping the outer loop of domain decomposition and pair search on the CPU turns out to be a clear advantage, since the index map building is a somewhat complex random access operation, but once complete the data is moved to the accelerator and reused across multiple simulation time steps.

# Algorithm details

## The cluster pair algorithm

The Verlet list and linked cell list algorithms for finding particles in spatial proximity and constructing lists of short-range neighbors were some of the first algorithms in the field and are cornerstones of MD. However, while the Verlet list exposes a high degree of parallelism its traditional formulation expresses this in an irregular form, which is ill-suited for SIMD-like architectures. To reduce the execution imbalance due to varying list lengths, padding or binning have been used. However, the community has largely converged on reformulating the problem by grouping interactions into fixed size work units instead.

A common approach is to assign different $i$-particles to each SIMT thread requiring a separate $j$ particle loaded for each pair interaction. This leads to memory access divergence which becomes a bottleneck in SIMD-style implementations, even with arithmetically intensive interactions. Sorting particles to increase spatial locality for better caching improves performance, but the inherently scattered access is still inefficient. Early efforts used GPU textures to improve data reuse, but this is hard to control as the effectiveness depends on the size of the $j$-lists relative to cache.

Given the need for increasing the arithmetic-to-memory operation ratio, we formulated an algorithm that regularizes the problem and increases data reuse. The goal is to load $j$-particle data as efficiently and rarely as possible and reuse it for multiple $i$-particles, roughly similar to blocking algorithms in matrix-matrix multiplication. Our cluster pair algorithm uses a fixed number of particles per cluster, and pairs of such clusters rather than individual particles are the unit of computing short-range interactions. Hence, we compute interactions between $i$-clusters of $N$ particles and a list of $j$-clusters each of $M$ particles. $M$ is adjusted to the SIMD width while $N$ allows balancing data reuse with register usage. In addition to a data layout that allows efficient access, and that $N \times M$ interactions are calculated for every load/store, the algorithm is easy to adapt to new architectures or SIMD widths. Since the algorithmic efficiency will be higher for smaller clusters, we can also place two sets of the $N$ $i$-cluster particles in a wide SIMD register of length $2M$, which our SIMD layer supports on all hardware where sub-register load/store operations are efficient. The clusters and pair list are obtained during search using a regular grid in $x$ and $y$ dimensions but binning the resulting $z$ columns into cells with fixed number of particles (in contrast to fixed-size cells) that define our clusters (Fig. , left). The irregular 3D grid is then used to build the cluster pair list.

Following the approach used for CPU SIMD, choosing $M$ to match the GPU execution width may seem suitable. However, as GPUs typically have large execution width, the resulting cluster size would greatly increase the fraction of zero interactions evaluated. The raw FLOP-rate would be high, but efficiency low. To avoid this, our GPU algorithm calculates interactions between all pairs of an $i-j$ cluster pair instead of assigning the same $i$- and different $j$-particles to each GPU hardware thread. Hence, we adjust $N \times M$ to the GPU execution width. However, evaluating all $N \times M$ interactions of a cluster pair in parallel would eliminate the $i$-particle data reuse. The arithmetic intensity required to saturate modern processors is quite similar across the board, so restoring data reuse is imperative. We achieve this by introducing a *super-cluster* grouping (Fig. ). A *joint pair list* is built for the super-cluster as the union of $j$-clusters that fall in the interaction sphere of any $i$-cluster. This improves arithmetic saturation at the cost of some pairs in the list not containing interacting particles, since all $j$-clusters are not shared. To minimize this overhead, the super-clusters are kept as compact as possible, and the search is optimized to obtain close to cubic cluster geometry – we use an eight-way grouping obtained from a $2 \times 2 \times 2$ cell grouping on the search grid. Even so, the joint $j$-list would lead to substantial overhead if interactions were computed with all clusters, similarly to the challenge with large regular tiling. We avoid this elegantly by skipping cluster pairs with no interacting particles based on an interaction bitmask stored in the the pair list. This is illustrated in Fig.  where lighter-shaded $j$-clusters do not interact with some of the $i$-clusters; e.g. $j_m$ can be skipped for $i_1$–$i_3$. As $M \times N$ is adjusted to match the execution width, the interaction masks allow efficient divergence-free skipping of entire cluster pairs. Our organization of pair-interaction calculation is similar to that used by others , with the key difference that those approaches rely on larger fixed size tiles and use other techniques to reduce the impact of large grouping.

The interaction mask describes a $j-i$ relationship, swapping the order of the standard $i-j$ formulation. Consequently, the loop order is also swapped and our GPU implementation uses an outer loop over the joint $j$-cluster list and inner loop over the eight $i$-clusters. This has two main benefits: First, 8 bits per $j$-cluster is sufficient to encode the interaction mask, instead of needing variable-length structure. Second, the force reduction becomes more efficient. Since we utilize Newton's third law to only calculate interactions once, we need to reduce forces both for $i$- and $j$-particles. At the end of an outer $j$ iteration, all interactions of the $j$-particles loaded will have been computed and the results can be reduced and stored. At the same time, accumulating the $i$-particle partial forces requires little memory ($8 \times 8$ forces) and can be done in registers. Self-exclusions are handled in the interaction kernels while force-field exclusions are stored in the list with $j$-clusters as bitmasks and enforced simultaneously with the interaction cut-off, just as the interaction masks. In our typical target systems, on average approximately four of the eight $i$-clusters contain interactions with $j$ particles. Hence, about half of the inner loop checks result in skips, and we have observed these to cost 8–12%, which is a rough estimate of the super-cluster overhead. In comparison, the normal cut-off check has at most 5–10% cost in the CUDA implementation.

For PME simulations, we calculate the real-space Ewald correction term in the kernel. On early GPU architectures (and CPUs with low FMA FLOPS) tabulated $F\cdot r$ is most efficient. On all recent architectures, we have instead developed an analytical function approximation of the correction force. This yields better performance as it relies on FMA arithmetics despite the $>15\%$ increase in kernel instruction count.

Multiprocessor-level parallelism is provided by assigning each thread block a list element that computes interactions of all particles in a super-cluster. To avoid conditionals, separate kernels are used for different combinations of electrostatics and Lennard-Jones interactions and cut-offs, and whether energy is required or not. The cluster algorithm has been implemented both in CUDA and OpenCL and tuned for multiple GPU architectures. On NVIDIA and recent AMD GPUs with 32-wide execution we use an $8 \times 4$ cluster setup, for 64-wide execution on AMD $8 \times 8$, and on Intel hardware with 8-wide execution a $4 \times 2$ setup, all with 8-way super-clustering.

## Algorithmic work efficiency and pair-list buffers

The cluster algorithm trades computing interactions known to evaluate to zero for improved execution efficiency on SIMD-style architectures. To quantify the amount of additional work, we calculate the parallel work efficiency of the algorithm as fraction of non-zero interactions evaluated, It is worth noting that this metric is $\leq 1$ even for the standard Verlet algorithm as any finite buffer contains non-interacting particles (Fig. ). In the cluster algorithm this is augmented with particles outside the buffered sphere, but where another particle in the cluster falls inside it (Fig. ). The work-efficiency depends on the cut-off, buffer size, and geometry/size of the clusters that is optimized during search. The cost of this search is the reason why absolute performance still benefits from larger buffers, just as kernel execution efficiency benefits from cluster size.

The improved computational efficiency offsets the algorithmic work-efficiency trade-off for all modern architectures. Moreover, particles in the pair list that fall outside the buffered cut-off can be made use of: these provide an extra *implicit buffer* that allows using a shorter explicit buffer when evaluating pair interactions while maintaining the accuracy of the algorithm. One example of a cluster contributing to the implicit buffer is illustrated in Fig. . Making use of this implicit buffer increases the practical parallel efficiency relative to the standard particle-based algorithm. With a box of SPC/E water where a $\SI{0.9}{nm}$ cut-off pair list is reconstructed every 40 steps, the $1 \times 1$ setup requires a buffer of $\SI{0.218}{nm}$ to reach the same error tolerance as the $8 \times 4$ cluster setup achieves with a $\SI{0.105}{nm}$ buffer.

We believe this is a striking example of the importance of moving to tolerance-based settings instead of rule-of-thumb or heuristics to control accuracy. All algorithms in a simulation affect the accuracy of the final results, and while the acceptable error varies greatly between problems, it will be dominated by the worst part of the algorithm – there is little benefit from evaluating only some parts more accurately. To control the effect of missing pair interactions close to the cut-off, our implementation estimates the Verlet buffer for a given upper bound for the error in the energy. The estimate is based on the particle masses, temperature, pair interaction functions and constraints, also taking into account the cluster setup and its implicit buffering . Since first introduced, we have refined the buffer estimate to account for constrained atoms rotating around the atom they are constrained to rather than moving linearly, which allows tighter estimates for long list lifetimes. The upper bound for the maximum drift can be provided by the user as a tolerance setting in the simulation input. We use $\SI{0.005}{kJ/mol/ps}$ per atom as default, but the tolerance and hence drift can be arbitrarily small. For the default setting, the implicit buffer turns out to be sufficient for a water system or solvated biomolecule with PME electrostatics and $\SI{20}{\femto \second}$ pair-list update intervals, and no extra explicit buffer is thus required in this case. The *actual* energy drift caused by these settings is $\SI{0.0001}{kJ/mol/ps}$ per atom, a factor of 5 smaller than the upper bound. For comparison, typical constraint algorithms result in drifts around 0.0002 kJ/mol/ps per atom, so being significantly more conservative than this will usually not improve the overall error in a simulation.

We see several advantages to this approach, and would argue the field in general should move to requested tolerances instead of heuristic settings. First, the user can set a single parameter that is easier to reason about and that will be valid across systems and temperatures. Second, it will enable innovation in new algorithms that maintain accuracy (instead of performance improvements becoming a race towards the least accurate implementation). Finally, since other parameters can be optimized freely for the input and run conditions, we can make use if this for advanced load balancing to safely deviate from classical setups by relying on maintaining accuracy rather than arbitrary settings as described below.

## Non-bonded pair interaction kernel throughput

The throughput of the cluster-based pair interaction algorithm depends on the number of interactions per particle, hence particle density. This varies across applications, from coarse-grained to all-atom bio-molecular systems or liquid-crystal simulations.

Hence, we investigate the performance of the pair interaction kernels as a function of particle density. We measure the pair throughput of the nonbonded kernel using a Lennard-Jones system consisting of argon atoms to facilitate comparison across a wide range of application areas from physical to biological systems.

The *effective* pair throughput (counting only non-zero interactions) is also influenced by the buffer length and the conditionally-enforced cut-off in the GPU kernels. When comparing CUDA GPU and AVX512 CPU kernels with same-size clusters (identical work-efficiency), the raw throughput reaches peak performance already from ${\sim}150$ pairs per particle on the CPU, while the GPU does not saturate until ${\sim}1000$ pairs (Fig. ). This is explained by the increasing $i$-particle data reuse with longer $j$-lists. The effective pair throughput shows a monotonic increase, since more pairs will be inside the cut-off with more particles in the interaction range, as expected from Fig. . For typical all-atom simulations, the effective GPU kernel throughput gets close to 100 Ginteractions/s while the corresponding throughput on a 20-core CPU is an order-of-magnitude lower.

To provide good performance for small systems and enable strong scaling, it is important to achieve high kernel efficiency already at limited particle counts. To illustrate performance as a function of system size for all-atom systems, Fig.  shows actual pair throughput for SPC/E water systems with 1 nm cut-off, Ewald electrostatics, 40 step search frequency and default tolerances.

Water typically represents up to 90% of biomolecular systems (hence it is the particle density of interest) and therefore nonbonded performance is critical for water. Historically, GROMACS and other codes used special kernels for water but this no longer works well with SIMD-style architectures. On the other hand, when some particles have only one type of interaction (hydrogens in water typically only have Coulomb interactions), this opens up the possibility for additional optimizations useful on some architectures.

The GROMACS CPU kernels achieve peak performance already around 3000 atoms (using up to 40 threads), and, apart from the largest devices, within 10% of peak GPU pair throughput is reached around 48k atoms. GPU throughput is up to seven-fold higher at peak than on CPUs and although it suffers significantly with smaller inputs (up to five-fold lower than peak), for all but the very smallest systems the GPU kernels reach higher absolute throughput. The challenge for small systems is the overhead incurred from kernel invocation and other fixed-cost operations. Pair list balancing also comes at a slight cost, and while it is effective when there are enough lists to split, it is limited by the amount of work relative to the size of the GPU. The sub-10k atom systems simply do not have enough parallelism to execute in a balanced manner on the largest GPUs. In contrast, CPU kernels exhibit a slight decrease for large input due to cache effects.

## The pair list generation algorithm

GROMACS uses a fixed pair list lifetime instead of heuristic updates based on particle displacement, since the Maxwell-Boltzmann distribution of velocities means the likelihood of requesting a pair list update at a step approaches unity as the size of the system increases. In addition, the accuracy-based buffer estimate allows the pair search frequency to be picked freely (it will only influence the buffer size), unlike the classical approach that requires it to be carefully chosen considering simulation settings and run conditions.

We early decided to keep the pair search on the CPU. Given the complex algorithms involved, our main reason was to ensure parallelization, portability and ease of maintenance, while getting good performance (reducing GPU idle time) through algorithmic improvements. The search uses a SIMD-optimized implementation and, to further reduce its cost, the hierarchical GPU list is initially built using cluster bounding-box distances avoiding expensive all-to-all particle distance checks. A particle-pair distance based list pruning is carried out on the GPU which eliminates non-interacting cluster pairs. This can also further adapt the setup to the hardware execution width by splitting $j$-clusters; e.g. for current Intel GPUs the search produces a $4 \times 4$ cluster setup (Fig. ) which is pruned into $4 \times 2$ for 8-wide execution. Initial pruning is done the first time the list is processed by a special version of the kernel (using warp collectives for low extra cost), and the pruned list is reused for consecutive MD steps. Depending on the cut-off and buffer length, pruning reduces the list size by 50–75%.

## Dual pair list with dynamic pruning

Domain decomposition and pair list generation rely on irregular data access, and their performance has not improved at the same rate as compute kernels. Trading their cost for more pair interaction work through increasing the search frequency has drawbacks. First, as the buffer increases the overhead becomes large (Fig. ). The trade-off is also sensitive to inputs and runtime conditions, with a small optimal window. In order to address this, we have developed an extension to the cluster algorithm with a dual pair list setup that uses a longer outer and a short inner list cut-off. The outer list is built very infrequently, while frequent pruning steps produce a pair list based on the inner cut-off, typically with close to zero explicit buffer. As pruning operates on regularized particle data layout produced by the pair search, it comes at a much lower cost (typically $<1\%$ of the total runtime) than using a long buffer-based pair list in evaluating pair interactions. This avoids the previous trade-off and reduces the cost of search and DD without force computation overhead. With GPUs, pruning is done in a rolling fashion scheduled in chunks between force computations of consecutive MD steps, which allows it to overlap other work such as CPU integration (Fig. ).

## Multi-level load balancing

Both data and task decomposition contribute to load imbalance on multiple levels of parallelism. Sources of data-parallel imbalance include inherently irregular pair interaction data (varying list sizes), non-uniform particle density (e.g. membrane protein simulations using united-atom lipids) resulting in non-bonded imbalance across domains, and non-homogeneously distributed bonded interactions (solvent does not have as many bonds as a protein). With task-parallelism when using MPMD or GPU offload, task load imbalance is also a source of imbalance between MPI ranks or CPU and GPU. Certain algorithmic choices like pair search frequency or electrostatics settings can shift load between parts of the computation, whether running in parallel or not.

In particular for small systems it is a challenge to balance work between the compute units of high-performance GPUs. Especially with irregular work, there will be a kernel "tail" where only some compute units are active, which leads to inefficient execution. In addition, with small domains per GPU with DD there may be too few cluster lists to process in a balanced manner. To reduce this imbalance and the kernel tail it would be desirable to control block scheduling, but this is presently not possible on GPUs. Instead, we tune scheduling indirectly through indexing order. The GPU pair interaction work is reshaped by sorting to avoid long lists being scheduled late. In addition, when the number of pair lists is too low for efficient execution for given hardware, we heuristically split lists to increase the available parallelism.

Kernel tail effects can be also be mitigated by overlapping compute kernels. This requires enough concurrent work available to fill idle GPU cores and that it is expressed such that parallel execution is possible. GPU APIs do not allow fine-grained control of kernel execution and instead the hardware scheduler decides on an execution strategy. By using multiple streams/queues with asynchronous event-based dependencies, our GPU schedule is optimized to maximize the opportunity for kernel overlap. This reduces the amount of idle GPU resources due to kernel tails as well as those caused by scheduling gaps during a sequence of short dependent kernels (e.g. the 3D-FFT kernels used in PME).

With PME electrostatics, the split into real and reciprocal space provides opportunities to rebalance work at constant tolerance by scaling the cut-off together with the PME grid spacing. This was first introduced as part of or MPMD approach with an external tool. This load balancing was later automated and implemented as an online load balancer, originally to allow shifting work from the CPU to the GPU. This approach now works remarkably well also with dedicated PME ranks both in CPU-only runs and when using multiple GPUs. The load balancer is automatically triggered at startup and scans through cut-off and PME grid combinations using CPU cycle counters to find the highest-performance alternative as illustrated in Fig.

using a biomolecular system typical for applications that aim to reduce the system size simulated.

The load balancer typically converges in a few thousand steps, apart from noisy environments such as multi-node runs with network contention. Significant efforts have been made to ensure the robustness of the algorithm. It accounts for measurement noise, avoids cache effects, mitigates interference of CPU/GPU frequency scaling, and it reduces undesirable interaction with the DD load balancer (which could change the domain size while the cut-off is scaled).

Nevertheless, load balancing comes with a trade-off in terms of increased communication volume. In addition, linearithmic (FFT) or linear (kernel) time-complexity reciprocal-space work is traded for quadratic time complexity real-space work (Fig. ). To mitigate waste of energy we impose a cut-off scaling threshold to avoid increasing GPU load in heavily CPU-bound runs. The performance gain from PME load balancing depends on the hardware; with balanced CPU–GPU setups it is up to 25%, but in highly imbalanced cases much larger speedups can be observed.

The dual pair list algorithm allows us to avoid most of the drawbacks of shifting work to direct space, since the list pruning is significantly cheaper than evaluating interactions. This makes the balancing less sensitive and easier to use, and where the previous approach saturated around pair list update intervals of 50-100 steps, the dual list with pruning can allow hundreds of steps, which is particularly useful in reducing CPU load to maximize GPU utilization in runs that offload the entire inner iteration (Fig. ).

Finally, the domain decomposition achieves load balancing by resizing spatial domains, thereby redistributing particles between domains and shifting work between MPI ranks. With force offload the use of GPUs is largely transparent to the code, but extensions to the DLB algorithm were necessary. Support for timing concurrent GPU tasks is limited, in particular in CUDA. We account for GPU work in DLB through the wall-time the CPU spends waiting for results, labeled accordingly on the CPU timeline in Fig. . However, this can introduce jitter when a GPU is assigned to multiple MPI ranks. GPUs are not partitioned across MPI ranks, but work is scheduled in an undefined order and executed until completion (unless preempted). Hence, the CPU wait can only be systematically measured on some of the ranks sharing a GPU while not on others. This leads to spurious imbalance and to avoid it we redistribute the CPU wall-time spent waiting for the GPU evenly across the MPI ranks assigned to the same GPU to reflect the average GPU load and eliminate execution order bias.

# Performance benchmarks

#### Benchmark systems

To illustrate the real-world performance of the GROMACS heterogeneous parallelization, we use a set of benchmark systems representative of typical biomolecular workloads both in term of size and force-fields. For single GPU benchmarks we evaluate performance using a small (RNAse) and a medium-sized (GluCl) biomolecular system, both using the AMBER force field. To show multi-GPU ensemble simulation throughput we use a medium-sized aquaporin membrane protein with coupled simulations that employ the Accelerated Weight Histogram (AWH) enhanced sampling algorithm , while strong benchmarks use a larger ${\sim 1}$ million atom satellite tobacco mosaic virus (STMV) system, still representative of common workloads and challenging for strong scaling. The latter two benchmark systems use the CHARMM force field and its characteristic settings which notably yield a different short- to long-range nonbonded workload balance, hence different performance behavior compared to AMBER-based simulations that use shorter cutoffs. Further details of the benchmark systems, including input files, can be found in the supplementary information.

Faster hardware has been a blessing for simulations, but as shown in Fig.  the improvements in algorithms and software described here have at least doubled performance *for the same hardware* even for older cost-efficient GPU hardware, and with latest-generation consumer cards the improvement is almost fourfold over the last five years. Given the low-end CPU and high-end GPU combination, new offload modes bring significant performance improvements when offloading either only PME or the entire inner iteration to the accelerator.

Figure  shows the impact of different offload setups for single-GPU runs. As expected, the CPU-only run scales with the number of cores. When the non-bonded task is offloaded, the performance increases significantly for both systems, but it does not saturate even when using all cores - indicating the CPU is oversubscribed. This is confirmed by the large jump in performance when the PME task too is offloaded. The GPU now becomes the bottleneck for computations, and the curves saturate when enough CPU cores are used - adding more will not aid performance. Consequently, when the bonded forces are also offloaded, there is a performance regression, in particular for the membrane protein system with lots of torsions. (Figure B). With GPU force tasks taking longer than the CPU force tasks, the data transfers between host and device are no longer effectively overlapping with compute tasks. This is solved by offloading the entire innermost MD loop, including coordinate constraining and updating. This leads to another significant jump in performance, despite the CPU now being mostly idle. To make use of this idle resource, one can move the bonded force evaluation from the GPU back to the CPU. This is beneficial when the entire cycle is faster than the evaluation of non-bonded and PME forces on the GPU. For all results the cross-over points will depend on the system, but it is generally faster to evaluate bonded forces on the CPU when apart from very dense systems where only a single CPU core is available per GPU.

Although it is common to run a single simulation per GPU, it is often not the best way of maximizing cumulative throughput since the options to overlap data transfers between CPU and GPU with computational tasks are limited. One way to increase the efficiency even further is to run many uncoupled or loosely coupled trajectories simultaneously on a single GPU. In this case, compute tasks from one trajectory can overlap with the data transfers in another. Figure  shows benchmarks for the case of ensemble simulations using the AWH method on a single node equipped with two CPUs and four GPUs. With medium-performance GPUs, the best performance is achieved when a CPU is used for computing the bonded forces, with everything else evaluated on the GPU (Figure A), and the most efficient throughput is obtained with four ensemble members running on each GPU. Using the GPU for the full MD loop is still the second best case when only a single run is executed on the GPU, but for many runs per GPU there are more options for overlapping transfer and compute tasks when using the CPU for updates (integration) and/or bonded forces. However, for the somewhat older GPUs the difference between the worst- and best-performing cases is only about 25%. When pairing the same older CPUs with recent GPUs ((Figure B), the balance changes appreciably, and it is no longer justified to use the CPUs even for the lightest compute tasks. Performing all tasks on the GPU more than doubles performance compared to leaving updates and bonded forces on the CPU. Another advantage is that the throughput does not change significantly with more ensemble members per GPU, which allows for greater flexibility, not to mention the absolute performance will always be highest when only a single ensemble member is assigned to each GPU.

Finally, the work on direct GPU communication now also enables quite efficient multi-GPU scaling combined with outstanding absolute performance. Fig.  illustrates the effect of the direct GPU communication optimizations on performance through results from running the Satellite tobacco mosaic virus (STMV) benchmark (1M atoms, 2 fs steps) on up to four compute nodes, each equipped with four NVIDIA Telsa V100 GPUs per node. Intra-node communication uses NVLink and inter-node communication MPI over Infiniband. We believe this configuration is a good match to emerging next-generation HPC systems, which have a focus on good balance for mixed workloads. When using reaction-field instead of PME, the scaling is excellent all the way to 16 GPUs. While this is a less common choice for electrostatics, it highlights the efficiency and benefits of the GPU halo exchange combined with GPU update, and shows the performance and scaling possible when avoiding the challenges with small 3D FFTs, extra communication between direct- and reciprocal-space GPUs, and task imbalance inherent to PME. When using PME electrostatics, the relative scaling is good up to 8 GPUs (50% efficiency) when the GPU halo exchange is combined with the direct GPU PME task communication, and there are again clear benefits from combination with the GPU update path. Beyond this we are currently limited by the restriction of a single PME GPU when offloading lattice summation, which becomes a bottleneck both in terms of communication and imbalance in computation. However, we believe the absolute performance of 55 ns/day is excellent.

# Discussion

Microsecond-scale simulations have not only become routine but eminently approachable with commodity hardware. However, it is only the barrier of entry that has been lowered. Bridging the time-scale gap from hundreds of microseconds to millisecond in single-trajectories still requires special-purpose hardware. Nevertheless, general-purpose codes have unique advantages in terms of flexibility, adaptability and portability to new hardware – not to mention it is relatively straightforward to introduce new special algorithms e.g. for including experimental constraints in simulations. There are also great opportunities with using massive-scale resources for efficient ensemble simulations where the main challenge is cost-efficient generation of trajectories. Hence, we believe that a combination of new algorithms, efficient heterogeneous parallelization and large-scale ensemble algorithms will characterize MD in the exascale era - and performance advances in the core MD codes will always multiply the advances obtained from new cleaver sampling algorithms.

The flexibility of the GROMACS engine comes with challenges both for developers and users. There is a range of options to tune, from algorithmic parameters to parallelization settings, and many of these are related to fundamental shifts towards much more diverse hardware that the entire MD community has to adapt to. Our approach has been to provide a broad range of heuristics-based defaults to ensure good performance out of the box, but by increasingly moving to tolerance-based settings we aim to both improve quality of simulations and make life easier for users. Still, efficiently using a multitude of different hardware combinations for either throughput or single long simulations is challenging. As described here, many of the steps have been automated, but to dynamically decide e.g. the algorithm to resource mapping in a complex compute node (or what code flavor to use) requires a fully dynamic auto-tuning approach. Developing a robust auto-tuning framework and integrating it with the multi-level load balancers is especially demanding due to complex feature set and broad use-cases of codes such as GROMACS, but it is something we are working actively on.

To reach performance for which ASICs are needed today, MD engines need to be capable of $<\SI{100}{\micro \second}$ iterations. Such iteration rates are possible with GROMACS for small systems, like the villin headpice (${\sim} 5000$ atoms)[^3]. Reaching this performance for larger systems like membrane proteins with hundreds of thousands of atoms will require a range of improvements. In terms of parallelization in GROMACS, improving the efficiency of GPU task scheduling, CPU tasking and better overlap of communication are necessary. When it comes to algorithms, we expect PME to remain the long-range interaction method of choice at low scale, but the limitations of the 3D FFT many-to-many communication for strong scaling requires a new approach. With recent extensions to the fast multipole method, we expect it to become the algorithm of choice for the largest parallel runs. Future technological improvements, including faster interconnects and closer on-chip integration as well as advances in both traditional and coarse-grained reconfigurable architectures could allow getting closer to this performance target.

We expect to see several of these advancements in the future. In the mean time, we believe the present GROMACS implementation provides a major step forward in terms of absolute performance as well as relative scaling by being able to use almost arbitrary-balance combinations of CPU and GPU functional units. It is our hope this will help enable a wide range of scientific applications on everything from cost-efficient consumer GPU hardware to large HPC resources.

# Data availability

The data that support the findings of this study are available in the supplementary material and in the following repositories: methodology, topologies, input data and parameters for benchmarks are published at <https://doi.org/10.5281/zenodo.3893789>; the source code for multi-GPU scaling is available at <https://doi.org/10.5281/zenodo.3890246>.

[^1]: To whom correspondence should be adressed; email@example.com, email@example.com

[^2]: To whom correspondence should be adressed; firstname.lastname@example.com, email@example.com

[^3]: $\SI{1.95}{\micro \second/day}$ ($<\SI{88}{\micro \second \per step}$) on an AMD R9-3900X CPU and NVIDIA RTX 2080 SUPER GPU; using 2 fs time step and 0.9 nm cut-off.


abstract: A high-temperature $YBa_2Cu_3O_{7-x}$ bulk ceramic superconductor with composite structure has revealed weak shielding properties against gravitational force while in a levitating state at temperatures below 70 $K$. A toroidal disk with an outer diameter of 275 $mm$ and a thickness of 10 $mm$ was prepared using conventional ceramic technology in combination with melt-texture growth. Two solenoids were placed around the disk in order to initiate the current inside it and to rotate the disk about its central axis. Samples placed over the rotating disk initially demonstrated a weight loss of 0.3-0.5%. When the rotation speed was slowly reduced by changing the current in the solenoids, the shielding effect became considerably higher and reached 1.9-2.1% at maximum.
  .
  .
  .-h High-$T_c$ cuprates.

Weak gravitation shielding properties of

composite bulk $YBa_2Cu_3O_{7-x}$ superconductor

below 70 $K$ under e.m. field.

E.E. Podkletnov

*Moscow Chemical Scientific Research Centre*

*113452 Moscow - Russia*

# Introduction.

The behavior of high-$T_c$ ceramic superconductors under high-frequency magnetic fields is of great interest for practical applications. Crystal structure seems to be the key factor determining all physical properties of bulk superconductors, and the interaction of this structure with external and internal e.m. fields might result in quite unusual effects. Despite a large number of studies the nature of these interactions still remains unresolved.

Our recent experimental work clearly indicated that under certain conditions single-phase bulk, dense $YBa_2Cu_3O_{7-x}$ revealed a moderate shielding effect against gravitational force. In order to obtain more information about this unusual phenomenon, a new installation was built, enabling operation with larger disks (275 $mm$ in diameter), in magnetic fields up to 2 $T$ and frequencies up to $10^8 \ Hz$ at temperatures from 40 to 70 $K$. A new experimental technique was employed to modify the structure of the ceramic superconductor. All these efforts yielded a larger value of the shielding effect (up to 0.5% in stationary conditions and to 2.1% for shorter periods), providing good hopes for technological applications.

A gravitational shielding effect of this strength has never been previously observed, and its implications present serious theoretical difficulties (see for references and an analysis of some hypotheses). Thus, great attention was devoted to the elimination of any possible source of systematic errors or of spurious non-gravitational effects. The small disturbances due to air flows pointed out by some authors were eliminated by weighing the samples in a closed glass tube (see Section ). The entire cryostat and the solenoids were enclosed in a stainless steel box. But probably the best evidence for the true gravitational nature of the effect is that the observed weight reductions (in %) were independent of the mass or chemical composition of the tested samples (Section ).

This work is organized as follows. Sections and describe our experimental setup. Section summarizes the main steps in the sinterization of the composite ceramic disk and contains information about the final properties of the disk ($T_c$ for the two layers, $J_c$ for the upper layer, etc.) and about the microscopic structure of the material. Section describes how we obtain and control the levitation and rotation of the disk, up to an angular speed of about 5000 $rpm$. In Section we describe the weight measurement procedure and analyze in detail possible error sources and parasitic effects. Several checks were performed to exclude any influence of spurious factors (Section ). In Section we give the maximum % shielding values obtained in dependence on the rotation speed of the disk and on the frequency of the applied magnetic field. Finally, Sections and contain a short discussion and our conclusions.

According to public information, a NASA group in Huntsville, Alabama, is now attempting to replicate our experiment. This is a difficult task, especially in view of the sophisticated technology involved in the construction of the large ceramic disk and in the control of its rotation. We are also aware, through unofficial channels, that other groups are working on similar experiments with smaller disks.

# Construction of the disk.

The shielding superconducting element was made of dense, bulk, almost single-phase $YBCO$ and had the shape of a toroidal disk with an outer diameter of 275 $mm$, an inner diameter of 80 $mm$, and a thickness of 10 $mm$. The preparation of the 123-compound and fabrication of the disk involved mixing the initial oxides, then calcining the powder at 930$^0$ C in air, grinding, pressing the disk at 120 $MPa$, and sintering it in oxygen at 930$^0$ C for 12 hours with slow cooling to room temperature. After that, the disk was put back in the furnace at 600$^0$ C, and the upper surface was quickly heated to 1200$^0$ C using a planar high-frequency inductor as shown in Figure 1. During this last heating, the gap between the disk and the inductor was chosen precisely so that heating would occur only in the top 2 $mm$-thick layer of the disk, although the material's high heat conductivity caused some heating below this region. Finally, the disk was slowly cooled down to room temperature in a flow of oxygen and treated mechanically in order to obtain good balance during rotation. A thin (1 $mm$) foil of magnetic material was attached (without electric contact) to the upper surface of the disk, using hot-melt adhesive, to facilitate rotating the disk as described below, especially at the initial stages of rotation.

The phase and crystal structure of the superconductor were studied using X-ray diffraction analysis (XRD) and a scanning electron microscope (SEM) equipped with an energy dispersive spectral (EDS) analyzer. The samples were cut layer by layer from the bulk ceramic disk.

The analysis of the cross-section of the ceramic $YBa_2Cu_3O_{7-x}$ disk revealed the existence of two zones with different crystal structures. The upper part of the disk (6-7 $mm$ thick) had an orthorhombic structure typical of the quench and melt growth process and consisted mainly of single-phase orthorhombic 123-compound. This material was dense, with uniformly fine grain boundaries, i.e. no impurities or secondary phases were found between the grains. Inter-grain boundaries were barely visible, indicating that there were good electrical contacts between the particles of the superconducting body and that the sintering of the material had produced a nearly perfect polycrystal lattice with no apparent defects.

The grains were less than 2 $\mu m$ wide and were oriented (about 75%) with c-axis parallel to the surface of the disk. The transition temperature $T_c$ for this region of the disk was found by direct measurements to be 94 $K$, with a width of 1.5-2 $K$. $T_c$ was determined from the resistive transition in a variable temperature cryostat, under zero magnetic field, using an AC current and sputtered golden contacts.

The lower part of the disk, which was in contact with a water-cooled base during the high-frequency heat treatment, had a markedly different structure: randomly oriented grains, with typical grain sizes between 5 and 15 $\mu m$. The porosity of this zone varied from 5 to 9% and the material contained about 40% of the tetragonal phase. The transition temperature $T_c$ was equal to 60 $K$, with a width of ca. 10 $K$. EDS analysis showed the presence of small inclusions of $Y_2BaCuO_5$ in the lower layer.

Crystal lattice parameters for these two layers, as calculated from XRD, are listed below. These are dimensions in $nm$:

Upper layer: a=0.381; b=0.387; c=1.165;

Lower layer: a=0.384; b=0.388; c=1.170 (orthorhombic phase);

cm a=0.387; c=1.183 (tetragonal phase).

The critical current density was measured for samples cut from the top of the superconducting disk. Measurements of $J_c$ were carried out at 75 $K$ using an AC current, four-probe method, and direct transport measurements. The accuracy for $J_c$ determination was defined as 1 $\mu V/cm$ in a zero magnetic field, with the sample immersed in liquid nitrogen. It turned out that $J_c$ exceeded 15000 $A/cm^2$. The value of $J_C$ for the lower part of the disk was not measured, since it is not superconducting at the temperature of operation.

We also estimated the current density in the upper part of the disk while subjected to the magnetic fields usually applied during the measurements. To measure this we made a thin radial cut through the sample disk and attached electric contacts to an ampermeter, with a tecnique allowing fast on/off switching. We calibrated the currents in the driving coils that correspond to the currents inside the disk. These currents are slightly different for each new disk as the thickness of the SC part is not the same in every new sintered ceramic toroid, but we estimate their density to range between 5000 and 7000 $A/cm^2$.

# Operation of the apparatus.

Two identical solenoids were placed around the superconductor using fibreglass supports, as shown in Figures 2, 4, 5. The gaps between these solenoids and the disk were large enough for it to move about 20 $mm$ in any lateral direction. The toroidal disk was placed inside a cryostat equipped with a set of three coils (Fig. 3) that could keep it levitating when it reached the superconducting state. The angle $\beta$ was between 5 and 15 degrees. This helped to keep the rotating disk in a stable position, otherwise it tended to slip aside.

A schematic of the electrical connections is shown in Fig. 6. High-frequency electric current ($10^5 \ Hz$) was first sent to the two main solenoids around the disk to initiate an internal current in the ceramics while the disk was still at room temperature. Then the system was slowly cooled down to 100 $K$ by liquid nitrogen, and then quickly cooled by liquid helium vapors. We estimate the temperature of the disk to be lower than 70 $K$ in stationary conditions. Thus the upper layer of the disk is superconducting in these conditions, while the lower layer is not. The main solenoids were switched off. After this, the high-frequency current was sent to the coils below the disk, and the superconductor raised up (at least 15 $mm$; see Section ).

Then a small current ($10^5 \ Hz$) was sent to the main solenoids, causing the disk to begin rotating counter-clockwise with increasing speed. The rotation speed was increased up to 5000 $rpm$. At this point the current in the rotating coils was of the order of 8-10 $A$. (The diameter of the wire of these coils is 1.2 $mm$). This current was supplied by powerful high-frequency generators usually employed for induction heating and quenching of metals.

Most weight measurements for various objects were taken in these conditions, which can be maintained in a stable way for quite long periods (10 minutes or more). Next, the disk's rotational speed was slowly reduced by changing the current in the main solenoids (Fig. 9). The speed of rotation was regulated by means of laser beam reflections off a small piece of plastic light-reflecting foil attached to the disk.

The frequency of the e.m. field was varied from $10^3$ to $10^8 \ Hz$. Samples made of various materials were tested, including metals, glass, plastic, wood and so on. All the samples were hung over the cryostat on a cotton thread connected to a sensitive balance. The distance between the samples and the cryostat varied from 25 to 3000 $mm$.

# Weight measurements. Error sources.

To measure the weight loss of the samples, we used a Dupont balance that is a part of the standard equipment for DTA and TGA (differential thermal analysis, thermo-gravimetric analysis). One of the two arms of the balance, holding the sample to be weighed, was lying within the vertical projection of the HTC disk (we call this region the "shielding cylinder"), while the other arm was well outside. The arms of the balance were up to 220 $cm$ long. The sensitivity of the balance for masses of 10-50 $g$, like those employed in the measurements, is on the order of $10^{-6} \ g$, which was sufficient to detect the observed weight loss. Three different balances were used for verification and are described below (see "Checks", Section ).

The main error sources in the weight measurements were the following.

1.  *Buoyancy and air flow.* The presence of the cryostat perturbs the air and causes weak local flows. Moreover, a much larger ascending flux is caused by the weight loss of the air in all the shielding cylinder; this also produces a pressure drop in the shielding cylinder (see Section ). The error introduced by this effect in the weight measurements has been substantially reduced by enclosing the samples to be weighed in a long glass tube, closed at the bottom. The samples had the form of elongated objects (like big pencils). In order to set an upper limit on the buoyancy effect, some flat samples were weighed too, without enclosing them in the glass tube, in vertical and horizontal positions, and the results for the two cases were compared. It turned out that the weight loss in horizontal position was larger (by approx. 10% of the total loss) than in vertical position.

2.  *Hydrostatic force.* This introduces a slight dependence of the weight loss on the density of the sample. For a material with density 1 $Kg/dm^3$, the Archimede pull amounts to about the 0.1% of the weight. Since the air density is slightly lower in the shielding cylinder (see above and Section ), the Archimede pull is lower itself, but this effect can be disregarded, being of the order of 0.001% of the total weight or less.

3.  *Diamagnetism.* It is known that molecular diamagnetism produces a small levitating force on samples placed in a magnetic field gradient. For a large class of materials, this force is essentially independent on the chemical composition of the material and is proportional to its weight. For instance, a standard value for the diamagnetic levitating force is the following: in a field gradient of 0.17 $T/cm$ the force exerted on a 1 $g$ sample of $NaCl$, $SiO_2$, $S$ or diamond is ca. equal to 16 $dyne$ (about 0.016 $g$, or 1.6% of the weight).

    Since the force is proportional to the square of the field gradient, one easily finds that the value of the field gradient corresponding to a percentage weight loss of 0.1% is about 0.04 $T/cm$ (for comparison, we recall that the maximum weight loss we observed in stationary conditions was 0.5%). A mapping of the static field produced by our apparatus yelded in all cases smaller values of the field gradient near the disk, and much smaller values at a height of 50 $cm$ or more above the disk, where the observed weight loss stays the same (compare Section ). We thus conclude that the effect of molecular diamagnetism can be completely disregarded.

4.  *RF fields.* The possibility of a slight levitating effect from an RF magnetic field cannot be excluded completely. However, such disturbance was attenuated, if not eliminated, by placing thick metal screens between the cryostat and the samples. Copper, aluminum, and steel screens were tested separately and in many different combinations. The individual screens had a diameter of 300 mm and a thickness of 50 mm. The presence of the screens never altered the effect.

# Checks.

We did several checks in order to correlate more clearly the appearence of the effect to specific experimental conditions.

1.  *Substituting a metal disk or a disk made totally from superconducting ceramic.* In all these cases, the shielding effect was not observed. This confirms, in our opinion, that the origin of the effect resides in the interaction between the upper (superconducting) part of the disk and the lower part, where considerable resistive phenomena take place.

2.  *Measurements in vacuum, or in different gases, or in a fluid.* These measurements show that the effect exists in these conditions, too, and has the same magnitude as in air. For these cases, however, we cannot furnish data as precise as the measurements taken in air yet, because the experimental conditions are more difficult and the necessary statistics are still being accumulated. Measurements in vacuum or in gases other than air ($N_2$, $Ar$) are hampered by the fact that the samples and the analytical balance must be enclosed in a sealed container. For the measurements in fluids ($H_2O$, $C_2H_5OH$) Archimede pull is more relevant and thus complicates measurements.

3.  *An AC field is indispensable.* The shielding effect was completely absent when only static magnetic fields were employed.

4.  *Other weighing techniques.* Although the most accurate measurements have been taken with the Dupont balance for precision differential thermogravimetric analysis, we employed for verification three other balances. In the latest runs, using heavier samples with weights varying from 100 $g$ to 250 $g$, a standard analytical balance was used. Given these masses and the balance's accuracy (0.01 $g$), weight losses of 0.01% were easy to detect. In addition, we used two other types of balances, sketched in Fig. 8. The first one is a torsion balance, whose oscillation period depends on the tension of the wire and thus on the weight of a suspended sample. This period can be measured with high accuracy through a laser beam reflected by a mirror attached to the wire. Finally, we employed a spring balance whose movements are detected by an induction transducer.

# Results.

The levitating disk revealed a clearly measurable shielding effect against the gravitational force even without rotation. In this situation, the weight-loss values for various samples ranged from 0.05 to 0.07%. As soon as the main solenoids were switched on and the disk began to rotate in the vapors of liquid helium, the shielding effect increased, and at 5000 $rpm$, the air over the cryostat began to rise slowly toward the ceiling. Particles of dust and smoke in the air made the effect clearly visible. The boundaries of the flow could be seen clearly and corresponded exactly to the shape of the toroid.

The weight of various samples decreased no matter what they were made from. Samples made from the same material and of comparable size, but with different masses, lost the same fraction of their weight. The best measurement gave a weight loss of 0.5% while the disk was spinning at 5000 $rpm$, with typical values ranging from 0.3 to 0.5%. Samples placed above the inner edge of the toroid (5-7 $mm$ from the edge) were least affected, losing only 0.1 to 0.25% of their weight. The external boundary of the shielding cylinder was quite clear (no more than 2 $cm$). The maximum values of weight loss were obtained when the levitation height of the disk was at its maximum value, about 30-35 $mm$ over the magnets. This condition cannot be reached above 70 $K$, although the disk had become superconducting already at 94 $K$.

During the time when the rotation speed was being decreased from 5000 to 3500 $rpm$, using the solenoids as braking tools, the shielding effect reached maximum values: the weight loss of the samples was from 1.9 to 2.1%, depending on the position of the sample with respect to the outer edge of the disk. These peak values were measured during a 25-30 seconds interval, when the rotational speed was decreasing to 3300 $rpm$. Because of considerable disk vibration at 3000-3300 $rpm$, the disk had to be rapidly braked in order to avoid unbalanced rotation, and further weight measurements could not be carried out.

The samples' maximum weight loss was observed only when the magnetic field was operating at high frequencies, on the order of 3.2 to 3.8 $MHz$. The following tables show how the shielding effect varied in response to changes in the disk's rotation speed or the current frequency.

*At constant frequency of 2 $MHz$:*

Rotation speed ($rpm$)       Weight loss (%)

4000 cm 0.17

4200 cm 0.19

4400 cm 0.20

4600 cm 0.21

4800 cm 0.22

5000 cm 0.23

*At constant rotation speed of 4300 $rpm$:*

Frequency ($MHz$)       Weight loss (%)

3.1 cm 0.22

3.2 cm 0.23

3.3 cm 0.24

3.4 cm 0.26

3.5 cm 0.29

3.6 cm 0.32

Remarkably, the effective weight loss was the same even when the samples, together with the balance, were moved upward to a distance of 3 $m$, but still within the vertical projection of the toroidal disk. No weight loss at all was observed below the cryostat.

We also observed a slight diminution of the air pressure inside the shielding cylinder. The difference between the external and the internal pressure was up to 5 $mm$ of mercury in stationary conditions (disk rotating at 4000-5000 $rpm$) and increased up to 8 $mm$ of mercury during the "braking" phase. Pressures were measured through a vacuum chamber barometer. We believe that the diminution of the pressure inside the shielding cylinder is originated by the pseudo-convective motion of the air, which being lighter tends to raise. This phenomenon is favoured by the fact that an entire cylinder of air get lighter at the same time and thus the ascending motion is amplified.

# Discussion.

The interaction of a superconducting ceramic body with the gravitational field is a complicated process and cannot be characterized by one single law or physical phenomenon. Also, a comprehensive explanation of the mechanism responsible for high-temperature superconductivity has not yet been found. Still, these facts do not make the observed phenomenon less interesting.

In our previous work the weight loss of samples over the levitating superconductor was smaller, varying from 0.05 to 0.3%. At that time it was difficult to exclude entirely any influence from the radio-frequency field because the sample was separated from the disk and the magnets by a thin plastic film. Now, the superconductor is situated in a stainless steel cryostat, so this influence should be negligible (see also Section ).

The modification of the superconductor's crystalline structure produced a composite body with a dense and highly oriented upper layer and a porous lower layer with random grain orientation. The upper layer is superconducting at the operation temperature and is able to carry high $J_c$ current under considerable magnetic field, while the lower layer cannot conduct high currents. The boundary between the two layers is likely to constitute a "transition" region in which the supercurrents, that are completely free to move in the upper layer, begin to feel some resistance.

It is also expected that a complex interaction between the composite ceramic body and the external magnetic field takes place. This interaction depends on the coherence length, the flux pinning, the field frequency and the field force, the penetration depth, and the parameters of the crystal lattice. These characteristics are interrelated in a complex way. According to the experimental data (compare also , where only a static field was applied), a levitating superconductor does not reveal any unusual shielding if it has no contact with the high frequency AC magnetic field.

As analyzed in , pinning centers with different origins may exist inside the superconducting disk, and fluxes will be trapped at some of them. Fluxes trapped at weak centers will begin to move first, while those trapped at strong centers will not move until the Lorentz force exceeds the pinning force. The overall current will be composed of the superposition of flux motions with different speeds.

There are no grounds to claim that the rotation momentum of the disk interacts with gravitation force, but it seems that fast rotation is favorable for stabilization of the shielding effect.

Finally, it is worth noting that the experimental equipment described above has much in common with magneto-hydro-dynamic (MHD) generators.

The first attempt at a theoretical explanation of the effect has been offered by G. Modanese . Further investigations now in progress may help to prove, change, or complete the present understanding of the observed phenomenon.

# Conclusions.

A levitating superconducting ceramic disk of $YBa_2Cu_3O_{7-x}$ with composite structure demonstrated a stable and clearly measurable weak shielding effect against gravitational force, but only below 70 $K$ and under high-frequency e.m. field. The combination of a high-frequency current inside the rotating toroidal disk and an external high-frequency magnetic field, together with electronic pairing state and superconducting crystal lattice structure, apparently changed the interaction of the solid body with the gravitational field. This resulted in the ability of the superconductor to attenuate the energy of the gravitational force and yielded a weight loss in various samples of as much as 2.1%.

Samples made of metals, plastic, ceramic, wood, etc. were situated over the disk, and their weight was measured with high precision. All the samples showed the same partial loss of weight, no matter what material they were made of. Obtaining the maximum weight loss required that the samples be oriented with their flat surface parallel to the surface of the disk. The overall maximum shielding effect (2.1%) was obtained when the disk's rotational speed and the corresponding centrifugal force were slightly decreased by the magnetic field.

It was found that the shielding effect depended on the temperature, the rotation speed, the frequency and the intensity of the magnetic field. At present it seems early to discuss the mechanisms or to offer a detailed analysis of the observed phenomenon, as further investigation is necessary. The experimentally obtained shielding values may eventually prove to have fundamental importance for technological applications as well as scientific study.

## Acknowledgments.

The author is grateful to the Institute for High Temperatures at the Russian Academy of Sciences for their help in preparing the unique superconducting ceramic disks and for being permitted to use their laboratory equipment. The effect was first observed and studied at Tampere University of Technology. The author would like to thank G. Modanese, F. Pavese and O. Port for advice and support.

**FIGURE CAPTIONS.**

1.  Schematic cross-section of the furnace for high-temperature treatment of the ceramic disk with planar hig-frequency inductors.

2.  General magnets and cryostat setup.

3.  Typical geometry and position of the disk over supporting solenoids.

4.  Schematic design of rotating solenoids. a, b: various configurations.

5.  Typical configuration of the tested set-up for the rotating solenoids.

6.  Block scheme of the electrical connections.

7.  Schematic design of the cryogenic system for the refrigeration of the superconducting disk.

8.  General configuration of the equipment for the weight loss measurements.

9.  Typical design of the three-point disk-braking system.