<HTML>
<CENTER><A HREF = "Section_packages.html">Previous Section</A> - <A HREF = "https://sparta.github.io">SPARTA WWW Site</A> -
<A HREF = "Manual.html">SPARTA Documentation</A> - <A HREF = "Section_commands.html#comm">SPARTA Commands</A> - <A HREF = "Section_howto.html">Next
Section</A> 
</CENTER>






<HR>

<H3>5. Accelerating SPARTA performance 
</H3>
<P>This section describes various methods for improving SPARTA
performance for different classes of problems running on different
kinds of machines.
</P>
<P>Currently the only option is to use the KOKKOS accelerator
packages provided with SPARTA that
contains code optimized for certain kinds of hardware, including
multi-core CPUs, GPUs, and Intel Xeon Phi coprocessors.
</P>
<UL><LI>5.1 <A HREF = "#acc_1">Measuring performance</A> 

<LI>5.2 <A HREF = "#acc_2">Accelerator packages with optimized styles</A> 

<LI>5.3 <A HREF = "#acc_3">KOKKOS package</A> 
</UL>
<P>The <A HREF = "https://sparta.github.io/bench.html">Benchmark page</A> of the SPARTA
web site gives performance results for the various accelerator
packages discussed in Section 5.2, for several of the standard SPARTA
benchmark problems, as a function of problem size and number of
compute nodes, on different hardware platforms.
</P>
<HR>

<H4><A NAME = "acc_1"></A>5.1 Measuring performance 
</H4>
<P>Before trying to make your simulation run faster, you should
understand how it currently performs and where the bottlenecks are.
</P>
<P>The best way to do this is run the your system (actual number of
particles) for a modest number of timesteps (say 100 steps) on several
different processor counts, including a single processor if possible.
Do this for an equilibrium version of your system, so that the
100-step timings are representative of a much longer run.  There is
typically no need to run for 1000s of timesteps to get accurate
timings; you can simply extrapolate from short runs.
</P>
<P>For the set of runs, look at the timing data printed to the screen and
log file at the end of each SPARTA run.  <A HREF = "Section_start.html#start_8">This
section</A> of the manual has an overview.
</P>
<P>Running on one (or a few processors) should give a good estimate of
the serial performance and what portions of the timestep are taking
the most time.  Running the same problem on a few different processor
counts should give an estimate of parallel scalability.  I.e. if the
simulation runs 16x faster on 16 processors, its 100% parallel
efficient; if it runs 8x faster on 16 processors, it's 50% efficient.
</P>
<P>The most important data to look at in the timing info is the timing
breakdown and relative percentages.  For example, trying different
options for speeding up the FFTs will have little impact
if they only consume 10% of the run time.  If the collide time is
dominating, you may want to look at the KOKKOS package, as discussed
below.  Comparing how the percentages change as
you increase the processor count gives you a sense of how different
operations within the timestep are scaling.
</P>
<P>Another important detail in the timing info are the histograms of
particles counts and neighbor counts.  If these vary widely across
processors, you have a load-imbalance issue.  This often results in
inaccurate relative timing data, because processors have to wait when
communication occurs for other processors to catch up.  Thus the
reported times for "Communication" or "Other" may be higher than they
really are, due to load-imbalance.  If this is an issue, you can
uncomment the MPI_Barrier() lines in src/timer.cpp, and recompile
SPARTA, to obtain synchronized timings.
</P>
<HR>

<H4><A NAME = "acc_2"></A>5.2 Packages with optimized styles 
</H4>
<P>Accelerated versions of various <A HREF = "collide_style.html">collide_style</A>,
<A HREF = "fix.html">fixes</A>, <A HREF = "compute.html">computes</A>, and other commands have
been added to SPARTA via the KOKKOS package, which may run faster than
the standard non-accelerated versions.
</P>
<P>All of these commands are in the KOKKOS package provided with SPARTA.
An overview of packages is give in <A HREF = "Section_packages.html">Section
packages</A>.
</P>
<P>SPARTA currently has acceleration support for three kinds of hardware,
via the KOKKOS package: Many-core CPUs, NVIDIA GPUs, and Intel Xeon
Phi.
</P>
<P>Whether you will see speedup for your hardware may depend on the size
problem you are running and what commands (accelerated and
non-accelerated) are invoked by your input script.  While these doc
pages include performance guidelines, there is no substitute for
trying out the KOKKOS package.
</P>
<P>Any accelerated style has the same name as the corresponding standard
style, except that a suffix is appended.  Otherwise, the syntax for
the command that uses the style is identical, their functionality is
the same, and the numerical results it produces should also be the
same, except for precision and round-off effects, and differences in
random numbers.
</P>
<P>For example, the KOKKOS package provides an accelerated variant of the
Temperature Compute <A HREF = "compute_temp.html">compute temp</A>, namely <A HREF = "compute_temp.html">compute
temp/kk</A>
</P>
<P>To see what accelerate styles are currently available, see <A HREF = "Section_commands.html#cmd_5">Section
3.5</A> of the manual.  The doc pages for
individual commands (e.g. <A HREF = "compute_temp.html">compute temp</A>) also list
any accelerated variants available for that style.
</P>
<P>To use an accelerator package in SPARTA, and one or more of the styles
it provides, follow these general steps:
</P>
<P>using make:
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR><TD >install the accelerator package </TD><TD >  make yes-fft, make yes-kokkos, etc </TD></TR>
<TR><TD >add compile/link flags to Makefile.machine in src/MAKE </TD><TD >  KOKKOS_ARCH=PASCAL60 </TD></TR>
<TR><TD >re-build SPARTA </TD><TD >  make kokkos_cuda
</TD></TR></TABLE></DIV>

<P>or, using CMake from a build directory:
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR><TD >install the accelerator package </TD><TD >  cmake -DPKG_FFT=ON -DPKG_KOKKOS=ON, etc </TD></TR>
<TR><TD >add compile/link flags </TD><TD >  cmake -C /path/to/sparta/cmake/presets/kokkos_cuda.cmake -DKokkos_ARCH_PASCAL60=ON </TD></TR>
<TR><TD >re-build SPARTA </TD><TD >  make
</TD></TR></TABLE></DIV>

<P>Then do the following:
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR><TD >prepare and test a regular SPARTA simulation </TD><TD >  lmp_kokkos_cuda -in in.script; mpirun -np 32 lmp_kokkos_cuda -in in.script </TD></TR>
<TR><TD >enable specific accelerator support via '-k on' <A HREF = "Section_start.html#start_7">command-line switch</A>, </TD><TD >  -k on g 1 </TD></TR>
<TR><TD >set any needed options for the package via "-pk" <A HREF = "Section_start.html#start_7">command-line switch</A> or <A HREF = "package.html">package</A> command, </TD><TD >  only if defaults need to be changed, -pk kokkos react/retry yes </TD></TR>
<TR><TD >use accelerated styles in your input via "-sf" <A HREF = "Section_start.html#start_7">command-line switch</A> or <A HREF = "suffix.html">suffix</A> command </TD><TD > lmp_kokkos_cuda -in in.script -sf kk
</TD></TR></TABLE></DIV>

<P>Note that the first 3 steps can be done as a single command with
suitable make command invocations. This is discussed in <A HREF = "Section_packages.html">Section
4</A> of the manual, and its use is illustrated in
the individual accelerator sections.  Typically these steps only need
to be done once, to create an executable that uses one or more
accelerator packages.
</P>
<P>The last 4 steps can all be done from the command-line when SPARTA is
launched, without changing your input script, as illustrated in the
individual accelerator sections.  Or you can add
<A HREF = "package.html">package</A> and <A HREF = "suffix.html">suffix</A> commands to your input
script.
</P>
<P>The <A HREF = "https://sparta.github.io/bench.html">Benchmark page</A> of the SPARTA
web site gives performance results for the various accelerator
packages for several of the standard SPARTA benchmark problems, as a
function of problem size and number of compute nodes, on different
hardware platforms.
</P>
<P>Here is a brief summary of what the KOKKOS package provides.
</P>
<LI>Styles with a "kk" suffix are part of the KOKKOS package, and can be
run using OpenMP on multicore CPUs, on an NVIDIA GPU, or on an Intel
Xeon Phi in "native" mode.  The speed-up depends on a variety of
factors, as discussed on the KOKKOS accelerator page. 


</UL>
<P>The KOKKOS accelerator package doc page explains:
</P>
<UL><LI>what hardware and software the accelerated package requires
<LI>how to build SPARTA with the accelerated package
<LI>how to run with the accelerated package either via command-line switches or modifying the input script
<LI>speed-ups to expect
<LI>guidelines for best performance
<LI>restrictions 
</UL>
<HR>

<H4><A NAME = "acc_3"></A>5.3 KOKKOS package 
</H4>
<P>Kokkos is a templated C++ library that provides abstractions to allow
a single implementation of an application kernel (e.g. a collision
style) to run efficiently on different kinds of hardware, such as
GPUs, Intel Xeon Phis, or many-core CPUs. Kokkos maps the C++ kernel
onto different backend languages such as CUDA, OpenMP, or Pthreads.
The Kokkos library also provides data abstractions to adjust (at
compile time) the memory layout of data structures like 2d and 3d
arrays to optimize performance on different hardware. For more
information on Kokkos, see
<A HREF = "https://github.com/kokkos/kokkos">Github</A>. Kokkos is part of
<A HREF = "http://trilinos.sandia.gov/packages/kokkos">Trilinos</A>. The Kokkos
library was written primarily by Carter Edwards, Christian Trott, and
Dan Sunderland (all Sandia).
</P>
<P>The SPARTA KOKKOS package contains versions of collide, fix, and
compute styles that use data structures and macros provided by the
Kokkos library, which is included with SPARTA in /lib/kokkos. The
KOKKOS package was developed primarily by Stan Moore (Sandia) with
contributions of various styles by others, including Dan Ibanez
(Sandia), Tim Fuller (Sandia), and Sam Mish (Sandia). For more
information on developing using Kokkos abstractions see the Kokkos
programmers' guide at /lib/kokkos/doc/Kokkos_PG.pdf.
</P>
<P>The KOKKOS package currently provides support for 3 modes of execution
(per MPI task). These are Serial (MPI-only for CPUs and Intel Phi),
OpenMP (threading for many-core CPUs and Intel Phi), and CUDA (for
NVIDIA GPUs). You choose the mode at build time to produce an
executable compatible with specific hardware.
</P>
<P>NOTE: Kokkos support within SPARTA must be built with a C++17
compatible compiler. For a list of compilers that have been tested with
the Kokkos library, see the Kokkos <A HREF = "https://github.com/kokkos/kokkos/blob/master/README.md">README</A>.
</P>
<P><B>Building SPARTA with the KOKKOS package with Makefiles:</B>
</P>
<P>To build with the KOKKOS package, start with the provided Kokkos
Makefiles in /src/MAKE/. You may need to modify the KOKKOS_ARCH
variable in the Makefile to match your specific hardware. For example:
</P>
<UL><LI>for Sandy Bridge CPUs, set KOKKOS_ARCH=SNB
<LI>for Broadwell CPUs, set KOKKOS_ARCH=BWD
<LI>for K80 GPUs, set KOKKOS_ARCH=KEPLER37
<LI>for P100 GPUs and Power8 CPUs, set KOKKOS_ARCH=PASCAL60,POWER8 
</UL>
<P><B>Building SPARTA with the KOKKOS package with CMake:</B>
</P>
<P>To build with the KOKKOS package, start with the provided preset files
in /cmake/presets/. You may need to set -D Kokkos_ARCH_<I>TYPE</I>=ON
to match your specific hardware. For example:
</P>
<UL><LI>for Sandy Bridge CPUs, set -D Kokkos_ARCH_SNB=ON
<LI>for Broadwell CPUs, set -D Kokkos_ARCH_BWD=ON
<LI>for K80 GPUs, set -D Kokkos_ARCH_KEPLER37=ON
<LI>for P100 GPUs and Power8 CPUs, set -D Kokkos_ARCH_PASCAL60=ON, -D Kokkos_ARCH_POWER8=ON 
</UL>
<P>See the <B>Advanced Kokkos Options</B> section below for a listing of all
Kokkos architecture options.
</P>
<P><B>Compile for CPU-only (MPI only, no threading):</B>
</P>
<P>Use a C++17 compatible compiler and set Kokkos architicture variable in as described above.  Then do the
following:
</P>
<P>using Makefiles:
</P>
<PRE>cd sparta/src
make yes-kokkos
make kokkos_mpi_only 
</PRE>
<P>using CMake:
</P>
<PRE>cd build
cmake -C /path/to/sparta/cmake/presets/kokkos_mpi_only.cmake
make 
</PRE>
<P><B>Compile for CPU-only (MPI plus OpenMP threading):</B>
</P>
<P>NOTE: To build with Kokkos support for OpenMP threading, your compiler
must support the OpenMP interface. You should have one or more
multi-core CPUs so that multiple threads can be launched by each MPI
task running on a CPU.
</P>
<P>Use a C++17 compatible compiler and set Kokkos architecture variable in
as described above.  Then do the
following:
</P>
<P>using Makefiles:
</P>
<PRE>cd sparta/src
make yes-kokkos
make kokkos_omp 
</PRE>
<P>using CMake:
</P>
<PRE>cd build
cmake -C /path/to/sparta/cmake/presets/kokkos_omp.cmake
make 
</PRE>
<P><B>Compile for Intel KNL Xeon Phi (Intel Compiler, OpenMPI):</B>
</P>
<P>Use a C++17 compatible compiler and do the following:
</P>
<P>using Makefiles:
</P>
<PRE>cd sparta/src
make yes-kokkos
make kokkos_phi 
</PRE>
<P>using CMake:
</P>
<PRE>cd build
cmake -C /path/to/sparta/cmake/presets/kokkos_phi.cmake
make 
</PRE>
<P><B>Compile for CPUs and GPUs (with OpenMPI or MPICH):</B>
</P>
<P>NOTE: To build with Kokkos support for NVIDIA GPUs, NVIDIA CUDA
software version 11.0 or later must be installed on your system.
</P>
<P>Use a C++17 compatible compiler and set Kokkos architecture variable in
for both GPU and CPU as described
above.  Then do the following:
</P>
<P>using Makefiles:
</P>
<PRE>cd sparta/src
make yes-kokkos
make kokkos_cuda 
</PRE>
<P>using CMake:
</P>
<PRE>cd build
cmake -C /path/to/sparta/cmake/presets/kokkos_cuda.cmake
make 
</PRE>
<P><B>Running SPARTA with the KOKKOS package:</B>
</P>
<P>All Kokkos operations occur within the context of an individual MPI
task running on a single node of the machine. The total number of MPI
tasks used by SPARTA (one or multiple per compute node) is set in the
usual manner via the mpirun or mpiexec commands, and is independent of
Kokkos. The mpirun or mpiexec command sets the total number of MPI
tasks used by SPARTA (one or multiple per compute node) and the number
of MPI tasks used per node. E.g. the mpirun command in OpenMPI does
this via its -np and -npernode switches. Ditto for MPICH via -np and
-ppn.
</P>
<P><B>Running on a multi-core CPU:</B>
</P>
<P>Here is a quick overview of how to use the KOKKOS package for CPU
acceleration, assuming one or more 16-core nodes.
</P>
<PRE>mpirun -np 16 spa_kokkos_mpi_only -k on -sf kk -in in.collide        # 1 node, 16 MPI tasks/node, no multi-threading
mpirun -np 2 -ppn 1 spa_kokkos_omp -k on t 16 -sf kk -in in.collide  # 2 nodes, 1 MPI task/node, 16 threads/task
mpirun -np 2 spa_kokkos_omp -k on t 8 -sf kk -in in.collide          # 1 node,  2 MPI tasks/node, 8 threads/task
mpirun -np 32 -ppn 4 spa_kokkos_omp -k on t 4 -sf kk -in in.collide  # 8 nodes, 4 MPI tasks/node, 4 threads/task 
</PRE>
<P>To run using the KOKKOS package, use the "-k on", "-sf kk" and "-pk
kokkos" <A HREF = "Section_start.html#start_7">command-line switches</A> in your
mpirun command.  You must use the "-k on" <A HREF = "Section_start.html#start_7">command-line
switch</A> to enable the KOKKOS package. It
takes additional arguments for hardware settings appropriate to your
system. Those arguments are <A HREF = "Section_start.html#start_7">documented
here</A>. For OpenMP use:
</P>
<PRE>-k on t Nt 
</PRE>
<P>The "t Nt" option specifies how many OpenMP threads per MPI task to
use with a node. The default is Nt = 1, which is MPI-only mode.  Note
that the product of MPI tasks * OpenMP threads/task should not exceed
the physical number of cores (on a node), otherwise performance will
suffer. If hyperthreading is enabled, then the product of MPI tasks *
OpenMP threads/task should not exceed the physical number of cores *
hardware threads.  The "-k on" switch also issues a "package kokkos"
command (with no additional arguments) which sets various KOKKOS
options to default values, as discussed on the <A HREF = "package.html">package</A>
command doc page.
</P>
<P>The "-sf kk" <A HREF = "Section_start.html#start_7">command-line switch</A> will
automatically append the "/kk" suffix to styles that support it.  In
this manner no modification to the input script is
needed. Alternatively, one can run with the KOKKOS package by editing
the input script as described below.
</P>
<P>NOTE: When using a single OpenMP thread, the Kokkos Serial backend (i.e. 
Makefile.kokkos_mpi_only) will give better performance than the OpenMP 
backend (i.e. Makefile.kokkos_omp) because some of the overhead to make 
the code thread-safe is removed.
</P>
<P>NOTE: The default for the <A HREF = "package.html">package kokkos</A> command is to
use "threaded" communication. However, when running on CPUs, it will
typically be faster to use "classic" non-threaded communication.  Use
the "-pk kokkos" <A HREF = "Section_start.html#start_7">command-line switch</A> to
change the default <A HREF = "package.html">package kokkos</A> options. See its doc
page for details and default settings. Experimenting with its options
can provide a speed-up for specific calculations. For example:
</P>
<PRE>mpirun -np 16 spa_kokkos_mpi_only -k on -sf kk -pk kokkos comm classic -in in.collide       # non-threaded comm 
</PRE>
<P>For OpenMP, the KOKKOS package uses data duplication (i.e. 
thread-private arrays) by default to avoid thread-level write conflicts 
in some compute styles. Data duplication is typically fastest for small 
numbers of threads (i.e. 8 or less) but does increase memory footprint 
and is not scalable to large numbers of threads. An alternative to data 
duplication is to use thread-level atomics, which don't require 
duplication. When using the Kokkos Serial backend or the OpenMP backend 
with a single thread, no duplication or atomics are used. For CUDA, the 
KOKKOS package always uses atomics in these computes when necessary. The 
use of atomics instead of duplication can be forced by compiling with the 
"-DSPARTA_KOKKOS_USE_ATOMICS" compile switch. 
</P>
<P><B>Core and Thread Affinity:</B>
</P>
<P>When using multi-threading, it is important for performance to bind
both MPI tasks to physical cores, and threads to physical cores, so
they do not migrate during a simulation.
</P>
<P>If you are not certain MPI tasks are being bound (check the defaults
for your MPI installation), binding can be forced with these flags:
</P>
<PRE>OpenMPI 1.8: mpirun -np 2 -bind-to socket -map-by socket ./spa_openmpi ...
Mvapich2 2.0: mpiexec -np 2 -bind-to socket -map-by socket ./spa_mvapich ... 
</PRE>
<P>For binding threads with KOKKOS OpenMP, use thread affinity
environment variables to force binding. With OpenMP 3.1 (gcc 4.7 or
later, intel 12 or later) setting the environment variable
OMP_PROC_BIND=true should be sufficient. In general, for best
performance with OpenMP 4.0 or better set OMP_PROC_BIND=spread and
OMP_PLACES=threads.  For binding threads with the KOKKOS pthreads
option, compile SPARTA the KOKKOS HWLOC=yes option as described below.
</P>
<P><B>Running on Knight's Landing (KNL) Intel Xeon Phi:</B>
</P>
<P>Here is a quick overview of how to use the KOKKOS package for the
Intel Knight's Landing (KNL) Xeon Phi:
</P>
<P>KNL Intel Phi chips have 68 physical cores. Typically 1 to 4 cores are
reserved for the OS, and only 64 or 66 cores are used. Each core has 4
hyperthreads, so there are effectively N = 256 (4*64) or N = 264
(4*66) cores to run on. The product of MPI tasks * OpenMP threads/task
should not exceed this limit, otherwise performance will suffer. Note
that with the KOKKOS package you do not need to specify how many KNLs
there are per node; each KNL is simply treated as running some number
of MPI tasks.
</P>
<P>Examples of mpirun commands that follow these rules are shown below.
</P>
<PRE>Intel KNL node with 64 cores (256 threads/node via 4x hardware threading):
mpirun -np 64 spa_kokkos_phi -k on t 4 -sf kk -in in.collide      # 1 node, 64 MPI tasks/node, 4 threads/task
mpirun -np 66 spa_kokkos_phi -k on t 4 -sf kk -in in.collide      # 1 node, 66 MPI tasks/node, 4 threads/task
mpirun -np 32 spa_kokkos_phi -k on t 8 -sf kk -in in.collide      # 1 node, 32 MPI tasks/node, 8 threads/task
mpirun -np 512 -ppn 64 spa_kokkos_phi -k on t 4 -sf kk -in in.collide  # 8 nodes, 64 MPI tasks/node, 4 threads/task 
</PRE>
<P>The -np setting of the mpirun command sets the number of MPI
tasks/node. The "-k on t Nt" command-line switch sets the number of
threads/task as Nt. The product of these two values should be N, i.e.
256 or 264.
</P>
<P>NOTE: The default for the <A HREF = "package.html">package kokkos</A> command is to
use "threaded" communication. However, when running on KNL, it will
typically be faster to use "classic" non-threaded communication.  Use
the "-pk kokkos" <A HREF = "Section_start.html#start_7">command-line switch</A> to
change the default <A HREF = "package.html">package kokkos</A> options. See its doc
page for details and default settings. Experimenting with its options
can provide a speed-up for specific calculations. For example:
</P>
<PRE>mpirun -np 64 spa_kokkos_phi -k on t 4 -sf kk -pk kokkos comm classic -in in.collide      # non-threaded comm 
</PRE>
<P>NOTE: MPI tasks and threads should be bound to cores as described
above for CPUs.
</P>
<P>NOTE: To build with Kokkos support for Intel Xeon Phi coprocessors
such as Knight's Corner (KNC), your system must be configured to use
them in "native" mode, not "offload" mode.
</P>
<P><B>Running on GPUs:</B>
</P>
<P>Use the "-k" <A HREF = "Section_commands.html#start_7">command-line switch</A> to
specify the number of GPUs per node, and the number of threads per MPI
task. Typically the -np setting of the mpirun command should set the
number of MPI tasks/node to be equal to the # of physical GPUs on the
node.  You can assign multiple MPI tasks to the same GPU with the
KOKKOS package, but this is usually only faster if significant
portions of the input script have not been ported to use Kokkos. Using
CUDA MPS is recommended in this scenario. As above for multi-core CPUs
(and no GPU), if N is the number of physical cores/node, then the
number of MPI tasks/node should not exceed N.
</P>
<PRE>-k on g Ng 
</PRE>
<P>Here are examples of how to use the KOKKOS package for GPUs, assuming
one or more nodes, each with two GPUs.
</P>
<PRE>mpirun -np 2 spa_kokkos_cuda -k on g 2 -sf kk -in in.collide          # 1 node,   2 MPI tasks/node, 2 GPUs/node
mpirun -np 32 -ppn 2 spa_kokkos_cuda -k on g 2 -sf kk -in in.collide  # 16 nodes, 2 MPI tasks/node, 2 GPUs/node (32 GPUs total) 
</PRE>
<P>NOTE: Use the "-pk kokkos" <A HREF = "Section_start.html#start_7">command-line
switch</A> to change the default <A HREF = "package.html">package
kokkos</A> options. See its doc page for details and default
settings. For example:
</P>
<PRE>mpirun -np 2 spa_kokkos_cuda -k on g 2 -sf kk -pk kokkos gpu/aware off -in in.collide      # set gpu/aware MPI support off 
</PRE>
<P>NOTE: Using OpenMP threading and CUDA together is currently not
possible with the SPARTA KOKKOS package.
</P>
<P>NOTE: For good performance of the KOKKOS package on GPUs, you must
have Kepler generation GPUs (or later). The Kokkos library exploits
texture cache options not supported by Telsa generation GPUs (or
older).
</P>
<P>NOTE: When using a GPU, you will achieve the best performance if your
input script does not use fix or compute styles which are not yet
Kokkos-enabled. This allows data to stay on the GPU for multiple
timesteps, without being copied back to the host CPU. Invoking a
non-Kokkos fix or compute, or performing I/O for <A HREF = "stat.html">stat</A> or
<A HREF = "dump.html">dump</A> output will cause data to be copied back to the CPU
incurring a performance penalty.
</P>
<P><B>Run with the KOKKOS package by editing an input script:</B>
</P>
<P>Alternatively the effect of the "-sf" or "-pk" switches can be
duplicated by adding the <A HREF = "package.html">package kokkos</A> or <A HREF = "suffix.html">suffix
kk</A> commands to your input script.
</P>
<P>The discussion above for building SPARTA with the KOKKOS package, the
mpirun/mpiexec command, and setting appropriate thread are the same.
</P>
<P>You must still use the "-k on" <A HREF = "Section_start.html#start_7">command-line
switch</A> to enable the KOKKOS package, and
specify its additional arguments for hardware options appropriate to
your system, as documented above.
</P>
<P>You can use the <A HREF = "suffix.html">suffix kk</A> command, or you can explicitly add a
"kk" suffix to individual styles in your input script, e.g.
</P>
<PRE>collide vss/kk air ar.vss 
</PRE>
<P>You only need to use the <A HREF = "package.html">package kokkos</A> command if you
wish to change any of its option defaults, as set by the "-k on"
<A HREF = "Section_start.html#start_7">command-line switch</A>.
</P>
<P><B>Speed-ups to expect:</B>
</P>
<P>The performance of KOKKOS running in different modes is a function of
your hardware, which KOKKOS-enable styles are used, and the problem
size.
</P>
<P>Generally speaking, when running on CPUs only, with a single thread per MPI task, the
performance difference of a KOKKOS style and (un-accelerated) styles
(MPI-only mode) is typically small (less than 20%).
</P>
<P>See the <A HREF = "https://sparta.github.io/bench.html">Benchmark page</A> of the
SPARTA web site for performance of the KOKKOS package on different
hardware.
</P>
<P><B>Advanced Kokkos options:</B>
</P>
<P>There are other allowed options when building with the KOKKOS package.
A few options are listed here; for a full list of all options,
please refer to the Kokkos documentation.
As above, these options can be set as variables on the command line,
in a Makefile, or in a CMake presets file. For default CMake values,
see cmake -LH | grep -i kokkos.
</P>
<P>The CMake option Kokkos_ENABLE_<I>OPTION</I> or the makefile setting KOKKOS_DEVICE=<I>OPTION</I> sets the 
parallelization method used for Kokkos code (within SPARTA). 
For example, the CMake option Kokkos_ENABLE_SERIAL=ON or the makefile setting KOKKOS_DEVICES=SERIAL
means that no threading will be used.  The CMake option Kokkos_ENABLE_OPENMP=ON or the 
makefile setting KOKKOS_DEVICES=OPENMP means that OpenMP threading will be
used. The CMake option Kokkos_ENABLE_CUDA=ON or the makefile setting
KOKKOS_DEVICES=CUDA means an NVIDIA GPU running CUDA will be used.
</P>
<P>As described above, the CMake option Kokkos_ARCH_<I>TYPE</I>=ON or the makefile setting KOKKOS_ARCH=<I>TYPE</I> enables compiler switches needed when compiling for a specific hardware:
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR><TD ><B>Arch-ID</B>	</TD><TD > <B>HOST or GPU</B> </TD><TD >	<B>Description</B> </TD></TR>
<TR><TD >NATIVE </TD><TD > HOST </TD><TD > Local machine </TD></TR>
<TR><TD >AMDAVX </TD><TD > HOST </TD><TD > AMD 64-bit x86 CPU (AVX 1) </TD></TR>
<TR><TD >ZEN </TD><TD > HOST </TD><TD > AMD Zen class CPU (AVX 2) </TD></TR>
<TR><TD >ZEN2 </TD><TD > HOST </TD><TD > AMD Zen2 class CPU (AVX 2) </TD></TR>
<TR><TD >ZEN3 </TD><TD > HOST </TD><TD > AMD Zen3 class CPU (AVX 2) </TD></TR>
<TR><TD >ARMV80 </TD><TD > HOST </TD><TD > ARMv8.0 Compatible CPU </TD></TR>
<TR><TD >ARMV81 </TD><TD > HOST </TD><TD > ARMv8.1 Compatible CPU </TD></TR>
<TR><TD >ARMV8_THUNDERX </TD><TD > HOST </TD><TD > ARMv8 Cavium ThunderX CPU </TD></TR>
<TR><TD >ARMV8_THUNDERX2 </TD><TD > HOST </TD><TD > ARMv8 Cavium ThunderX2 CPU </TD></TR>
<TR><TD >A64FX </TD><TD > HOST </TD><TD > ARMv8.2 with SVE Support </TD></TR>
<TR><TD >ARMV9_GRACE </TD><TD > HOST </TD><TD > ARMv9 NVIDIA Grace CPU </TD></TR>
<TR><TD >SNB </TD><TD > HOST </TD><TD > Intel Sandy/Ivy Bridge CPU (AVX 1) </TD></TR>
<TR><TD >HSW </TD><TD > HOST </TD><TD > Intel Haswell CPU (AVX 2) </TD></TR>
<TR><TD >BDW </TD><TD > HOST </TD><TD > Intel Broadwell Xeon E-class CPU (AVX 2 + transactionalmem) </TD></TR>
<TR><TD >SKL </TD><TD > HOST </TD><TD > Intel Skylake Client CPU </TD></TR>
<TR><TD >SKX </TD><TD > HOST </TD><TD > Intel Skylake Xeon Server CPU (AVX512) </TD></TR>
<TR><TD >ICL </TD><TD > HOST </TD><TD > Intel Ice Lake Client CPU (AVX512) </TD></TR>
<TR><TD >ICX </TD><TD > HOST </TD><TD > Intel Ice Lake Xeon Server CPU (AVX512) </TD></TR>
<TR><TD >SPR </TD><TD > HOST </TD><TD > Intel Sapphire Rapids Xeon Server CPU (AVX512) </TD></TR>
<TR><TD >KNC </TD><TD > HOST </TD><TD > Intel Knights Corner Xeon Phi </TD></TR>
<TR><TD >KNL </TD><TD > HOST </TD><TD > Intel Knights Landing Xeon Phi </TD></TR>
<TR><TD >POWER8 </TD><TD > HOST </TD><TD > IBM POWER8 CPU </TD></TR>
<TR><TD >POWER9 </TD><TD > HOST </TD><TD > IBM POWER9 CPU </TD></TR>
<TR><TD >RISCV_SG2042 </TD><TD > HOST </TD><TD > SG2042 (RISC-V) CPU </TD></TR>
<TR><TD >KEPLER30 </TD><TD > GPU </TD><TD > NVIDIA Kepler generation CC 3.0 GPU </TD></TR>
<TR><TD >KEPLER32 </TD><TD > GPU </TD><TD > NVIDIA Kepler generation CC 3.2 GPU </TD></TR>
<TR><TD >KEPLER35 </TD><TD > GPU </TD><TD > NVIDIA Kepler generation CC 3.5 GPU </TD></TR>
<TR><TD >KEPLER37 </TD><TD > GPU </TD><TD > NVIDIA Kepler generation CC 3.7 GPU </TD></TR>
<TR><TD >MAXWELL50 </TD><TD > GPU </TD><TD > NVIDIA Maxwell generation CC 5.0 GPU </TD></TR>
<TR><TD >MAXWELL52 </TD><TD > GPU </TD><TD > NVIDIA Maxwell generation CC 5.2 GPU </TD></TR>
<TR><TD >MAXWELL53 </TD><TD > GPU </TD><TD > NVIDIA Maxwell generation CC 5.3 GPU </TD></TR>
<TR><TD >PASCAL60 </TD><TD > GPU </TD><TD > NVIDIA Pascal generation CC 6.0 GPU </TD></TR>
<TR><TD >PASCAL61 </TD><TD > GPU </TD><TD > NVIDIA Pascal generation CC 6.1 GPU </TD></TR>
<TR><TD >VOLTA70 </TD><TD > GPU </TD><TD > NVIDIA Volta generation CC 7.0 GPU </TD></TR>
<TR><TD >VOLTA72 </TD><TD > GPU </TD><TD > NVIDIA Volta generation CC 7.2 GPU </TD></TR>
<TR><TD >TURING75 </TD><TD > GPU </TD><TD > NVIDIA Turing generation CC 7.5 GPU </TD></TR>
<TR><TD >AMPERE80 </TD><TD > GPU </TD><TD > NVIDIA Ampere generation CC 8.0 GPU </TD></TR>
<TR><TD >AMPERE86 </TD><TD > GPU </TD><TD > NVIDIA Ampere generation CC 8.6 GPU </TD></TR>
<TR><TD >ADA89 </TD><TD > GPU </TD><TD > NVIDIA Ada Lovelace generation CC 8.9 GPU </TD></TR>
<TR><TD >HOPPER90 </TD><TD > GPU </TD><TD > NVIDIA Hopper generation CC 9.0 GPU </TD></TR>
<TR><TD >AMD_GFX908 </TD><TD > GPU </TD><TD > AMD GPU MI100 </TD></TR>
<TR><TD >AMD_GFX90A </TD><TD > GPU </TD><TD > AMD GPU MI200 </TD></TR>
<TR><TD >AMD_GFX942 </TD><TD > GPU </TD><TD > AMD GPU MI300 </TD></TR>
<TR><TD >AMD_GFX1030 </TD><TD > GPU </TD><TD > AMD GPU V620/W6800 </TD></TR>
<TR><TD >AMD_GFX1100 </TD><TD > GPU </TD><TD > AMD GPU RX7900XTX </TD></TR>
<TR><TD >INTEL_GEN GPU SPIR64-based devices, e.g. Intel GPUs, using JIT </TD><TD >INTEL_DG1 </TD><TD > GPU </TD></TR>
<TR><TD > Intel Iris XeMAX GPU </TD><TD >INTEL_GEN9 </TD><TD > GPU </TD></TR>
<TR><TD > Intel GPU Gen9 </TD><TD >INTEL_GEN11 </TD><TD > GPU </TD></TR>
<TR><TD > Intel GPU Gen11 </TD><TD >INTEL_GEN12LP </TD><TD > GPU </TD></TR>
<TR><TD > Intel GPU Gen12LP </TD><TD >INTEL_XEHP </TD><TD > GPU </TD></TR>
<TR><TD > Intel GPU Xe-HP </TD><TD >INTEL_PVC </TD><TD > GPU </TD></TR>
<TR><TD > Intel GPU Ponte Vecchio </TD><TD >
</TD></TR></TABLE></DIV>

<P>The CMake option Kokkos_ENABLE_CUDA_<I>OPTION</I> or the makefile setting KOKKOS_CUDA_OPTIONS=<I>OPTION</I> are 
additional options for CUDA. For example, the CMake option Kokkos_ENABLE_CUDA_UVM=ON or the makefile setting KOKKOS_CUDA_OPTIONS="enable_lambda,force_uvm" enables the use of CUDA "Unified Virtual Memory" (UVM) in Kokkos. UVM allows to one to use the host CPU memory to supplement the memory used on the GPU (with some performance penalty) and thus enables running larger problems that would otherwise not fit into the RAM on the GPU. Please note, that the SPARTA KOKKOS package must always be compiled with the CMake option Kokkos_ENABLE_CUDA_LAMBDA=ON or the makefile setting KOKKOS_CUDA_OPTIONS=enable_lambda when using GPUs. The CMake configuration will thus always enable it.
</P>
<P>The CMake option Kokkos_ENABLE_DEBUG=ON or the makefile setting KOKKOS_DEBUG=yes is useful
when developing a Kokkos-enabled style within SPARTA. This option enables printing of run-time debugging
information that can be useful and also enables runtime bounds
checking on Kokkos data structures, but may slow down performance.
</P>
<P><B>Restrictions:</B>
</P>
<P>Currently, there are no precision options with the KOKKOS package. All
compilation and computation is performed in double precision.
</P>
</HTML>
