<HTML>
<CENTER><A HREF = "Section_commands.html">Previous Section</A> - <A HREF = "https://sparta.github.io">SPARTA WWW Site</A> -
<A HREF = "Manual.html">SPARTA Documentation</A> - <A HREF = "Section_commands.html#comm">SPARTA Commands</A> - <A HREF = "Section_accelerate.html">Next
Section</A> 
</CENTER>






<HR>

<H3>4. Packages 
</H3>
<P>This section gives an overview of the optional packages that extend
SPARTA functionality with instructions on how to build SPARTA with
each of them.  Packages are groups of files that enable a specific set
of features.  For example, the KOKKOS package provides styles that
can run on different hardware such as GPUs.  You can see the list of all
packages and "make" commands to manage them by typing "make package"
from within the src directory of the SPARTA distribution or
"cmake -DSPARTA_LIST_PKGS" from within a build directory.  <A HREF = "Section_start.html#start_3">Section
2.3</A> gives general info on how to install
and un-install packages as part of the SPARTA build process.
</P>
<P>Packages may require some
additional code compiled located in the lib folder, or may require
an external library to be downloaded, compiled, installed, and SPARTA
configured to know about its location and additional compiler flags.
</P>
<P>Following the next two tables is a sub-section for each package.  It
lists authors (if applicable) and summarizes the package contents.  It
has specific instructions on how to install the package, including (if
necessary) downloading or building any extra library it requires. It
also gives links to documentation, example scripts, and
pictures/movies (if available) that illustrate use of the package.
</P>
<P>NOTE: To see the complete list of commands a package adds to SPARTA,
just look at the files in its src directory, e.g. "ls src/KOKKOS".
Files with names that start with fix, compute, etc correspond to
commands with the same style names.
</P>
<P>In these two tables, the "Example" column is a sub-directory in the
examples directory of the distribution which has an input script that
uses the package.  E.g. "fft" refers to the examples/fft
directory; The "Library" column indicates whether an extra library is needed to build
and use the package:
</P>
<UL><LI>dash = no library
<LI>sys = system library: you likely have it on your machine
<LI>int = internal library: provided with SPARTA, but you may need to build it
<LI>ext = external library: you will need to download and install it on your machine 
</UL>
<HR>

<A NAME = "pkg_1"></A><P><B>SPARTA packages</B> 
</P>
<DIV ALIGN=center><TABLE  BORDER=1 >
<TR ALIGN="center"><TD  ALIGN ="left">Package</TD><TD > Description</TD><TD > Doc page</TD><TD > Example</TD><TD > Library</TD></TR>
<TR ALIGN="center"><TD  ALIGN ="left"><A HREF = "#FFT">FFT</A></TD><TD > fast Fourier transforms</TD><TD > <A HREF = "compute_fft_grid.html">compute_style compute/fft/grid</A></TD><TD > fft</TD><TD > int or ext</TD></TR>
<TR ALIGN="center"><TD  ALIGN ="left"><A HREF = "#KOKKOS">KOKKOS</A></TD><TD > Kokkos-enabled styles</TD><TD > <A HREF = "Section_howto.html#acc_3">Section 5.3</A></TD><TD > <A HREF = "https://sparta.github.io/bench.html">Benchmarks</A></TD><TD > - 
</TD></TR></TABLE></DIV>

<HR>

<A NAME = "FFT"></A><H4>FFT package 
</H4>
<P><B>Contents:</B>
</P>
<P>Apply Fast Fourier Transforms (FFTs) to simulation data. The FFT library is
specified in the Makefile.machine using the FFT_INC, FFT_PATH, and FFT_LIB
variables. Supported external FFT libraries that can be specified include FFTW3
and MKL. If no FFT library is specified in the Makefile, SPARTA will use the
internal KISS FFT library that is included with SPARTA.
</P>
<P>Similarly an external FFT library can be specified for the KOKKOS package.
Options are CUFFT, HIPFFT, FFTW3, and MKL. If no FFT library is specified in
the Makefile, SPARTA will use the internal Kokkos version of the KISS FFT
library that is included with SPARTA.
</P>
<P>See the see discussion in doc/Section_start.html#2_2 (step 6).
</P>
<P><B>Install or un-install with make:</B>
</P>
<PRE>make yes-fft
make machine 
</PRE>
<PRE>make no-fft
make machine 
</PRE>
<P><B>Install or un-install with CMake:</B>
</P>
<PRE>cd build
cmake -C /path/to/sparta/cmake/presets/machine.cmake -DPKG_FFT=ON /path/to/sparta/cmake
make 
</PRE>
<PRE>cmake -C /path/to/sparta/cmake/presets/machine.cmake -DPKG_FFT=OFF /path/to/sparta/cmake
make 
</PRE>
<P><B>Supporting info:</B>
</P>
<UL><LI><A HREF = "compute_fft_grid.html">compute fft/grid</A>
<LI>examples/fft 
</UL>
<HR>

<A NAME = "KOKKOS"></A><H4>KOKKOS package 
</H4>
<P><B>Contents:</B>
</P>
<P>Styles adapted to compile using the Kokkos library which can convert
them to OpenMP or CUDA code so that they run efficiently on multicore
CPUs, KNLs, or GPUs.  All the styles have a "kk" as a suffix in their
style name.  <A HREF = "Section_accelerate.html#acc_3">Section 5.3</A> gives details
of what hardware and software is required on your system, and how to
build and use this package.  Its styles can be invoked at run time via
the "-sf kk" or "-suffix kk" <A HREF = "Section_start.html#start_7">command-line
switches</A>.
</P>
<P>You must have a C++17 compatible compiler to use this package.
</P>
<P><B>Authors:</B> The KOKKOS package was created primarily by Stan Moore (Sandia),
with contributions from other folks as well.
It uses the open-source <A HREF = "https://github.com/kokkos">Kokkos library</A>
which was developed by Carter Edwards, Christian Trott, and others at
Sandia, and which is included in the SPARTA distribution in
lib/kokkos.
</P>
<P><B>Install or un-install:</B>
</P>
<P>For the KOKKOS package, you have 3 choices when building.  You can
build with either CPU or KNL or GPU support.  Each choice requires
additional settings in your Makefile.machine or machine.cmake file 
for the KOKKOS_DEVICES and KOKKOS_ARCH settings. See the 
src/MAKE/OPTIONS/Makefile.kokkos* or cmake/presets/*kokkos*.cmake
files for examples. For CMake, it's best to start by copying
cmake/presets/kokkos_cuda.cmake to cmake/presets/machine.cmake.
</P>
<P><B>For multicore CPUs using OpenMP:</B>
</P>
<P>Using Makefiles:
</P>
<PRE>KOKKOS_DEVICES = OpenMP
KOKKOS_ARCH = HSW           # HSW = Haswell, SNB = SandyBridge, BDW = Broadwell, etc 
</PRE>
<P>Using CMake:
</P>
<P>-DKokkos_ENABLE_OPENMP=ON
-DKokkos_ARCH_HSW=ON
</P>
<P><B>For Intel KNLs using OpenMP:</B>
</P>
<P>Using Makefiles:
</P>
<PRE>KOKKOS_DEVICES = OpenMP
KOKKOS_ARCH = KNL 
</PRE>
<P>Using CMake:
</P>
<PRE>-DKokkos_ENABLE_OPENMP=ON
-DKokkos_ARCH_KNL=ON 
</PRE>
<P><B>For NVIDIA GPUs using CUDA:</B>
</P>
<PRE>KOKKOS_DEVICES = Cuda
KOKKOS_ARCH = PASCAL60,POWER8     # P100 hosted by an IBM Power8, etc
KOKKOS_ARCH = KEPLER37,POWER8     # K80 hosted by an IBM Power8, etc 
</PRE>
<P>Using CMake:
</P>
<PRE>-DKokkos_ENABLE_CUDA=ON
-DKokkos_ARCH_PASCAL60=ON -DKokkos_ARCH_POWER8=ON 
</PRE>
<P>For make with GPUs, the following 2 lines define a nvcc wrapper compiler, which will use
nvcc for compiling CUDA files or use a C++ compiler for non-Kokkos, non-CUDA
files.
</P>
<PRE>KOKKOS_ABSOLUTE_PATH = $(shell cd $(KOKKOS_PATH); pwd)
export OMPI_CXX = $(KOKKOS_ABSOLUTE_PATH)/bin/nvcc_wrapper
CC =		mpicxx 
</PRE>
<P>For CMake, copy cmake/presets/kokkos_cuda.cmake so OMPI_CXX and CC are set
properly.
</P>
<P>Once you have an appropriate Makefile.machine or machine.cmake, you can
install/un-install the package and build SPARTA in the usual manner.
Note that you cannot build one executable to run on multiple hardware
targets (CPU or KNL or GPU).  You need to build SPARTA once for each
hardware target, to produce a separate executable.
</P>
<P>Using make:
</P>
<PRE>make yes-kokkos
make machine 
</PRE>
<PRE>make no-kokkos
make machine 
</PRE>
<P>Using CMake:
</P>
<PRE>cmake -C /path/to/sparta/cmake/presets/machine.cmake /path/to/sparta/cmake
make 
</PRE>
<PRE>cmake -C /path/to/sparta/cmake/presets/machine.cmake -DPKG_KOKKOS=OFF /path/to/sparta/cmake
make 
</PRE>
<P><B>Supporting info:</B>
</P>
<UL><LI>src/KOKKOS: filenames -> commands
<LI>src/KOKKOS/README
<LI>lib/kokkos/README
<LI>the <A HREF = "Section_accelerate.html#acc_3">Accelerating SPARTA</A> section
<LI><A HREF = "Section_accelerate.html#acc_3">Section 5.3</A>
<LI><A HREF = "Section_start.html#start_7">Section 2.6 -k on ...</A>
<LI><A HREF = "Section_start.html#start_7">Section 2.6 -sf kk</A>
<LI><A HREF = "Section_start.html#start_7">Section 2.6 -pk kokkos</A>
<LI><A HREF = "package.html">package kokkos</A>
<LI><A HREF = "https://sparta.github.io/bench.html">Benchmarks page</A> of web site 
</UL>
</HTML>
