%% $Id$

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\documentclass[english]{article}
\usepackage[latin1]{inputenc}
\usepackage{babel}
\usepackage{verbatim}

%% do we have the `hyperref package?
\IfFileExists{hyperref.sty}{
   \usepackage[bookmarksopen,bookmarksnumbered]{hyperref}
}{}

%% do we have the `fancyhdr' or `fancyheadings' package?
\IfFileExists{fancyhdr.sty}{
\usepackage[fancyhdr]{latex2man}
}{
\IfFileExists{fancyheadings.sty}{
\usepackage[fancy]{latex2man}
}{
\usepackage[nofancy]{latex2man}
\message{no fancyhdr or fancyheadings package present, discard it}
}}

%% do we have the `rcsinfo' package?
\IfFileExists{rcsinfo.sty}{
\usepackage[nofancy]{rcsinfo}
\rcsInfo $Id$
\setDate{\rcsInfoLongDate}
}{
\setDate{ 2014/03/05}
\message{package rcsinfo not present, discard it}
}

\setVersionWord{Version:}  %%% that's the default, no need to set it.
\setVersion{=PACKAGE_VERSION=}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{document}

\begin{Name}{1}{hpcrun}{The HPCToolkit Performance Tools}{The HPCToolkit Performance Tools}{hpcrun:\\ Statistical Profiling}

\Prog{hpcrun} is a call path profiler based on statistical sampling.
It supports multiple sample sources during one execution.
\Prog{hpcrun} profiles complex applications (forks, execs, threads and dynamic linking) and may be used in conjunction with parallel process launchers such as MPICH's \texttt{mpiexec} and SLURM's \texttt{srun}.

See \HTMLhref{hpctoolkit.html}{\Cmd{hpctoolkit}{1}} for an overview of \textbf{HPCToolkit}.


\end{Name}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Synopsis}

\Prog{hpcrun} \oOpt{profiling-options} \Arg{command} \oOpt{command-arguments}

\Prog{hpcrun} \oOpt{info-options}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Description}

\Prog{hpcrun} profiles the execution of an arbitrary command \Arg{command} using statistical sampling (rather than instrumentation).
It collects per-thread call path profiles that represent the full calling context of sample points.
Sample points may be generated from multiple simultaneous sampling sources.
\Prog{hpcrun} profiles complex applications that use forks, execs, threads, and dynamic linking/unlinking; it may be used in conjuction with parallel process launchers such as MPICH's \texttt{mpiexec} and SLURM's \texttt{srun}.

To profile a statically linked executable, make sure to link with \HTMLhref{hpclink.html}{\Cmd{hpclink}{1}}.

To configure \Prog{hpcrun}'s sampling sources, specify events and periods using the \texttt{-e/--event} option.
For an event \emph{e} and period \emph{p}, after every \emph{p} instances of \emph{e}, a sample is generated that causes \Prog{hpcrun} to inspect the and record information about the monitored \Arg{command}.

When \Arg{command} terminates, a profile measurement databse will be written to the directory:\\
\SP\SP\SP hpctoolkit-\Arg{command}-measurements[-\Arg{jobid}]\\
where \Arg{jobid} is a PBS or Sun Grid Engine job identifier.

\Prog{hpcrun} enables a user to abort a process and write the partial profiling data to disk by sending the Interrupt signal (INT or Ctrl-C).
This can be extremely useful on long-running or misbehaving applications.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Arguments}

\begin{Description}
\item[\Arg{command}] The command to profile.
\item[\Arg{command-arguments}] Arguments to the command to profile.
\end{Description}

Default values for an option's optional arguments are shown in \{\}.

\subsection{Options: Informational}

\begin{Description}
\item[\Opt{-l}, \Opt{-L}, \Opt{--list-events}]
List available events. (N.B.: some may not be profilable)

\item[\Opt{-V}, \Opt{--version}]
Print version information.

\item[\Opt{-h}, \Opt{--help}]
Print help.
\end{Description}

\subsection{Options: Profiling}

\begin{Description}

\item[\OptArg{-e}{event\Lbr@period\Rbr}, \OptArg{--event}{event\Lbr@period\Rbr}]
An event to profile and its corresponding sample period. \Arg{event} may be either a PAPI, native processor event, or WALLCLOCK (microseconds).  May pass multiple times as implementations permits.  \{\verb+WALLCLOCK@5000+\}
\begin{itemize}
  \item It is recommended to always specify the sampling period for each profiling event.
  \item The special event WALLCLOCK may be used to profile the `wall clock.'  It may be used only \emph{once} and cannot be used with another event. Its period is in microseconds.
  \item Hardware and drivers 1) limit the maximum number of events that can be monitored simultaneously, and 2) forbid certain combinations of events. 
  \item See the ``Sample sources'' under \textbf{NOTES} for additional details.
\end{itemize}

\item[\Opt{-t}, \Opt{--trace}]
Generate a call path trace (in addition to a call path profile).

\item[\OptArg{-f}{frac}, \OptArg{-fp}{frac}, \OptArg{--process-fraction}{frac}]
Measure only a fraction \Arg{frac} of the execution's processes.
For each process, enable measurement (of all threads) with probability \Arg{frac}; \Arg{frac} is a real number (0.10) or a fraction (1/10) between 0 and 1.
(To minimize perturbations, when a process is disabled, all threads in a process still receive sampling interrupts, but they are ignored.)

\item[\Opt{-r}, \Opt{--retain-recursion}]
Normally, hpcrun will collapse (simple) recursive call chains
to a single node. This option disables that behavior: all
elements of a recursive call chain are recorded
NOTE: If the user employs the RETCNT sample source, then this
      option is enabled: RETCNT implies *all* elements of
      call chains, including recursive elements, are recorded.



\item[\OptArg{-o}{outpath}, \OptArg{--output}{outpath}]
Directory for output data.
\{\verb+hpctoolkit-<command>-measurements[-<jobid>]+\}

\begin{itemize}
 Bug: Without a \Arg{jobid} or an output option, multiple profiles of the same <command> will be placed in the same output directory.
\end{itemize}

\end{Description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Environment Variables}
For most systems, \Prog{hpcrun} requires no special environment variable settings.
There are two situations, however, where \Prog{hpcrun}, to function correctly,
\emph{must} refer to environment variables. These environment variables, and
corresponding situations are:
\begin{Description}
  \item[\verb+HPCTOOLKIT+] To function correctly, \Prog{hpcrun} must know
       the location of the \Prog{HPCToolkit} top-level installation directory.
       The \Prog{hpcrun} script uses elements of the installation \File{lib} and
       \File{libexec} subdirectories. For most systems, the 
       installation procedure ensures that \Prog{hpcrun} can find the requisite
       components. Some parallel job launchers, however, will \emph{copy} the
       \Prog{hpcrun} script to a different location from the installed base. If your
       system uses this copying mechanism, you must set the \verb+HPCTOOLKIT+
       environment variable to the top-level installation directory.
\end{Description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Examples}

Assume we wish to profile the application \texttt{zoo}.
The following examples lists some useful events for different processor architectures.
In each case, the special option \texttt{--} is used to clearly demarcate the end of \Prog{hpcrun} options.

\begin{enumerate}
\item \verb+hpcrun -e WALLCLOCK@5000 zoo+

\item Opteron, (Rev B-F)
  \begin{enumerate}
    \item \verb+hpcrun -e DC_L2_REFILL@1300013 -e PAPI_L2_DCM@510011 -e PAPI_STL_ICY@5300013 -e PAPI_TOT_CYC@13000019 zoo+ (\verb+DC_L2_REFILL+ is an approximation of L1 D-cache misses).
    \item \verb+hpcrun -e PAPI_L2_DCM@510011 -e PAPI_TLB_DM@510013 -e PAPI_STL_ICY@5300013 -e PAPI_TOT_CYC@13000019 zoo+
  \end{enumerate}

\end{enumerate}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Notes}

\subsection{Sample sources}

\Prog{hpcrun} uses the PAPI library to provide access to hardware performance counter events.
If you have not configured HPCToolkit to use the PAPI library, you will be unable to measure hardware performance counter events.

The PAPI library supports a large collection of hardware counter events.
Some events have standard names across all platforms, e.g. \verb+PAPI_TOT_CYC+, the event that measures total cycles.
In addition to events whose names begin with the \verb+PAPI_+ prefix, platforms also provide access to a set of native events with names that are specific to the platform's processor.
A complete list of events supported by the PAPI library for your platform may be obtained by using the \texttt{--list-events} option.
Any event whose name begins with the \verb+PAPI_+ prefix that is listed as "Profilable" can be used as an event in a sampling source --- provided it does not conflict with another event.

The precise rules for selecting good events and periods are complex.
\begin{itemize}

\item \textbf{Choosing sampling events.}
We recommend using PAPI events in general.
However, some PAPI events are not profilable because of PAPI implementation details.
Also, PAPI's standard event list may not cover an architectural feature you are interested in.
In such cases, it is necessary to resort to native events.
In many cases, you will have to consult the architecture's manual to fully understand what the event means: there is no standard event list or naming scheme and events sometimes have unusual meanings.

\item \textbf{Number of sampling events.}
Currently, hpcrun does not multiplex hardware counters.
This means that the number of events that you may concurrently profile against is limited by your architecture's performance monitoring unit.
Note that some architectures hard-wire one or more counters to a specific event (such as cycles).

\item \textbf{Choosing sampling periods.}
The key requirement in choosing sampling periods is that you obtain enough samples to provide statistical significance.
We usually recommend a sampling rate between 100s-1000s of samples per second.
This usually only produces 1-5\% execution time overhead.

Choosing sampling rates depends on the architecture and sometimes the application.

Choosing periods for cycle and instruction-related events are usually easy.
Cycles directly relates to the clock speed.
Instruction-related events relates to the issue rate and width.

Choosing periods for other events seems harder because different applications uses resources differently.
For example, some applications are memory intensive and others are not.
However, if the goal is to identify rate-limiting factors of the architecture, then it is not necessary to consider the application.
For example, if the goal is to determine whether L2 D-cache latency is a limiting factor, then it is only necessary to work backward from the architecture's specifications to determine what number of L2 D-cache misses per second would be problematic.

\item \textbf{Architectural event conflicts.}
With some performance monitoring units, certain events may not be concurrently used with other events.

\item \textbf{Architectural interrupt delay.}
On most out-of-order pipelined architectures, a hardware counter interrupt is not precisely attributed to the instruction that induced the counter to overflow.
The gap is commonly 50-70 instructions.
This means that one should not assume that aggregation at the source line level is fully precise.
(E.g., if a L1 D-cache miss is attributed to a statement that has been compiled to register-only operations, assume the miss is attributed to a nearby load.)
However, aggregation at the procedure and loop level is reliable.

\item \textbf{System itimer (WALLCLOCK).}
On Linux systems, the kernel will not deliver itimer interrupts faster than the unit of a jiffy, which defaults to 4 milliseconds; see the itimer man page.
One can configure the kernel to use a value as small as 1 millisecond, but it is unlikely the kernel will actually deliver itimer signals at that rate when a period of 1000 microseconds is requested.

However, on Linux one can get quite close to the kernel Hz rate by setting the itimer interval to something less than the Hz rate.
For example, if the Hz rate is 1000 microseconds, one can use 500 microseconds (or just 1) and obtain about 999 interrupts per second.
\end{itemize}

\subsection{Platform-specific notes}
\subsubsection{Cray XE and XK}
When using dynamically linked binaries on Cray XE and XK systems, you
should add the \verb+HPCTOOLKIT+ environment variable to your launch
script.  Set \verb+HPCTOOLKIT+ to the top-level \verb+HPCToolkit+ install
prefix (the directory containing the \File{bin}, \File{lib} and
\File{libexec} subdirectories) and export it to the environment.  This is
only needed for running dynamically linked binaries.  For example:

\begin{verbatim}
#!/bin/sh
#PBS -l mppwidth=#nodes
#PBS -l walltime=00:30:00
#PBS -V

export HPCTOOLKIT=/path/to/hpctoolkit/install/directory

    ...Rest of Script...
\end{verbatim}

If \verb+HPCTOOLKIT+ is not set, you may see errors such as the
following in your job's error log.

\begin{verbatim}
/var/spool/alps/103526/hpcrun: Unable to find HPCTOOLKIT root directory.
Please set HPCTOOLKIT to the install prefix, either in this script,
or in your environment, and try again.
\end{verbatim}

The problem is that the Cray job launcher copies the \Prog{hpcrun}
script to a directory somewhere below \File{/var/spool/alps/} and runs
it from there.  By moving \Prog{hpcrun} to a different directory, this
breaks \Prog{hpcrun}'s method for finding its own install directory.
The solution is to add \verb+HPCTOOLKIT+ to your environment so that
\Prog{hpcrun} can find its install directory.

\subsection{Miscellaneous}

\begin{itemize}
  \item \Prog{hpcrun} uses preloaded shared libraries to initiate profiling.  For this reason, it cannot be used to profile setuid programs.
  \item \Prog{hpcrun} may not be able to profile programs that themselves use preloading.
\end{itemize}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{See Also}

\HTMLhref{hpctoolkit.html}{\Cmd{hpctoolkit}{1}}.\\
\HTMLhref{hpclink.html}{\Cmd{hpclink}{1}}.


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Version}

Version: \Version\ of \Date.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{License and Copyright}

\begin{description}
\item[Copyright] \copyright\ 2002-2015, Rice University.
\item[License] See \File{README.License}.
\end{description}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Authors}

\noindent
<NAME> \\
<NAME> \\
<NAME> \\
<NAME> \\
Rice HPCToolkit Research Group \\
Email: \Email{hpctoolkit-forum =at= rice.edu} \\
WWW: \URL{http://hpctoolkit.org}.

\LatexManEnd

\end{document}

%% Local Variables:
%% eval: (add-hook 'write-file-hooks 'time-stamp)
%% time-stamp-start: "setDate{ "
%% time-stamp-format: "%:y/%02m/%02d"
%% time-stamp-end: "}\n"
%% time-stamp-line-limit: 50
%% End:

