On using SCALEA for performance analysis of distributed and parallel programs.
In this paper we give an overview of SCALEA, which is a new performance analysis tool for OpenMP, MPI, HPF, and mixed parallel/distributed programs. SCALEA instruments, executes and measures programs and computes a variety of performance overheads based on a novel overhead classification. Source code and HW-profiling is combined in a single system which significantly extends the scope of possible overheads that can be measured and examined, ranging from HW-counters, such as the number of cache misses or floating point operations, to more complex performance metrics, such as control or loss of parallelism. Moreover, SCALEA uses a new representation of code regions, called the dynamic code region call graph, which enables detailed overhead analysis for arbitrary code regions. An instrumentation description file is used to relate performance information to code regions of the input program and to reduce instrumentation overhead. Several experiments with realistic codes that cover MPI, OpenMP, HPF, and mixed OpenMP/MPI codes demonstrate the usefulness of SCALEA.
Introduction
As hybrid architectures (e.g., SMP clusters) become the
mainstay of distributed and parallel processing in the
market, the computing community is busily developing
languages and software tools for such machines. Besides
OpenMP [27], MPI [13], and HPF [15], mixed programming
paradigms such as OpenMP/MPI are increasingly
being evaluated.
In this paper we introduce a new performance analysis
system, SCALEA, for distributed and parallel programs
that covers all of the above mentioned programming
paradigms. SCALEA is based on a novel classica-
tion of performance overheads for shared and distributed
memory parallel programs which includes data move-
ment, synchronization, control of parallelism, additional
computation, loss of parallelism, and unidentied over-
heads. SCALEA is among the rst performance analysis
tools that combines source code and HW proling
in a single system, signicantly extending the scope of
possible overheads that can be measured and examined.
These include the use of HW counters for cache analysis
to more complex performance metrics such as control or
loss of parallelism. Specic instrumentation and performance
analysis is conducted to determine each category
of overhead for individual code regions. Instrumentation
can be done fully automatically or user-controlled
through directives. Post-execution performance analysis
is done based on performance trace-les and a novel
representation for code regions named dynamic code region
call graph (DRG). The DRG re
ects the dynamic
relationship between code regions and its subregions and
enables a detailed overhead analysis for every code re-
gion. The DRG is not restricted to function calls but
also covers loops, I/O and communication statements,
etc. Moreover, it allows arbitrary code regions to be an-
alyzed. These code regions can vary from a single statement
to an entire program unit. This is in contrast to existing
approaches that frequently use a call graph which
considers only function calls.
A prototype of SCALEA has been implemented. We
will present several experiments with realistic programs
including a molecular dynamics application (OpenMP
version), a nancial modeling (HPF, and OpenMP/MPI
versions) and a material science code (MPI version) that
demonstrate the usefulness of SCALEA.
The rest of this paper is structured as follows: Section
describes an overview of SCALEA [24]. In Section
3 we present a novel classication of performance overheads
based on which SCALEA instruments a code and
analyses its performance. The dynamic code region call
graph is described in the next section. Experiments are
shown in Section 5. Related work is outlined in Section 6.
Conclusions and future work are discussed in Section 7.
SCALEA is a post-execution performance tool that in-
struments, measures, and analyses the performance behavior
of distributed memory, shared memory, and mixed
parallel programs.

Figure

1 shows the architecture of SCALEA which consists
of two main components: SCALEA instrumentation
system (SIS) and a post execution performance analysis
tool set. SIS is integrated with VFC [3] which is a
compiler that translates Fortran programs
MPI, OpenMP, HPF, and mixed programs) into For-
tran90/MPI or mixed OpenMP/MPI programs. The
input programs of SCALEA are processed by the compiler
front-end which generates an abstract syntax tree
(AST). SIS enables the user to select (by directives or
command-line options) code regions of interest. Based
on pre-selected code regions, SIS automatically inserts
probes in the code which will collect all relevant performance
information in a set of prole/trace les during
execution of the program on a target architecture. SIS
also generates an instrumentation description le (see
Section 2.2) that enables all gathered performance data
to be related back to the input program and to reduce
instrumentation overhead.
SIS [25] targets a performance measurement system
based on the TAU performance framework. TAU is an
integrated toolkit for performance instrumentation, mea-
surement, and analysis for parallel, multi-threaded pro-
grams. The TAU measurement library provides portable
proling and tracing capabilities, and supports access to
hardware counters. SIS automatically instruments parallel
programs under VFC by using the TAU instrumentation
library and builds on the abstract syntax tree of
VFC and on the TAU measurement system to create the
dynamic code region call graph (see Section 4). The main
functionality of SIS is given as follows:
Automatic instrumentation of pre-dened code regions
(loops, procedures, I/O statements, HPF INDEPENDENT
loops, OpenMP PARALLEL loops,
OpenMP SECTIONS, MPI send/receive, etc.) for
various performance overheads by using command-line
options.
Manual instrumentation through SIS directives
which are inserted in the program. These directives
also allow to dene user dened code regions for instrumentation
and to control the instrumentation
overhead and the size of performance data gathered
during execution of the program.
Post-execution
analysis with
sisprofile,
sisoverhead, pprof,
racy, vampir,
Dynamic code
region call graph
Raw
performance
data
instrumentation
description file
Pre-processing
profile/trace
files
Compilation
Linking
instrumented
code
Automatic
instrumentation
with SIS
Manual
instrumentation
Input program:
Fortran MPI program
OpenMP program
Hybrid program
executable
program
User's
instrumentation
control
Execution
SIS runtime system:
SISPROFILING, TAUSIS,
PAPI
Load imbalance
Timing result
Hardware counter result
Visualization
with racy,
vampir
Target
machine
subselect me
Select command and
visualize performance
data
Intermediate
database
Training sets
database
Performance
database
data repository
data object
physical resource
data processing
data flow
external input, control
control flow
Diagram legend

Figure

1: Architecture of SCALEA
Manual instrumentation to turn on/o proling for
a given code region.
A pre-processing phase of SCALEA lters and extracts
all relevant performance information from proles/trace
les which yields ltered performance data and the dynamic
code region call graph (DRG). The DRG re
ects
the dynamic relationship between code regions and its
subregions and is used for a precise overhead analysis
for every individual sub-region. This is in contrast to
existing approaches that are based on the conventional
call graph which considers only function calls but not
other code regions. Post-execution performance analysis
also employs a training set method to determine specic
information (e.g. time penalty for every cache miss over-
head, overhead of probes, time to access a lock, etc.)
for every target machine of interest. In the following we
describe the SCALEA instrumentation system and the
instrumentation description le. More details about SIS
and SCALEA's post-execution performance analysis can
be found in [24, 25].
2.1 SCALEA Instrumentation System
Based on user-provided command-line options or di-
rectives, SIS inserts instrumentation code in the program
which will collect all performance data of inter-
est. SIS supports the programmer to control prol-
ing/tracing and to generate performance data through
selective instrumentation of specic code region types
(loops, procedures, I/O statements, HPF INDEPENDENT
loops, OpenMP PARALLEL loops, OpenMP
SECTIONS, OpenMP CRITICAL, MPI barrier state-
ments, etc. SIS also enables instrumentation of arbitrary
code regions. Finally, instrumentation can be
turned on and o by a specic instrumentation directive.
In order to measure arbitrary code regions SIS provides
the following instrumentation:
code region
The directive !SIS$ CR BEGIN and !SIS$ CR END
must be, respectively, inserted by the programmer before
and after the region starts and nishes. Note that
there can be several entry and exit nodes for a code re-
gion. Appropriate directives must be inserted by the
IDF Entry Description
id code region identier
type code region types
le source le identier
unit program unit identier that encloses this region
line start line number where this region starts
column start column number where this starts
line end line number where this ends
column end column number where this ends
performance data performance data collected or
computed for this region
aux auxiliary information

Table

1: Contents of the instrumentation description le (IDF)
programmer in every entry and exit node of a given code
region. Alternatively, compiler analysis can be used to
automatically determine these entry and exit nodes.
Furthermore, SIS provides specic directives in order
to control tracing/proling. The directives MEASURE
ENABLE and MEASURE DISABLE allow the
programmer to turn on and o tracing/proling of a program

code region
For instance, the following example instruments a portion
of an OpenMP pricing code version (see Section 5.2),
where for the sake of demonstration, the call to function
RANDOM PATH is not measured by using the facilities
to control proling/tracing as mentioned above.
END DO
Note that SIS directives are inserted by the programmer
based on which SCALEA automatically instruments
the code.
2.2 Instrumentation Description File
A crucial aspect of performance analysis is to relate performance
information back to the original input program.
During instrumented of a program, SIS generates an
instrumentation description le (IDF) which correlates
proling, trace and overhead information with the corresponding
code regions. The IDF maintains for every
instrumented code region a variety of information (see

Table

1).
A code region type describes the type of the code re-
gion, for instance, entire program, outermost loop, read
statement, OpenMP SECTION, OpenMP parallel loop,
MPI barrier, etc. The program unit corresponds to a
subroutine or function which encloses the code region.
The IDF entry for performance data is actually a link to
a separate repository that stores this information. Note
that the information stored in the IDF can actually be
made a runtime data structure to compute performance
overheads or properties during execution of the program.
IDF also helps to keep instrumentation code minimal, as
for every probe we insert only a single identier that allows
to relate the associated probe timer or counter to
the corresponding code region.
3 Classication of Temporal
According to Amdahl's law [1], theoretically the best sequential
algorithm takes time T s to nish the program,
and T p is the time required to execute the parallel version
with p processors. The temporal overhead of a parallel
program is dened by T re
ects the
dierence between achieved and optimal parallelization.
T can be divided into T i and T u such that T
where T i is the overhead that can be identied and T u
is the overhead fraction which could not be analyzed in
detail. In theory T can never be negative, which implies
that the speedup T s =T p can never exceed p [16]. How-
ever, in practice it occurs that temporal overhead can
become negative due to super linear speedup of applica-
tions. This eect is commonly caused by an increased
available cache size. In Figure 2 we give a classication
of temporal overheads based on which the performance
analysis of SCALEA is conducted:
Temporal
overheads
Data movement
Local memory
access
Remote memory
access
Level 2 to level 1
Level 3 to level 2
Level n to level
Send/Receive
Put/Get
Synchronisation
Barriers
Locks
Conditional
variable
Control of
parallelism
Scheduling
Inspector/
Executor
Fork/join
Additional
computation
Algorithm change
Compiler change
Front-end
normalization
Loss of parallelism
Unparallelised
code
Replicated code
Partial parallelised
code
Unidentified

Figure

2: Temporal overheads classication
Data movement corresponds to any data transfer
within a single address space of a process (local
memory access) or between processes (remote memory
access).
Synchronization (e.g. barriers and locks) is used
to coordinate processes and threads when accessing
data, maintaining consistent computations and
data, etc.
Control of parallelism (e.g. fork/join operations and
loop scheduling) is used to control and manage the
parallelism of a program and can be caused by run-time
library, user, and compiler operations.
Additional computation re
ects any change of the
original sequential program including algorithmic or
compiler changes to increase parallelism (e.g. by
eliminating data dependences) or data locality (e.g.
through changing data access patterns).
Loss of parallelism is due to imperfect parallelization
of a program which can be further classied as
follows: unparallelized code (executed by only one
processor), replicated code (executed by all proces-
sors), and partially parallelized code (executed by
more than one but not all processors).
Unidentied overhead corresponds to the overhead
that is not covered by the above categories.
Note that the above mentioned classication has been
stimulated by [6] but diers in several respects. In [6],
synchronization is part of information movement, load
imbalance is a separate overhead, local and remote memory
are merged in a single overhead class, loss
of parallelism is split into two classes, and unidentied
overhead is not considered at all. Load imbalance in our
opinion is not an overhead but represents a performance
property that is caused by one or more overheads.
4 Dynamic Code Region
Call Graph
Every program consists of a set of code regions which can
range from a single statement to the entire program unit.
A code region can be, respectively, entered and exited by
multiple entry and exit control
ow points (see Figure 3).
In most cases, however, code regions are single-entry-
single-exit code regions.
In order to measure the execution behavior of a code
region, the instrumentation system has to detect all entry
and exit nodes of a code region and insert probes
at these nodes. Basically, this task can be done with the
support of a compiler or guided through manual insertion
of directives. Figure 3 shows an example of a code region
with its entry and exit nodes. To select an arbitrary
code region, the user, respectively, marks two statements
as the entry and exit statements { which are at the same
time entry and exit nodes { of the code region (e.g., by
using SIS directives [25]). Through a compiler analysis,
SIS then automatically tries to determine all other entry
and exit nodes of the code region. Each node represents
a statement in the program. Figure 3 shows an example
code region with multiple entry and exit nodes. The
instrumentation tries to detect all these nodes and automatically
inserts probes before and after all entry and
exit nodes, respectively.
Code regions can be overlapping. SCALEA currently
does not support instrumentation of overlapped code re-
gions. The current implementation of SCALEA supports
mainly instrumentation of single-entry multiple-exit code
regions. We are about to enhance SIS to support also
multiple-entry multiple-exit code regions.
4.1 Dynamic Code Region Call Graph
SCALEA has a set of predened code regions which are
classied into common (e.g. program, procedure, loop,
function call, statement) and programming paradigm
specic code regions (MPI calls, HPF INDEPENDENT
loops, OpenMP parallel regions, loops, and sections,
etc. Moreover, SIS provides directives to dene arbitrary
code regions (see Section 2.1) in the input program.
Based on code regions we can dene a new data structure
called dynamic code region call graph (DRG):
A dynamic code region call graph (DRG) of a
program Q is dened by a directed
ow graph
with a set of nodes R and a set of
edges E. A node r 2 R represents a code region
which is executed at least once during runtime
of Q. An edge (r
indicates that a code
region r 2
is called inside of r 1
during execution
of Q and r 2
is a dynamic sub-region of r 1
. The
rst code region executed during execution of
Q is dened by s.
The DRG is used as a key data structure to conduct a
detailed performance overhead analysis under SCALEA.
Notice that the timing overhead of a code region r with
explicitly instrumented sub-regions r 1 ; :::; r n is given by
where T (r i ) is the timing overhead for an explicitly
instrumented code region r i (1  i  n). T (Start r ) and
correspond to the overhead at the beginning
(e.g. fork threads, redistribute data, etc.) and at the end
(join threads, barrier synchronization, process reduction
operation, etc.) of r. T (Remain) corresponds to the
code regions that have not been explicitly instrumented.
Entry point
Statement that begins
the selected code
region
Statement that ends
the selected code
region
Exit point
Statement
Control flow
Control flow
Exit point
Entry point

Figure

3: A code region with several entry and exit points
However, we can easily compute T (Remain) as region r
is instrumented as well.

Figure

4 shows an excerpt of an OpenMP code together
with its associated DRG.
Call graph techniques have been widely used in performance
analysis. Tools such as Vampir [20], gprof [11, 10],
CXperf [14] support a call graph which shows how much
time was spent in each function and its children. In [7] a
call graph is used to improve the search strategy for automated
performance diagnosis. However, nodes of the call
graph in these tools represent function calls [10, 14]. In
contrast our DRG denes a node as an arbitrary code region
(e.g. function, function call, loop, statement, etc.
INTEGER::X, A,N
PRINT *, "Input N="
READ *,N
call
A =0
call
DO I=1,N
A =A+1
END DO
call
call SISF_START(5)
call SISF_STOP(5)
call
END PROGRAM
R 4
R 5
R 4
R 5

Figure

4: OpenMP code excerpt with DRG
4.2 Generating and Building the
Dynamic Code Region Call Graph
Calling code region r 2
inside a code region r 1
during execution
of a program establishes a parent-children relationship
between r 1 and r 2 . The instrumentation library
will capture these relationships and maintain them during
the execution of the program. If code region r 2 is
called inside r 1 then a data entry representing the relationship
between r 1 and r 2 is generated and stored in
appropriate proles/trace les. If a code region r is encountered
that isn't child of any other code region (e.g.,
the code region that is executed rst), an abstract code
region is assigned as its parent. Every code region has a
unique identier which is included in the probe inserted
by SIS and stored in the instrumentation description le.
The DRG data structure maintains the information of
code regions that are instrumented and executed. Every
thread of each process will build and maintain its own
sub-DRG when executing.
In the pre-processing phase (cf. Figure 1) the DRG
of the application will be built based on the individual
sub-DRGs of all threads. A sub-DRG of each thread is
computed by processing the proles/trace les that contain
the performance data of this thread. The algorithm
for generating DRGs is described in detail in [24].

Figure

5: Execution time(s) of the MD application
5 Experiments
We have implemented a prototype of SCALEA which is
controlled by command-line options and user directives.
Code regions including arbitrary code regions can be selected
through specic SIS directives that are inserted
in the input program. Temporal performance overheads
according to the classication shown in Figure 2 can be
selected through command-line options. Our visualization
capabilities are currently restricted to textual out-
put. We plan to build a graphical user interface by the
end of 2001. The graphical output except for tables of the
following experiments have all been generated manually.
More information of how to use SIS and post-execution
analysis can be found in [25, 24].
Overhead 2CPUs 3CPUs 4CPUs
Loss of parallelism 0.025 0.059 0.066
Control of parallelism 1.013 0.676 0.517
Synchronization 1.572 1.27 0.942
Total execution time 146.754 98.438 74.079

Table

2: Overheads (sec) of the MD application. T i ,
are identied, unidentied and total overhead,
respectively.
In this section, we present several experiments to
demonstrate the usefulness of SCALEA. Our experiments
have been conducted on Gescher [23] which is an
SMP cluster with 6 SMP nodes (connected by FastEth-
ernet) each of which comprises 4 Intel Pentium III
Xeon 700 MHz CPUs with 1MB full-speed L2 cache,
2Gbyte ECC RAM, Intel Pro/100+Fast Ethernet, Ul-
tra160 36GB hard disk is run with Linux 2.2.18-SMP
patched with perfctr for hardware counters performance.
We use MPICH [12] and pgf90 compiler version 3.3. from
the Portland Group Inc.
5.1 Molecular Dynamics (MD) Applica-
tion
The MD program implements a simple molecular dynamics
simulation in continuous real space. This program obtained
from [27] has been implemented as an OpenMP
program which was written by Bill Magro of Kuck and
Associates, Inc. (KAI).
The performance of the MD application has been measured
on a single SMP node of Gescher. Figure 5 and

Table

2 show the execution time behavior and measured
overheads, respectively. The results demonstrate a good
speedup behavior (nearly linear). As we can see from Table
2, the total overhead is very small and large portions
of the temporal overhead can be identied.
The time of the sequential code regions (unparal-
lelized) doesn't change as it is always executed by only

Figure

The L2 cache misses/cache accesses ratio of
OMP DO regions in the MD application
one processor. Loss of parallelism for an unparallelized
code region r in a program q is dened as t r
processors are used to execute q and t r is the sequential
execution time of r. By increasing p it can be easily
shown that the loss of parallelism increases as well which
is also conrmed by the measurements shown in Table
2.
Control of parallelism { mostly caused by loop
scheduling { actually decreases for increasing number of
processors. A possible explanation for this eect can be
that for larger number of processors the master thread
processes less loop scheduling phases than for a smaller
number of processors. The load balancing improves by
increasing the number of processors/threads in one SMP
node which at the same time decreases synchronization
time.
We then examine the cache miss ratio { dened by the
number of L2 cache misses divided by the number of L2
cache accesses { of the two most important OMP DO
code regions namely OMP DO COMPUTE and OMP
DO UPDATE as shown in Figure 6. This ratio is nearly
when using only a single processor which implies very
good cache behavior for the sequential execution of this
code. All data seem to t in the L2 cache for this
case. However, in a parallel version, the cache miss ratio
increases substantially as all threads process data of
global arrays that are kept in private L2 caches. The
cache coherency protocol causes many cache lines to be

Figure

7: Execution times of the HPF+ and OpenMP/MPI version for the backward pricing application
exchanged between these private caches which induces
cache misses. It is unclear, however, why the master
thread has a considerably higher cache miss ratio then
all other threads. Overall, the cache behavior has very
little impact on the speedup of this code.
5.2 Backward Pricing Application
The backward pricing code [8] implements the backward
induction algorithm to compute the price of an interest
rate dependent nancial product, such as a variable
coupon bond. Two parallel code versions have been
created. First, an HPF+ version that exploits only
data parallelism and is compiled to an MPI program,
and second, a mixed version that combines HPF+ with
OpenMP. For the latter version, VFC generates an
OpenMP/MPI program. HPF+ directives are used to
distribute data onto a set of SMP nodes. Within each
node an OpenMP program is executed. Communication
among SMP nodes is realized by MPI calls.
The execution times for both versions are shown
in

Figure

7. The term \all" in the legend denotes
the entire program, whereas \loop" refers to the main
computational loops (HPF INDEPENDENT loop and
an OpenMP parallel loop for version 1 and 2, re-
spectively). The HPF+ version performs worse than
the OpenMP/MPI version which shows almost linear
speedup for up to 2 nodes (overall 8 processors). Tables
3 and 5 display the overheads for the HPF+ and
mixed OpenMP/MPI version, respectively. In both cases
the largest overhead is caused by the control of parallelism
overhead which rises signicantly for the HPF+
version with increasing number of nodes. This eect is
less severe for the OpenMP/MPI version. In order to
nd the cause for the high control of parallelism overhead
we use SCALEA to determine the individual components
of this overhead (see Tables 4 and 6). Two routines
(Update HALO and MPI Init) are mainly responsible
for the high control of parallelism overhead of the
version. Update HALO updates the overlap areas
of distributed arrays which causes communication if
one process requires data that is owned by another process
in a dierent node. MPI Init initializes the MPI
runtime system which also involves communication. The
version implies a much higher overhead for these
two routines compared to the OpenMP/MPI reason because
it employs a separate process on every CPU of each
SMP node. Whereas the OpenMP/MPI version uses one
process per node.
5.3 LAPW0
LAPW0 [4] is a material science program that calculates
the eective potential of the Kohn-Sham eigenvalue
problem. LAPW0 has been implemented as a Fortran
MPI code which can be run across several SMP
nodes. The pgf90 compiler takes care of exchanging data
between processors both within and across SMP nodes.
We used SCALEA to localize the most important code
regions of LAPW0 which can be further subdivided into
sequentialized code regions: FFT REAN0,
FFT REAN3, FFT REAN4
parallelized code regions: Interstitial Potential,
Loop 50, ENERGY, OUTPUT
The execution time behavior and speedups (based on
the sequential execution time of each code region) for
each of these code regions are shown in Figures 8 and 9,
respectively. LAPW0 has been examined for a problem
size of 36 atoms which are distributed onto the processors
of a set of SMP nodes. Clearly when using 8, 16, and 24
processors we can't reach optimal load balance, whereas
processors display a much better
load imbalance. This eect is conrmed by SCALEA
(see

Figure

for the the most computationally intensive
routines of LAPW0 (Interstitial Potential and Loop 50 ).
scales poorly due to load imbalances
and large overheads due to loss of parallelism, data move-
ment, and synchronization; see Table 7. LAPW0 uses
many BLAS and SCALAPACK library calls that are currently
not instrumented by SCALEA which is the reason
for the large fraction of unidentied overhead (see Table
7). The main sources of the control of parallelism overhead
is caused by MPI Init (see Figure 10). SCALEA
also discovered the main subroutines that cause the loss
of parallelism overhead: FFT REAN0, FFT REAN3,
and FFTP REAN4 all of which are sequentialized.
6 Related Work
Paraver [26] is a performance analysis tool for
OpenMP/MPI tools which dynamically instruments binary
codes and determines various performance param-
eters. This tool does not cover the same range of performance
overheads supported by SCALEA. Moreover,
tools that use dynamic interception mechanisms commonly
have problems to relate performance data back to
the input program.
Ovaltine [2] measures and analyses a variety of performance
overheads for Fortran77 OpenMP programs.
Paradyn [18] is an automatic performance analysis tool
that uses dynamic instrumentation and searches for performance
bottlenecks based on a specication language.
A function call graph is employed to improve performance
tuning [7].
Recent work on an OpenMP performance interface [19]
based on directive rewriting has similarities to the SIS instrumentation
approach in SCALEA and to SCALA [9]
{ the predecessor system of SCALEA. Implementation
of the interface (e.g., in a performance measurement library
such as TAU) allows proling and tracing to be per-
formed. Conceivably, such an interface could be used to
generate performance data that the rest of the SCALEA
system could analyze.
The TAU [22, 17] performance framework is an integrated
toolkit for performance instrumentation, mea-
surement, and analysis for parallel, multithreaded pro-
grams. SCALEA uses TAU instrumentation library as
one of its tracing libraries.
PAPI [5] species a standard API for accessing hardware
performance counters available on most modern mi-
croprocessors. SCALEA uses the PAPI library for measuring
hardware counters.
gprof [11, 10] is a compiler-based proling framework
that mostly analyses the execution behavior and counts
of functions and function calls.
VAMPIR [20] is a performance analysis tool that processes
trace les generated by VAMPIRtrace [21]. It supports
various performance displays including time-lines
and statics that are visualized together with call graphs
and the source code.
7 Conclusions and Future Work
In this paper, we described SCALEA which is a performance
analysis system for distributed and parallel pro-
grams. SCALEA currently supports performance analysis
for OpenMP, MPI, HPF and mixed parallel programs
(e.g. OpenMP/MPI).
SCALEA is based on a novel classication of performance
overheads for shared and distributed memory
parallel programs. SCALEA is among the rst performance
analysis tools that combines source code and HW-
proling in a single system which signicantly extends
the scope of possible overheads that can be measured and
examined. Specic instrumentation and performance
analysis is conducted to determine each category of overhead
for individual code regions. Instrumentation can
be done fully automatically or user-controlled through
directives. Post-execution performance analysis is done
based on performance trace-les and a novel representation
for code regions named dynamic code region call
graph (DRG). The DRG re
ects the dynamic relationship
between code regions and its subregions and enables
a detailed overhead analysis for every code region. The
DRG is not restricted to function calls but also covers
loops, I/O and communication statements, etc. More-
over, it allows to analyze arbitrary code regions that can
vary from a single statement to an entire program unit.
Processors Sequential 1N, 1P 1N, 4P 2N, 4P 3N, 4P 4N,4P 5N,4P 6N,4P
Data movement
Control of parallelism 0 0.244258 6.59928 17.2419 28.9781 41.4966 56.4554 70.7302
Tu 3.139742 1.726465 1.835957 2.047059 2.3549 2.99739 2.5173
To 3.384 8.33775 19.10787 31.0459 43.8749 59.48315 73.2829
Total execution time 316.417 319.801 87.442 58.66 57.414 63.651 75.304 86.467

Table

3: Overheads of the HPF+ version for the backward pricing application. T i , T u , T are identied, unidentied
and total overhead, respectively. 1N, 4P means 1 SMP node with 4 processors.
Processors 1N, 1P 1N, 4P 2N, 4P 3N, 4P 4N,4P 5N,4P 6N,4P
Inspector
Work distribution 0.000258 0.000285
Update HALO 0.149 3.114 9.110 16.170 24.060 33.868 43.830
MPI Init 0.005 3.462 8.113 12.784 17.420 22.568 26.860
Other

Table

4: Control of parallelism overheads for the HPF+ version for the backward pricing application.
This is in contrast to existing approaches that frequently
use a call graph which considers only function calls.
Based on a prototype implementation of SCALEA we
presented several experiments for realistic codes implemented
in MPI, HPF, and mixed OpenMP/MPI. These
experiments demonstrated the usefulness of SCALEA to
nd performance problems and their causes.
We are currently integrating SCALEA with a database
to store all derived performance data. Moreover, we plan
to enhance SCALEA with a performance specication
language in order to support automatic performance bottleneck
analysis.
The SISPROFILING measurement library for DRG
and overhead proling is an extension of TAU's prol-
ing capabilities. Our hope is to integrate these features
into the future releases of the TAU performance system
so that these features can be oered more portably and
other instrumentation tools can have access to the API.



--R

Validity of the single processor approach to achieving large scale computing capabili- ties
Automatic overheads pro
VFC: The Vienna Fortran Compiler.

A scalable cross-platform infrastructure for application performance tuning using hardware counters
A hierarchical classi
A callgraph-based search strategy for automated performance diagnosis
Pricing Constant Maturity Floaters with Embeeded Options Using Monte Carlo Simulation.

GNU gprof.
A call graph execution pro

The MPI standard for message pass- ing
CXperf User's Guide
High Performance Fortran Forum.
Introduction to Parallel Comput- ing:design and analysis of parallel algorithms
Performance technology for complex parallel and distributed sys- tems

Towards a performance tool interface for openmp: An approach based on directive rewriting.
VAMPIR: Visualization and analysis of MPI resources.
Vampirtrace 2.0 Installation and User's Guide
Portable pro
Gescher system.

Scalea version 1.0: User's guide.


--TR
Introduction to parallel computing
A high-performance, portable implementation of the MPI message passing interface standard
Portable profiling and tracing for parallel, scientific applications using C++
Execution-driven performance analysis for distributed and parallel systems
Performance technology for complex parallel and distributed systems
A scalable cross-platform infrastructure for application performance tuning using hardware counters
The Paradyn Parallel Performance Measurement Tool
A Callgraph-Based Search Strategy for Automated Performance Diagnosis (Distinguished Paper)
A hierarchical classification of overheads in parallel programs
The MPI Standard for Message Passing
Gprof

--CTR
Thomas Fahringer , Clvis Seragiotto Jnior, Modeling and detecting performance problems for distributed and parallel programs with JavaPSL, Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM), p.35-35, November 10-16, 2001, Denver, Colorado
Ming Wu , Xian-He Sun, Grid harvest service: a performance system of grid computing, Journal of Parallel and Distributed Computing, v.66 n.10, p.1322-1337, October 2006
distributed and parallel systems;performance analysis;performance overhead classification