Cactus Tools for Grid Applications.
Cactus is an open source problem solving environment designed for scientists and engineers. Its modular structure facilitates parallel computation across different architectures and collaborative code development between different groups. The Cactus Code originated in the academic research community, where it has been developed and used over many years by a large international collaboration of physicists and computational scientists. We discuss here how the intensive computing requirements of physics applications now using the Cactus Code encourage the use of distributed and metacomputing, and detail how its design makes it an ideal application test-bed for Grid computing. We describe the development of tools, and the experiments which have already been performed in a Grid environment with Cactus, including distributed simulations, remote monitoring and steering, and data handling and visualization. Finally, we discuss how Grid portals, such as those already developed for Cactus, will open the door to global computing resources for scientific users.
Introduction
Cactus [1], [2] is an open source problem solving environment designed to
provide a unied modular and parallel computational framework for physicists
and engineers. The Cactus Code was originally developed to provide a framework
for the numerical solution of Einstein's Equations [3], one of the most complex
sets of partial dierential equations in physics. These equations govern such cataclysmic
events as the collisions of black holes or the supernova explosions of stars.
The solution of these equations with computers continues to provide challenges
in the elds of mathematics, physics and computer science. The modular design
of Cactus enables people and institutes from all these disciplines to coordinate
their research, using Cactus as the collaborating and unifying tool.
The name Cactus comes from the design of a central core (or
which
connects to application modules (or thorns) through an extensible interface.
Thorns can implement custom developed scientic or engineering applications,
such as the Einstein solvers, or other applications such as computational
uid
dynamics. Other thorns from a standard computational toolkit provide a range
of capabilities, such as parallel I/O, data distribution, or checkpointing.
Cactus runs on many architectures. Applications, developed on standard
workstations or laptops, can be seamlessly run on clusters or supercomputers.
Parallelism and portability are achieved by hiding the driver layer and features
such as the I/O system and calling interface under a simple abstraction API. The
Cactus API supports C/C++ and F77/F90 programming languages for the thorns.
Thus thorn programmers can work in the language they nd most convenient,
and are not required to master the latest and greatest computing paradigms.
This makes it easier for scientists to turn existing codes into thorns which can
then make use of the complete Cactus infrastructure, and in turn be used by
other thorns within Cactus.
Cactus provides easy access to many cutting edge software technologies being
developed in the academic research community, such as the Globus Meta-computing
Toolkit, HDF5 parallel le I/O, the PETSc scientic computing library,
adaptive mesh renement, web interfaces, and advanced visualization tools.
2. The Need for the Grid
Of many applications using the Cactus framework, an important one which
continues to drive its development is the solution of the Einstein Equations. The
large and varied computational requirements of solving these equations for scenarios
such as black hole or neutron star collisions, make them a good example
for demonstrating the need for Grid computing, and an ideal testbed for developing
new techniques. In developing the Cactus infrastructure to make full use
of the Grid for such problems these advances are then immediately available for
all applications.
Implementing the full Einstein Equations in a nite dierence code amounts
to a memory requirement of around one hundred 3D arrays, and a CPU requirement
of thousands of
oating point operations per grid point and timestep.
Considering that a su-ciently accurate solution of a full 3D black hole problem
will require at least 1000 grid points in each spatial dimension, this implies
computers. Further, to analyze the large data sets created
during a simulation requires advanced techniques in le management and visualization

To date, the resources of the individual supercomputers available have limited
simulations to around 200-300 grid points in each spatial dimension. Even
simulations generate huge amounts of data, and negotiating the curiosities
of dierent supercomputers, such as batch queues and le systems, is not
something that physicists relish.
The Grid provides a way to access the resources needed for these simulations.
It provides a uniform access layer to supercomputers and computing resources,
making these resources far more useful to scientists who want to use them for
simulations. Given the appropriate permissions, networks, allocations and Grid
enabling software, a scientist could in principle run a simulation on a set of
supercomputers, all connected by the Grid, and thus be able to run far larger
problems than would be possible on a routine basis without the Grid. With
proper software tools, the Grid provides the necessary infrastructure to connect
the machines, and to deal with the resulting large data sets, and, ultimately, the
resources to analyze such volumes of data.
The dream for physicists is that Grid computing will provide a scientic
programming environment similar to that shown in Figure 1, allowing working
scenarios such as:
A physicist, sitting in a cafe in Berlin has an idea for a colliding black hole run,
maybe to try a new initial data set, or to test a new evolution method. She uses
a web portal on her PDA to select the needed Cactus thorns, and to estimate the
required computer resources. Her Grid software takes over, selecting the most
appropriate machine or set of machines to use from those available to her. This
Grid software automatically creates or transfers executables and parameter les
and starts the run on the remote resources. After several coees, she connects
to the running Cactus simulation, using one of the remote access thorns, and
sees that things are going better than expected. She rings up colleagues in the
who watch the isosurfaces being streamed out from Cactus. They want
to save some 3D data sets to analyze later, so they connect to the Cactus run
using their web browser, and turn on output for the grid functions they are
interested in.
As futuristic as such a scenario sounds, all the pieces already exist in a
prototype form, and are being further developed and integrated, as described
below and in [1], [4], [5] and [6].

Figure

1. The dream of Grid computing: Grid infrastructure provides a transparent and
exible working environment providing access to global computing resources.
3. Grid Computing with Cactus
Cactus was designed with the Grid and Grid applications in mind. It provides
a layer on top of the Grid, giving a programming interface which allows
the user to be completely ignorant of the nature of the machine or machines
that the simulation runs on. The code provides access to Grid resources such as
distributed I/O and parallelization across any number of supercomputers with
precisely the same interface as it does to the resources of a single machine [7].
Cactus thus provides a convenient laboratory for computer scientists to develop
metacomputing techniques, which can then be tested with real physics
applications and also by real users without requiring changes to the physics application
code. When a new technique is perfected, it can immediately be made
available to the whole community of Cactus Users.
Grid Computing developments and experiments have been performed using
Cactus for several years, some of which are described in the sections below.
Capabilities are being further developed in connection with Cactus through several
projects. A DFN-Verein project [4] at the AEI in Germany is funded to
exploit high speed networks for colliding black hole simulations, and is concentrating
on remote visualization [8], remote steering and distributed I/O [9]. A
project funded by the so-called KDI program of the American National Science
Foundation (NSF) joins ve institutes to develop an Astrophysics Simulation
Collaboratory [5] which will provide an environment for physicists to utilize Grid
computing for problems such as the collisions of neutron stars. The GrADs
project [10], also funded by the NSF in the USA, is using Cactus as one of its
applications for developing a Grid based computing environment. The European
Grid Forum [11] has chosen Cactus as one of its applications running on the
European Grid-TestBed. These technologies are being brought into the scientic
and engineering communities as they are developed.
3.1. Distributed Simulations on the Grid
We are actively working to develop techniques which allow researchers to
harness computational resources wherever they may be on the Grid. This could
include a distributed set of machines connected by high speed networks, allowing
larger or faster simulations than would be possible on a single machine. At Super-computing
a neutron star collision was run with Cactus, using the Globus [6]
metacomputing toolkit to split the domain across two T3Es on dierent sides of
the Atlantic, one in Munich, Germany and one in San Diego, California. In this
simulation the neutron stars collided somewhere in cyberspace, over the Atlantic
Ocean. The simulations were launched, visualized, and steered from the show
oor in Orlando. The scaling across the two machines used for this simulation
was roughly 50%, which we believe is excellent considering the large amount of
communication required between the machines to solve these equations and the
latencies in the transatlantic link.
The latency and bandwidth are characteristic features, which determine the
speed of a network. Cactus is aware of these features and can be ne-tuned in order
to optimize communication. For example, if a network has a high latency but
also a high bandwidth, many small messages can be coalesced into fewer bigger
ones. When running in a metacomputing environment, one has to deal with dier-
ent types of networks (shared-memory, distributed-memory, high-speed-network,
LAN, WAN/internet) with dierent latency/bandwidth characteristics. Here, we
also have the possibility to distinguish between these dierent types of network-
connections in one single distributed run and tune Cactus communication patterns
adequately. Partly this is already achieved by using MPICH-G2, the next-generation
MPI-implementation, which can distinguish between processors located
on one host (with native MPI installed) and processors separated by a LAN or
WAN. According to the location, MPICH-G2 can choose dierent protocols (TCP
or vendor's MPI) for communication in one single distributed parallel run. Cactus
can be used with this new technology without problem, which was demonstrated
in many metacomputing experiments last year, including Supercomputing 2000
in Dallas.
A further aspect is load-balancing. Since dierent architectures provide
dierent types of processors at dierent speeds, Cactus provides the ability to
decompose the whole computational problem into sub-problems of dierent size,
which t the local processor power.
For running in a metacomputing environment at a real production-level, a
User Portal has been built (described in Section 5), making it possible to congure
and start Cactus runs from one machine via a special web-based GUI. This greatly
simplies the situation for the scientist, since she does not have to deal with every
detail of the local supercomputer, such as batch-systems, username/password.
The portal provides automatic staging and compilation of the code on the local
supercomputer, the distributed machines appearing as a single virtual machine.
3.2. Checkpointing Distributed Simulations on the Grid
Grid computing events in the past have been often understood as one-time,
well prepared attempts to harness several machines at the same time. In a more
realistic setting, compute resources of considerable size can not be expected to
be available at a given time, instead their availability is a dynamic process. A
true Grid application has to be capable of dealing with such dynamic allocations
of resources.
The Cactus framework addresses this challenge by providing a sophisticated
cross-platform checkpointing mechanism. In general, checkpointing technology
allows the user to freeze the state of an application by writing a checkpoint le to
disk, from which the application can be restored and continued at a later time.
single MPP supercomputer
cluster or compute farm
tightly coupled heterogeneous supercomputers

Figure

2. Migration scenario for a distributed simulation: the simulation starts on three
tightly coupled supercomputers from where it is checkpointed and migrated to a single
machine. The computation migrates again to nish on a cluster.
In Cactus the checkpoint is not just the memory image of the application
written to disk, as found in several other checkpointing systems, but the total set
of user dened objects (variables, scalars, etc. While memory images tend to be
quite huge and are only compatible within the same class of operating systems and
architectures, this approach allows for smaller, architecture-independent check-
points. The cross-platform checkpoints of Cactus can be transferred between
arbitrary architectures, operating systems and numbers of processors for restarting
and continuing simulations.
The checkpointing mechanism is completely transparent to the user, who can
request a checkpoints to be written at regular timestep intervals, at the end of the
requested compute time allocation, or using a steering interface immediately at
the current timestep. All of the internal technicalities of parallel I/O are hidden
from the user. The user can control checkpoint behavior (as frequency or parallel
I/O) by means of steerable parameters.
The checkpoint mechanism allows for the output of a single, global check-point
le as well as multiple checkpoint les for each of the distributed machines.
The mechanism makes use of parallel I/O where possible. For restarting, the
multiple checkpoint les can be recombined into a single le which can be used
to restart on an arbitrary set of machines. The parallel restart operation from
multiple les is currently restricted to the same topology of machines. Future developments
will add intelligent components to immediately restart from multiple
checkpoint les across arbitrary machine topologies.
With respect to distributed simulations, a Cactus user has the ability to
perform a distributed run and checkpoint this simulation even though is it being
run on a heterogeneous machine set. A checkpoint le can then be transferred
to a new conguration of machines to continue the simulation. The new pool
of machines can dier from the previous one in type and number of machines
involved as well as the number of processors. This
exible chain of distributed
simulations is illustrated in Figure 2: an initial simulation run across three tightly
coupled supercomputers is checkpointed. The checkpoint le is transferred to a
single MPP machine and restarted. After a second checkpointing event the third
stage of the simulation is continued on a cluster system.
4. Grid-enabled Communication and I/O Techniques
The parallel driver layer in Cactus, which manages the allocation and domain
decomposition of grid variables as well as their synchronization between
processor boundaries, is provided by a thorn. This means that dierent thorns
can be used to implement dierent parallel paradigms, such as PVM, Pthreads,
OpenMP, CORBA, etc. Cactus can be compiled with as many driver thorns as required
(subject to availability), with the one actually used chosen by the user at
run time through the parameter le.
The current standard driver thorn is called PUGH, which uses MPI to provide
parallelism. In order to perform distributed Cactus simulations on the Grid,
this PUGH thorn is simply linked against the Grid-enabled MPICH-G [12] implementation
of MPI which is available with the Globus toolkit. Thus, preparing
a Grid-enabled version of Cactus is a compilation choice, and it is completely
transparent for application thorn programmers to add their own code to a Grid-enabled
9Cactus. Using the Globus job submission tools, Cactus users can start
their Cactus runs in a Grid environment just as easily as they do on a single
machine.
The Cactus I/O subsystem is implemented in a similar, generic manner: the
esh provides a runtime interface for arbitrary I/O thorns to register their own,
specic I/O methods. These methods can then in turn be invoked by the
esh
or any application thorn to read external data into Cactus variables or dump
their contents to a storage medium for postprocessing analysis and visualization
purposes.
The I/O thorns currently available in the computational toolkit provide
methods to write simulation data in dierent formats (1D traceline plots, 2D
slices and JPEG images, full N-dimensional arrays, arbitrary hyperslabs of N-dimensional
arrays, reduction scalars (e.g. minimum/maximum values), isosurface
geometry data, particle trajectories, runtime standard output) also using
dierent I/O libraries (FlexIO [13], HDF5 [14], JPEG, ASCII). Further methods
or libraries can easily be added by thorn programmers.
In the following sections we will describe in more detail the Grid software
techniques we have developed to date which allow Cactus users to easily perform
postprocessing analysis on data produced by a remote Cactus simulation, and
also to monitor and steer running Cactus jobs remotely. A general overview
of the nal proposed architecture of our Grid-enabled I/O system is shown in

Figure

3.
The Hierarchical Data Format version 5 (HDF5) plays a key role in this
overall picture. HDF5 has become a widely accepted standard in the scientic
computing community for storing data. It denes a very
exible le format and
provides an e-cient software library for managing arbitrary multidimensional
datasets of various types. Raw data access is accomplished via a generic Virtual
File Driver (VFD) layer in HDF5. Beneath this abstraction layer exists a set of
low-level I/O drivers which provide dierent ways of accessing the raw data of
an HDF5 le, either located on a local disk or on other storage media. We have
added our own drivers to this layer which enable existing applications to have the
additional capability of accessing remote data residing anywhere on the Grid.
4.1. Direct Remote File Access
HDF5 already has a GASS driver (Global Access to Secondary Storage) which
Local File System
Virtual File Driver Layer
Amira Data
DPSS
GridFTP
Virtual File Driver Layer
GASS
GridFTP
Unix DPSS Stream
Handling
Cactus Simulation
Thorns
Thorn Hyperslab
I/O Methods
Methods
Visualization (Amira)
Amira Modules
Steering GUI
Steering
DPSS Server
GridFTP Server
HDF5 I/O Library HDF5 I/O Library

Figure

3. General overview of the Grid-enabled I/O architecture
automatically stages complete remote les rst to the local machine and then
operates on their local copies via standard UNIX le I/O. This method is feasible
for small or medium sized data les. However, large-scale computer simulations
often generate large-scale data sets | single simulations may generate
les containing several hundreds of GBytes, up to the order of a TByte as
machine resources increase. Conventional postprocessing analysis then becomes
prohibitively resource-intensive when remote simulation data must be staged for
local processing. Further, in many cases, for example for rst-sight visualization
purposes, only a small fraction of the overall data is really needed. For example,
in a simulation of the evolution of two colliding black holes, the output may contain
a dozen variables representing the state of the gravitational eld at perhaps
1000 time steps during the evolution. For visualization one might want to analyze
only the rst time step of one or two of the variables. Or, in order to perform a
quick pre-analysis of high-resolution data, it might be su-cient to downsample
the array variables and fetch data at only every other grid point.
By enhancing the HDF5 VFD layer with a driver that builds on top of the
Data Grid software components [15] from the Globus toolkit we enable existing
I/O layers to operate on remote HDF5 les directly. These are uniquely addressed
by their URL, and after opening them with the appropriate driver, all read and
write operations are performed as network transactions on the Grid | completely
transparent to the application. Using the data selection capabilities of
(dening so-called hyperslabs as arbitrary rectangular subregions in the
multidimensional data sets, optionally with downsampling and type conversion
applied) individual time steps and zones of interesting data can be read and
visualized in a very e-cient and convenient way.
The Data Grid client software only supports remote partial le access to
Distributed Parallel Storage Systems (DPSS) [16]. During Supercomputing 1999
in Portland and at CeBIT 2000 in Hannover we successfully demonstrated the
feasibility of such a DPSS Data Grid Infrastructure. In these demonstrations,
Cactus simulation data residing on remote DPSS data servers was visualized
by an HDF5-enabled version of the visualization package Amira [17]. This is
illustrated in Figure 4.
Remote Visualization
with Amira
CeBIT 2000 Hannover
AEI Potsdam
HDF5 datasets
on DPSS prototype
Rechenzentrum Garching
Cactus Simulation on T3E
Infrastructure
Gigabit Network
German

Figure

4. Remote le access and visualization demo presented at CeBIT 2000
Remote access to les which are located anywhere on the Grid will soon be
provided by a GridFtp driver [18] which supports the standard FTP protocol,
enhanced with partial le access, parallel streaming capabilities, and Grid security
mechanisms.
Another challenge occurs when simulations are carried out on a distributed
computer and generate physically distributed les. This would occur, for exam-
ple, in order to exploit parallel I/O methods. It is desirable to access and transfer
such distributed data sets as consistent single les, using a global address space
having pointers to pieces at other locations. We plan to also tackle these problems
with the Data Grid components, by organizing related les as collections of
logical le instances. The DataGrid project of the Globus group is investigating
such techniques [15].
4.2. Remote Online Data Streaming and Visualization
Cactus also provides the capability to stream online-data from a running
simulation via TCP/IP socket communications. This can be used for many pur-
poses. To date, the most common use for live data streaming is for remote
visualization, which is our focus here. However, in our vision of future Grid sim-
ulations, we expect running simulations to communicate with each other, migrate
from machine to machine, spawn additional processes on the Grid, etc. Hence,
we expect that data streaming will be a fundamental enabling technology for future
Grid simulations. We are now exploiting the data streaming capabilities
we describe here to enable such advanced Grid simulations, as demonstrated in
our \Cactus Worm" scenario where a running Cactus simulation was able to migrate
itself, using the data streaming techniques described below, from site to site
across the European Egrid [19]. This is a simple example of more sophisticated
types of Grid simulations, based on data streaming, that we will be developing
in the future. But in the remainder of this section we focus on data streaming
for use in remote visualization.
Multiple visualization clients can then connect to a running Cactus executable
via a socket from any remote machine on the Grid, request arbitrary
data from the running simulation, and display simulation results in real-time,
visualizing for example photons falling into a black hole, or isosurfaces of gravitational
waves which are emitted during a black hole collision.
Data streaming is integrated into Cactus in several dierent ways. One
method is to access Cactus output les while they are being written by the
running simulation. Those les are registered with the HTTP control interface,
described in the following section, and can be downloaded to any web browser.
For example, simple 1D data graphs can be viewed by simply clicking on a down-load
le and ring o, for example, an xgraph program. Two-dimensional JPEG
images can be viewed directly in a web browser, and continuous time sequences
of JPEGs can be displayed using the auto-refresh option of the capable browsers.
Another technique implements a proprietary communication protocol for
sending specic geometry data such as isosurfaces or particle trajectories down
a raw socket connection to a visualization program [20]. This is illustrated in

Figure

6. Precomputing such data at the simulation side not only allows for
parallel rendering of images but also reduces the amount of data to be transferred
to remote visualization clients.

Figure

5. Online Visualization: The Amira visualization toolkit [17] allows a user to visualize
slices through a complete 3D data set streamed from a running Cactus simulation,
and at the same time to display an isosurface obtained online for the same 3D eld.
The most generic approach for streaming arbitrary data of any type is again
based on the HDF5 I/O library and its VFD layer. We have developed a Stream
driver which holds the HDF5 data to be streamed out of the Cactus simulation as
an in-memory HDF5 le. On a
ush/close operation the entire le is sent through

Figure

6. Trajectories of freely falling particles in the vicinity of a rotating black hole. The
particle positions are streamed to the visualization tool in real-time during computation.
This was demonstrated with Cactus and Amira at IGrid 2000 in Yokohama [20].
a socket to the connected client. In the client application, the same driver is used
to reconstruct the in-memory le which then can be accessed as usual to read the
datasets.
Since the VFD layer hides all low-level I/O operations from the upper layers
of the HDF5 library and from the application that builds on top of it, applications
can use their existing HDF5 le-based I/O methods immediately for online
remote data access without changing their I/O interfaces. This has been demonstrated
using dierent visualization toolkits, including Amira [17], the IBM Data
Explorer [21], and LCA Vision [22].
The Stream driver is capable of sending data simultaneously to multiple
clients. This is one key component for building a collaborative visualization
environment where scientists at dierent sites can analyze the results of a remote
simulation either by looking simultaneously at the same data or by requesting
dierent views of it. We are working on a design for a more sophisticated I/O
request protocol and the implementation of an external data server which will
handle multiple clients and can also serve requests individually. By integrating
intelligent data management and caching strategies, such a server would relieve
the simulation from communication overhead and help to reduce data tra-c in
general.
4.3. Remote Monitoring and Steering
The Cactus Computational Toolkit contains a thorn HTTPD which can be
added to any Cactus simulation to provide an inbuilt HTTP server. Pointing their
web browsers to a URL identifying a running Cactus job on a remote machine,
any number of collaborators can connect to monitor and steer the simulation
online.
The provided Cactus web interface allows users to query certain information
about the run, such as the current iteration step, a list of available thorns and
variables, and a full description of all parameters and their current settings.
After successful authorization a user can also interactively change parameters
which are marked as steerable. At each simulation cycle these parameters are
checked, and the appropriate thorns may react on changes individually. Most of
the I/O parameters are steerable. This enables users to selectively switch on or
specic output at runtime, dynamically choosing which variables are output
using which I/O method. I/O options such as hyperslabbing or downsampling
parameters may also be modied in order to adjust online data streaming to
remote visualization clients. The web interface can also be used to pause the
simulation, optionally when a chosen condition is satised, and to advance the
simulation by single timesteps.
The Web interface provided by thorn HTTPD is dynamically extensible in
that any thorn can register and update its own HTML pages at runtime. Besides
a download page for Cactus output les there is a also viewport available which
embeds dynamically generated JPEG images.
Another steering interface again builds on top of HDF5 and the Stream driver
described above. For this interface the data streaming is simply used in a bidirectional
way: Cactus writes parameters into an HDF5 le which is then streamed to
any connected steering client. After some user interaction, this client sends back
a modied version of the parameter le which is read and evaluated by Cactus.
Because of its self-describing data format and the
exibility to add addi-
tional, user-dened information, HDF5 also provides the possibility to build more
advanced steering clients with graphical user interfaces. As an example, mini-
mum/maximum values could be assigned to numerical parameters to create sliders
gfor more convenient user interactions. Parameters belonging to one thorn
could be sorted into a group hierarchy for building menus.
These features of HDF5 make it relatively easy to implement dynamic graphical
user interfaces for arbitrary Cactus parameter sets which adapt themselves
to the current Cactus conguration. We are actively working on including such
user interfaces into existing visualization clients. Their steering capabilities would
then not only be limited to exchanging HDF5 parameter les but could also be
extended to feed back any kind of data elds into Cactus, for instance to add
photons to a black hole simulation to locate an event horizon.
5. Portals onto the Grid
The Grid is only useful as a concept if its services can be used to create
the illusion that all of the resources are centralized to the user's workstation.
So the most successful distributed applications on the Grid will paradoxically
be those which make the user least aware that they are in fact operating in
distributed fashion. The motivation for producing a Grid Portal interface to a
like Cactus is derived from the desire to hide distributed applications and
immensely complex distributed/parallel software architectures behind a single
point of presence, and to make them accessible through comparatively simple
client-side interfaces.
A portal is a single point of presence (typically hosted on the web) which
can be customized for a particular user and remembers particular aspects of the
customizations regardless of where the user accesses it from. Yahoo and HotMail
are typical consumer-oriented examples of this capability and are in fact the
originators of this new meaning for the term portal. It doesn't matter where you
are, when you login to the URL of these portals, you get access to the same view
of your personalized environment and data (ie. your email).
Placing the PSE like Cactus within a portal creates a universally accessible
interface to your scientic computing platform. The GUI is user-customizable,
as if it were a desktop application on the user's own workstation, except that the
same customized environment is accessible from virtually any location by simply
connecting to the same URL address. A science portal has the additional implied
function of automating the entire work
ow for a particular scientic application,
from initial data generation, to selecting resources to run the application, to
archival storage management and analysis of the results of those simulations.
This replaces a series of perhaps loosely (or usually poorly) integrated tools with a
comprehensive environment which is customized around a particular application.
Finally a collaboratory provides additional facilities for sharing information either
online or asynchronously among users of the portal.
Cactus has several key advantages which make it very suitable as the basis
for a portal design. Its modular design supports dynamic assembly of applications
online through a simplied GUI. Its sophisticated multiplatform compilation
makes it very simple to run the code on any available Grid resource without
the complexities of Imake or the performance penalty of a Virtual Machine. Its
centralized revision control mechanism permits e-cient sharing of code, software
updates, and bug xes. Finally the integrated visualization for remote monitoring
and steering of the code through a web interface allows seamless integration
of these capabilities with the portal's web-GUI.
The Astrophysics Simulation Collaboratory Portal [5] is a concrete use of
Cactus within a web portal GUI which leverages o of technology which was
developed originally for e-commerce applications. The architecture utilizes a
commercial-grade StrongHold (Apache) webserver which oers SSL encryption
using a site certicate from a commercial Certicate Authority. Running side-
by-side with the webserver is a TomCat JSP engine which oers a cleaner means
to manage automation in an elegant and easily maintainable fashion. JSP allows
us to directly execute methods of server-side Java beans rather than the typical
CGI-script methodology of parsing the state of form elements individually after
an HTTP 'POST' event. The Java beans directly call Java CoG [6] which is a
pure Java implementation of the Globus toolkit to extend the automation to
Grid resources. The user-state within the system is maintained by a back-end
database system (OpenLDAP or mySQL) which allows simple replication of portal
state allowing the web services to be scaled through server replication.
The science portals and collaboratories will play an increasingly important
role in HPC as the Grid evolves. The natural point of organization for
user communities in an HPC environment is a particular eld of science or a
particular application code, just as experimental laboratories bring together top
researchers who are interested in the same or similar lines of research. The internet
has provided us with access to enormous remotely located resources, but
this has shifted the center of focus to the particular HPC site and its operating
environment, batch queues and security policies rather than the science that is
computed there. The single point of presence oered by an Grid portal recreates
gthe traditional laboratory environment where scientists who share similar
interests and applications are brought together under the umbrella of a shared
a collaboratory. The portal itself is a distributed Grid application
for a specic community of scientists rather than a general-purpose resource. So
unlike traditional web-portals where you have an implicit oer that if you go
to www.<my portal location>.org we'll do everything for you here using our
compute resources, the Grid portal's business plan can be simply stated as go
to www.<my portal application>.org and we will do everything you need for
<my application> regardless of your location and that of your compute resources.
This returns the focus of a scientic community to the scientic application rather
than the location of the HPC resources.
If the Grid is really working, in another 5 years we will no longer think of
the, for example, NSF supercomputing centers as distinct sites like SDSC, NCSA,
or PSC. We will instead think only of the particular application collaboratories
which have been set up to study dierent scientic applications. That resource
will merely be a name rather than a place.

Summary

The Cactus Code and Computational Toolkit and the large scale applications
which it serves provide a ideal laboratory for developing and testing new
Grid techniques and working practices. Cactus can be very easily congured
and run in a Grid environment, and the tools developed so far already provide
many capabilities for exploiting global computing resources. The infrastructure
and tools developed are immediately available to the user community for testing,
and many are already being successfully and benecially used by collaborations
researching computationally intensive problems such as black hole and neutron
star collisions.

Acknowledgements

The development of the Cactus Code is a highly collaborative eort, and
we are indebted to a great many experts at dierent institutions for their advice,
visions and support. The original design of Cactus was by Joan Masso and Paul
Walker, since when it has been extensively developed at the AEI, NCSA and
Washington University.
It is a pleasure for us to thank Ian Foster, Steve Tuecke, Warren Smith,
Brian Toonen, and Joe Bester from the Globus team at Argonne National Labs
(ANL) for their Globus and Data Grid work; Mike Folk and his HDF5 development
group at NCSA who helped us in implementing the requirements of remote le
access into their HDF5 code; Brian Tierney from Lawrence Berkeley Labs for his
DPSS support; Jason Novotny at NLANR for his help with Globus and graphical
user interfaces; and Michael Russell at the University of Chicago for his Portal
work. Computing resources and technical support have been provided by AEI,
ANL, NCSA, Rechenzentrum Garching/Germany, and ZIB.
We greatly acknowledge nancial support for Andre Merzky and Thomas
Radke as well as provision of a gigabit network infrastructure in the course of the
TIKSL research project by DFN (German Research Network).



--R

http://www.

Numerical Relativity As A Tool For Computational Astrophysics J.
DFN Gigabit Project:
Toolkit: http://www.



Grid Adaptive Development Software (GrADS): http://www.
The European Grid-Forum: http://www
Implementation: http://www.
FlexIO: http://zeus.
Hierachical Data Format Version 5: http://hdf.

Distributed Parallel Storage System http://www-didc

The Globus Project: GridFTP: Universal Data Transfer for the Grid.

Geodesics in Kerr Space-Time
IBM Data Explorer: http://www.
http://zeus.
--TR

--CTR
Michael Russell , Gabrielle Allen , Greg Daues , Ian Foster , Edward Seidel , Jason Novotny , John Shalf , Gregor von Laszewski, The Astrophysics Simulation Collaboratory: A Science Portal Enabling Community Software Development, Cluster Computing, v.5 n.3, p.297-304, July 2002
Henri Casanova , Francine Berman , Thomas Bartol , Erhan Gokcay , Terry Sejnowski , Adam Birnbaum , Jack Dongarra , Michelle Miller , Mark Ellisman , Marcio Faerman , Graziano Obertelli , Rich Wolski , Stuart Pomerantz , Joel Stiles, The Virtual Instrument: Support for Grid-Enabled Mcell Simulations, International Journal of High Performance Computing Applications, v.18 n.1, p.3-17, February  2004
Gabrielle Allen , David Angulo , Ian Foster , Gerd Lanfermann , Chuang Liu , Thomas Radke , Ed Seidel , John Shalf, The Cactus Worm: Experiments with Dynamic Resource Discovery and Allocation in a Grid Environment, International Journal of High Performance Computing Applications, v.15 n.4, p.345-358, November  2001
James A. Kohl , Torsten Wilde , David E. Bernholdt, Cumulvs: Interacting with High-Performance Scientific Simulations, for Visualization, Steering and Fault Tolerance, International Journal of High Performance Computing Applications, v.20 n.2, p.255-285, May       2006
Katarzyna Rycerz , Alfredo Tirado-Ramos , Alessia Gualandris , Simon F. Portegies Zwart , Marian Bubak , Peter M. A. Sloot, Regular Paper: Interactive N-Body Simulations On the Grid: HLA Versus MPI, International Journal of High Performance Computing Applications, v.21 n.2, p.210-221, May       2007
Lingyun Yang , Jennifer M. Schopf , Ian Foster, Conservative Scheduling: Using Predicted Variance to Improve Scheduling Decisions in Dynamic Environments, Proceedings of the ACM/IEEE conference on Supercomputing, p.31, November 15-21,
Wes Bethel , John Shalf, Grid-Distributed Visualizations Using Connectionless Protocols, IEEE Computer Graphics and Applications, v.23 n.2, p.51-59, March
Ming Wu , Xian-He Sun, Grid harvest service: a performance system of grid computing, Journal of Parallel and Distributed Computing, v.66 n.10, p.1322-1337, October 2006
grid portals;cactus;grid computing