Warm Standby in Hierarchically Structured Process-Control Programs.
We classify standby redundancy design space in process-control programs into the following three categories: cold standby, warm standby, and hot standby. Design parameters of warm standby are identified and the reliability of a system using warm standby is evaluated and compared with that of hot standby. Our analysis indicates that the warm standby scheme is particularly suitable for long-lived unmaintainable systems, especially those operating in harsh environments where burst hardware failures are possible. The feasibility of warm standby is demonstrated with a simulated chemical batch reactor system.
Introduction
Process-control programs, such as those for controlling
manufacturing systems, can often be organized in a multi-level
hierarchical control structure where higher level processes
formulate long-term control strategies, e.g., optimizing
resource management, whereas lower level processes
perform real-time control functions [1,13]. The long-term
nature of the decisions made by non real-time upper level
processes means that the system may be able to tolerate
temporary loss of such processes, e.g., by using suboptimal
strategies. However, the loss of critical real-time processes
can disrupt the whole system. This suggests that different
fault tolerance techniques should be adopted for upper and
lower level processes since their reliability requirements are
quite different.
The standby replacement approach [2,6,7,12] is an economical
and efficient way of achieving fault tolerance at
reasonable cost for processes in the control hierarchy. In
one scheme, termed "cold standby," only one copy of each
process is active at a time, and each copy is allocated to
a processor that is designed to be fail-stop [11], i.e., able
to detect if it contains a fault during the normal course
of operation by using redundant hardware such as a self-checking
circuit [8,10]. When a processor is faulty, processes
which reside on the failed processor are assigned to
a spare processor or other functional processors if no spare
processors are available. The main disadvantage of this
"cold standby" scheme is its long recovery time to load
and restart a backup copy. This problem is overcome by
This work was supported in part by the National Science Foundation
under Grant CCR-9110816 and the US Nuclear Regulatory Commission
under award NRC-04-92-090. The opinions, findings, conclusions, and
recommendations expressed herein are the authors' and do not necessarily
reflect the views of the NRC.
I.R. Chen is with Department of Computer and Information Science,
University of Mississippi, Weir 302, University, MS 38677.
F.B. Bastani is with Department of Computer Science, University of
Houston, Houston,
using the "hot standby" scheme where two or more copies
are allowed to run at the same time on different fail-stop
processors, with one copy serving as the primary and the
others serving as active backups. When the primary copy
fails (due to the failure of the processor on which it resides),
a backup copy running on another processor can take over
instantaneously without any recovery time delay. Howev-
er, this "hot standby" approach requires up to date copies
of a process and may not be cost-effective for upper level
processes which do not require instantaneous recovery.
This paper develops a warm standby scheme in which
the copies of a process may be partial copies instead of
full copies as in the "hot standby" scheme. In the design
space of replication, we envision that (a) for cold standby,
there is only a single active copy of the process and, hence,
there are no other active copies; (b) for hot standby, there
are multiple active, full copies of a process; (c) for warm
standby, there are also multiple active copies of a process,
some of which are partial copies. Warm standby is suitable
for upper level processes because it incurs medium cost
and moderate recovery time delay as compared with other
standby schemes, although it potentially can also be used
for lower level processes that are less time-critical.
The rest of the paper is organized as follows. Section II
defines the meaning of a warm standby copy in a hierarchically
structured system as opposed to a hot standby copy,
and identifies the design parameters of the warm standby
scheme. Section III presents a reliability analysis of warm
standby and a simulation evaluation using a simulated
chemical batch reactor. Finally, Section IV concludes the
paper and outlines some future research areas.
II. Definition of Warm Standby Copies
We first define our fault model. We assume that if a processor
fails then its failure is detected by redundant hardware
and it ceases operation. In no cases does a machine
behave unexpectedly. This assumption can be satisfied using
techniques based on fail-stop processors [11].
As an example of warm standby copies, consider a part
of a process-control system where a temperature profile is
controlled by a control process according to a prescribed
optimal time-temperature curve. This control process monitors
the temperature sensor input and calculates the actuator
output for effecting temperature changes (with a goal
of minimizing the mean square error between the actual
temperature profile and the optimal temperature profile).
To tolerate possible failure of the control process, a stand-by
copy is created in another computer. The standby copy

Figure

1. Different Detailed View of A Temperature Profile

Figure

2. An Abstract Hierarchy.
can be implemented in three ways: (a) it has the same view
of the control information as that of the primary copy and
the frequency of receiving the temperature sensor input is
the same as that of the primary copy, (b) it only has a
partial view of the control information and, hence, the frequency
of receiving the sensor input is less than that of the
primary copy, and (c) it does not have any view of the control
information and, hence, the frequency of receiving the
sensor input is zero. These three implementations correspond
to the hot standby, warm standby, and cold standby
schemes, respectively. Figure 1 illustrates the sensor input
temperature profile as perceived by the standby copy
using hot, warm, and cold standby schemes, respectively.
In effect, the sensor input temperature profile is viewed at
different levels of detail, ranging from the most detailed
one corresponding to the use of maximum sensor sampling
frequency, to the least detailed one corresponding to the
use of minimum sensor sampling frequency.
There are two implications in this example which must
be pointed out. First, a warm standby copy, although possessing
only a partial view of the sensor input temperature
profile, still has a useful and summarized view of the temperature
profile, e.g., it may still know what the maximum
temperature is, when it was attained, etc. This allows a
warm standby to immediately take charge using its summarized
information without having to start from scratch
as in the cold standby scheme. Second, the amount of
processing power to create and maintain a standby copy
is proportional to the sampling frequency and, hence, a
warm standby copy will not consume as much processing
power as a hot standby copy since the sampling rate is low-
er. This means that for the same hardware cost a higher
degree of replication may be achieved by using warm standby
instead of hot standby copies. This higher degree
of replication for the same hardware cost can provide the
system with a better reliability, particularly for long-lived
unmaintainable systems, and those operating in harsh environments
where burst hardware failures are possible, because
now a process has more copies to tolerate multiple
processor failures. A reliability analysis will be performed
later in Section 3 to illustrate this point.
In the following, we give a more formal definition of a
primary or standby copy of a control process, and its interaction
with other control processes in a hierarchically
structured system. The definitions are illustrated using
the system shown in Figure 2. It consists of 4 application
processes, a; b; c, and d.
2.1 Seniority Function
A copy of a process may impose only a partial load on
a processor depending on a design parameter called the
seniority of that copy. Formally, let A denote the set of
processes in the hierarchical control system (e.g., a, b, c,
and d in Figure 2) and let P denote the set of available
processors. Then, let
f;g be the parent function (e.g.,
a in

Figure

for the hierarchical structure;
be the load function (e.g., instr/sec) of
processes in A;
be the capacity (e.g., instr/sec) of processors
in P .
The seniority function allocates copies of processes to processors

Thus, if a copy of a process a's seniority oe(a; p) is 1, then
it means that the copy is either a primary copy or a hot
standby copy and it runs at its full load, l(a), on processor
indicates that processor p does not
execute a at all. A value between 0 and 1 means that
this copy of process a is a warm standby copy and imposes
oe(a; p)l(a) load on processor p. The allocation must satisfy
the following constraint,
a2A
2.2 Logical Communication Link
In a conventional hierarchical structure, for each par-
ent/child process-pair, we have one parent-to-child logical
communication link (for sending control instructions) and
one child-to-parent logical communication link (for transmitting
status information). In the hierarchical structure
with warm standby copies, similar logical communication
links are used. However, the logical communication links
need not be of the same capacity. Formally, let
In other words, allocation(a) is the set of processors having
at least one copy of a and primary(a) is the set of processors
having the most senior copies of a. There are two sets of
active logical communication links in the hierarchy:
ffl A parent-to-child link from x to y iff 9a 2 A such that
allocation(a). The capacity of the link is proportional
to oe(a; y).
ffl A child-to-parent link from x to y iff 9a 2 A such that
allocation(-(a)). The capacity of the link is proportional
to oe(-(a); y).
Notice that the load on the communication subsystem is
the same for both the warm standby and hot standby
schemes since the source of all information is the set of
primary nodes while the destination is the set of allocated
nodes. If the receiver is a full copy (one that is allocated
to a primary node) then it gets complete information,
otherwise it receives only partial information.
III. Evaluation
We first show that the use of warm standby copies instead
of just hot standby copies can enhance the system
reliability. Design conditions under which the above statement
is true are investigated. Then, we present a simulation
evaluation of the warm standby scheme using a case
study.
3.1 Reliability of Partial Replication
As pointed out in Section II, since a warm standby (i.e.,
a partial) copy requires less processing power than a full
copy, more standby copies for the same hardware cost can
be used to tolerate hardware failures, resulting in a system
that is less vulnerable to hardware failure. A direct consequence
of this effect is enhanced reliability. At the same
time, there is no increase in software complexity since all
copies of a process run the same program. Nevertheless,
a design tradeoff associated with the use of warm standby
copies over just hot standby copies is that there exists a
possibility that a warm standby copy may not be able to
deal with a control situation when it takes control. The
period that is required for a partial copy to advance its seniority
to become a full copy when it initially takes charge
is called a vulnerable period, which depends on the copy's
seniority. In the following, we present an analysis that illustrates
conditions under which the warm standby scheme
may be favored over the hot standby scheme.
Consider the case of allocating 2 processes, a and b, to 4
processors, a (a is the parent
of b and thus both are important system functions and
cannot fail at any time) and c(p i
(each processor has the processing capability of loading up
to one full copy, either a or b). We assume that a processor
functions for an exponentially distributed time with
rate -; once it fails it stays down because there is no repair
capability in the system. Now, consider the following
two ways of achieving fault tolerance by means of standby
redundancy:
1. Using only hot standby copies, e.g.,
1, and oe(b; p 3 It consists of a series
structure of two subsystems with one consisting of p 1
and each containing a full copy of a, in a parallel
structure, and the other consisting of p 3 and p 4 , each
containing a full copy of b, also in a parallel structure.
The reliability of the system is given by
2. Using warm standby copies, e.g.,
and oe(a; p 3
seniority of 0.5 means that a copy runs only at its one-half
load on the processor it is allocated to. The reliability
of this system is bounded from above by the reliability
of a 2-out-of-4 system. Let x i be 1 if p i is alive
and let it be 0 if p i has failed, for
denote the complement of x i . Then the structure function
for the system [3] is
From this, the upper bound on the reliability
of the warm standby system is given by:
r(t)j upper bound
which is better than the reliability of the hot standby
system. However, the lower bound on the reliability is
given by
r(t)j lower bound
The reason the reliability is less than the upper bound
is because of the probability of a faulty control decision
while the warm standby is in the process of gathering
sufficient information to become the primary controller
after the primary fails. We have developed a
detailed reliability model [5] that assumes that when
a partial copy of process a or process b takes over,
it takes an exponentially distributed time (this is the
vulnerable period) with rate - a or - b , respectively, to
become a primary copy. Moreover, it models the fact
that during this vulnerable period, there is a software
failure rate, ', representing the rate at which a partial
copy fails to deal with a control task when it takes
over.
Detailed calculations [5] show that the reliability of the
system using warm standby copies is better than that just
using hot standby copies as ' (software failure rate of a
partial copy) decreases and as - (recovery rate of a partial
copy) increases (here - observation
is that when ' -, the reliability of the warm standby system
is always better than that of the hot standby system.

Figure

3 compares the reliability of these two systems with
all parameters varying proportionately. When ' is comparable
in magnitude to -, the warm standby system can
provide a better reliability than the hot standby system as
the underlying hardware becomes more unreliable. An explanation
of this is because state transitions that could lead
to system failure in the warm standby system are mostly
due to ' rather than - and the probability that a state
transition can lead to system failure in the warm standby
system is less than that of the hot standby system since
there are more states in the warm standby system. Con-
sequently, the reliability of the warm standby system will
decline by a lesser extent than that of the hot standby system
as - increases since increasing - only increases ' by
the same order of magnitude. Conversely, when ' is an
order of magnitude higher than - (e.g., 10-),
the warm standby system will suffer more from increasing
- since this increases ' by an extra order of magnitude
times) and the probability of state transitions that
can lead to system failure for the warm standby system is
greatly increased. In summary, we conclude that the warm
standby scheme is most favorable when ' is of the same order
of magnitude as - and this favorable situation is most
likely when the underlying hardware is unreliable and/or
the recovery rate (-) is high.
3.2 Simulation Evaluation

Figure

3. Reliabilities of Hot Standby and Warm Standby
with Lower and Upper Bounds.

Figure

4. A Batch Reactor.
In this section, we first develop a process-control program
for a simulated experimental chemical batch reactor
system to illustrate the warm standby technique in prac-
tice. Then, we present the simulation results and analyze
the effect of various parameters on the reliability of the
recovery procedure. In this case study, the physical environment
of the batch reactor in which the control processes
are embedded is simulated; however, the control processes
are completely implemented, instead of being simulated,
and operate in real-time. The environment simulator sends
sensor data in every 4t interval to the control processes;
when it receives control actions in response to a sensor
event (e.g., opening a fraction of a steam valve) from the
control processes, the simulator updates the state of the
environment (e.g., temperature and pressure) to that at
t +4t based on the state at t, and advances its simulation
clock to t +4t.
3.2.1 A Chemical Batch Reactor System
Consider the batch reactor sketched in Figure 4 where
first-order consecutive reactions take place in the reactor
as time proceeds. Reactant A (with a corresponding solution
concentration CA ) is initially charged into the vessel.
Steam is fed into the jacket to bring the reactor up to a temperature
at which the consecutive reactions begin. Cooling
water is later added to the jacket to remove the exothermic
heat of reactions. The product that is desired is component
(with a solution concentration CB ). If the control
process lets the reaction go on for too long, too much B will
react to form compound C (with a solution concentration
CC ) and consequently the yield of B will be low. On the
other hand, if the control process stops the reaction too
early, too little A will have reacted and the conversion and
yield of B will again be low. Therefore, the control process
needs to control the batch reaction to follow a specific temperature
profile (i.e., time vs temperature profile) in order
to optimize the yield. The actual temperature is adjusted
by a controller which controls two split-range valves, a
steam valve and a water valve. The fraction of the steam
valve which is open, X s , and the fraction of the water valve
which is open, Xw , are determined by an output signal, P c ,
produced by the temperature actuator. The steam valve is
wide open when P and is closed when P c - 9 while
the water valve is closed when P c - 9 and wide open when
3. Hence, the control process needs to communicate
closely with the temperature controller to properly adjust
the temperature in the reactor.

Figure

5 shows an instance of the optimum temperature
and concentration profiles with Tmax representing the maximum
temperature and C j representing the concentration
of A, B, or C. If the reaction runs longer than t opt , the yield

Figure

5. Batch Profiles.

Figure

6. A Control Hierarchy for the Batch Reactor System

of B decreases. A complete set of equations that describe
the kinetics of the first-order consecutive reactions can be
found in [9].
3.2.2 A Hierarchically Structured Control Program
The control process described above can be implemented
as a 4-level control hierarchy (Figure
Level 4: This level controls the inventory of chemicals,
the scheduling of batch reactions, the maintenance of
production level, etc.
Level 3: This level governs the kinetics of different batch
reactions (e.g. formulating optimal temperature pro-
files).
Level 2: This level consists of processes (i.e., master control
processes) each of which controls a batch reaction
using an optimal temperature profile provided by a
level 3 process. The responsibilities of a master control
process include (a) minimizing the mean square error
(MSE) between the actual and the desired reactor
temperature profiles such that CB;desired \GammaC B;actual
CB;desired -
3% for the final concentration of product B, and (b)
dynamically formulating desired jacket temperature
profiles one segment at a time to be followed by a
level 1 process. The MSE is defined as
batch
batch X
desired
actual (t)j
where t batch is the total batch reaction time in minutes.
Level 1: This level consists of specialized processes (i.e.,
jacket temperature control processes) each of which is
responsible for regulating jacket temperature changes
(by controlling the steam and cooling water valves and
flow rates) such that the prescribed jacket temperature
profile formulated by a level 2 process is followed.
We focus our attention on a level 2 process (the master
control process) and a level 1 process (the jacket temperature
control process) of the control hierarchy. (In a hierarchical
control structure as such, levels 1 and 2 are normally
made fault tolerant because on-line repair is not practical
for lower level real-time controllers.) We assume that a
level 3 process which formulates the desired temperature
profile to be followed by the master control process is allocated
to some processor in the system and there are four
processors, to which the master and the
jacket processes can be allocated for controlling the batch
reaction. Also, we assume that a level 2 or a level 1 process
will consume a fraction of the processing power of a
processor in a ratio that is equivalent to its seniority. For
example, if a copy's seniority is 1, then it will consume the
full processing power of a processor that it is allocated to.
Seniority Function. A level 2 master control process, m,
is replicated on three processors,
functions

Figure

7. The Data Structure for Describing
A Temperature Profile.
Only the copy with oe(m; primary copy)
provides direct control to the batch reactor with the others
serving as partial copies. On the other hand, a level 1
jacket temperature control process, j, is replicated on three
processors,
to 1.0 and oe(j; p 4 only the copy with
primary copy) provides direct control to
the jacket temperature. Recall that a copy consumes a fraction
of the processing power of a processor in a ratio that
is equivalent to its seniority, so that oe(m; p 3
and oe(m; p 4 Furthermore, when a junior
copy becomes a primary copy, other partial copies residing
in the same processor will be deprived of their processing
power. For example, if oe(m;
the junior copy with oe(m; advances its seniority
to oe(m; p 3 due to detection of a failure of the primary
copy of m, the junior copy with oe(j;
be deprived of its processing power from processor p 3 . In
our implementation, this is achieved by scheduling it to
die at the same time when failure of the primary copy of
occurs.
Knowledge Representation. We use the same knowledge
representation to describe the desired and actual reactor
(or jacket) temperature profiles for all copies of m (or
j).

Figure

7 shows the data structure used to describe a
temperature profile. There are 60=t s slots which could be
filled per minute, where t s represents the sensor sampling
interval in seconds. The degree to which these slots are
filled is proportional to a copy's seniority. For example,
every nth slot is filled for a copy of a control process with
a seniority equal to 1=n. This data structure allows a temperature
profile to be described at different levels of detail,
i.e., as the number of filled slots increases, the temperature
profile is known to a greater detail. Consequently, a copy
with a low seniority will probably have more up-to-date information
about less frequently updated information such
as phase, rate of change of temperature (slope), etc., and
have less up-to-date information about temperature slots.
Note that with this implementation, a primary copy of m
does not send different information to different copies of j
and, hence, broadcast protocols with sampling by copies of
(with a sampling rate proportional to their seniorities)
could be used.
Simulating the Batch Reactor Environment. The
control environment in which the master and jacket processes
are embedded is simulated by the following three
processes running on other processors:
1. an environment process which simulates the physical
environment of the batch reactor and the sensor sub-system

2. a channel process which simulates the underlying communication
subsystem with varying degrees of chan-
Table

1. Parameters of Batch Reactor Control Program.

Table

2. MSE and CB for Single, Double and Triple Processor
Failures.
nel capacities simulated by changing its input queue
lengths (this parameter is called C channel ); and
3. a supervisor control process which simulates a level 3
process.
These three simulated processes, together with the three
copies of the master control process and the three copies of
the jacket temperature control process, communicate with
one another through the channel process.
Fault Recovery Procedure. Process failure detection is
implemented by "are you alive" and "I am alive" messages.
Specifically, if a process does not respond to an "are you
alive" message for more than N broadcast (a program param-
eter) consecutive broadcasting intervals (a broadcasting interval
a sensor sampling interval in our implementation),
then the process is considered dead. The recovery action
taken by a junior copy of m upon detection of a failure of
its senior counterpart is (a) advancing its seniority from
either 0.2 to 0.6, or 0.6 to 1.0, and (b) acquiring more detail
control information from the supervisor control process
(for the desired temperature profile) and the environment
process (for the sensor data) in a frequency that is proportional
to its new seniority. If the new seniority is 1.0,
the junior copy takes charge immediately while gradually
acquiring information from the environment. The recovery
action taken by a junior copy of j is the same except that
the parent process from which it acquires the desired jacket
temperature profile is the primary copy of m.
3.2.3 Simulation Results
The parameters of our control program are shown in Table
1.

Table

2 compares the cases of single, double, and
triple processor failures in terms of the MSE (mean square
error) between the actual and optimal temperature profiles
and the final yield of CB . The time of processor failure is
chosen to be at the most critical moment of the batch reac-
tion, namely, at the time when a phase change occurs (at
the 14th minute mark).
From

Table

2, we see that when all the copies of m or
(for example, all copies of j process fail when processors
the batch reaction goes on
unattended and the final yield of CB becomes quite low
(in fact it is equal to zero) because too much B is con-
sumed. On the other hand, for other cases (even for the
case of a triple failure, e.g., (p
copy still survives, the yield of CB is quite good. This
is partly due to the fact that the batch reactor is intrinsically
a set-point based reactor system and thus a temporary
out-of-sync between the control and environment processes
can be tolerated in a short period of time without causing
catastrophic damages. However, it also points out the importance
of using partial copies to provide fault tolerance
- even a partial copy with seniority equal to 0.2 can make
a significant difference. Our other experimental results [4]
show that in all cases when at least one partial copy survives
failures (for both the level 1 and level 2 processes),
the yield of CB is good and the MSE is never more than
Comparison of the simulation results with the case of
using full copies is obvious. In the latter case, p 1 and p 2
may be allocated for the master control process, and p 3 and
may be allocated for the jacket control process. Hence,
when a critical double failure occurs, namely, (p 1 ,p 2 ) or
(p 3 ,p 4 ), the batch reaction will go on unattended. This is
in contrast with the results for warm standby where we observe
that when a double failure occurs, the batch reaction
is always under proper control. Of particular interest is
the simulation result as compared to the case of using cold
standby processes. In the former case, there is no disruption
of continuity of control when a warm standby process
takes over whereas in the latter case a cold standby process
would require a loading and restart period (e.g., to
load the temperature history logged in some stable storage
before restart) and during that period the system is left
uncontrolled.
IV.

Summary

In this paper, we have developed a fault-tolerant technique
that can be used in a variety of process-control sys-
tems. This technique provides good reliability in a cost-effective
way by incorporating the concept of warm stand-
by. Our case study shows that a surviving warm standby
copy with a seniority as low as 0.2 can make a significant
difference in providing continuity of control when failures
occur. Our comparative study of the reliability of partial
and hot standby techniques suggests that warm standby
appears to have its greatest advantage when it is applied
to systems whose underlying hardware is unreliable.
There are several research areas which include (1) developing
a decentralized management methodology for hierarchically
structured process-control programs with warm
standby and analyzing the effects of local and global factors
which influence the distribution of oe per process (influenced
by local factors, e.g., importance of a process) and per processor
(affected by global factors, e.g., load balancing re-
quirements), (2) using a frame-structure-based knowledge
representation technique to facilitate self-learning capability
of control processes, (e.g., via peer-to-peer communications
which can exist among copies of a process), and
extending it to cases where the knowledge base represented
by the frame structure is large, and (3) comparing the
performance of warm standby using unreliable communication
protocols (e.g., datagram services) with hot standby
using reliable protocols based on timeout and retransmission

Acknowledgments

The authors wish to thank the five anonymous reviewers
for their detailed comments which have significantly
improved the quality of this paper.



--R

"Theory and practice of hierarchical control,"
"The STAR (self-testing-and repairing) com- puter: an investigation into the theory and practice of fault tolerant computing,"
Statistical Theory of Reliability and Life Testing

"Reliability of fully and partially replicated systems,"
"Making processing fail-safe,"
"Fault tolerant computer system for the A129 helicopter,"
Design and Analysis of Fault Tolerant Systems

"Evaluation of a self-checking version of the M- C68000 microprocessor,"
"Fail-stop processors: An approach to designing fault tolerant computing systems,"
"Fault-tolerant design of local ESS processors,"
"The development of reliability in industrial control systems"
--TR
Fail-stop processors
Process Modeling, Simulation, and Control for Chical Engineers
An ai-based architecture of self-stabilizing fault-tolerant distributed process control programs and its analysis
software reliability;process computer control;hierarchically structured process-control programs;software maintenance;burst hardware failures;redundancy;warm standby;process-control programs;simulated chemical batch reactor system;system reliability;long-lived unmaintainable systems;hot standby;cold standby;computer integrated manufacturing;standby redundancy design space;fault tolerant computing