Coverage criteria for GUI testing.
A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing techniques do not directly apply to GUIs. This paper's focus is on coverage critieria for GUIs, important rules that provide an objective measure of test quality. We present new coverage criteria to help determine whether a GUI has been adequately tested. These coverage criteria use events and event sequences to specify a measure of test adequacy. Since the total number of permutations of event sequences in any non-trivial GUI is extremely large, the GUI's hierarchical structure is exploited to identify the important event sequences to be tested. A GUI is decomposed into GUI components, each of which is used as a basic unit of testing. A representation of a GUI component, called an event-flow graph, identifies the interaction of events within a component and intra-component criteria are used to evaluate the adequacy of tests on these events. The hierarchical relationship among components is represented by an integration tree, and inter-component coverage criteria are used to evaluate the adequacy of test sequences that cross components. Algorithms are given to construct event-flow graphs and an integration tree for a given GUI, and to evaluate the coverage of a given test suite with respect to the new coverage criteria. A case study illustrates the usefulness of the coverage report to guide further testing and an important correlation between event-based coverage of a GUI and statement coverage of its software's underlying code.
INTRODUCTION
The importance of graphical user interfaces (GUIs) as critical
components of today's software is increasing with the
recognition of their usefulness. The widespread use of GUIs
has led to the construction of more and more complex GUIs.
Although the use of GUIs continues to grow, GUI testing
has, until recently, remained a neglected research area. Because
GUIs have characteristics different from conventional
software, techniques developed to test conventional software
cannot be directly applied to GUI testing. Recent advances
in GUI testing have focused on the development of test case
generators [8, 11, 12, 14, 18, 6] and test oracles [9] for GUIs.
However, development of coverage criteria for GUIs has not
been addressed.
Coverage criteria are sets of rules used to help determine
whether a test suite has adequately tested a program and
to guide the testing process. The most well-known coverage
criteria are statement coverage, branch coverage, and path
coverage, which require that every statement, branch and
path in the program's code be executed by the test suite
respectively. However such criteria do not address the adequacy
of GUI test cases for a number of reasons. First, GUIs
are typically developed using instances of precompiled elements
stored in a library. The source code of these elements
may not always be available to be used for coverage evalu-
ation. Second, the input to a GUI consists of a sequence of
events. The number of possible permutations of the events
may lead to a large number of GUI states and for adequate
testing, a GUI event may need to be tested in a large number
of these states. Moreover, the event sequences that the
GUI must be tested for are conceptually at a much higher
level of abstraction than the code and hence cannot be obtained
from the code. For the same reason, the code cannot
be used to determine whether an adequate number of these
sequences have been tested on the GUI.
The above challenges suggest the need to develop coverage
criteria based on events in a GUI. The development of such
coverage criteria has certain requirements. First, since there
are a large number of possible permutations of GUI events,
the GUI must be decomposed into manageable parts. GUIs,
by their very nature, are hierarchical and this hierarchy may
be exploited to identify groups of GUI events that can be
tested in isolation. Hence, each group forms a unit of test-
ing. Such a decomposition also allows coverage criteria to
be developed for events within a unit. Intuitively, a unit of
testing has a well-defined interface to the other parts of the
software. It may be invoked by other units when needed and
then terminated. For example, when performing code-based
testing, a unit of testing may be a basic block, procedure,
an object, or a class, consisting of statements, branches, etc.
Next, interactions among units must be identified and coverage
developed to determine the adequacy of tested interac-
tions. Second, it should be possible to satisfy the coverage
criterion by a finite-sized test suite. The finite applicability
[20] requirement holds if a coverage criterion can always
be satisfied by a finite-sized test suite. Finally, the test designer
should recognize whether a coverage criterion can ever
be fully satisfied [16, 17]. For example, it may not always
be possible to satisfy path coverage because of the presence
of infeasible paths, which are not executable because of the
context of some instructions. Detecting infeasible paths in
general is a NP complete problem. No test case can execute
along an infeasible path, perhaps resulting in loss of
coverage. Infeasibility can also occur in GUIs. Similar to
infeasible paths in code, static analysis of the GUI may not
reveal infeasible sequences of events. For example, by performing
static analysis of the menu structure of MS Word-
pad, one may construct a test case with Paste as the first
event. However, experience of using the software shows that
such a test case will not execute since Paste is highlighted
only after a Cut or Copy. 1
In this paper, we define a new class of coverage criteria called
event-based coverage criteria to determine the adequacy of
tested event sequences, focusing on GUIs. The key idea is
to define the coverage of a test suite in terms of GUI events
and their interactions. Since the total number permutations
of event sequences in any non-trivial GUI is extremely
large, the GUI's hierarchical structure is exploited to identify
the important event sequences to be tested. The GUI
is decomposed into GUI components, 2 each of which is a
unit of testing. Events within a component do not interleave
with events in other components without explicit invocation
or termination events. Because of this well-defined
behavior, a component may be tested in isolation. Two
kinds of coverage criteria are developed from the decomposition
intra-component coverage criteria for events within a
component and inter-component coverage criteria for events
among components. Intra-component criteria include event,
event-selection, and length-n event-sequence coverage. Inter-component
criteria include invocation, invocation-termination
and length-n event-sequence coverage. A GUI component is
represented by a new structure called an event-flow graph
that identifies events within a component. The interactions
among GUI components are captured by a representation
called the integration tree. We present algorithms to
automatically construct event-flow graphs and the integra-
1 Note that Paste will be available if the ClipBoard is not
empty, perhaps because of an external software. External
software is ignored in this simplified example.
components should not be confused with GUI elements
that are used as building blocks during GUI develop-
ment. We later provide a formal definition of a GUI component

tion tree for a given GUI and to evaluate intra- and inter-component
coverage for a given test suite. We present a
case study to demonstrate the correlation between event-based
coverage of our version of WordPad's GUI and the
statement coverage of its underlying code for a test suite.
The important contributions of the coverage method presented
in this paper include:
1. a class of coverage criteria for GUI testing in terms of
GUI events.
2. the identification of a GUI component, useful for GUI
testing.
3. a representation of a GUI component called an event-
flow graph that captures the flow of events within a
component and a representation called the integration
tree to identify interactions between components.
4. an automated technique to decompose the GUI into
interacting components and coverage criteria for intra-component
and inter-component testing.
5. a technique to compute the coverage of a given test
suite.
6. a case study demonstrating the correlation between
coverage in terms of events and code.
In the next section we present a classification of GUI events
and use the classification to identify GUI components. In
Section 3 we present coverage criteria for event interactions
within a component and between components. Section 4
presents algorithms to construct event-flow graphs and an
integration tree for a given GUI and then evaluate intra- and
inter-component coverage of the GUI for a given test suite.
In Section 5, we present details of a case study conducted
on our version of the WordPad software. Lastly, Section 6
presents related work and in Section 7 we conclude with a
discussion of ongoing and future work.
2. STRUCTURE OF GUIS
A GUI uses one or more metaphors for objects familiar in
real life, such as buttons, menus, a desktop, and the view
through a window. The software user performs events to interact
with the GUI, manipulating GUI objects as one would
real objects. These events may cause deterministic changes
to the state of the software that may be reflected by a change
in the appearance of one or more GUI objects. Moreover,
GUIs, by their very nature, are hierarchical. This hierarchy
is reflected in the grouping of events in windows, dialogs,
and hierarchical menus. For example, all the "options" in
MS Internet Explorer can be set by interacting with events
in one window of the software's GUI.
The important characteristics of GUIs include their graphical
orientation, event-driven input, hierarchical structure,
the objects they contain, and the properties (attributes) of
those objects. Formally, a GUI may be defined as follows:
Graphical User Interface (GUI) is a hierar-
chical, graphical front-end to a software that accepts
user-generated and system-generated events, from a
fixed set of events, as input and produces deterministic
graphical output. A GUI contains graphical objects;
each object has a fixed set of properties. At any time
during the execution of the GUI, these properties have
discrete values, the set of which constitutes the state
of the GUI. 2
The above definition specifies a class of GUIs that have a
fixed set of events with deterministic outcome that can be
performed on objects with discrete valued properties. This
paper develops coverage criteria for the class of GUIs defined
above.
In this section, a new unit of testing called a GUI component
is defined. It consists of a number of events, selections,
invocations, and terminations that restrict the focus of a
GUI user. The user interacts with a component by explicitly
invoking it, performing events, and then terminating
the component. Note that since events within a component
cannot be interleaved with events in other components, the
interaction among events within a component may be tested
independently of other components. A classification of GUI
events is used to identify GUI components. GUI events may
be classified as:
Menu-open events open menus, i.e., they expand the set
of GUI events available to the user. By definition,
menu-open events do not interact with the underlying
software. The most common example of menu-open
events are generated by buttons that open pull-down
menus, e.g., File and Edit.
Restricted-focus events open modal windows, i.e., windows
that have the special property that once invoked,
they monopolize the GUI interaction, restricting the
focus of the user to a specific range of events within the
window until the window is explicitly terminated by a
termination event. Preference setting is an example
of restricted-focus events in many GUI systems; the
user clicks on Edit and Preferences, a window opens
and the user then spends time modifying the prefer-
ences, and finally explicitly terminates the interaction
by either clicking OK or Cancel.
Unrestricted-focus events open modeless windows that
do not restrict the user's focus; they merely expand
the set of GUI events available to the user. Note
that the only difference between menu-open events and
unrestricted-focus events is that the latter open windows
that have to be explicitly terminated. For exam-
ple, in the MS PowerPoint software, the Basic Shapes
are displayed in an unrestricted-focus window.
System-interaction events interact with the underlying
software to perform some action; common examples
include cutting and pasting text, and opening object
windows.
Termination events close modal windows; common examples
include Ok and Cancel.
At all times during interaction with the GUI, the user interacts
with events within a limited focus. This limited focus
consists of a restricted-focus window X and a set of
unrestricted-focus windows that have been invoked, either
directly or indirectly by X. The limited focus remains in
place until X is explicitly terminated using a termination
event such as OK, Cancel, or Close. Intuitively, the events
within the limited focus form a GUI component.
component C is an ordered pair (RF,
UF), where RF represents a modal window in terms
File
Edit
Help
Open Save
Cut Copy Paste
and Help

Figure

1: An Event-flow Graph for a Part of MS
WordPad.
of its events and UF is a set whose elements represent
modeless windows also in terms of their events. Each
element of UF is invoked either by an event in UF or
A common example of a GUI component is the FileOpen
modal window (and its associated modeless windows) found
in most of today's software. The user interacts with events
within this component, selects a file and terminates the component
by performing the Open event (or sometimes the
Cancel event).
Formally, a GUI component can be represented as a flow
graph.
An event-flow graph for a GUI component C is
a 4-tuple !V, E, B, I? where:
1. V is a set of vertices representing all the events in
the component. Each v 2V represents an event
in C.
2. is a set of directed edges between
vertices. We say that event e i follows e j iff e j
may be performed immediately after e i . An edge
the event represented by vy follows
the event represented by vx .
3. is a set of vertices representing those
events of C that are available to the user when
the component is first invoked.
4. I ' V is the set of restricted-focus events of the
component.An example of an event-flow graph for a part of the Maincomponent of MS WordPad is shown in Figure 1. At the
top are three vertices (File, Edit, and Help) that represent
part of the pull-down menu of MS WordPad. They are
menu-open events that are available when the Main component
is first invoked. Hence they form the set B. Once
File has been performed in WordPad any of Edit, Help,
Open, and Save may be performed. Hence there are edges
in the event-flow graph from File to each of these events.
Note that Open is shown with a dashed oval. We use this
representation for restricted-focus events, i.e., events that
invoke components. Similarly, About and Contents are also
3 We assume that all GUIs have a Main component, i.e., the
component that is presented to the user when the GUI is
first invoked.
Main
FileNew FileOpen Print FormatFont
Properties
FileSave PageSetup ViewOptions

Figure

2: An Integration Tree for a Part of MS
WordPad.
restricted-focus events, i.e., for this component, I = fOpen,
About, Contentsg. Other events (i.e., Save, Cut, Copy, and
Paste) are all system-interaction events. After any of these
is performed in MS WordPad, the user may perform File,
Edit, or Help, shown as edges in the event-flow graph.
Once all the components of the GUI have been represented
as event-flow graphs, the remaining step is to identify their
interactions. Testing interactions among components is also
an area of research in object-oriented software testing [5] and
inter-procedural data-flow testing [4]. The identification of
interactions among objects and procedures is aided by structures
such as function-decomposition trees and call-graphs
[4]. Similarly, we develop a structure to identify interactions
among components. We call this structure an integration
tree because it shows how the GUI components are
integrated to form the GUI. Formally, an integration tree is
defined as:
An integration tree is a 3-tuple
where N is the set of components in the GUI, R 2 N
is a designated component called the Main component.
We say that a component Cx invokes component Cy
if Cx contains a restricted-focus event ex that invokes
Cy . B is the set of directed edges showing the invokes
relation between components, i.e., (Cx ; Cy
invokes Cy . 2

Figure

2 shows an example of an integration tree representing
a part of the MS WordPad's GUI. The nodes represent
the components of the MS WordPad GUI and the edges
represent the invokes relationship between the components.
Main is the top-level component that is available when Word-
Pad is invoked. Other components' names indicate their
functionality. For example, FileOpen is the component of
WordPad used to open files. The tree in Figure 2 has an
edge from Main to FileOpen showing that Main contains an
event, namely Open (see Figure 1) that invokes FileOpen.
3. COVERAGE CRITERIA
Having created representations for GUI components and
events among components, we are ready to define the coverage
criteria. We will first define coverage criteria for events
within a component, i.e., intra-component coverage criteria
and then for events among components, i.e., inter-component
criteria.
3.1 Intra-component Coverage
In this section, we define several coverage criteria for events
and their interactions within a component. We first formally
define an event sequence.
An event-sequence is en ? where
All the new coverage criteria that we define next are based
on event-sequences.
3.1.1 Event Coverage
Intuitively, event coverage requires each event in the component
to be performed at least once. Such a requirement is
necessary to check whether each event executes as expected.
set P of event-sequences satisfies the event
coverage criterion if and only if for all events v 2 V,
there is at least one event-sequence
event v is in p. 2
3.1.2 Event-selection Coverage
Another important aspect of GUI testing is to check the
interactions among all possible pairs of events in the com-
ponent. However, we want to restrict the checks to pairs of
events that may be performed in a sequence. We focus on
the possible implicit selection of events that the user may
encounter during interaction with the GUI.
Definition: The event-selections for an event e is the set
In this criterion, we require that after an event e has been
performed, all event-selections of e should be executed at
least once. Note that this requirement is equivalent to requiring
that each element in E be covered by at least one
test case.
set P of event-sequences satisfies the event-
selection coverage criterion if and only if for all elements
there is at least one event-sequence
contains
3.1.3 Length-n Event-sequence Coverage
In certain cases, the behavior of events may change when
performed in different contexts. In such cases event coverage
and event-selection coverage on their own are weak
requirements for sufficient testing. We now define a criterion
that captures the contextual impact. We first formally
define a context.
Definition: The context of an event en in the event-sequence
Intuitively, the context for an event e is the sequence of
events performed before e. An event may be performed in
an infinite number of contexts. For finite applicability, we
define a limit on the length of the event-sequence. Hence,
we define the length-n event-sequence criterion.
set P of event-sequences satisfies the length-n
event-sequence coverage criterion if and only if P
contains all event-sequences of length equal to n. 2
Note the similarity of this criterion to the length-n path coverage
criterion defined by Gourlay for conventional software
[2], which requires coverage of all subpaths in the program's
flow-graph of length less than or equal to n. As the length
of the event-sequence increases, the number of possible contexts
also increases.
3.2 Subsumption
A coverage criterion C1 subsumes criterion C2 if every
test suite that satisfies C1 also satisfies C2 [13]. Since event
coverage and event-selection coverage are special cases of
length-n event-sequence coverage, i.e., length 1 event-sequence
and length 2 event-sequence coverage respectively, it follows
that length-n event-sequence coverage subsumes event and
event-selection coverage. Moreover, if a test suite satisfies
event-selection coverage, it must also satisfy event coverage.
Hence, event-selection subsumes event coverage.
3.3 Inter-component Criteria
The goal of inter-component coverage criteria is to ensure
that all interactions among components are tested. In GUIs,
the interactions take the form of invocation of components,
termination of components, and event-sequences that start
with an event in one component and end with an event in
another component.
3.3.1 Invocation Coverage
Intuitively, invocation coverage requires that each restricted-
focus event in the GUI be performed at least once. Such a
requirement is necessary to check whether each component
can be invoked.
set P of event-sequences satisfies the invocation
coverage criterion if and only if for all restricted-
focus events i 2 I, where I is the set of all restricted-
focus events in the GUI, there is at least one event-
sequence event i is in p. 2
Note that event coverage subsumes invocation coverage since
it requires that all events be performed at least once, including
restricted-focus events.
3.3.2 Invocation-termination Coverage
It is important to check whether a component can be invoked
and terminated.
Definition: The invocation-termination set IT of a GUI
is the set of all possible length 2 event sequences !
component Cx and e j terminates
component Cx , for all components Cx 2 N .Intuitively, the invocation-termination coverage requires that
all length 2 event sequences consisting of a restricted-focus
event followed by the invoked component's termination events
be tested.
set P of event-sequences satisfies the invocation-
termination coverage criterion if and only if for all i 2
IT , there is at least one event-sequence
that i is in p. 2
Satisfying the invocation-termination coverage criterion assures
that each component is invoked at least once and then
terminated immediately, if allowed by the GUI's specifica-
tions. For example, in WordPad, the component FileOpen
is invoked by the event Open and terminated by either Open
or Cancel. Note that WordPad's specification do not allow
Open to terminate the component unless a file has been se-
lected. On the other hand, Cancel can always be used to
terminate the component.
v: Vertex or Event)f 1
system-interaction 7
return(B of Invoking
Invoked component); 12
return(B of Invoked component); 14

Figure

3: Computing follow set(v) for a Vertex v.
3.3.3 Inter-component Length-n Event-sequence Cov-
erage
Finally, the inter-component length-n event-sequence coverage
criterion requires testing all event-sequences that start
with an event in one component and end with an event in
another component. Note that such an event-sequence may
use events from a number of components. A criterion is
defined to cover all such interactions.
set P of event-sequences satisfies the inter-component
length-n event-sequence coverage criterion
for components C1 and C2 if and only if P contains
all length-n event-sequences
that
may belong to C1 or C2 or any other
component C i . 2
Note that the inter-component length-n event-sequence coverage
subsumes invocation-termination coverage since length-n
event sequences also include length 2 sequences.
4. EVALUATING COVERAGE
Having formally presented intra- and inter-component coverage
criteria, we now present algorithms to evaluate the
coverage of a test suite using these criteria. In this section,
we present algorithms to evaluate the coverage of the GUI
for a given test suite. We show how to construct an event-
flow graph and use it to evaluate intra-component coverage.
Then we show how to construct an integration tree and use
it to evaluate inter-component coverage.
4.1 Construction of Event-flow Graphs
The construction of event-flow graphs is based on the structure
of the GUI. The classification of events in the previous
section aids the automated construction of the event-flow
graphs, which we describe next.
For each v 2 V, we define follow set(v) as the set of all
events vx such that vx follows v. Note that follow set(v)
is the set of outgoing edges in the event-flow graph. We
determine follow set(v) using the algorithm in Figure 3
for each vertex v. The recursive algorithm contains a switch
structure that assigns follow set(v) according to the type
of each event. If the type of the event v is a menu-open event
(line represents events that are
available when a component is invoked) then the user may
either perform v again, its sub-menu choices, or any event
in B (line 4). However, if v 62 B then the user may either
perform all sub-menu choices of v, v itself, or all events in
follow set(parent(v)) (line 6). We define parent(v) as
any event that makes v available. If v is a system-interaction
event, then after performing v, the GUI reverts back to the
events in B (line 8). If v is an exit event, i.e., an event that
terminates a component, then follow set(v) consists of all
the top-level events of the invoking component (line 10).
If the event type of v is an unrestricted-focus event then
the available events are all top-level events of the invoked
component available as well as all events of the invoking
component (line 12). Lastly, if v is a restricted-focus event,
then only the events of the invoked component are available.
4.2 Evaluating Intra-component Coverage
Having constructed an event-flow graph, we are now ready
to evaluate the intra-component coverage of any given test
suite using the elements of this graph. Figure 4 shows a
dynamic programming algorithm to compute the percentage
of length-n event-sequences tested. The final result of the
algorithm is Matrix, where Matrix i;j is the percentage of
length-j event-sequences tested on component i.
The main algorithm is ComputePercentageTested. In this
algorithm, two matrices are computed (line 6,7). Count i;j
is the number of length-j event-sequences in component i
that have been covered by the test suite T (line 6). Total i;j
is the total number of all possible length-j event-sequences
in component i (line 7). The subroutine ComputeCounts
calculates the elements in count matrix. For each test
case in T, ComputeCounts finds all possible event-sequences
of different lengths (line 19.21). The number of event-
sequences of each length are counted in (lines 22, 23). In-
tuitively, the ComputeTotals subroutine starts with single-length
event-sequences, i.e., individual events in the GUI
(lines 31.33). Using follow set (line 38), the event-
sequences are lengthened one event at each step. A counter
keeps track of the number of event-sequences created (line
39). Note that since ComputeCounts takes a union of the
event sequences, there is no danger of counting the same
event sequence twice.
The result of the algorithm is Matrix, the entries of which
can be interpreted as follows:
Event Coverage requires that individual events in the GUI
be exercised. These individual events correspond to
length 1 event-sequences in the GUI. Matrix j;1 j 2 S
represents the percentage of individual events covered
in each component.
Event-selection Coverage requires that all the edges of
the event-flow graph be covered by at least one test
case. Each edge is effectively captured as a length-2
event-sequence. Matrix j;2 j 2 S represents the percentage
of branches covered in each component j.
Length-n Event-sequence Coverage is available directly
from Matrix. Each column i of Matrix represents the
number of length-i event-sequences in the GUI.
S: Set of Components; 2
T: Test
M: Maximum Event-sequence Length) 4
fcount /\Gamma ComputeCounts(T, S, M); 5,6
count i;j is the tested number
of length-j event-sequences in component i */
total
/* total i;j is the total number
of length-j event-sequences in component i */
Matrix i;j /\Gamma (count i;j /total i;j ) \Theta 100; 10
return(Matrix)g 11
T: Test
M: Maximum Event-sequence Length) 14
A /\Gamma fg; /* Empty Set */ 17
A
/* count number of sets of length j */
count i;j /\Gamma NumberOfSetsOfLength(S, j); 23
return(count)g
S: Set of Components; 26
M: Maximum Event-sequence Length) 27
28
x
total j;k /\Gamma total j;k
newfreq i /\Gamma 0; 44
return(total)g

Figure

4: Computing Percentage of Tested Length-n
Event-sequences of All Components.
4.3 Evaluating Inter-component Coverage
Once all the components in the GUI have been identified,
the integration tree may be constructed by adding, for each
restricted-focus event ex , the element (Cx ; Cy ) to B where
Cx is the component that contains ex and Cy is the component
that it invokes. The integration tree may be used in
several ways to identify interactions among components. For
example, in Figure 2 a subset of all possible pairs of components
that interact would be f (Main, FileNew), (Main,
FileOpen), (Main, Print), (Main, FormatFont), and (Print,
g. To identify sequences such as the ones from
Main to Properties, we traverse the integration tree in a
bottom-up manner, identifying interactions among Print
and Properties. We then merge Print and Properties to
form a super-component called PrintProperties. We then
check interactions among Main and PrintProperties. This
process continues until all components have been merged
into a single super-component.
Evaluating the inter-component coverage of a given test suite
requires computing the (1) invocation coverage, (2) invocation-
termination coverage, and (3) length-n event sequence cover-
age. The total number of length 1 event sequences required
to satisfy the invocation coverage criterion is equal to the
number of restricted-focus events available in the GUI. The
percentage of restricted-focus events actually covered by the
test cases is (x=I) \Theta 100, where x is the number of restricted-
focus events in the test cases, and I is the total number of
restricted-focus events available in the GUI. Similarly, the
total number of length 2 event sequences required to satisfy
the invocation-termination criterion is
and T i are the number of restricted-focus and termination
events that invoke and terminate component C i respectively.
The percentage of invocation-termination pairs actually covered
by the test cases is (x=
where x is the
number of invocation-termination pairs in the test cases.
Computing the percentage of length-n event sequences is
slightly more involved. The algorithm shown in Figure 5
computes the percentage of length-n event sequences tested
among GUI components. Intuitively, the algorithm obtains
the number of event sequences that end at a certain restricted-
focus event. It then counts the number of event sequences
that can be extended from these sequences into the invoked
component. The main algorithm called Integrate is recursive
and performs a bottom-up traversal of the integration
tree T (line 2). Other than the recursive call (line 8),
Integrate makes a call to ComputeTotalInteractions that
takes two components as parameters (lines 13,14). It initializes
the vector Total for all path lengths i (1  i
M) (line 16,17). We assume that a freq matrix has been
stored for each component. The freq matrix is similar to
the freq vector already computed in the algorithm in Figure
4. freq i;j is the number of event-sequences that start
with event i and end with event j. After obtaining both
frequency matrices for both C1 and C2 , for all path lengths
(lines 21,26), the new vector Total is obtained by adding
the frequency entries from F1 and F2 (lines 28.30). A
new frequency matrix is computed for the super-component
This new frequency matrix will be utilized
by the same algorithm to integrate "C1C2 " to other
components.
The results of the above algorithm are summarized in Ma-
trix. Matrix i;j is the percentage of length-j event-sequences
that have been tested in the super-component represented
by the label i.
5. CASE STUDY
We performed a case study on our version of WordPad to
determine the (1) total number of event sequences required
T: Integration Tree) 2
ComputeTotalInteractions(newT, c); 9
MatrixnewT+c
Component
Component
Total
x
/* get freq table of C1 for event-seq of length i */ 20
/* Add all values in column x */ 22
/* get freq table of C2 for event-seq of length j */ 25
28
Total i+j /\Gamma Total i+j
ComputeFreqMatrix(C1 , C2 ); 31

Figure

5: Computing Percentage of Tested Length-n
Event-sequences of All Components.
to test the GUI and hence enable a test designer to compute
the percentage of event sequences tested, (2) correlation between
event-based coverage of the GUI and statement coverage
of the underlying code, and (3) time taken to evaluate
the coverage of a given test suite and usefulness of the coverage
report to guide further testing.
In the case study, we employed our own specifications and
implementation of the WordPad software. The software consists
of 36 modal windows, and 362 events (not counting
short-cuts). Our implementation of WordPad is similar to
Microsoft's WordPad except for the Help menu, which we
did not model.
5.1 ComputingTotalNumber of Event-sequences
for WordPad
In this case study, we wanted to determine the total number
of event sequences that our new criteria specify to test parts
of WordPad. We performed the following steps:
Components and Events: Individual Word-
Pad components and events within each component
were identified. Table 1 shows some of the components
of WordPad that we used in our case study. Each row
represents a component and each column shows the
Component
Name
Open
System
Interaction
Restricted
Focus
Unrestricted
Focus Termination Sum
Main 7
FileOpen
FileSave
Properties
Sum 7 78
Event Type

Table

1: Types of Events in Some Components of
MS WordPad.
different types of events available within each component

Creating Event-flow Graphs: The next step was to construct
an event-flow graph for each component. In

Figure

1 we showed a part of the event-flow graph of
the most important component, i.e., Main. Recall that
each node in the event-flow graph represents an event.
Computing Event-sequences: Once the event-flow graphs
were available, we computed the total number of possible
event-sequences of different lengths in each component
by using the computeTotals subroutine in Figure
4. Note that these event-sequences may also include
infeasible event-sequences. The total number of
event-sequences is shown in Table 2. The rows represent
the components and the shaded rows represent
the inter-component interactions. The columns represent
different event-sequence lengths. Recall that
an event-sequence of length 1 represents event coverage
whereas an event-sequence of length 2 represents
event-selection coverage. The columns 1' and 2' represent
invocation and invocation-termination coverage
respectively.
The results of this case study show that the total number
of event sequences grows with increasing length. Note that
longer sequences subsume shorter sequences; e.g., if all event
sequences of length 5 are tested, then so are all sequences of
length-i, where 4. It is difficult to determine the maximum
length of event sequences needed to test a GUI. The
large number of event sequences show that it is impractical
to test a GUI for all possible event sequences. Rather, depending
on the resources, a subset of "important" event sequences
should be identified, generated and executed. Identifying
such important sequences requires that they be ordered
by assigning a priority to each event sequence. For
example, event sequences that are performed in the Main
component may be given higher priority since they will be
used more frequently; all the users start interacting with
the GUI using the Main component. The components that
are deepest in the integration tree may be used the least.
This observation leads to a heuristic for ordering the testing
of event sequences within components of the GUI. The
structure of the integration tree may be used to assign priorities
to components; Main will have the highest priority, decreasing
for components at the second level, with the deepest
components having the lowest priority. A large number
Component Name 1'
Main 56 791 14354 255720 4490626 78385288
FileOpen
FileSave
Print 12 108 972 8748 78732 708588
Properties 13 143 1573 17303 190333 2093663
PageSetup 11 88 704 5632 45056 360448
FormatFont 9 63 441 3087 21609 151263
Print+Properties
Main+FileSave
Main+FormatFont
Main+Print+Properties 12 145 1930 28987 466578
Event-sequence Length

Table

2: Total Number of Event-sequences for Selected
Components of WordPad. Shaded Rows
Show Number of Interactions Among Components.
of event sequences in the high priority components may be
tested first; the number will decrease for low priority components

5.2 Correlation Between Event-based Coverage
and Statement Coverage
In this case study, we wanted to determine exactly which
percentage of the underlying code is executed when event-
sequences of increasing length are executed on the GUI. We
wanted to see whether testing longer sequences adds to the
coverage of the underlying code. We performed the following
steps:
Code Instrumentation: We instrumented the underlying
code of WordPad to produce a statement trace, i.e., a
sequence of statements in the order in which they are
executed. Examining such a trace allowed us to determine
which statements are executed by a test case.
Event-sequence Generation: We wanted to generate all
event-sequences up to a specific length. We modified
ComputeTotals in Figure 4 resulting in an event-
sequence generation algorithm that constructs event
sequences of increasing length. The dynamic programming
algorithm constructs all event sequences of length
1. It then uses follow set to extend each event sequence
by one event, hence creating all length 2 event-
sequences. We generated all event-sequences up to
length 3. In all we obtained 21659 event-sequences.
Controlling GUI's State: Bringing a software to a state
S i in which a test case T i may be executed on it is
traditionally known as the controllability problem [1].
This problem also occurs in GUIs and for each test
case, appropriate events may need to be performed
on the GUI to bring it to the state S i . We call this
sequence of events the prefix, P i , of the test case. Although
generating the prefix in general may require the
development of expensive solutions, we used a heuristic
for this study. We executed each test case in a
fixed state S0 in which WordPad contains text, part
of the text was highlighted, the clipboard contains a
text object, and the file system contains two text files.
We traversed the event-flow graphs and the integration
tree to produce the prefix of each test case. We
do, however, note that using this heuristic may render
Event-sequence Length
Percentage
of
Statements
Executed

Figure

The Correlation Between Event-based
Coverage and Statement Coverage of WordPad.
some of the event sequences non-executable because of
infeasibility. We will later see that such sequences do
exist but are of no consequence to the results of this
study. We have modified WordPad so that no statement
trace is produced for P i .
Test-case Execution: After all event-sequences up to length
3 were obtained, we executed them on the GUI using
our automated test executor [10] and obtained all
the execution traces. The test case executor executed
without any intervention for hours. We note that
(or 19.3%) of the test cases could not be executed
because of infeasibility.
Analysis: In analyzing the traces for our study, we determined
the new statements executed by event-sequences
of length 1, i.e., individual events. The graph in Figure
6 shows that almost 92% of the statements were
executed by just these individual events. As the length
of the event sequences increases, very few new statements
are executed (5%). Hence, a high statement
coverage of the underlying code may be obtained by
executing short event sequences.
The results of this case study can be explained in terms of
the design of the WordPad GUI. Since the GUI is an event-driven
software, a method called an event handler is implemented
for each event. Executing an event caused the execution
of its corresponding event handler. Code inspection
of the WordPad implementation revealed that there were
few or no branch statements in the code of the event han-
dler. Consequently, when an event was performed, most of
the statements in the event-handler were executed. Hence
high statement coverage was obtained by just performing
individual events. Whether other GUIs exhibit similar behavior
requires a detailed analysis of a number of GUIs and
their underlying code.
The result shows that statement coverage of the underlying
code can be a misleading coverage criterion for GUI test-
ing. A test designer who relies on statement coverage of
the underlying code for GUI testing may test only short
event sequences. However, testing only short sequences is
not enough. Longer event sequences lead to different states
of the GUI and that testing these sequences may help detect
a larger number of faults than short event sequences.
Component Name 1'
Main
FileOpen 9
FileSave 9 33 132
Print 11 37 313 787 3085 1314
Properties 12
Print+Properties
Main+FileSave
Main+Print+Properties 6 56 123 189 423
Event-sequence Length

Table

3: The Number of Event-sequences for Selected
Components of WordPad Covered by the Test
Cases.
For example, in WordPad, the event Find Next (obtained
by clicking on the Edit menu) can only be executed after at
least 6 events have been performed; the shortest sequence
of events needed to execute Find Next is !Edit, Find,
TypeInText, FindNext2, OK, Edit, Find Next?, which has
7 events. If only short sequences (! are executed on the
GUI, a bug in Find Next may not be detected. Extensive
studies of the fault-detection capabilities of executing short
and long event sequences for GUI testing are needed, and is
targeted for future work. Another possible extension to this
experiment is to determine the correlation between event-based
coverage and other code-based coverage, e.g., branch
coverage.
5.3 Evaluating the Coverage of a Test Suite
We wanted to determine the time taken to evaluate the
coverage of a given test suite and how the resulting coverage
report could guide further testing. We used our earlier
developed planning-based test case generation system
called Planning Assisted Tester for grapHical user interface
Systems(PATHS) to generate test cases [8]. We performed
the following steps:
Tasks: In PATHS, commonly used tasks were
identified. A task is an activity to be performed by
using the events in the GUI. In PATHS, the test designer
inputs tasks as pairs (I, G), where I is the initial
GUI state before the task is performed and G is
the final GUI state after the task has been performed.
We carefully identified 72 different tasks, making sure
that each task exercised at least one unique feature of
WordPad. For example, in one task we modified the
font of text, in another we printed the document on
size paper.
Generating Test Cases: Multiple test cases were generated
using a plan generation system to achieve these
tasks. In this manner, we generated 500 test cases
(multiple cases for each task).
Coverage Evaluation: After the test cases were available,
we executed the algorithms of Figures 4 and 5. The algorithms
were implemented using Perl and Mathematica
[19] and were executed on a Sun UltraSPARC workstation
(Sparc Ultra running SunOS 5.5.1. Even
Component Name 1'
Main
FileOpen 90 56 17.50 0.72 0.06
FileSave 90 41 20.63 1.27 0.47 0.02
Print
Properties
PageSetup 91
FormatFont
Print+Properties 100 0 46 51.15 8.18 3.87
Main+FileOpen
Main+FileSave 100 0 20 13.00 8.64 1.26 0.28
Main+FormatFont 100 0 33 28.40 5.17 0.97 0.10
Main+Print+Properties 50 38.62 6.37 0.65 0.09
Event-sequence Length

Table

4: The Percentage of Total Event-sequences
for Selected Components of WordPad Covered by
the Test Cases.
with the inefficiencies inherent in the Perl and Mathematica
implementation, we could process the 500 test
cases in 47 minutes (clock time). The results of applying
the algorithms are summarized as coverage reports
in

Tables

and 4. Table 3 shows the actual number
of event-sequences that the test cases covered. Table 4
presents the same data, but as a percentage of the total
number of event-sequences. Column 1 in Table 4
shows close to 90% event coverage. The remaining 10%
of the events (such as Cancel) were never used by the
planner since they did not contribute to a goal. Column
2 shows the event-selection coverage and the test
cases achieved 40-55% coverage. Note that since all
the components were invoked at least once, 100% invocation
coverage (column 1') was obtained. However,
none of the components were terminated immediately
after being invoked. Hence, no invocation-termination
coverage (column 2') was obtained.
This result shows that the coverage of a large test suite can
be obtained in a reasonable amount of time. Looking at
columns 4, 5, and 6 of Table 4, we note that only a small
percentage of length 4, 5, and 6 event sequences were tested.
The test designer can evaluate the importance of testing
these longer sequences and perform additional testing. Also,
the two-dimensional structure of Table 4 helps target specific
components and component-interactions. For example,
60% of of length 2 interactions among Main and PageSetup
have been tested whereas only 11% of the interactions among
Main and FileOpen have been tested. Depending on the relative
importance of these components and their interactions,
the test designer can focus on testing these specific parts of
the GUI.
6. RELATED WORK
Very little research has been reported on developing coverage
criteria for GUI. The only exception is the work by Ostrand
et al. who briefly indicate that a model-based method may
be useful for improving the coverage of a test suite [12].
However, they have deferred a detailed study of the coverage
of the generated test cases using this type of GUI model to
future work.
There is a close relationship between test-case generation
techniques and the underlying coverage criteria used. Much
of the literature on GUI test case generation focuses on describing
the algorithms used to generate the test cases [14,
6]. Little or no discussion about the underlying coverage
criteria is presented. In the next few paragraphs, we
present a discussion of some of the methods used to develop
test cases for GUIs and their underlying coverage criteria.
We also present a discussion of automated test case generation
techniques that offer a unique perspective of GUI
coverage.
The most commonly available tools to aid the test designer
in the GUI testing process include record/playback tools
[15, 3]. These tools record the user events and GUI screens
during an interactive session. The recorded sessions are later
played back whenever it is necessary to generate the same
GUI events. Record/playback tools provide no functionality
to evaluate the coverage of a test suite. The primary reason
for no coverage support is that these tools lack a global
view of the GUI. The test cases are constructed individually
with a local perspective. Several attempts have been made
to provide more sophisticated tools for GUI testing. One
popular technique is programming the test case generator
[7]. The test designer develops programs to generate test
cases for a GUI. The use of loops, conditionals, and data
selection switches in the test case generation program gives
the test designer a broader view of the generated test cases'
coverage.
Several finite-state machine (FSM) models have also been
proposed to generate test cases [14]. Once an FSM is built,
coverage of a test suite is evaluated by the number of states
visited by the test case. This method of evaluating coverage
of a test suite needs to be studied further as an accurate
representation of the GUI's navigation results in an infinite
number of states.
White et al. presents a new test case generation technique
for GUIs [18]. The test designer/expert manually identifies
a responsibility, i.e., a GUI activity. For each responsibility,
a machine model called the complete interaction sequence
(CIS) is identified manually. To reduce the size of the test
suite, the CIS is reduced using constructs/patterns in the
CIS. The example presented therein showed that testing
could be performed by using 8 test cases instead of 48. How-
ever, there is no discussion of why no loss of coverage will
occur during this reduction. Moreover, further loss of coverage
may occur in identifying responsibilities and creating
the CIS. The merit of the technique will perhaps be clearer
when interactions between the CIS are investigated.
7. CONCLUSION
In this paper, we presented new coverage criteria for GUI
testing based on GUI events and their interactions. A unit of
testing called a GUI component was defined. We identified
the events within each component and represented them as
an event-flow graph. Three new coverage criteria were de-
fined: event, event-selection, and length-n event-sequence
coverage. We defined an integration tree to identify events
among components and defined three inter-component coverage
criteria: invocation, invocation-termination and inter-component
length-n event-sequence coverage.
In the future we plan to examine the effects of the GUI's
structure on its testability. As GUIs become more struc-
tured, the integration tree becomes more complex and inter-component
testing becomes more important.
We also plan to explore the possibility of using the event-based
coverage criteria for software other than GUIs. We
foresee the use of these criteria for (1) object-oriented soft-
ware, which use messages/events for communication among
objects, (2) networking software, which use messages for
communication, and (3) the broader class of reactive soft-
ware, which responds to events.
8.



--R

A framework for testing database applications.
A mathematical framework for the investigation of testing.
Integrated data capture and analysis tools for research and testing an graphical user interfaces.
Interprocedual data flow testing.

Toward automatic generation of novice user test scripts.
The black art of GUI testing.
Using a goal-driven approach to generate test cases for GUIs
Automated test oracles for GUIs.
A planning-based approach to GUI testing
Hierarchical GUI test case generation using automated planning.
A visual test development environment for GUI systems.
Selecting software test data using data flow information.
A method to automate user interface testing using variable finite state machines.

The applicability of program schema results to programs.
Translatability and decidability questions for restricted classes of program schemas.
Generating test cases for GUI responsibilities using complete interaction sequences.
A System for Doing Mathematics by Computer.
Test data adequacy measurements.
--TR
Selecting software test data using data flow information
Mathematica: a system for doing mathematics by computer
Integrated data capture and analysis tools for research and testing on graphical user interfaces
Test data adequacy measurement
Object-oriented integration testing
Toward automatic generation of novice user test scripts
A visual test development environment for GUI systems
Using a goal-driven approach to generate test cases for GUIs
A framework for testing database applications
Automated test oracles for GUIs
Hierarchical GUI Test Case Generation Using Automated Planning
A Method to Automate User Interface Testing Using Variable Finite State Machines
Generating Test Cases for GUI Responsibilities Using Complete Interaction Sequences

--CTR
Yanhong Sun , Edward L. Jones, Specification-driven automated testing of GUI-based Java programs, Proceedings of the 42nd annual Southeast regional conference, April 02-03, 2004, Huntsville, Alabama
Aine Mitchell , James F. Power, An approach to quantifying the run-time behaviour of Java GUI applications, Proceedings of the winter international synposium on Information and communication technologies, January 05-08, 2004, Cancun, Mexico
Philippe Palanque , Regina Bernhaupt , Ronald Boring , Chris Johnson, Testing Interactive Software: A Challenge for Usability and Reliability, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montral, Qubec, Canada
Ping Li , Toan Huynh , Marek Reformat , James Miller, A practical approach to testing GUI systems, Empirical Software Engineering, v.12 n.4, p.331-357, August    2007
Christopher J. Howell , Gregory M. Kapfhammer , Robert S. Roos, An examination of the run-time performance of GUI creation frameworks, Proceedings of the 2nd international conference on Principles and practice of programming in Java, June 16-18, 2003, Kilkenny City, Ireland
Geoffrey R. Gray , Colin A. Higgins, An introspective approach to marking graphical user interfaces, ACM SIGCSE Bulletin, v.38 n.3, September 2006
Atif M. Memon , Mary Lou Soffa, Regression testing of GUIs, ACM SIGSOFT Software Engineering Notes, v.28 n.5, September
Mikael Lindvall , Ioana Rus , Paolo Donzelli , Atif Memon , Marvin Zelkowitz , Aysu Betin-Can , Tevfik Bultan , Chris Ackermann , Bettina Anders , Sima Asgari , Victor Basili , Lorin Hochstein , Jrg Fellmann , Forrest Shull , Roseanne Tvedt , Daniel Pech , Daniel Hirschbach, Experimenting with software testbeds for evaluating new technologies, Empirical Software Engineering, v.12 n.4, p.417-444, August    2007
Qing Xie , Atif M. Memon, Designing and comparing automated test oracles for GUI-based software applications, ACM Transactions on Software Engineering and Methodology (TOSEM), v.16 n.1, p.4-es, February 2007
Atif Memon , Adithya Nagarajan , Qing Xie, Automating regression testing for evolving GUI software: Research Articles, Journal of Software Maintenance and Evolution: Research and Practice, v.17 n.1, p.27-64, January 2005
Atif Memon , Adithya Nagarajan , Qing Xie, Automating regression testing for evolving GUI software, Journal of Software Maintenance: Research and Practice, v.17 n.1, p.27-64, January 2005
component testing;integration tree;event-flow graph;event-based coverage;GUI testing;GUI test coverage