Further Experiences with Scenarios and Checklists.
Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. This replication is broadly supportive of the results from the original experiment, namely, that the Scenario approach is superior to the Checklist approach; and that the meeting component of a software inspection is not an effective defect detection mechanism. This experiment also tentatively proposes additional relationships between general academic performance and individual inspection performance; and between meeting loss and group inspection performance.
Introduction
Software inspection is a method for statically verifying documents. It was first
described by Michael Fagan in 1976 [6]. Since then there have been many variations
and experiences described, but a typical inspection involves a team of
three to five people reviewing and understanding a document to find defects.
The benefits of inspection are generally accepted, with success stories regularly
published. In addition to Fagan's papers describing his experiences [6, 7], there
are many other favourable reports. For example, Doolan [4] reports industrial
experience indicating a times return on investment for every hour devoted to
inspection of software requirement specifications. Russell [20] reports a similar
return of 33 hours of maintenance saved for every hour of inspection invested.
This benefit is derived from applying inspection early in the lifecycle. By inspecting
products as early as possible, major defects are caught sooner and are
not propagated through to the final product, where the cost of removal is far
greater.
One point of disagreement between inspection advocates is the role of detection
aids. Many practitioners do not use any (The Ad Hoc approach), while
others use Checklists to aid and regularise their process. Recently, Parnas and
Weiss[16] have argued that even Checklists are not sufficiently focused to reliably
aid inspectors, and have suggested that inspectors should be allocated
specific responsibilities. Parnas argues that the Checklist approach results in
an inspector repeatedly covering the same ground, while ignoring other ar-
eas, resulting in reduced coverage. This idea has been further developed by
19], who has developed the Scenario approach in an attempt to
address this problem. As well as developing the idea, Porter has attempted to
show, via subject-based experimentation, that Scenarios are indeed superior to
both Ad Hoc and Checklist approaches.
To prove any hypothesis, results of experimental studies must be reproducible
and hence, much to their credit, Porter et al. have made their experimental
material available. This paper reports on our experiences and results in
undertaking a partial replication of Porter's experiment. It will also draw upon
experiences and results from another partial replication undertaken by Lanubile
and Visaggio[13]. The paper will attempt to 'mirror' the organisation of
Porter et al.[19] for ease of comparison. Hence the hypothesis of this replicated
experiment is the same as the hypothesis in the original experiment:
. that nonsystematic techniques with general reviewer responsibility
and no reviewer coordination, lead to overlap and gaps, thereby
lowering the overall inspection effectiveness; while systematic approaches
with specific coordinated responsibilities reduce gaps, thereby
increasing the overall effectiveness of the inspection. (Porter et
2 The experiment
To further evaluate this hypothesis, the authors have replicated the original
experiment. As with all subject-based experiments the design of Porter's experiment
can be criticised, and hence a modified experimental design was undertaken
to explore the hypothesis from a slightly different viewpoint. The alterations
to the experiment will be discussed as they arise within the paper. As
with the original experiment, the study uses students taking a formal university
course 1 and commenced with a training phase in which the subjects were taught
inspection techniques and the experimental procedures, during which they inspected
two sample software requirements specification documents 2 ; this was
followed by the experimental phase.
2.1 Experimental design
This partial replication will focus on only 2 detection methods, Checklists (non-
systematic) and Scenarios (systematic). This will simplify the design allowing
greater focusing at the cost of being unable to generalise any results to include
the Ad Hoc approach (nonsystematic), and hence our real hypothesis is less
generic than the original experiment.
1 The original experiment used graduate students, this study used undergraduate students
(3rd Year).
2 The original only used one sample document. All the documents inspected in all three
experiments are software requirements specification documents.
Variables
The experiment manipulates three independent variables 3 :
ffl The detection technique - Checklist or Scenario (Treatment Variable).
ffl The specification to be inspected - two were used during the experiment
(CRUISE, WLMS - see Section 2.2 for a brief description of these specifications

ffl The subjects or groups of subjects - 50 subjects organised into 16 groups.
The original experiment included another independent variable, the order in
which the specifications were inspected. It was decided that it was unfeasible to
allow variation in the order in which the specifications were inspected; this was
because the inspections were part of an assessed class and hence the possibility
of subject plagiarism corrupting the experimental data could not be discounted.
The original experiment included this variation, but does not discuss how it
avoids threats from plagiarism.
The original experiment also measured four dependent variables:
ffl The individual fault detection rate,
ffl The team fault detection rate,
ffl The percentage of faults first identified at the collection meeting (meeting
gain rate), and
ffl The percentage of faults first identified by an individual, but never reported
at the collection meeting (meeting loss rate).
The replication also measured the above variables, plus 3 additional variables

ffl The time spent by each participant for the individual inspection,
ffl The time spent by each group for the collection meeting, and
ffl Average academic performance of each subject.
Also the replication collected additional data from 4 debriefing questionnaires
per specification - one individual and one group).
Design
Unfortunately the traditional fractional factorial design is unsuitable for this
experiment because, as stated by Porter[19]:
3 The original experiment included an internal replication and thus included another independent
variable.
Round 1 Round 2
WLMS CRUISE WLMS CRUISE
Ad Hoc 1B, 1D, 1G 1A, 1C, 1E 1A 1D, 2B
1H, 2A 1F, 2D
Checklist 2B 2E, 2G 1E, 2D, 2G 1B, 1H
Scenario 2C, 2F 2H 1F, 1C, 2E 1G, 2A, 2C

Table

1: Subject allocation in the original experiment
. they require some teams to use the Checklist method after they
have used the Scenario method. Since the Checklist reviewers create
their own fault detection techniques during the inspection (based on
their understanding of the Checklist), our concern was that using
the Scenario method in an early round might imperceptibly distort
the use of the other methods in later rounds. Such influences would
be undetectable because, unlike the Scenario method, the Checklist
method does not require reviewers to perform specific, auditable
tasks.
Hence it was decided that a very straightforward design was the only alternative
available. Each subject would participate in two inspections using the
same technique. This design suffers from the problem that any ability effect
between the two groups can distort any results applied to the treatment vari-
able. Unfortunately, we have no direct way of measuring the subject's ability
at software inspection, and hence it was decided that a random allocation of
subjects to each group was the best alternative. In an attempt to estimate any
effect, the average academic performance for each group was calculated. This
indicated that the groups were extremely well balanced in terms of academic
performance further post-analysis has revealed
a strong linear association between average academic performance and average
inspection performance (Pearson, hence it is believed
that the experimental design has had little or no impact on the results
affecting the treatment variable.
The original experiment used a partial factorial design in which each group
participated in two inspections, using some combination of the three detection
methods, provided the above rule is not broken. Table 1 details the group
allocation (A-H), where a 1 denotes the first run of the experiment and a 2
denotes the second run (or internal replication).
Threats to internal validity
All experiments suffer from the problem that some factor may affect the dependent
variables without the experimenter's knowledge. This possibility must be
minimised. In the original experiment Porter discusses five such threats:
ffl Selection Effects
The replication's approach is described in the previous section. The original
experiment employed the following allocation strategy, in conjunction
with the partial factorial design detailed above:
. Create teams with equal skills. For example, rate each par-
ticipant's background knowledge and experience as either low,
medium, or high and then form teams of three by selecting one
individual at random from each experience category.
ffl Maturation Effects
In the replication, each subject uses the same detection method through-out
the experiment and the order of attempting the specifications is fixed,
hence any learning effect should either be symmetrical across the two
groups (i.e. derived from a common source, e.g. experimentation proce-
dure) or directly related to the treatment variable (i.e. increased understanding
of, or ability with, the detection method). The situation with
the original experiment is more complicated, due to the asymmetrical allocation
of subjects and the variation in the order of both applying the
detection methods and inspecting the specifications, see Table 1.
ffl Replication Effects
Since this external replication does not contain an internal replication
component, this potential threat is of no concern.
ffl Instrumentation Effects
This variation is impossible to avoid and is controlled by having each
subject inspect both specifications.
ffl Presentation Effects
In the replication, the CRUISE specification is presented before the WLMS
specification. Hence it is possible that the subjects can apply knowledge
from their CRUISE inspection experience to their WLMS inspection. (See
below for an explanation of why a fixed order of presentation was chosen
- Plagiarism.) It is believed that any such effect should be symmetric
between the groups and presents a far lesser risk than the risk of plagia-
rism. The original experiment varies the order to control this effect, as
described in Table 1.
In considering the experiment and conducting initial trials with the experimental
material, it was found that a larger list of internal threats exist than
discussed in the original.
ffl Plagiarism
It was decided that it was unfeasible to allow variation in the order in
which the specifications were inspected; this was because the inspections
were part of an assessed class and hence the possibility of subject plagiarism
corrupting the experimental data could not be discounted and was
considered to be a greater risk than any asymmetric maturation effects.
ffl Time Effects
The original experiment allocated two hours for an individual inspection.
It is believed that this fixed time period could have introduced a large
bias into the results. The two specification documents are 24 and 31
pages long. Hence an inspection rate of 15.5 pages per hour is required
for the longer document. This is well above the figure quoted in many
inspection texts, e.g. Ebenau and Strauss[5] recommend 5 to 7 pages for
specification documents, and Gilb and Graham[8] are even more conservative
recommending 0.5 to 1.5 pages for software development documents.
These figures by themselves do not show that the inspection rate in the
original experiment was incorrect because of variations in the assumptions
- the specifications are in a semi-formal notation, which the texts don't
explicitly address and no precise definition of what constitutes a page are
given. But they are a cause for concern. Also in their replication Lanubile
and Visaggio[13] state that "the time limit was too short" and during our
training of the subjects, it was found that the subjects required more than
2 hours to inspect a 16 page specification document.
Hence to minimise this threat to validity, the original experiments strict
time limit was abandoned and the subjects were given as much time as
they required. Post-analysis shows that the average subject took 374 and
430 minutes to inspect the 24 and 31 page specifications respectively 4 .
ffl Effort Effect
The original experiment asked the subjects to either inspect the
document using a Checklist or 2) inspect the document using one of the
three Scenarios. These two tasks require different amounts of time and
effort, the former requiring significantly more effort than the latter. This
problem is compounded, as in the original, when this division of labour
is used in conjunction with the fixed time limit. Asymmetric effort requirements
in the application of the different detection methods leads to
results parameterised by the percentage of task completed rather than a
dimensionless quantity. This viewpoint is also supported by Cheng and
Jeffery[2], who have carried out independent studies investigating the relationship
between Scenarios and Checklists, and is consistent with the
work of Kelly et al.[11], who found that increasing the amount to be
inspected caused a decrease in inspector's performance.
Hence to minimise this internal threat, the subjects using the Scenario
technique were asked to apply all three Scenarios to balance the effort.
Post-analysis shows that this alteration does balance the effort. A comparison
of times taken can be found in Table 2.
ffl Measuring multiple events
The hypothesis of the original and this replication compares the various
detection methods. The original experiment waited until after the group
4 Note: These times will have a small error associated with them. Given the length of the
study, the subjects had to self-time parts of the study.
5 ignoring the Ad Hoc approach.
Checklist Scenario Checklist Scenario
Mean 460 403 379 370
St. Dev. 180 168 158 137
F ratio 1.34 0.04
Prob.

Table

2: Average time taken to complete each inspection
collection meeting to collect data for this comparison. Data collected
at this point will be distorted by meeting gains and losses which are
independent of the method used.
Hence to minimise this threat, it was decided to collect data from before
the collection meeting.
ffl Statistical power
All experiments should consider this most important design factor. Although
statistical power is difficult to estimate, it is important to consider
it, often in conjunction with effect size, in any well designed experiment.
Given that the sample size was fixed (size of class undertaking the course)
and assuming a standard setting for the power level (0.8), an experimenter
can find out what size of effect must exist for their experiment to be considered
well designed 6 . In the case of the original experiment, a large 7
effect size must be found, i.e. a large difference must be expected for the
experiment to have an appropriate design. Unfortunately we do not know
the estimated effect size of the original experiment, as it is not contained
within the paper.
Hence to minimise this threat, it was decided to reduce the required effect
size by increasing the sample size. This was achieved in two ways: first
increased subject numbers and second by measuring for the effect at the
individual rather than the group level. This is especially appropriate given
the decision on "Measuring multiple events". Hence with the increased
sample size the replication can now reliably find significant results for
small to medium 8 effect sizes.
ffl Group dynamics
Obviously the effectiveness of the group meeting component cannot be
divorced from the dynamics of the group undertaking the inspections,
and since the experiment has a repeated measure design, concern exists
that the group dynamics will alter between the two inspection phases.
Hence to minimise this threat, it was decided to have different group
allocations for each phase, and therefore the group component resembles
6 The real situation is slightly more complicated than this, see Miller et al.[15] for a discussion
on the impact of Statistical power on empirical research.
7 about 0.75, for a definition of large, medium and small power, see Cohen[3].
an experiment with an internal replication.
There also exists some internal threats which we were unable to minimise fully.
ffl Degree of application of the detection aids
It is impossible to strictly enforce application of the detection aids all
of the time. In fact, that situation is unnatural. Inspectors start their
inspection by reading the document and obviously may well find defects
during this initial preparation. Hence it is reasonable to assume that
a proportion of the reported defects are not directly attributable to the
application of the detection aid, even if the subject using it applies it
fully. But which proportion? Should the defects for the initial read of
the specification be discounted? The inspector may already be implicitly
applying the detection aid while reading. Applying the detection aid to
check a particular fact on a particular page leads the inspector to uncover
an indirectly related fact, should this be discounted? What about an
inspector applying an aid in a slightly non-standard fashion again leading
to an error not directly attributable to the detection aid. Further imagine
an inspector who chooses to augment their detection aided work with
some additional Ad Hoc based inspection, during which they find errors
directly related to the detection aids, how could these defects to reliably
found and discounted? The authors believe that this is a very grey area
with few distinguishing boundaries, and hence to attempt to subdivide the
reported errors is highly dangerous and unadvisable, without large data
sets to help to account for these potential variations. The experiment
has relatively small data sets, especially with regard to defect coverage
of individual Scenarios, e.g. as reported in the original experiment, the
missing or ambiguous functionalities Scenario has a strict coverage of one
defect in the CRUISE specification, which was not discovered by any
subject, and hence all defects reported by application of this aid are at
best indirect or secondary effects of applying the aid.
Hence, rather than attempting to subdivide the recorded defects, the
subjects will be asked to answer questions in the debriefing questionnaire
about their level of application of the detection aid. Answers to these
and other related questions should provide evidence for the validity of the
undertaken experiment.
ffl Typographical errors in specifications
The specifications from the original experiment contained a significant
number of typographical errors. Despite removing many of these errors,
some still existed during the experiment and may have caused confusion
for some of the subjects. Also these typographical errors preclude any
analysis of the false positives reported. This is a major deficiency in the
original and subsequent replications.
ffl Initial Data Analysis
Initial data analysis consists of the experimenter deciding on which reported
defects are correct and which are incorrect. Defects are unstructured
English descriptions of what the subjects' believe to be incorrect,
and they are required to supply an accurate description of the fault. This
is a inexact process and has potentially a large degree of variability. To
illustrate, the experimenter checks possible defects against the master defect
list (a list of all the defects contained within the document); Lanubile
and Visaggio[13] state that they believe that this list from the original
experiment is in error and provide several additional defects which they
believe are missing from the original list, i.e. the researchers can't even
agree on the perfect solution. It is concluded that this experiment has
a natural in-built risk which is difficult, if not impossible, to safeguard
against, unless the experimenter has the resources to employ multiple
researchers cross-checking the initial analysis.
ffl Learning Curve - Novice Subjects
All the subjects had to learn both the inspection process and the specification
language. It is quite possible that the experiment has experienced
effects due to subjects lack of understanding of either of these topics.
Apart from careful teaching of this material, it is difficult to employ further
measures to safegaurd against this effect. Instead any effect will try
to be 'measured' via the debriefing questionnaire and may subsequently
alter the analysis of the independent variables.
Threats to External Validity
Threats to external validity limit the ability to generalise the results from an
experiment to a wider population, under different conditions. The threats to
external validity for the replication are the same as for the original experiment:
ffl The subjects in our replication may not be representative of the general
software engineering population, e.g. this study used students rather than
software professionals. This threat is always a problem, because of the
lack of sampling frame, and hence even studies using professionals will be
exposed to this threat.
ffl The specification documents may not be representative of industrial prob-
lems. The documents used in this study are smaller and less complex than
industrial specifications.
ffl The inspection process may not be representative of industrial software
development practice. Despite using a well known and widely used inspection
technique[8], many other inspection processes exist in industry
which pose a threat to the ability to generalise from this experiment.
2.2 Experiment Instrumentation
This experiment has re-used, except as described above, the material from the
original experiment, see Porter et al.[19] for a full description of the materials.
The following is an abridged description of the materials to enhance the reader's
appreciation of the experimental results.
Software Requirements Specifications
The specifications used in this experiment are: an automatic cruise control
system[12] (CRUISE) and a water level monitoring system[22] (WLMS). The
CRUISE specification is 31 pages long and the WLMS specification is 24 pages
long. Each specification has four sections: overview, specific function require-
ments, external interfaces, and a glossary. The overview is written in En-
glish, whereas the other sections are specified using a tabular requirements
All the faults in both documents are natural, with the CRUISE
specification believed to contain 26 technical faults and the WLMS specification
believed to contain 42 technical faults.
Fault detection methods
The two detection methods are designed to search for a well-defined population
of faults. The original experiment used a general fault taxonomy to
define the responsibilities of Ad Hoc reviewers and consequently directly derived
a set of Checklist responsibilities from the Ad Hoc responsibilities. The
individual Checklist questions were selected from several industrial checklists.


Appendix

A gives a complete description of the Checklist. The Scenarios were
then derived from the Checklist by replacing individual Checklist items with
procedures designed to implement them. The Scenario procedures have been
further grouped into three distinct sub-groups: data type inconsistencies, incorrect
functionalities, missing or ambiguous functionalities. Hence the Scenarios
search for an exact subset of the faults defined by the Checklist's responsibili-
ties. A post-experiment investigation by the original experimenters estimated
that the Scenarios include about half of the faults covered by the Checklist.


Appendix

contains a complete list of all the Scenarios.
3 Data and Analysis
Due to the alterations of the design of the experiment, the collected data cannot
be analysed using the same strategy as the original experiment. Having said
this, comparisons will be drawn whenever possible.
3.1 Verification of experimental validity

Table

3 shows the average defect group detection rates for both the original,
the first external replication 9 and this subsequent replication. This table shows
that the subjects in this experiment performed at least as well as the subject's
in the other experiments, and subsequently if we assume that the alterations in
the experimental procedure have not increased the defect detection rate, this
verifies these subjects suitability for this study. As stated earlier, it is believed
that this group defect detection rate represents a composite result of using
the detection method plus a collection meeting component. Hence rather than
9 Figures for this experiment are approximate, as Lanubile and Visaggio only give a graphical
representation of this result.
Checklist Scenario Checklist Scenario
Original 0.24 0.45 0.41 0.57
1st. Rep. 0.24 0.20 0.32 0.33
This Rep. 0.39 0.42 0.48 0.49

Table

3: Average proportion of defects found in the three experiments, at the
group level.
1. Do you think you understand Software Inspections?
(a) Completely
(b) Well
(c) Reasonably Well
(d) Not too sure
(e) Not at all
2. Do you think you understand the notation?
(a) Completely
(b) Well
(c) Reasonably Well
(d) Not too sure
(e) Not at all

Figure

1: Questions regarding validity of experiment
examining the above figures to analyse the treatment variable, the individual
defect detection rates will be considered. Neither of the other experiments
report these figures. This analysis is described in the following section.
The experiment has one further source of data for analysis: the debriefing
questionnaire. Each individual was asked two questions, one about their
understanding of the inspection process and the other about the semi-formal
specification language; Figure 1 details these questions and Table 4 summarises
the results. The results are sufficiently positive, to verify these aspects of the
experiment, i.e. the subjects have gained sufficient ability in these two areas to
limit the impact of the learning curve. Further questions with a direct impact
on assessing validity are discussed in future sections. It is believed that all these
questions yield a sufficiently positive picture of the experimental procedures as
to add further weight to the belief that no unknown external factor or factors
have made the experiment invalid.
3.2 Individual Inspection Performance
The following two figures (Figures 2 and
the two specifications) inspection performance given a detection methodology.
As can be seen, on average the Scenario technology out performs the Checklist

Table

4: Summary of answers to questions regarding validity
Normalised Defects
Number
of
Subjects
S. D.11

Figure

2: Histogram of average subject performance using Scenarios
technology, but taking a closer look (Table 5), despite a noticeable difference in
the means the experiment falls just short of providing a statistically significant
result. Although not directly comparable, it is worth noting at this point, the
treatment variable analysis from the other experiments: the original,
and the first replication, Hence it is reasonable to
conclude that the result from this experiment is more supportive of the original
experiment than the first replication. We will return to the question of whose
treatment variable results (at the group level) later in this paper.
The only other variable to provide a significant result in either of the experiments
was the specification variable. Unfortunately this experiment's specification
variable could also hide a maturation effect, but given the experiences
of the other two experiments and the fact that nothing has been altered which
might directly impact on this effect, it is believed that this is unlikely. The specification
variable shows a statistically significant effect, in line with the other
Normalised Defects
Number
of
Subjects
S. D.11
Figure

3: Histogram of average subject performance using Checklists
Detection Method
Scenario Checklist
Subjects 26 24
Mean 0.36 0.31
St. Dev. 0.11 0.11
St.
F Ratio 2.80
F Prob. 0.10

Table

5: Basic detection method analysis
two experiments (original first replication:

Table

6. What is also of interest, is that this replication experienced
an asymmetric effect with respect to detection method, again see Table
6, whereas the original experiment did not experience
any 11 .
These results suggest that a closer investigation is required and suggest that
the treatment variable should be investigated against each specification sepa-
rately. Table 7 shows this information. As can be seen, a large difference exists
between the relative performance of the detection aids across the two specifications
- with the first phase of the experiment (using the CRUISE specification)
Remember these figures include results from using the Ad hoc approach and hence are
not directly comparable.
11 The first replication does not quote any figure.
Both Detection Method
Methods Scenario Checklist
Subjects 50 26 24
Mean -0.05 -0.01 -0.08
St. Dev. 0.11 0.12 0.12
St.
F Ratio 7.81 4.85

Table

Specification effect: independent and dependent of detection method;
the negative means indicate a preference for the WLMS specification
Scenario Checklist Scenario Checklist
Subjects 26 24 26 24
Mean 0.35 0.27 0.37 0.35
St. Dev. 0.14 0.13 0.10 0.12
St.
F Prob. 0.03 0.63

Table

7: Detection method analysis by specification
showing a strong effect, while the second phase (using the WLMS specification)
shows an extremely weak effect. The same analysis can be conducted for the
original experiment, albeit at the group level. This analysis is shown in Table
8, and is derived from Table 3 (pp. 569) in the paper by Porter et al.[19] Again
we see a strong performance bias towards one specification, further analysis is
not possible given the reported data 12 . Hence further analysis was undertaken
to attempt to explain this effect more fully.
One possibility is an asymmetric change in the effort or commitment between
the two groups. Obviously it is impossible to measure these effects di-
rectly, and hence we must rely on the related measure of time spent when
looking for an explanation. Table 9 summarises the time spent on individual
inspections, it analyses the difference in time taken by each subject (CRUISE -
WLMS), grouped by detection method. Note three subjects using the Scenario
method failed to complete their time estimates, and are hence excluded from
this analysis. This table shows that a small asymmetric effect does exist, but it
is believed that it is too small to account for the effect. Taking the difference in
means (31.3) and comparing it with the average time to complete (403), shows
that the effect accounts for 7.5% of the average time spent on inspecting, and
hence by itself is rejected as having sufficient likely impact to explain all of the
difference.
12 The first experiment does not supply sufficient figures to allow a complementary analysis.
Scenario Checklist Scenario Checklist
Subjects
Mean 0.45 0.24 0.57 0.41
St. Dev. 0.07
St.
F Ratio 22.06 4.92
F Prob. !!0.01 0.06

Table

8: Detection method analysis by specification of the original experiment
at the group level
Scenario Checklist
Subjects
Mean 50.2 81.5
St. Dev. 152.6 162.7
St.
F Prob. 0.50

Table

9: Difference in time spent between inspecting each specification
Two further possible explanations were explored 1) that particular faults or
particular types of faults are easier to find using one technique or the other; and
differences in coverage, between the two detection aids. The original experiment
states that in effect due to differences in coverage have been minimised
by:
" . deriving the Scenarios from the Checklist by replacing individual
Checklist items with Scenario procedures designed to implement
them."
Unfortunately the experiment has only transposed a subset of the Checklist
items into Scenario procedures and hence the possibility of a coverage effect
cannot be ignored. For the CRUISE specification which has 26 potential defects,
the Checklist items cover 24 of the defects, whereas the Scenario procedures
only cover 14 (10 Data type inconsistencies, 1 Incorrect functionalities and 3
Missing or Ambiguous functionalities) defects; and for the WLMS specification
which has 42 potential defects, the Checklist items cover 38 of the defects,
whereas the Scenario procedures only cover 24 (14 Data type inconsistencies,
5 Incorrect functionalities and 5 Missing or Ambiguous functionalities) defects.
These facts could be considered to add extra weight to the claim that the
Scenario procedures are superior - increased performance, for lower coverage.
This is probably true given the decision to give the subjects as much time
as they required. It is more difficult to estimate the potential impact on the
original with its fixed time limit.
Mean
Difference
(Scenario
Checklist)Defect

Figure

4: Difference between technologies against individual defects in CRUISE-8Mean
Difference
(Scenario
Checklist)
Defect

Figure

5: Difference between technologies against individual defects in WLMS
The following figures (Figures 4 and 5) shows the difference in performance
between the two technologies (Scenario - Checklist) against the individual de-
fects. The first figure illustrates the behaviour in the CRUISE specification. As
can be seen, subjects using the Scenario technique perform better than Check-list
users for nearly every defect and by a relatively consistent amount, with
little difference between the areas explicitly covered by the Scenario procedures
and those which are not. Turning our attention to the second figure, detailing
the results from the WLMS specification, we see that this consistent picture
disappears, and that the Checklist users out-perform the Scenario users in 36%
of the defects. Here many of the items covered by the Checklist but not by
the Scenario procedures, show results in favour of the Checklist technique. But
again there is no consistency in the interaction between the detection aids and
the specifications.
A further possibility that exists is that these defects fall into a single type of
3. Did you use your technique (Scenario or Checklist)?
(a) Always
(b) Most of the time
(c) Sometimes
(d) Occasionally
Never
4. Do you understand your technique?
(a) Completely
(b) Well
(c) Sort of
(d) Not to sure
(e) Not at all
5. Do you think your technique was better or worse than the Ad Hoc approach
previously employed?
(a) Definitely Better
(b) Probably Better
(c) About the same
(d) Probably Worse

Figure

Questions regarding technique used.
defect. To investigate this, the defects were characterised using the taxonomy
developed in the original experiment, which is a composite of two schemes
developed by Schneider et al.[21] and Basili and Weiss[1]. Unfortunately this
classification failed to reveal any insights into the effects, as the defects are
spread across the taxonomy.
Finally, the questionnaires were investigated to address this issue. The
questionnaires contain three questions directly relevant to this point (Figure
6).
Question 3 attempts to find the rate of use of each technique; Question 4
how well each technique is understood by the relevant subjects; and Question 5
elicits the subject's opinion on each techniques performance against a common
control technique. Table 10 summarises the subjects responses. Analysing
table yields two relevant facts. Firstly the more regular use of the Scenario
technique, compared with the Checklist technique; in fact the latter might be
more correctly named Checklist plus Ad Hoc. It is impossible to estimate the
impact this would have on the previous calculations, but any impact should be
symmetrical with regard to the specifications. The second observation is the
drop in usage of the Scenario technique between inspecting the CRUISE and
WLMS specifications, while the Checklist technique does not experience any
decline in application. This could well lead to an asymmetric effect, especially
if the Scenario technique is superior to simply adopting an Ad Hoc approach
to inspection.
To summarise the asymmetric effect between the two specifications is difficult
to explain. A number of factors have been found each of which could have
Q. 3 Q. 4 Q. 5 Q. 3 Q. 4 Q. 5
Checklist C%

Table

10: Detection method analysis by questionnaire (Figure 6), all percentages
rounded to 5%.
Scenario Checklist Scenario Checklist
No. of groups 8 8 8 8
Mean 0.42 0.39 0.49 0.48
St. Dev. 0.08 0.09 0.07 0.11
St.
F Prob. 0.49 0.75

Table

11: Initial analysis of group performance
a small asymmetric impact on the experiment. The most likely explanation of
the asymmetric effect is a combination of these factors, namely:
ffl Asymmetric effort effect
ffl Differences in coverage
ffl Decline in application of the Scenario technique
ffl Natural variation within the experiment
ffl The performance of the detection aids is dependent on the nature of the
specification
3.3 Analysis of Group Meeting Data
Earlier it is stated that this experiment deviated from the original design because
of concern of multiple events obscuring the basic hypothesis. Table 11
investigates the original hypothesis at the group level. Comparing this table
with the same analysis at the individual level (Tables 5 and 7) clearly shows
Scenario Checklist Scenario Checklist
No. of groups 8 8 8 8
Mean 0.62 0.48 0.64 0.59
St. Dev. 0.09 0.11 0.08 0.09
St.
F Ratio 7.46 1.14
F Prob. 0.02 0.30

Table

12: Analysis of Potential maximum group results
Scenario Checklist Scenario Checklist
No. of groups 8 8 8 8
Mean -5.13 -2.38 -6.00 -4.75
St. Dev. 2.30 3.46 4.20 3.69
St.
F Prob. 0.29 0.60

Table

13: Analysis of the relationship between meeting effects and detection
aid. Figures are quoted as per defect, rather than as percentages, due to the
small numbers involved.
that the group component has had a large effect upon the analysis of the treatment
variable, especially with regard to analysing the results of the CRUISE
specification. Recalling the results for the original experiment (p ! 0.01) and
the first replication (p ! 0.92), the results at the group level are now more supportive
of the first replication rather than the original experiment. This change
of support is directly attributable to (some aspects of) the group inspection.
This can be clearly seen by comparing Table 11 with Table 12, which estimates
the group score by forming a union of the three individual scores. In Table 12,
the familiar pattern from the individual analysis re-establishes itself, if anything
the effect (pro-Scenario) is even stronger than at the individual level. So why
the change? It is believed that the meeting component is introducing an effect
unrelated to the detection aids, and this effect is demonstrated in the difference
between Tables 11 and 12. The difference can be characterised by two variables:
meeting loss and meeting gain. Meeting gain represents new defects found at
the meeting, whereas meeting loss represents those defects found during the individual
phase which are 'lost' during the meeting session. (Meeting loss occurs
for a number of reasons which will be discussed later in the paper.)

Table

13 explores this possible relationship further, directly exploring the
possibility of a relationship between the meeting component (meeting gain -
meeting loss) and the detection aid. It shows that no significant result exists
between these two concepts and the larger negative results experienced by the
Meeting Gain Meeting Loss Meeting Gain Meeting Loss
No. of groups
Total Defects 26 26 42 42
Mean 1.13 4.88 1.63 7.00
St. Dev. 0.81 2.80 1.20 3.27

Table

14: Meeting Gain and Loss by Specification
Score Maximum
Loss p 0.14 !!0.01
WLMS r -0.51 0.52
Loss

Table

15: Correlation (Pearson) between Meeting Loss, Group Score and Maximum
Possible Group Score
Scenario subjects is believed to be directly related to the fact that these subjects,
on average, found more defects than any indirect association with the detection
aid. This conjuncture is discussed more fully in the following section and Table
15 shows the correlation between the meeting effects and the number of defects
found. Examining Tables 11-13 and 15 shows that a strong case can be made
that the meeting component is independent of the detection aid used, and
hence this experiments choice to explore the treatment variable at the individual
level rather than at the group level, as conducted in the original, is the more
appropriate experimental design.
3.4 Analysis of Meeting Losses and Gains
Another outcome of the original experiment was the rejection of the meeting
component as a fault detection technique. The original experiment reports
that the number of meeting losses outweighs the number of meeting gains, this
viewpoint is also supported by the first replication. The results here are no
different, with meeting losses comfortably outweighing meeting gains. Given
the low resolution in these values, complex statistical evaluation is inappropri-
ate, for example the median value of the meeting gain is one defect for both
specifications, i.e. one unit of resolution. Table 14 gives a brief summary of
meeting gains and meeting losses for each specification. The figures in this table
broadly agree with the original and the first replication, that on average meeting
losses are greater than meeting gains, and are also in line with the results
reported by McCarthy et al.[14], who reported that inspection techniques which
rely more on the individual component were more productive than those which
relied more upon the meeting component. Further, with regard to the meeting
gain component, these results are in line with the results reported by Votta[10].
Votta reported on his experiences observing a series of professional inspection
meetings and found that the meeting component only contributed 4%, on av-
erage, to the number of defects found. This experiment produced an average
meeting gain of 9% from our novice inspectors, i.e. over twice the professional
rate. It is clearly highly unlikely that our subjects are able to perform at twice
the level of seasoned professionals. Much more likely, and certainly the view of
the authors, is that this increase is due to an experimental process, and shows
the danger of extrapolating from such small scale, low resolution data. The
meeting loss figures broadly follow the trend of the other experiments, but at
a more extreme level, i.e. this experiment has larger meeting loss values than
the other two experiments. So why the large meeting loss? Table 15 shows
that the loss is a systematic effect, and that the meeting loss correlates with
the meeting score, and correlates extremely strongly with the potential maximum
group score, i.e. the greater the diversity of the defects discovered by the
individuals in the group, the greater the number of defects lost. One aspect of
the experimental material which may be influencing this is the typographical
errors contained within the specification. The subjects were told these errors
were not defects and to discard them. The typographical errors don't cause
any problem with the correctness of the specifications, but they do contribute
an additional difficultly in understanding the document. Given that the experiment
uses novice inspectors, it is believed that many groups had some difficulty
distinguishing between defects and typographical errors, leading to some genuine
defects being discarded at the meeting. Unfortunately it is impossible to
measure the impact of the typographical errors on the meeting component, and
even on the individual component, but it is believed that the greatest potential
impact will be on the meeting loss estimation, especially given the lower sample
size at the group level.
This numerical concern of the effectiveness of the meeting component is
not shared by the subjects. The subjects were asked if they thought that the
meeting component was worthwhile, via the questions in Figure 7.
In general, their responses must be categorised as positive (Table 16). Looking
in more detail we can see that the responses to the second questionnaire
are significantly less positive than the first. That is, there exists a large negative
trend from the first exercise to the second exercise in the subjects opinions
about the effectiveness of the meeting component as a vehicle for defect detec-
tion. This trend, rather than the individual responses, is more in line with the
quantifiable data from the inspection meeting- unfortunately there is no way
of predicting if this trend would continue into subsequent phases of such an
experiment.
Conclusions
Software inspection are undoubtedly one of the best ways to verify software
documents. Having said this, the technique remains relatively unexplored, in
terms of which variations upon the basic theme work best. This paper explores
one such variation - detection aids, specifically it attempts to compare Scenarios
6. In considering the effective use of an inspector's time,
how would you rate the collection meeting against the individual component?
A. Far superior
B. Useful
C. Of Similar Worth
D. Of limited use
E. A waste of time
7. Which (you may answer yes to more than one) of the following objectives
were achieved by your collection meeting?
A. Defect Merging
B. Extra Defect Detection
C. Group Bonding/Team Spirit
D. Education of Weak group Members
E. Ensure all individuals adequately prepare their defect lists
F. Ensure common practices amongst individual inspectors

Figure

7: Questions regarding technique used.
and Checklists, to see which approach is better, if any.
Importantly this paper is not a one-off study, but is part of a large piece
of work involving several other researchers investigating the same hypothesis.
Multiple independent studies of the same hypothesis are essential if software
engineering is going to produce empirically evaluated theories and procedures.
The paper attempts to compare its results with the other studies whenever
possible.
With regard to the hypothesis of the experiment, the results are ambiguous,
but on balance are generally supportive of the results in the original experiment.
Hence, given the current weight of evidence, from the three experiments, there
seems to be emerging support for the conjuncture that the Scenario approach
is better than the Checklist approach. More work is required to finally confirm
this conjecture.
As normal when conducting an experiment several other effects, or more
accurately possible effects, were revealed. These additional effects include:
ffl a strong correlation between general academic ability and the ability to
successfully inspect software.
ffl an interaction between the detection aids and the specifications.
ffl that the subjects were more willing to use Scenarios compared with Check-list

ffl that the effectiveness of the meeting component is independent of the
detection aid used.
ffl that meeting losses outweigh meeting gains, suggesting the meeting component
is not an effective defect detection mechanism.
Q. 6 Q. 7 Q. 6 Q. 7
A%
B%
C%

Table

Analysis of Questions 6 and 7 (Figure 7)
ffl that meeting losses correlate strongly with various measures of the number
of defects found by a group.
These new potential effects also require further investigation to verify their
validity. Looking ahead and assuming that they exist, then the above results
suggested several new lines of research within the software inspection area, e.g.
inspection models with alternatives to the meeting component.
A Checklist Method
ffl General
- Are the goals of the system defined?
- Are the requirements clear and unambiguous?
- Is a functional overview of the system provided?
- Is an overview of the operational modes provided?
- Have the software and hardware environments been specified?
- If assumptions that affect implementation have been made, are they
stated?
- Have the requirements been stated in terms of input, output, and
processing for each function?
- Are all functions, devices, constraints traced to requirements and vice
versa?
- Are the required attributes, assumptions and constraints of the system
completely listed?
ffl Omission
Missing Functionality
* Are the described functions sufficient to meet the system objectives

* Are all inputs to a function sufficient to perform the required
* Are undesired events considered and their required responses specified

* Are the initial and special states considered (e.g. system initia-
tion, abnormal termination)?
Missing Performance
* Can the system be tested, demonstrated, analyzed, or inspected
to show that it satisfies the requirements?
* Have the data type, rate, units, accuracy, resolution, limits, range
and critical values for all internal data items been specified?
* Have the accuracy, precision, range, type, rate, units, frequency,
and volume of inputs and outputs been specified for each function

Missing Interface
* Are the inputs and outputs for all interfaces sufficient?
* Are the interface requirements between hardware, software, per-
sonnel, and procedures included?
Missing Environment
* Have the functionality of hardware or software interacting with
the system been properly specified?
ffl Comission
Ambiguous Information
* Are the individual requirements stated so that they are discrete,
unambiguous, and testable?
* Are the transitions specified deterministicly?
Inconsistent Information
* Are the requirements mutually consistent?
* Are the functional requirements consistent with the overview?
* Are the functional requirements consistent with the actual operating
environment?
- Incorrect or Extra Functionality
* Are all the described functions necessary to meet the system objectives

* Are all inputs to a function necessary to perform the required
* Are the inputs and outputs for all interfaces necessary?
* Are all the outputs produced by a function used by another function
or transferred across an external interface?
Wrong Section
* Are all the requirements, interfaces, constraints, etc. listed in the
appropriate sections.
B.1 Data Type Consistency Scenario
1. data object's mentioned in the overview (e.g. hardware com-
ponents, application variable, abbreviated term or function)
(a) Are all data objects mentioned in the overview listed in the external
interface section?
2. For each data object appearing in the external interface section determine
the following information:
ffl Object name:
ffl Class: (e.g. input port, output port, application variable, abbreviated
ffl Data type: (e.g. integer, time, boolean, enumeration)
ffl Acceptable values: Are there any constraints, ranges, limits for the
values of this object
ffl Failure value: Does the object have a special failure value?
ffl Units or rates
ffl Initial value:
(a) Is the object's specification consistent with its description in the
overview?
(b) If object represents a physical quantity, are its units properly specified

(c) If the object's value is computed, can that computation generate a
non-acceptable value?
3. For each functional requirement identify all data object references:
(a) Do all data object references obey formatting conventions?
(b) Are all data objects referenced in this requirement listed in the input
or output sections?
(c) Can any data object use be inconsistent with the data object's type,
acceptable values, failure values, etc.?
(d) Can any data object definition with the data object's type, acceptable
values, failure value, etc.?
B.2 Incorrect Functionality Scenario
1. For each functional requirement identify all input/output data objects:
(a) Are all values written to each output data object consistent with its
intended function?
(b) Identify at least one function that uses each output data object.
2. For each functional requirement identify all specified system events:
(a) Is the specification of these events consistent with their intended
3. Develop an invariant for each system mode (i.e. Under what conditions
must the system exit or remain in a given mode)?
(a) Can the system's initial conditions fail to satisfy the initial mode's
invariant?
(b) Identify a sequence of events that allows the system to enter a mode
without satisfying the mode's invariant.
(c) Identify a sequence of events that allows the system to enter a mode,
but never leave (deadlock).
B.3 Ambiguities Or Missing Functionality Scenario
1. Identify the required precision, response time, etc. for each functional
requirement.
(a) Are all required precisions indicated?
2. For each requirement, identify all monitored events.
(a) Does a sequence of events exist for which multiple output values can
be computed?
(b) Does a sequence of events exist for which no output value will be
3. For each system mode, identify all monitored events.
(a) Does a sequence of events exist for which transitions into two or more
modes is allowed?



--R

Evaluation of software requirements document by analysis of change data.
Comparing inspection strategies for software requirement specifications.
Statistical Power Analysis for the Behavioral Sciences.
Experience with Fagan's inspection method.
Software Inspection Process.
Design and code inspections to reduce errors in program development.
Advances in software inspection.
Software Inspection.
Specifying software requirements for complex systems.
Votta Jr.
An analysis of defect densities found during software inspections.
Example NRL/SCR software requirements for an automobile cruise control and monitoring system.
Assessing defect detection methods for software requirements inspections through external replication.
An experiment to assess cost-benefits of inspection meetings and their alternatives
Statistical power and its subcomponents - missing and misunderstood concepts in emprical software enginerring research
Active design reviews: Principles and practices.
An experiment to assess different defect detection methods for software requirements inspections.
Comparing detection methods for software requirements inspections: A replicated experiment.
Comparing detection methods for software requirements inspections: a replicated experiment.
Experience with inspection in ultralarge-scale develop- ments
An experimental study of fault detection in user requirements.

--TR

--CTR
Stefan Biffl , Bernd Freimut , Oliver Laitenberger, Investigating the cost-effectiveness of reinspections in software development, Proceedings of the 23rd International Conference on Software Engineering, p.155-164, May 12-19, 2001, Toronto, Ontario, Canada
J. Miller, On the independence of software inspectors, Journal of Systems and Software, v.60 n.1, p.5-10, 15 January 2002
Stefan Biffl , Wilfried Grossmann, Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles, Proceedings of the 23rd International Conference on Software Engineering, p.145-154, May 12-19, 2001, Toronto, Ontario, Canada
Bente Anda , Dag I. K. Sjberg, Towards an inspection technique for use case models, Proceedings of the 14th international conference on Software engineering and knowledge engineering, July 15-19, 2002, Ischia, Italy
Bjrn Regnell , Per Runeson , Thomas Thelin, Are the Perspectives Really Different?  FurtherExperimentation on Scenario-Based Reading of Requirements, Empirical Software Engineering, v.5 n.4, p.331-356, December 2000
Hidetake Uwano , Masahide Nakamura , Akito Monden , Ken-ichi Matsumoto, Analyzing individual performance of source code review using reviewers' eye movement, Proceedings of the 2006 symposium on Eye tracking research & applications, March 27-29, 2006, San Diego, California
Stefan Biffl, Using Inspection Data for Defect Estimation, IEEE Software, v.17 n.6, p.36-43, November 2000
Stefan Biffl , Michael Halling, Investigating the Defect Detection Effectiveness and Cost Benefit of Nominal Inspection Teams, IEEE Transactions on Software Engineering, v.29 n.5, p.385-397, May
Stefan Biffl , Walter J. Gutjahr, Using a Reliability Growth Model to Control Software Inspection, Empirical Software Engineering, v.7 n.3, p.257-284, September 2002
Oliver Laitenberger , Thomas Beil , Thilo Schwinn, An Industrial Case Study to Examine a Non-Traditional Inspection Implementation for Requirements Specifications, Empirical Software Engineering, v.7 n.4, p.345-374, December 2002
Susan S. Brilliant , John C. Knight, Empirical research in software engineering: a workshop, ACM SIGSOFT Software Engineering Notes, v.24 n.3, p.44-52, May 1999
F. MacDonald , J. Miller, A Comparison of Tool-Based and Paper-Based Software Inspection, Empirical Software Engineering, v.3 n.3, p.233-253, September 1998
James Miller , Fraser Macdonald , John Ferguson, ASSISTing Management Decisions in the Software Inspection Process, Information Technology and Management, v.3 n.1-2, p.67-83, January 2002
Thomas Thelin , Per Runeson , Claes Wohlin, An Experimental Comparison of Usage-Based and Checklist-Based Reading, IEEE Transactions on Software Engineering, v.29 n.8, p.687-704, August
Victor R. Basili , Forrest Shull , Filippo Lanubile, Building Knowledge through Families of Experiments, IEEE Transactions on Software Engineering, v.25 n.4, p.456-473, July 1999
Andreas Zendler, A Preliminary Software Engineering Theory as Investigated by Published Experiments, Empirical Software Engineering, v.6 n.2, p.161-180, June 2001
Oliver Laitenberger , Khaled El Emam , Thomas G. Harbich, An Internally Replicated Quasi-Experimental Comparison of Checklist and Perspective-Based Reading of Code Documents, IEEE Transactions on Software Engineering, v.27 n.5, p.387-421, May 2001
Oliver Laitenberger , Dieter Rombach, (Quasi-)experimental studies in industrial settings, Lecture notes on empirical software engineering, World Scientific Publishing Co., Inc., River Edge, NJ,
partial replication;experiments;software inspection;defect detection aids