Understanding the sources of variation in software inspections.
In a previous experiment, we determined how various changes in three structural elements of the software inspection process (team size and the number and sequencing of sessions) altered effectiveness and interval. Our results showed that such changes did not significantly influence the defect detection rate, but that certain combinations of changes dramatically increased the inspection interval. We also observed a large amount of unexplained variance in the data, indicating that other factors must be affecting inspection performance. The nature and extent of these other factors now have to be determined to ensure that they had not biased our earlier results. Also, identifying these other factors might suggest additional ways to improve the efficiency of inspections. Acting on the  hypothesis that the inputs into the inspection process (reviewers, authors, and code units) were significant sources of variation, we modeled their effects on inspection performance. We found that they were responsible for much more variation in detect detection than was process structure. This leads us to conclude that better defect detection techniques, not better process structures, are the key to improving inspection effectiveness. The combined effects of process inputs and process structure on the inspection interval accounted for only a small percentage of the variance in inspection interval. Therefore, there must be other factors which need to be identified.
Introduction
Software inspection has long been regarded as a simple, effective, and inexpensive way of detecting
and removing defects from software artifacts. Most organizations follow a three-step procedure
of Preparation, Collection, and Repair. First, each member of a team of reviewers reads the
artifact, detecting as many defects as possible (Preparation). Next, the review team meets, looks
for additional defects, and compiles a list of all discovered defects (Collection). Finally, these defects
are corrected by the artifact's author (Repair).
Several variants of this method have been proposed in order to improve inspection performance.
Most involve restructuring the process, e.g., rearranging the steps, changing the number of people
working on each step, or the number of times each step is executed. Some of these variants
have been evaluated empirically. However, focus has been on their overall performance. Very few
investigations attempted to isolate the effects due specifically to structural changes. However, we
must know which effect are caused by which changes in order to determine the factors that drive
inspection performance, to understand why one method may be better than another, and to focus
future research on high-payoff areas.
Therefore, we conducted a controlled experiment in which we manipulated the structure of the
inspection process[20]. We adjusted the size of the team and the number of sessions. Defects
were sometimes repaired in between multiple sessions and sometimes not. Comparing the effects of
different structures on inspection effectiveness and interval 1 indicated that none of the structural
changes we investigated had a significant impact on effectiveness, but some changes dramatically
increased the inspection interval.
Regardless of the treatment used, both the effectiveness and interval data seemed to vary widely. To
strengthen the credibility of our previous study and to deepen our understanding of the inspection
process, we must now study this variation.
1.1 Problem Statement
We are asking two questions: (1) Are the effects of process structure obscured by other sources of
variation, i.e., is the "signal" swamped by "noise"? (2) Are the effects of other factors more influential
than the effects of process structure, i.e., are researchers focusing on the wrong mechanisms?
To answer the first question, we will attempt to separate the effects of some external sources of
variation from the effects due to changes in the process structure. By eliminating the effects of
external variation we will have a more accurate picture of the effects of our experimental treatments.
Also, by understanding the external variation we may be able to evaluate how well our experimental
design controlled for it, which will aid the design of future experiments.
To answer the second question, we will compare the variation due to process structure with that
due to other sources. If the other sources are more influential than process structure, then it may
Inspections have many different costs and benefits. In this study we restricted our discussion of benefits to the
number of defects found, and costs to inspection interval (the time from the start of the inspection to its completion)
and person effort.
be possible to significantly improve inspections by properly manipulating them. We expect that
identifying and understanding these sources will aid the development of better inspection methods.
Therefore, we have extended the results of our experiment by identifying some sources of variation
and modeling their influence on inspection effectiveness and interval. We will show that our previous
results do not change even after these sources of variation are accounted for. This analysis also
suggests some improvements for the inspection process and raises some implications about past
research and future studies.
1.2 Analysis Philosophy
We hope to identify mechanisms that drive the costs and benefits of inspections so that we can
engineer better inspections. To do this we will rely heavily on statistical modeling techniques.
However, these techniques are not completely automated. Therefore, we must make judgments
about which variables or combinations of variables to allow in the models. These choices are
guided by our desire to create models that are robust and interpretable.
To improve robustness we avoided fitting the data with too many factors. Doing so could result in
a model that explains much of the variation in the current data, but has no predictive power when
used on a different set of data.
To improve interpretability we omitted factors for which we have no readily available measure. We
also omitted factors whose effects were known to be confounded with other factors in the model.
Finally, we rejected models for which, based on our experience, we could not argue that their
variables were causal agents of inspection performance. Specifically, there are four conditions that
must be satisfied before factor A can be said to cause response B[12]:
1. A must occur before B.
2. A and B must be correlated.
3. There is no other factor C that accounts for the correlation between A and B.
4. A mechanism exists that explains how A affects B.
One implication of all these is that the "best" model for our purpose is not necessarily the one that
explains the largest amount of variation. Throughout this research we have chosen certain models
over others. Some were rejected because a smaller, but equally effective model could be found, or
because one variable was strongly confounded with another, or because a variable failed to show a
causal relationship with inspection performance. We will point out these cases as they arise.
2 Summary of Experiment
With the cooperation of professional developers working on an actual software project at Lucent
Technologies (formerly AT&T Bell Labs), we conducted a controlled experiment to compare the
costs and benefits of making several structural changes to the software inspection process. (See
et al.[20] for details.) The project was to create a compiler and environment to support
developers of Lucent Technologies' 5ESS(TM) telephone switching system. The finished compiler
contains over 55K new lines of C++ code, plus 10K which was reused from a prototype. (See


Appendix

A for a description of the project.)
The inspector pool consisted of the 6 developers building the compiler plus 5 developers working
on other projects. 2 They had all been with the organization for at least 5 years and had similar
development backgrounds. In particular, all had received inspection training at some point in their
careers. Data was collected over a period of (June 1994 to December 1995), during
which 88 code inspections were performed.
2.1 Experimental Design
We hypothesized that (1) inspections with large teams have longer intervals, but find no more
defects than smaller teams; (2) multiple-session inspections 3 are more effective than single-session
inspections, but at the cost of a significantly longer interval; and (3) although repairing the defects
found in each session of a multiple-session inspection before starting the next session will catch
even more defects, it will also take significantly longer than multiple sessions meeting in parallel.
We manipulated these independent variables: the number of reviewers (1, 2, or 4); the number
of sessions (1 or 2); and, for multiple sessions, whether to conduct the sessions in parallel or
in sequence. The treatments were arrived at by selecting various combinations of these (e.g., 1
session/4 reviewers, 2 sessions/2 reviewers without repair, etc.
Among the dependent variables measured were inspection effectiveness-in terms of observed number
of defects, as explained in Appendix B-and inspection interval-in terms of working days from
the time the code was made available for inspection up to the collection meeting. 4
2.2 Conducting the Experiment
To support the experiment, one of us joined the development team in the role of inspection quality
engineer (IQE). He was responsible for tracking the experiment's progress, capturing and validating
data, and observing all inspections. He also attended the development team's meetings, but had
no development responsibilities.
When a code unit was ready for inspection, the IQE randomly assigned a treatment and randomly
drew the review team from the inspector pool. In this way, we attempted to control for differences
in natural ability, learning rate, and code unit quality.
In addition, 6 more developers were called in at one time or another to help inspect 1 or 2 pieces of code, mostly
to relieve the regular pool during the peak development periods. It is common practice to get non-project developers
to inspect code during peak periods.
3 In this experiment, we used the term "session" to mean one cycle of the preparation-collection-repair process. In
multiple-session inspections, different teams inspect the same code unit.
4 For 2-session inspections, the longer interval of the two is selected.
The names of the reviewers were then given to the author, who scheduled the collection meeting.
If the treatment called for 2 sessions, the author scheduled 2 separate collection meetings. If repair
was required between the 2 sessions, then the second collection meeting was not scheduled until
the author had repaired all defects found in the first session.
The reviewers were expected to prepare sufficiently before the meeting. During preparation, reviewers
did not merely acquaint themselves with the code, but carefully examined it for defects.
They were not given any specific technical roles (e.g., tester or end-user) nor any checklists. On an
individual preparation form, they recorded the time spent on preparation, and the page and line
number and the description of each issue (each "suspected" defect). 5 The experiment placed no
limit on preparation time.
For the collection meeting one reviewer was selected as the moderator and another as the reader.
The moderator ran the meeting and recorded administrative data on a moderator report form.
This comprised the name of the author, lines of code inspected, hours spent testing the code before
inspection, and inspection team members. The reader paraphrased the code. During this activity,
reviewers brought up any issues found during preparation or briefly discussed newly discovered
issues. On a collection form, the code unit's author recorded the page and line number and description
of each issue regarded as valid, as well as the start and end time of the collection meeting.
Each valid issue was tagged with a unique Issue ID. If a reviewer had found that particular issue
during preparation, he or she recorded that ID next to the issue on his or her preparation form.
This enabled us to trace issues back to the reviewers who found them. No limit was placed on
meeting duration, although most lasted less than 2 hours.
After the collection meeting, the author kept the collection form and resolved all issues. In the
process he or she recorded on a repair form the disposition (no change, fixed, deferred), nature (non-
issue, optional, requires change not affecting execution, requires change affecting execution), locality
(whether repair is isolated to the inspected code), and effort spent (-
each issue. Afterwards, the author returned all paperwork to us. We used the information from
the repair form and interviews with the author to classify each issue as a true defect (if the author
was required to make an execution affecting change to resolve it), soft maintenance issue (any other
issue which the author fixed), or false positive (any issue which required no action).
In the course of the experiment, several treatments were discontinued because they were either not
performing effectively, or were taking too long to complete. These were the 1-session, 1-person
treatment and all 2-session treatments which required repair between sessions.
After months, we managed to collect data from 88 inspections, with a combined total of 130
collection meetings and 233 individual preparation reports. The entire data set may be examined
online at http://www.cs.umd.edu/users/harvey/variance.html.
2.3 Self-Reported Data
Self-reported data tend to contain systematic errors. Therefore we minimized the amount of self-reported
data by employing direct observation[19] and interviews[2]. The IQE attended 125 of the
5 A sample of this, and all other forms we used may be found at http://www.cs.umd.edu/users/harvey/
variance.html.
collection meetings 6 to make sure the meeting data was reported accurately and that reviewers
do not mistakenly add to their preparation forms any issues that were not found until collection.
We also made detailed field notes to corroborate and supplement some of the data in the meeting
forms. The repair information was verified through interviews with the author, who completed the
form. Our defect classification was not made available to the reviewers or the authors to avoid
biasing them.
Among the data that remained self-reported were the amount of preparation time and pre-inspection
testing time expended. We had two concerns in dealing with these data: a participant might
deliberately fail to tell the truth (e.g., reporting 2 hours preparation time when he or she really
did not prepare at all); participants might make errors in recording data (e.g., reporting 2 hours of
preparation time when the correct figure was 1.9 hours).
During the experiment, the IQE had an office next to those of the compiler development team,
and after working with the team for months, a great deal of trust was built up. Also, the
development environment routinely collects self-reported data, which is unavailable to management
at the individual level. Thus developers are conditioned to answer as reliably as they can. We
therefore see no reason to suspect that participants ever deliberately misrepresented their data.
As for the element of error, previous observational studies on time usage conducted in this environment
have shown that although there are always inaccuracies in self-reported data, the self-reported
data is generally within 20% of the observed data[18].
2.4 Results of the Experiment
Our experiment produced three general results:
1. Inspection interval and effectiveness of defect detection were not significantly affected by team
size (large vs. small).
2. Inspection interval and effectiveness of defect detection were not significantly affected by
number of sessions (single vs. multiple).
3. Effectiveness of defect detection was not improved by performing repairs between sessions of
two-session inspections. However, inspection interval was significantly increased.
From this we concluded that single-session inspections by small teams were the most efficient, since
their defect detection rate was as good as that of other formats, and inspection interval was the
same or less.
The observed number of defects and the intervals per treatment are shown as boxplots 7 in Figures 1
and 2, respectively. The treatments are denoted [1,or 2] sessions X [1,2, or 4] persons [No-
6 The unattended ones are due to schedule conflicts and illness.
7 We have made extensive use of boxplots to represent data distributions. Each data set is represented by a box
whose height spans the central 50% of the data. The upper and lower ends of the box marks the upper and lower
quartiles. The data's median is denoted by a bold line within the box. The dashed vertical lines attached to the
box indicate the tails of the distribution; they extend to the standard range of the data (1.5 times the inter-quartile
range). All other detached points are "outliers."[5]
OBSERVED
1sX1p 1sX4p 2sX1pR 2sX2pR

Figure

1: Observed Number of Defects by Treatment. The treatment labels are interpreted
as follows: the first digit stands for the number of sessions, the second digit stands for the number
of reviewers per session, and, for 2-session inspections, the 'R' or `N' suffix indicates "with repair"
or "no repair". As seen here, the distributions all seem to be similar except for 1sX1p and 2sX2pR,
which were discontinued after 7 and 4 data points, respectively.
repair,Repair]. (For example, the label 2sX1pN indicates a two-session, one-person, without-repair
inspection.) It can be seen that most of the treatment distributions are similar but that they vary
widely within themselves.
3 Sources of Variation
3.1 Process Inputs as Sources of Variation
In addition to the process structure, we see that differences in process inputs (e.g., code unit and
reviewers) also affects inspection outcomes. Therefore, we will attempt to separate the effects of
process inputs from the effects of the process structure. To do this we will estimate the amount of
variation contributed by these process inputs.
Thus, our first question from Section 1.1 may be refined as, (1) How will our previous results
change when we eliminate the contributions due to variability in the process inputs? (2) Did our
experimental design spread the variance in process inputs uniformly across treatments?
Our second question then becomes, (1) Are the differences due to process inputs significantly larger
than the differences in the treatments? (2) If so, what factors or attributes affecting the variability
of these process inputs have the greatest influence?

Figure

3 is a diagram of the inspection process and associated inputs, e.g., the code unit, the
1sX1p 1sX4p 2sX1pR 2sX2pR
INTERVAL
(working
days)

Figure

2: Pre-meeting Interval by Treatment. As seen here, the distributions all seem to be
similar except for 2sX2pR, which was significantly higher.
Process Inputs
Collection Output
Inspection Output
Reviewers
Preparation
Collection
Issues
Code Unit
Author
Total Defects
Repair and
Classification
Issues
Suppressed
Issues
Meeting
Gains
Observed
Defects
Maintenance
False
Positives

Figure

3: A Cause and Effect Diagram of the Inspection Process. The inputs to the process
(reviewers, author, and code unit) are shown in grey rectangles on the left, the solid ovals represent
process steps, the grouped boxes in between steps show the intermediate outputs of the process.
Time flows left to right.
reviewers, and the author. It shows how these inputs interact with each process step. This is an
example of a cause-and-effect diagram, similar to the ones used in practice[13], but customized here
for our use.
The number and types of issues raised in the preparation step are influenced by the reviewers
selected and by the number of defects originally in the code unit (which in turn may be affected
by the author of the code unit). The number and types of issues recorded in the collection step
are influenced by the reviewers on the inspection team and the author (who joins the collection
meeting), the number of issues raised in preparation, and the number remaining undetected in the
Process Inputs
Collection Output
Inspection Output
Reviewers
Preparation
Collection
Issues
Prep Time
Code Unit
Author
Total Defects
Repair and
Classification
Discussions
Meeting
Duration
Issues
Suppressed
Issues
Meeting
Gains
Observed
Defects
Maintenance
False
Positives
Language
Familiarity
Application
Experience
Inspection
Experience
Type of
Change
Functionality
Code
Structure
Code Size
Pre-inspection
Testing

Figure

4: The Refined Cause and Effect Diagram. This figure extends the inspection model
with some of the factors which we believe to affect reviewer and author performance and code unit
quality.
code unit.
3.2 Factors Affecting Inspections
We considered the factors affecting reviewer and author performance and code unit quality that
might systematically influence the outcome of the inspection. (Some of these are shown in Figure 4.)
In 3.2.1 through 3.2.3, we examine these factors, explain how they might influence the number of
defects, and discuss confounding issues. 8 As we examine them, we caution the reader from making
conclusions about the significance of any factor as a source of variation. The goal here is to establish
possible mechanisms, not to test significance of correlation. Each plot is meant to be descriptive,
showing the relationship of a factor against the number of defects, without eliminating the influence
of potentially confounding factors. The actual test of a factor's significance will be carried out when
the model is built. (See Section 4.)
3.2.1 Code Unit Factors
Some of the possible variables affecting the number of defects in the code unit include: size, author,
time period when it was written, and functionality.
8 We did not have any readily available measure of experience nor code complexity, so we did not include them in
our analysis.
OBSERVED
Figure

5: Size vs. Defects Found. This is a scatter plot showing the relation between the size
of the code and the number of defects found (cor = 0.40). The line indicates the trend of the data.
Note that the plot was "jittered" - a small, random offset was added to each point - to expose
overlapping points. (In fact, every scatter plot in this paper that may have overlapping points was
jittered.)
Code Size. The size of a code unit is given in terms of non-commentary source lines (NCSL). It
is natural to think that, as the size of the code increases, the more defects it will contain. From

Figure

5 we see that there is some correlation between size and number of defects found
0.4). 9
Author. The author of the code may inadvertently inject defects into the code unit. There were
6 authors in the project. Figure 6 is a boxplot showing the number of defects found, grouped
according to the code unit's author. The number of defects could depend on the author's level of
understanding and implementation experience.
Development Phase. The performance of the reviewers and the number of defects in the code
unit at the time of inspection might well depend also on the state of the project when the inspection
was held. Figure 7 is a plot of the total defects found in each inspection, in chronological order.
Each point was plotted in the order the code unit became available for inspection. There are two
distinct distributions in the data. The first calendar quarter of the project (July - September 1994)
- which has about a third of the inspections - has a significantly higher mean than the remaining
period. This coincided with the project's first integration build. With the front end in place, the
development team could incrementally add new code units to the system, possibly with a more
precise idea of how the new code is supposed to interact with the integrated system, resulting in
fewer misunderstandings and defects. In our data, we tagged each code unit as being from "Phase
1" if they were written in the first quarter and "Phase 2" otherwise.
9 Correlations calculated in this paper are Pearson correlation coefficients.
OBSERVED

Figure

In Authors' Code Units. These boxplots show the total defects
found in each inspection, grouped according to the code unit's author.
At the end of Phase 1, we met with the developers to evaluate the impact of the experiment on
their quality and schedule goals. We decided to discontinue the 2-session treatments with repair
because they effectively have twice the inspection interval of 1-session inspections of the same team
size. We also dropped the 1-session, 1-person treatment because inspections using it found the
lowest number of defects.

Figure

8 shows a time series plot of the number of issues raised for each code unit inspection. While
the number of true defects being raised dropped as time went by, the total number of issues did
not. This might indicate that either the reviewers' defect detection performance were deteriorating
in time, or the authors were learning to prevent the true defects but not the other kinds of issues
being raised.
Functionality. Functionality refers to the compiler component to which the code unit belongs,
e.g., parser, symbol table, code generator, etc. Some functionalities may be more straightforward
to implement than others, and, hence, will have code units with lower number of defects. Figure 9
is a boxplot showing the number of defects found, grouped according to functionality.

Table

1 shows the number of code units each author implemented within each functional area.
Because of the way the coding assignments were partitioned among the development team, the
effects of functionality are confounded with the author effect. For example, we see in Figure 9
that SymTab has the lowest number of defects found. However, Table 1 shows that almost all the
code units in SymbTab were written by author 6, who has the lowest number of reported defects.
Nevertheless, we may still be able to speculate about the relative impact of the two factors by
examining those functionalities with more than one author (CodeGen) and authors implementing
more than one functionality (author 6).
In addition, functionality is also confounded with development phase as Phase 1 had most of the
INSPECTIONS
OBSERVED
DEFECTS515Jul 94
Dec 94
Mar 95
Jun 95

Figure

7: Defects Detected Over Time. This is a plot of the inspection results in chronological
order showing the trends in number of defects found over time. The vertical lines partition the
plot into calendar quarters. Within each quarter, the solid horizontal line marks the mean of that
quarter's distribution. The dashed lines mark one standard deviation above and below the mean.
Author
CodeGen 8 6 8 6 28
Report 3 3
I/O 9 9
Library 12 12
Misc 11 11
Parser 4 4

Table

1: Assignment of Authors to Functionality. Each cell gives the number of code units
implemented by an author for a functionality.
code for the front end functionalities (input-output, parser, symbol table) while Phase 2 had the
back end functionalities (code generation, report generation, libraries).
Because author, phase, and functionality are related, they cannot all be considered in the model
as they account for much of the same variation. In the end, we selected functionality as it is the
easiest to explain.
Pre-inspection Testing. The code development process employed by the developers allowed
Inspections
Total
Issues
Recorded2060Jul 94
Dec 94
Mar 95
Jun 95

Figure

8: Number of Issues Recorded Over Time. This is a time series plot showing the trends
in number of issues being recorded over time. The vertical lines partition the plot into quarters.
Within each quarter, the solid horizontal line marks the mean of that quarter's distribution. The
dashed lines mark one standard deviation above and below the mean. Note that the scale of the
y-axis is different from the previous figure.
them to perform some unit testing before the inspection. Performing this would remove some of
the defects prior to the inspection. Figure 10 is a scatter plot of pre-inspection testing effort against
observed defects in inspection (cor = 0.15). One would suspect that the number of observed defects
would go down as the amount of pre-inspection testing goes up, but this pattern is not observed in

Figure

10.
A possible explanation to this is that testing patterns during code development may have changed
across time. As the project progressed and a framework for the rest of the code was set up, it may
have become easier to test the code incrementally during coding. This may result in code which
has different defect characteristics compared to code that was written straight through. It would
be interesting to do a longitudinal study to see if these areas had high maintenance cost.
3.2.2 Reviewer Factors
Here we examine how different reviewers affect the number of defects detected. Note that we only
look at their effect on the number of defects found in preparation, because their effect as a group
is different in the collection meeting's setting.
Reviewer. Reviewers differ in their ability to detect defects. Figure 11 shows that some reviewers
find more defects than others. 10 Even for the same code unit, different reviewers may find different
In addition to the 11 reviewers, 6 more developers were called in at one time or another to help inspect 1 or 2
pieces of code, mostly to relieve the regular pool during the peak development periods. We did not include them in
CodeGen Report I/O Library Misc Optimizer Parser SymTab
FUNCTIONALITY
OBSERVED

Figure

9: Defects Found In Code Units Classified by Functionality. These boxplots show
the total defects found in each inspection, grouped according to the code unit's functionality. Note
that specific authors were assigned to implement specific portions of the project's functionalities, so
the effects of functionality is usually not separable from that of authors - the independent factors
of author and functionality are confounded. For example, SymTab, which has the lowest number of
defects found, was implemented by author 6, who has the lowest number of reported defects.
numbers of defects (Figure 12). This may be because they were looking for different kinds of
issues. Reviewers may raise several kinds of issues, which may either be suppressed at the meeting,
or classified as true defects, soft maintenance issues (issues which required some non-behavior-
affecting changes in the code, like adding comments, enforcing coding standards, etc.), or false
positives (issues which were not suppressed at the meeting, but which the author later regarded
as non-issues). Figure 13 shows the mean number of issues raised by each reviewer as well as the
percentage breakdown per classification. We see that some of the reviewers with low numbers of
true defects (see Figure 11), like Reviewers H and I, simply do not raise many issues in total.
Others, like Reviewers J and K, raise many issues but most of them are suppressed. Still others,
like Reviewers E and G, raise many issues but most turn out to be soft maintenance issues. The
members of the development team (Reviewers A to F) raise on average more total issues (see left
plot in Figure 13), though a very high percentage turn out to be soft maintenance issues (see right
plot in Figure 13), possibly because, as authors of the project, they have a higher concern for its
long-term maintainability than the rest of the reviewers. An exception is Reviewer F, who found
almost as many true defects as soft maintenance issues.
Preparation Time. The amount of preparation time is a measure of the amount of effort the
reviewer put into studying the code unit. For this experiment, the reviewers were not instructed
to follow any prescribed range of preparation time, but to study the code in as much time as they
think they need. Figure 14 plots preparation time against defects found, showing a positive trend
but little correlation
this analysis because they each had too few data points.
PRE-INSPECTION TESTING (HOURS)
OBSERVED
Figure

10: Pre-inspection Testing Effort vs. Defects Found. This is a scatter plot showing
how the amount of pre-inspection testing related to the number of defects found in inspection (cor
0.15). Note that the pre-inspection testing data was self-reported by the author. Points cluster
at the quarter hours because we asked the authors to only record to that precision.
Even if preparation time is found to be a significant contributor, it must be noted that preparation
time depends not only on the amount of effort the reviewer is planning to put into the preparation,
but also on factors related to the code unit itself. In particular, it is influenced by the number of
defects existing in the code, i.e., the more defects he finds, the more time he spends in preparation.
Hence, high preparation time may be considered a consequence, as well as a cause, of detecting a
large number of defects. Further investigation is needed to quantify the effect of preparation time
on defects found as well as the effect of defects found on preparation time. Because there is no
way to tell how much of the preparation time was due to reviewer effort or number of defects, we
decided not to include it in the model. This is also in keeping with our analysis philosophy to only
consider factors that occur strictly before the response. (See Section 1.2.)
3.2.3 Team Factors
Team-specific variables also add to the variance in the number of meeting gains.
Team Composition. Since different reviewers have different abilities and experiences, and possibly
interact differently with each other, different teams also differ in combined abilities and experiences

Apparently, this mix tended to form teams with nearly the same performance. This is illustrated
in

Figure

which shows number of defects found by different 2-person teams in each 2sX2pN
inspection. Most of the time, the two teams found nearly the same number of defects. This may
be due to some interactions going on between team members. However, because teams are formed
randomly, there are only a few instances where teams composed of the same people were formed
REVIEWER
IN
PREPARATION

Figure

11: Number of Defects in Preparation per Reviewer. This plot shows the number
of true defects found in preparation by each reviewer.
more than once, not enough to study the interactions.
We incorporated the team composition into the model by representing it as a vector of boolean
variables, one variable per reviewer in the reviewer pool. When a particular reviewer is in that
collection meeting, his corresponding variable is set to "True".
Meeting Duration. The meeting duration is the number of hours spent in the meeting. In the
one person is appointed the reader, and he reads out the code unit, paraphrasing each
chunk of code. The meeting revolves around him. At any time, reviewers may raise issues related to
the particular chunk being read and a discussion may ensue. All these contribute toward the pace
of the meeting. The meeting duration is positively correlated with the number of meeting gains,
as shown in Figure 16 (cor = 0.57). As with the case of preparation time, the meeting duration is
partly dependent on the number of defects found, as detection of more defects may trigger more
discussions, thus lengthening the duration. It is also dependent on the complexity or readability
of the code. Further investigation is needed to determine how much of the meeting duration is
due to the team effort independent of the complexity and quality of the code being inspected. For
similar reasons as with preparation time (see the previous discussion on preparation time), we did
not include this in the model.
Combined Number of Defects Found in Preparation. The number of defects already found
going into the meeting may also affect the number of defects found at the meeting. Each reviewer
gets a chance to raise each issue he found in preparation as a point of discussion, possibly resulting
in the detection of more defects. Figure 17 shows some correlation between number of defects found
in the preparation and in the meeting (cor = 0.4).
INSPECTIONS
OBSERVED
IN
PREPARATION
PER

Figure

12: Reviewer Performance Per Inspection. This shows the number of defects found in
preparation by each reviewer, grouped according to inspection. Each column represents one inspec-
tion. The points in that column represent the number of true defects reported during preparation
by each reviewer in that inspection. The columns were ordered according to increasing means.
4 A Model of Inspection Effectiveness
4.1 Building the Model
To explain the variance in the defect data, we built statistical models of the inspection process,
guided by what we knew about it. Model building involves formulating the model, fitting the
model, and checking that the model adequately characterizes the process. We built the models in
the S programming language[3, 6].
Using the factors described in the previous section, we modeled the number of defects found with
a generalized linear model (GLM) from the Poisson family. 11 We started with a model which had
all code unit factors, all reviewers, and the original treatment factors, represented by the following
RA +RB +RC +RD +RE +R F +RG +RH +R I +R J +RK (1)
In this model, Functionality and Author are categorical variables represented in S as sets of dummy
11 The generalized linear model and the rationale for using it are explained in Appendix C.
We used S language notation to represent our models[6, pp. 24-31]. For example, the model formula y -
is read as, "y is modeled by a, b, and c."
SUPPRESSED ISSUES
NUMBER
OF
ISSUES
RAISED
IN
PREPARATION
REVIEWER
OF
ISSUE

Figure

13: Classification of Issues Found in Preparation. The bar graph on the left shows
the mean number of issues found in preparation by each reviewer, broken down according to issue
classification. The bar graph on the right shows the percentage breakdown.
PREPARATION TIME (HOURS)
OBSERVED
IN
PREPARATION
Figure

14: Preparation Time vs. Defects Found In Preparation. This is a scatter plot
showing how the amount of preparation time related to the number of defects found in preparation
pp. 20-22,32-36]. They have 7 and 5 degrees of freedom, respectively.
Stepwise model selection heuristic 13 selected the following model.
13 Stepwise model selection techniques are a heuristic to find the best-fitting models using as few parameters as
possible. To avoid overfitting the data, the number of parameters must always be kept small or the residual degrees
of freedom high. To perform stepwise model selection we used the step() function in S[6, pp. 233-238].
INSPECTIONS (2sX2pN only)
OBSERVED
PER
Figure

15: Team Performance Per Inspection (2sX2pN only). This shows the total number
of defects found per session in each 2sX2pN inspection. Each column represents one inspection.
The points in that column represent the total number of true defects reported in preparation and
meeting by each team. "1" and "2" plot the number of defects found by the first and second teams,
respectively. The columns are ordered by mean defects found.
RB +RC +R F +RG +RH +R I
This resulting model is not satisfactory because it retained many factors, making it difficult to
interpret. Also, even though these factors were considered important by the stepwise selection
criteria, some of them do not explain a lot of the variance. So we increased the selection threshold
to produce a smaller model. 14 Increasing the selection threshold did not simplify the model initially,
until, at one point, a large number of factors were suddenly dropped. The resulting model then
was:
It must be noted that the factors left out of the model are not necessarily unimportant. We believe
that there are other possible models for our data. In particular, Phase was considered important.
Phase is a surrogate variable representing the change in defects being found over time. Figure 7
clearly showed that something had changed over time but it is not clear what caused it. The
reason why this change over time explains a significant part of the variability may be attributable
to other factors. It is not clear which mechanism explains why Phase affects the number of defects.
14 In S, increase the scale parameter of the step() function.
OBSERVED
IN

Figure

Meeting Duration vs. Defects Found in Meeting. This is a scatter plot showing
how the amount of time spent in the meeting related to the number of defects found in the meeting
We also knew that Phase was confounded with Functionality (e.g., parser was implemented before
code generator). Since we knew also that some parts of the compiler are harder to implement than
others, the effects due to Functionality are easier to interpret than the effects due to Phase. Thus
we replaced Phase by Functionality in our final model:
The analysis of variance for this model is in Table 2. For comparison, the treatment factors were
added to the model. See Appendix C for details on calculating the significance values. The resulting
model explains - 50% of the variance using just 10 degrees of freedom.
In this model, Defects is the number of defects found in each of the 88 inspections. Note that
the presence of certain reviewers (Reviewers B and F) in the inspection team strongly affects the
outcome of the inspection. (See Table 2.) Note also the log transformation on the Code Size factor.
We do not really know what the actual underlying functional relationship is between Code Size
and Defects and so we applied square root, logarithmic, and linear transformations. Code Size
explained more variance under the log transformation than under other transformations.

Figure

diagnostic plots of the model's goodness-of-fit. The left plot shows the values
estimated by the model compared to the original values. It shows that the model reasonably
estimates the number of defects. The right plot shows the values estimated by the model compared
to the residuals. The residuals appear to be independent of the fitted values, suggesting that the
residuals are random.
DEFECTS OBSERVED IN PREPARATION
OBSERVED
IN

Figure

17: Defects Found in Preparation vs. Defects Found in Meeting. This is a scatter
plot showing how the combined amount of defects found in the preparation related to the number
of defects found in the meeting (cor = 0.4).
4.2 Lower Level Models
The inspection model is a high level description of the inspection defect detection process. The
effects of the process input and of the process structure can be compared using this model. But we
also know that defect detection in inspections is performed in two steps: preparation and collection.
These two steps may be considered as independent processes which can be modeled separately.
Doing so has several advantages. We can understand the resulting models of the simpler separate
processes better than the model for the composite inspection process. In addition, there are more
data points to fit - 233 individual preparations and 130 collection meetings, as opposed to 88
inspections.
4.2.1 A Model for Defect Detection During Preparation
To build the preparation model, we started with the same variables as in inspection model 1. Since
the same code unit was inspected several times, we added a categorical variable, CodeUnit, to the
regression model. CodeUnit is a unique ID for each code unit inspected.
Using stepwise model selection, we selected the variables that significantly affect the variance in the
preparation data. These were Functionality, Size, and Reviewers B, E, F, and J. This is represented
by the model formula:
repDefects
In this model, P repDefects is the number of defects found in each of the 233 preparation reports.
Factor Degrees of Sum of F Value Pr(F) Effect
Freedom Squares
Treatment Team Size 2 2.65 0.50 0.6062
factors Sessions 1 1.12 0.43 0.5146
Input log(Code Size) 1 59.63 22.66
factors Functionality 7 43.76 2.38 0.0303
Residuals 73 192.11

Table

2: Factors Affecting Inspection Effectiveness. The sum of squares measure the relative
contribution of each factor to the variance of the defect data. The probabilities indicate the significance
of the contribution. The last column for each significant scalar factor indicates whether
the factor was a positive or negative contributor to the number of defects. (Functionality had 7
degrees of freedom and different functionalities had different effects.)
SQRT(FITTED VALUE)
Figure

18: Examining the fit of the model. The left plot compares the values estimated by
the model with the original values (a perfect fit would imply that everything is on the line
There is a substantial correlation between the two 0.69). The right plot shows the relation
of the fitted values to the residuals. The residuals appear to be independent of the fitted values.
The presence of all the significant factors from the overall model at this level gives us more confidence
on the validity of the overall model.
4.2.2 A Model for Defect Detection During Collection
We started with the same variables as in preparation model. (See previous section.) Using stepwise
model selection to select the variables that significantly affect the meeting data we ended up with
Functionality, Size, and the presence of Reviewers B, F, H, J, and K. This is represented by the
RESIDUAL
(a)
NUMBER OF SESSIONS
RESIDUAL
(b)
REPAIR POLICY
RESIDUAL
(c)

Figure

19: Examining the Significance of the Experimental Treatment Factors. These
three panels depict the distribution of the residual data grouped according to Team Size, Sessions,
and Repair.
model formula:
MeetingGains
In this model, MeetingGains is the number of defects found in each of the 130 collection meetings.
This is again consistent with the previous two models.
4.3 Answering the Questions
We are now in a position to answer the questions raised in Section 3.1 with respect to inspection
effectiveness.
4.3.1 Will previous results change when process inputs are accounted for?
In this analysis, we build a GLM composed of the significant process input factors plus the treatment
factors and check if their contributions to the model would be significant.
The effect of increasing team size is suggested by plotting the residuals of the overall inspection
model, grouped according to Team Size (Figure 19(a)). We observe no significant difference in the
distributions. When we included the Team Size factor into the model, we saw that its contribution
was not significant (p = 0:6, see Table 2). 15

Appendix

C Section C.3.1 describes how Tables 2 and 3 were constructed.
The effect of increasing sessions is suggested by plotting the residuals of the overall inspection
model, grouped according to Session (Figure 19(b)). We observe no significant difference in the
distributions. When we included the Session factor into the model, we saw that its contribution
was not significant (p = 0:5).
The effect of adding repair is suggested by plotting the residuals of the overall inspection model
(for those inspections that had 2 sessions), grouped according to Repair policy (Figure 19(c)). We
observe no significant difference in the distributions. When we included the Repair factor into the
model, we saw that its contribution was not significant (p = 0:2).
4.3.2 Did design spread process inputs uniformly across treatments?
We want to determine if the factors of the process inputs which significantly affect the variance are
spread uniformly across treatments. This is useful in evaluating our experimental design. Although
randomization guarantees that the long run distribution of the factors will be independent of the
treatments, we had a single set of 88 data points. Thus we felt it is important to know of any
imbalances in this particular randomization.
As an informal sanity check we took each of the significant factors in the overall inspection model
and tested if they are independent of the treatments. For each factor, we built a contingency table,
showing the frequency of occurrence of each value of that factor within each treatment. We then
used Pearson's - 2 -test for independence[4, pp. 145-150]. If the result is significant, then the factor
is not independently distributed across the treatments. Although the counts in the table cells are
too low for this - 2 -test to be valid, we use it as informal means to indicate gross nonuniformities
in the assignment of treatments.
Results show that the distribution of Reviewer B is independent of treatment
Functionality may be unevenly assigned to treatments.
Examining further shows us that Reviewer F never got to do any 1sX1p inspections, and that
Functionality was not distributed evenly because some functionalities were implemented earlier
than others, when there were more treatments.
Contingency tables only work with data which have discrete values. To test the independence of
log(Size) to treatment, we modeled it instead with a linear model, log(Size) - T reatment, to
determine if treatment contribution to log(Size) is significant. The ANOVA result
that it is not, indicating that there is no dependence between code sizes and treatment.
4.3.3 Are differences due to process inputs larger than differences due to process
structure?

Table

2 shows the analysis of variance for our model. The significance of the treatment factors'
contribution were included for comparison.
The table shows that differences in code units and reviewers drive inspection performance more
than differences in any of our treatment variables. This suggests that relatively little improvement
in effectiveness can be expected of additional work on manipulating the process structure.
4.3.4 What factors affecting process inputs have the greatest influence?
The dominance of process inputs over process structure in explaining the variance also suggests
that more improvements in effectiveness can be expected by studying the factors associated with
reviewers and code units that drive inspection effectiveness.
Differences in code units strongly affect defect detection effectiveness. Therefore, it is important to
study the attributes that influence the number of defects in the code unit. Of the code unit factors
we studied, code size was the most important in all the models. This is consistent with the accepted
practice of normalizing the defects found by the size of the code. The next most important factor
is functionality. This may indicate that code functionalities have different levels of implementation
difficulty, i.e., some functionalities are more complex than others. Because functionality is confounded
with authors, it may also be explained by differences in authors. And because it is also
confounded with development phase, another possible explanation is that code functionalities implemented
later in the project may have less defects due to improved understanding of requirements
and familiarity with implementation environment.
The choice of people to use as reviewers strongly affects the defect detection effectiveness of the
inspection. The presence of certain reviewers (in particular, Reviewer F) is a major factor in all
the models. It suggests that improvements in effectiveness may be expected by selecting the right
reviewers or by studying the characteristics and background of the best reviewers and the implicit
techniques by which they study code and detect defects.
5 A Model of Inspection Interval
Using the same set of factors, we also built a statistical model for the interval data. We measured
the interval from submission of the code unit for inspection up to the holding of the collection
meeting. Unlike defect detection, we do not see any further decomposition of the inspection process
that drives the interval. The author schedules the collection meeting with the reviewers and the
reviewers spend some time before the meeting to do their preparation. So instead of splitting the
inspection process into preparation and collection, we just modeled the interval from submission
to meeting.
A linear model was constructed from the factors described in the previous section. 16 We started
by modeling interval with the same initial set of factors as in the previous section. Using stepwise
model selection heuristic we arrived at the following model.
Even though we ended up with a small set of factors, the model was hard to interpret. It did not
make sense for Functionality to be an important factor influencing the length of the inspection
interval. In addition Functionality and Phase were confounded so they may be explaining part
of the same variance. Our belief was that they were masking the effect of the other confounded
The linear model was used here rather than the generalized linear model because the original interval data
approximates the normal distribution.
Factor Degrees of Sum of F Value Pr(F) Effect
Freedom Squares
Treatment Team Size 2 206.6 0.85 0.4308
factors Sessions 1 161.6 1.28 0.2619
Input Author 5 2195.0 3.62 0.0054
factors R I 1 242.1 2.00
Residuals 77 9340.86

Table

3: Factors Affecting Interval. The sum of squares measure the deviation contributed
by each factor to the mean of the interval data. The probabilities indicate the significance of the
contribution. The last column for each scalar factor in the model indicates whether the factor was
a positive or negative contributor to the interval. (Author had 5 degrees of freedom and different
authors had different effects.)
factor, Author. It makes more sense for Author to be in the model since he is the central person
coordinating the inspection. So we re-ran the stepwise model selection heuristic, instructing it to
always retain the Author factor. The result was:
Interval - Author +R I +Repair
In this model, Interval is the number of days from availability of code unit for inspection up to
the last collection meeting.
The analysis of variance for this model is in Table 3. For comparison, all the treatment factors were
added to the model. The model explains - 25% of the variance using just 7 degrees of freedom.
The low explanatory power of the model indicates the limited extent to which structure and inputs
affect interval and suggests that other factors (that were not observed in this study) are more
important in determining the interval. The presence of Repair confirms our earlier experimental
result stating that adding repair in between inspections increases the interval.
5.1 Model Checking

Figure

20 gives diagnostic plots of the model's goodness-of-fit. The left plot shows the values
estimated by the model compared to the original values. Because the model only explains 25% of
the variance, it has limited predictive capabilities. The right plot shows the values estimated by
the model compared to the residuals. The residuals appear to be independent of the fitted values.
5.2 Answering the Questions
We are now in a position to answer the questions raised in Section 3.1, with respect to inspection
FITTED VALUE
FITTED VALUE
Figure

20: Examining the fit of the model. The left plot compares values estimated by the
model with the original values (a perfect fit would imply that everything is on the line
There is some correlation between the two 0.48). The right plot shows the relation of the
fitted values to the residuals. The residuals appear to be independent of the fitted values.
5.2.1 Will previous results change when process inputs are accounted for?
In this analysis, we build a linear model, composed of the significant process input factors plus the
treatment factors and check if their contributions to the model are significant.
The effect of increasing team size is suggested by plotting the residuals of the interval model
consisting only of input factors, grouping them according to Team Size (Figure 21(a)). We observe
no significant difference in the distributions. When we included the Team Size factor into the
model, we saw that its contribution was not significant (p = 0:4, see Table 3).
The effect of increasing sessions is suggested by plotting the residuals of the interval model consisting
only of input factors, grouping them according to Session (Figure 21(b)). We observe no significant
difference in the distributions. When we included the Session factor into the model, we saw that
its contribution was not significant (p = 0:3).
The effect of adding repair is suggested by plotting the residuals of the interval model consisting
only of input factors (for those inspections that had 2 sessions), grouping them according to Repair
policy (Figure 21(c)). We have already seen that Repair has a significant contribution
to the model in the previous section and this is supported by the plot.
5.2.2 Are differences due to process inputs larger than differences due to process
structure?

Table

3 shows the factors affecting inspection interval and the amount of variance in the interval
that they explain. We can see that some treatment factors and some process input factors
contribute significantly to the interval. Among treatment factors Repair contributes significantly
to the interval. This shows that while changes in process structure do not seem to affect defect
detection, it does affect interval.
RESIDUAL
INTERVAL
(a)
NUMBER OF SESSIONS
RESIDUAL
INTERVAL
(b)
REPAIR POLICY
RESIDUAL
INTERVAL
(c)

Figure

21: Examining the Significance of the Experimental Treatment Factors. These
three panels depict the distribution of the residual data grouped according to Team Size, Sessions,
and Repair.
5.2.3 What factors affecting process inputs have the greatest influence?
The results of modeling interval show that process inputs explain only - 25% of the variance in
inspection interval even after accounting for process structure factors. Clearly, other factors, apart
from the process structure and inputs affect the inspection interval. Some of these factors may
stem from interactions between multiple inspections, developer and reviewer calendars, and project
schedule and may reveal a whole new class of external variation which we will call the process
environment. These are beyond the scope of the data we observed for this study but they deserve
further investigation.
6 Conclusions
6.1 Intentions and Cautions
Our intention has been to empirically determine the influence upon defect detection effectiveness
and inspection interval resulting from changes in the structure of the software inspection process
(team size, number of sessions, and repair between multiple sessions). We have extended the
analysis to study as well the influence of process inputs.
All our results were obtained from one project, in one application domain, using one language
and environment, within one software organization. Therefore we cannot claim that our conclusions
have general applicability until our work has been replicated. We encourage anyone
interested to do so, and to facilitate their efforts we have described the experimental conditions
as carefully and thoroughly as possible and have provided the instrumentation online. (See
http://www.cs.umd.edu/users/harvey/variance.html.)
6.2 The Ratio of Signal to Noise in the Experimental Data
Our proposed models of the inspection process proved useful in explaining the variance in the data
gathered from our previous experiment. From them we could show that the variance was caused
mainly by factors other than the treatment variables. When the effects of these other factors were
removed, the result was a data set with significantly reduced variance across all of the treatments,
which improved the resolution of our experiment. After accounting for the variance (noise) caused
by the process inputs, we showed that the results of our previous experiment do not change (we
see the same signal).
This has several implications for the design and analysis of industrial experiments. Past studies
have cautioned that wide variation in the abilities of individual developers may mask effects due to
experimental treatments[8]. However, even with our relatively crude models, we managed to devise
a suitable means of accounting for individual variation when analyzing the experimental results.
But ultimately, we will get better results only if we can identify and control for factors affecting
reviewer and author performance.
Note also that the overall drop in defect data over time (see Figure 7) underscores the fact that
researchers doing long term studies must be aware that some characteristics of the processes they
are examining may change during the study.
6.3 The Need for a New Approach to Software Inspection
When process inputs are accounted for, the results of the experiment show that differences in
process structure have little effect on defect detection. This reinforces the results of our previous
experiment. That work showed that single session inspection by a small team is the most efficient
structure for the software inspection process (fewest personnel and shortest interval, with no loss
of effectiveness-see summary in Section 2.4 above).
If this is the case, and we believe that it is, then further efforts to increase defect detection rates
by modifying the structure of the software inspection process will produce little improvement.
Researchers should therefore concentrate on improving the small-team-single-session process by
finding better techniques for reviewers to carry it out (e.g., systematic reading techniques[1] for the
preparation step, meetingless techniques[9, 17, 11] for the collection step, etc.
7 Future Work
7.1 Framework For Further Study
Our study revealed a number of influences affecting variation in the data, some internal and some
external to the inspection process.
Internal sources included factors from the process structure (the manner in which the steps are
organized into a process, e.g., team sizes, number of sessions, etc.), and from the process techniques
(the manner in which each step is carried out, the amount of effort expended, and the methods
used, e.g., reading techniques, computer support, etc.
External sources included factors from the process inputs (differences in reviewers' abilities and in
code unit quality) and from the process environment (changes in schedules, priorities, workload,
etc.
7.2 Premise for Improving Inspection Effectiveness
We believe that to develop better inspection methods we no longer need to work on the way the
steps in the inspection process are organized (structure), but must now investigate and improve
the way they are carried out by reviewers (technique).
7.3 Need for Continued Study of Inspection Interval
We have not yet adequately studied the factors affecting interval data. Some of the factors are
found in process structure (specifically repairing in between sessions) and process inputs, but much
of its variance is still unaccounted for. To address this, we must examine the process environment,
including workloads, deadlines, and priorities.

Acknowledgments

We would like to recognize Stephen Eick and Graham Wills for their contributions to the statistical
analysis. Art Caso's editing is greatly appreciated.



--R

IEEE Trans.
A methodology for collecting valid software engineering data.
The New S Language.

Graphical Methods For Data Analysis.
Hastie, editors. Statistical Models in S.
Model uncertainty
Substantiating programmer variability.
Computer brainstorms: More heads are better than one.
Estimating software fault content before coding.
An instrumented approach to improving software quality through formal technical review.
Correlation and Causality.
Successful Industrial Experimentation
Software research and switch software.
Two application languages in software produc- tion
Generalized Linear Models.
Electronic meeting systems to support group work.
Experimental software engineer- ing: A report on the state of the art
Understanding and improving time usage in software development.
An experiment to assess the cost-benefits of code inspections in large scale software development
Assessing software designs using capture-recapture methods
--TR
A Two-Person Inspection Method to Improve Programming Productivity
Electronic meeting systems
An experimental study of fault detection in user requirements documents
Estimating software fault content before coding
An improved inspection technique
Assessing Software Designs Using Capture-Recapture Methods
An experiment to assess the cost-benefits of code inspections in large scale software development
Experimental software engineering
An instrumented approach to improving software quality through formal technical review
Active design reviews
Statistical Models in S
Comparing Detection Methods for Software Requirements Inspections

--CTR
Frank Padberg, Empirical interval estimates for the defect content after an inspection, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida
Miyoung Shin , Amrit L. Goel, Empirical Data Modeling in Software Engineering Using Radial Basis Functions, IEEE Transactions on Software Engineering, v.26 n.6, p.567-576, June 2000
Dewayne E. Perry , Adam Porter , Michael W. Wade , Lawrence G. Votta , James Perpich, Reducing inspection interval in large-scale software development, IEEE Transactions on Software Engineering, v.28 n.7, p.695-705, July 2002
Trevor Cockram, Gaining Confidence in Software Inspection Using a Bayesian Belief Model, Software Quality Control, v.9 n.1, p.31-42, January 2001
Stefan Biffl , Michael Halling, Investigating the Defect Detection Effectiveness and Cost Benefit of Nominal Inspection Teams, IEEE Transactions on Software Engineering, v.29 n.5, p.385-397, May
Oliver Laitenberger , Colin Atkinson, Generalizing perspective-based inspection to handle object-oriented development artifacts, Proceedings of the 21st international conference on Software engineering, p.494-503, May 16-22, 1999, Los Angeles, California, United States
Oliver Laitenberger , Thomas Beil , Thilo Schwinn, An Industrial Case Study to Examine a Non-Traditional Inspection Implementation for Requirements Specifications, Empirical Software Engineering, v.7 n.4, p.345-374, December 2002
James Miller , Fraser Macdonald , John Ferguson, ASSISTing Management Decisions in the Software Inspection Process, Information Technology and Management, v.3 n.1-2, p.67-83, January 2002
Lionel C. Briand , Khaled El Emam , Bernd G. Freimut , Oliver Laitenberger, A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content, IEEE Transactions on Software Engineering, v.26 n.6, p.518-540, June 2000
Bruce C. Hungerford , Alan R. Hevner , Rosann W. Collins, Reviewing Software Diagrams: A Cognitive Study, IEEE Transactions on Software Engineering, v.30 n.2, p.82-96, February 2004
Andreas Zendler, A Preliminary Software Engineering Theory as Investigated by Published Experiments, Empirical Software Engineering, v.6 n.2, p.161-180, June 2001
software process;software inspection;empirical studies;statistical models