abstract: Requirements Engineering (RE) is a means to reduce the risk of delivering a product that does not fulfill the stakeholders' needs. Therefore, a major challenge in RE is to decide how much RE is needed and what RE methods to apply. The quality of such decisions is strongly based on the RE expert's experience and expertise in carefully analyzing the context and current state of a project. Recent work, however, shows that lack of experience and qualification are common causes for problems in RE. We trained a series of Bayesian Networks on data from the NaPiRE survey to model relationships between RE problems, their causes, and effects in projects with different contextual characteristics. These models were used to conduct (1) a post-mortem (diagnostic) analysis, deriving probable causes of sub-optimal RE performance, and (2) to conduct a preventive analysis, predicting probable issues a young project might encounter. The method was subject to a rigorous cross-validation procedure for both use cases before assessing its applicability to real-world scenarios with a case study.
author:
;
;
bibliography: references.bib
title: Data-driven Risk Management for Requirements Engineering: An Automated Approach based on Bayesian Networks

# Introduction

The purpose of Requirements Engineering (RE) is to elicit, document, analyze, and manage requirements to minimize the risk of delivering a system that does not meet the stakeholders' desires and needs . Over the last 30 years, a number of methods, processes, tools, and best practices have been proposed to support this goal. However, there is no silver-bullet method or process that fits every project. In fact, a large part of the job of a requirements engineer in practice is to observe and analyze the context and current state of a project carefully and decide how much and what kind of RE is beneficial. As already addressed in the above-mentioned definition of RE, this decision is often a matter of controlling risks. Conducting RE tasks always comes with costs that ideally pay off in the sense that they lower a particular risk for a project . Making such decisions demands social and technical skills but also a lot of experience. Recent studies have shown that lack of experience and lack of qualification of RE team members are the second and third most common causes for problems in RE (lack of time being the top cause) . As a result, a number of projects fail either because of too little RE leading to stakeholder dissatisfaction or too much RE leading to high costs and developer frustration.

In this paper, we propose a data-driven approach to risk management in RE. Our goal is to predict RE problems, their causes, and effects for a given project. Intuitively, these three phenomena depend on each other in the sense that causes lead to problems, which result in certain effects. Bayesian Networks can be used to characterize such dependencies quantitatively by conditional probabilities and update the probability of certain phenomena when other phenomena are observed. Therefore, we evaluated different versions of Bayesian Networks that model the relations between causes, problems, and effects in RE. We trained the models on data that was collected through two surveys with answers from 228 and 488 practitioners, respectively, about problems, causes, and effects encountered in real projects. These surveys also provide data on the context of the projects.

We use the trained models for the following two use cases:

-   **Post-Mortem Analysis:** Given a set of problems and effects observed in a failing or failed project, the approach diagnoses the most likely causes leading to these issues (known as *diagnostic reasoning* in literature ).

-   **Preventive Analysis:** Given a set of causes and effects observed in a new or running project, the approach predicts the most likely problems to be faced (known as *predictive reasoning* in literature ).

We implemented the approach as an easily consumable web service, on which we based a graphical user interface to enter evidence and analyze the resulting predictions.

We performed two types of evaluations for our approach. Firstly, we performed cross-validation to compare the predictive power of different models. We achieved the best results for both use cases with surprisingly simple models, which ignore the causal structure implied by the original survey but include a set of context factors. For varying probability thresholds $t \in \{0.3, 0.5, 0.7\}$, the best diagnostic model achieves recalls of 0.6, 0.48, 0.44 and precisions of 0.76, 0.92, 0.99, respectively. The best predictive model achieves recalls of 0.84, 0.69, 0.59 and precisions of 0.71, 0.89, 0.99. A ranking-based output of the top-5 predictions results in a recall of 0.81 and a precision of 0.38 for the best diagnostic model and a recall of 0.73 and a precision of 0.71 for the best predictive model.

Secondly, we conducted a case study in industry to evaluate the external validity of the approach. We compared and discussed the predictions of the tool with the expectations of an RE expert for the diagnostic reasoning use case. Furthermore, we elicited feedback regarding the importance of recall vs. precision for the problem and how the tool should be tailored in detail to support practitioners best. In a nutshell, the case study showed that the method achieves good congruence between its predictions and the results expected by the expert, but requires additional tuning towards high precision.

We conclude that such data-driven approaches are very likely to be practical and advantageous, but that the remaining potentials in the underlying data and the user interface should be realized first.

# Related Work

## NaPiRE Initiative

The survey data used for our analysis originates from the NaPiRE project, which was presented on several occasions . Formerly a German initiative, it has incorporated a variety of teams of other nationalities since its inception and is now supported by RE researchers from all over the world. Most analyses run on the data have so far been of a descriptive nature, e.g., comparing summary statistics from different countries , or trying to find the most prevalent problems, their causes, and their effects in RE projects . Two studies, however, applied Bayesian Networks to analyze the relationship between these items, with the purpose of either supporting the so-called *Defect Causal Analysis*  (the diagnostic reasoning use case) or allowing data-driven risk-management (the predictive reasoning use case) . Both studies relied on the commercial Netica tool and lack a sophisticated validation procedure including an evaluation of their predictive power.

## Bayesian Networks in SE and RE

While certainly not a common tool for software engineers, Bayesian Networks have seen a variety of applications in Software and Requirements Engineering according to a mapping study , ranging (in declining importance) from software fault detection over software project management to design and testing. We would position this work in the second category, software project management. The survey also examines the methodological approach taken in the field: 80% of the 117 works rely solely on categorical variables, just as we do, while empirical data is used in only about 45% of the cases to learn the parameters of the model. In 24% of the cases, the network layout is inferred from the data as well, which we do heuristically. According to a follow-up survey by the same authors , there has been a trend towards data-driven methods and continuous variables. These claims should, however, be seen in the light of the very low sample size of only 10 papers.

A survey of 20 studies matching our application in RE a bit more closely can be found in , confirming similar trends: the network layout is usually constructed manually, while some approaches use quantitative, data-driven methods for parameter learning. Two of these studies treat the more specific topic of the RE process: Tang et al.  use a Bayesian Network based on survey data to propose a set of requirements engineering techniques for different phases of the RE process. Nagy et al.  employ a network with a manually specified layout and parameters for development release planning and project health monitoring. None of the two approaches was systematically validated and different model designs were not evaluated.

## Use Cases

The diagnostic reasoning use case is inspired by concepts such as *Root Cause Analysis* or *Defect Causal Analysis* with original works stemming from the early 90s , focusing mainly on quality management techniques. An interesting list of data-driven approaches is presented by Solé et al.  (unfortunately only as a pre-print), including a variety of applications of Bayesian Networks to the topic.

The preventive analysis use case aids risk management by estimating likely problems. These risks can then be tackled by choosing matching RE methods, so it is possible to implement Just-in time-RE , Just enough RE , or Value-oriented RE  in an empirically founded way. The model-based risk management tools available so far  neither apply specifically to RE nor do they incorporate larger data sets.

# Bayesian Networks

Bayesian Networks (also called belief networks) are part of a class of stochastic models called *graphical models*, also including hidden Markov models and conditional random fields, which are popular in machine vision applications. They encode the joint stochastic distribution of a set of discrete and dependent random variables ${X = \{X_1, \ldots, X_n\}}$ in a directed acyclic graph such as the one depicted in Figure . Each node in the graph represents a random variable $X_i$ whose distribution is dependent on its predecessors $pa(X_i)$, and this conditional distribution is stored along with the node. The explicit specification of dependencies strongly reduces the size of the parameter vector $\boldsymbol{\theta}$: a toy network of $n = 4$ binomial variables $X_1, X_2, X_3, X_4$ with edges pointing from $X_i$ to $X_{i+1}$ would require a vector of $2^n = 16$ scalars in a naive joint distribution specification, while a Bayesian Network formulation only requires $2 \cdot n = 8$ scalars.

It follows from the above definition that a variable $X_i$ in such a network is (conditionally) independent from any other variable in the graph given its predecessors $pa(X_i)$, so the joint probability function of the model is

$$\begin{aligned}
 P(X_1, \ldots, X_n) = \prod_i^n P(X_i\;|\;pa(X_i), \boldsymbol{\theta})
\end{aligned}$$

Given $m$ instantiations of all variables $x_1^j, \ldots, x_n^j, 1 \leq j \leq m$ , it is thus possible to calculate the likelihood of such a model:

$$\begin{aligned}
 L(\boldsymbol{\theta}\;|\;x^1, \ldots, x^m) = \prod_{j=1}^m P(X_1 = x_1^j, \ldots, X_n = x_n^j\;|\;\boldsymbol{\theta})
\end{aligned}$$

The optimization of this function with (usually infeasible) analytical or (usually gradient-based) numerical methods yields the maximum likelihood point estimate of $\boldsymbol{\theta}$[^1].

If the actual instantiations of any other random variables are known, this evidence $E,\;{E \cap V_o = \emptyset}$ can be introduced into the network by manually modifying the conditional distributions at the respective nodes to generate a prediction tailored to the situation. $v(X)$ being the set of all possible assignments to a set of random variables, the conditional probability is calculated as follows:

$$\begin{aligned}
 P(V_o = v_o\;|\;E = e) &= \frac{P(V_o = v_o \wedge E = e)}{P(E = e)} \\
                      &= \frac{\sum_{v(X_i/E/V_o)} P(E = e, V_o = v_o)}{\sum_{v(X_i/E)} P(E = e, X_i = x_i)}
\end{aligned}$$

This marginalization operation is more expensive than it would be for a naive joint distribution formulation, but the reduced memory requirements and statistical advantages of the smaller parameter space generally outweigh this concern. In addition, a variety of optimized approximate algorithms (such as Belief Propagation, or Gibbs Sampling, which we used) exploit the specific structure of the inference problem to reduce the overall computation time considerably.

For a more in-depth introduction to the theoretical foundations of Bayesian Networks, we suggest the textbook by Koski and Noble .

# Approach

## Assumptions and Design Decisions

Our contribution to RE risk management is an approach that is able to infer and report probabilities of certain RE problems, their effects, and their likely causes. The approach *learns* this inference from a large set of observations recorded in other RE projects in the past. Conceptually, we assume that these three classes of phenomena are dependent in the sense that causes lead to problems, which result in effects. Therefore, we assume that the learning approach will benefit from "knowing" this causal structure, which is a form of domain knowledge. We used Bayesian Networks since they are a well-suited to include and model such causal domain knowledge.

## Preprocessing

| Parameter               |    2014 | 2018 |
|:------------------------|--------:|-----:|
| Participants            |     228 |  488 |
| …from Africa            |       0 |    3 |
| …from Asia              |       0 |   24 |
| …from Europe            |     126 |  208 |
| …from North America     |      28 |   40 |
| …from South America     |      74 |  185 |
| …from elsewhere         |       0 |   28 |
| Group size              | company | team |
| …$\;x \leq 50$          |      69 |  443 |
| …$\;51 \leq x \leq 250$ |      33 |   39 |
| …$\;251 \leq x$         |     114 |    4 |
| …unknown                |       2 |    2 |
| Development method      |         |      |
| …agile                  |      92 |  194 |
| …hybrid                 |      58 |  161 |
| …plan-driven            |      46 |  124 |
| …unknown                |      22 |    0 |

Descriptive statistics of the NaPiRE data sets

|                      | Source     | $V$  | 2014 | 2018 |
|:---------------------|:-----------|:-----|-----:|-----:|
| problems             | predefined | `P`  |   21 |   20 |
| causes               | coded      | `C`  |   92 |  120 |
| cause categories^1^  | predefined | `CC` |    5 |  n/a |
| effects              | coded      | `E`  |   49 |   55 |
| effect categories^2^ | predefined | `EC` |    5 |  n/a |

Available problems, causes, and effects per data set

^1^Input, Method, Organization, People, Tools
^2^Implementation, Organization, Product, Customer, Validation

The 2014 NaPiRE data was obtained from Figshare, as proposed by the project . On request, the NaPiRE team provided the most recent data from the 2018 edition of the survey. A short summary of both data sets in terms of descriptive statistics is available in Table ; for more details, please refer to Méndez Fernández et al. . Both consist of a set of context factor variables for each subject, which was generated from closed questions, and five answers to the problems-causes-effects question: the participants were asked to think of a recent project and to select five problems experienced in the project from a closed list. Afterwards, they were asked to assign a rank $r \in \{1 \ldots 5\}$, a cause, and an effect to each of these problems. The latter two were coded manually, mostly in accordance with the principles established by Grounded Theory . The 2014 data set provided coarse categories into which causes and effects were grouped by the authors. Merging both data sets would have been a natural step to increase the statistical foundation of our approach, but was unfortunately prevented by incompatibilities due to survey improvements and the distinct manual coding processes.

We assigned each of the available variables to a set $V$ (the *variable type*, as depicted in Tables  and ) and then transformed it into one or more binary variables $v_i \in V$ according to the following variable type specific rules.

-   Problem, cause, and effect questions: Add one binary variable per possible answer. The variable is true if the subject selected this answer in the survey, false otherwise.

-   Cause and effects categories: Add one binary variable per category. True if the cause selected by the subject belongs to this category, false otherwise.

-   For each context factor, we added a variable type $V_c$, transforming it according to the data type mentioned in Table :

    -   Binary: Add one binary indicator variable.

    -   Categorical: Add one binary indicator variable per value.

    -   Ordinal: Add one binary indicator variable per value.

    -   Continuous: Discretize into a set of equiprobable intervals and add one binary indicator variable per interval.

This resulted in eight variable types for both, the 2014 and the 2018 data set with a total of 196 (2014) or 216 binary variables (2018). Given that only 28 (2014)[^2] or 20 (2018)[^3] of the $v_i$ can actually be true for each participant, the input data matrix is relatively sparse.

| 2014                         | $V$   | type        | indicators |
|:-----------------------------|:------|:------------|-----------:|
| company size                 | `CS`  | categorical |          8 |
| development method^1^        | `CDM` | categorical |          5 |
| distributed projects         | `CD`  | binary      |          1 |
| 2018                         |       | type        |            |
| team size                    | `CS`  | continuous  |          6 |
| development method^2^        | `CDM` | ordinal     |          5 |
| distributed project          | `CD`  | binary      |          1 |
| quality of customer relation | `CR`  | ordinal     |          5 |
| system type^3^               | `CT`  | categorical |          3 |

Available context factors per data set

^1^Waterfall, V-Model XT, Scrum, XP, RUP
^2^Agile, rather agile, hybrid, rather plan-driven, plan-driven
^3^Embedded system, business information system, hybrid

## Network Construction

In this work, the graph representing a Bayesian Network is defined by its architecture $\mathcal{A}$, which is a set of tuples of variable types. Each $(V_i, V_j) \in  \mathcal{A}$ indicates that (1) all binary variables $v \in V_i \cup V_j$ are contained in the graph as a node and (2) that an edge is added from each $v_i \in V_i$ to each $v_j \in V_j$. For example, an architecture $\mathcal{A} = \{(\texttt{C}, \texttt{P})\}$ specifies that all cause nodes are connected to all problem nodes, edges pointing to the problem nodes. This would result in $92 \cdot 21 = 1932$ (2014) or $120 \cdot 20 = 2400$ (2018) edges. More nodes and edges to other variable types can be specified by appending more tuples to $\mathcal{A}$ as long as the restrictions the inference algorithm places on the graph are respected.

These considerations show that even for simple architectures, the resulting graph quickly suffers from the curse of dimensionality, which is aggravated by the manually coded cause/effect statements in the survey leading to a high number of nodes. Learning of and inference on such models would require prohibitively large amounts of memory and CPU time, so we introduced two simple filter mechanisms to reduce complexity:

-   **Minimum Variable Occurrence Filter:** The number of true values for each variable in the data is counted. If this number is less than $f(V)$ (i.e., less than $f$ subjects reported this fact), its node is excluded from the graph. With this filter, variables with very little support in the dataset can be excluded.

-   **Minimum Relation Occurrence Filter:** The number of times the two variables connected by an edge are both true is counted. If this number is less than $g(V_1, V_2)$ (i.e., the dependence was reported by less than $g$ subjects), the edge is excluded from the graph. With this filter, relations with very little support in the dataset can be excluded.

More fine-grained control was achieved by not summing the number of occurrences of nodes or edges, but of the inverse rank $r_{\text{inv}} = 5 - r$ of a given cause-problem-effect triple. These heuristics worked well in our case, although it might be an option to explore more sophisticated approaches like the K2 structure learning algorithm .

In the above notation, a use case is equivalent to a set of binary variables $V_o$ which constitutes the output of a model. The choice of $\mathcal{A}$ is independent of the use case, as long as all variables in $V_o$ are included in $\mathcal{A}$. Any model can thus be applied to any use case; early experimentation has shown, however, that $\mathcal{A}$ has a major influence on the quality of the predictions. We formally define our use cases as follows:

Diagnostic reasoning
$$\begin{aligned}
 V_{D} = \{ v \in \texttt{C}\;|\;v \text{ is contained in }\mathcal{A} \}
\end{aligned}$$

Predictive reasoning
$$\begin{aligned}
 V_{P} = \{ v \in \texttt{P}\;|\;v \text{ is contained in }\mathcal{A} \}
\end{aligned}$$

In other words, diagnostic reasoning is the prediction of causes ($\texttt{C}$) and predictive reasoning is the prediction of problems ($\texttt{P}$).

An example, which will later be called the *Survey architecture*, is inspired by the causality assumptions of the NaPiRE survey:

$$\begin{aligned}
 \mathcal{A} = \{ (\texttt{C}, \texttt{P}), (\texttt{P}, \texttt{E}) \}
\end{aligned}$$

It is depicted graphically in Figure  with unrealistically high filter values to allow for a readable representation including individual binary variables. The following visual representations will be limited to variable types only.

## Implementation and Reproducibility

In order to circumvent the performance problems verbally reported by other authors , we implemented this method with Julia , relying on the (formidable) BayesNets.jl library  for inference and on Graphviz for visualization . The algorithms are wrapped in a simple web service to allow for easy integration into RE support tools and surveys. We are committed to ensuring the reproducibility of the following results, so our code and the data are freely accessible. We cordially invite other researchers to verify and extend this work, available at our GitHub repository[^4] or as a static reproduction package[^5].

# Evaluation

We validated our approach in two steps. First, to evaluate whether we produce predictions that are consistent with the data, we performed a 10-fold cross-validation (leaving out 30 samples in each iteration, Section ). Second, we selected the architecture yielding the best performance and conducted a case study to test its applicability to real-world scenarios (Section ).

## Internal Validity

```latex
\begin{table*}
\centering
\caption{Results of the internal validation \label{tab:internal-eva}}
\begin{tabular}{cccccrrrrrrrr}
 \toprule
 use case & \multicolumn{2}{l}{architecture}    & dataset & \(|V_o|\)\textsuperscript{1} & \(\overline{acc}\)  & \(\overline{rec}\)  & \(\overline{pre}\)  & \(rre(5)\) & \(rpr(5)\) \\
 \midrule
 \(V_D\) & \(\mathcal{A}_0\) & Baseline                    & 2018    & 29       &  0.89    & 0.07     & 0.04     &  0.32   &  0.16   \\
 \(V_D\) & \(\mathcal{A}_1\) & Kalinowski            & \textbf{2014} & 28       &  0.83    & 0.10     & 0.02     &  0.30   &  0.10   \\
 \(V_D\) & \(\mathcal{A}_2\) & Inverse Kalinowski    & \textbf{2014} & 28       &  0.92    & 0.12     & 0.37     &  0.48   &  0.15   \\
 \(V_D\) & \(\mathcal{A}_3\) & Survey                      & 2018    & 25       &  0.89    & 0.13     & 0.23     &  0.47   &  0.21   \\
 \(V_D\) & \(\mathcal{A}_4\) & Inverse Survey              & 2018    & 25       &  0.89    & 0.14     & 0.32     &  0.53   &  0.21   \\
 \(V_D\) & \(\mathcal{A}_5\) & Simple                      & 2018    & 26       &  0.89    & 0.18     & 0.35     &  0.57   &  0.26   \\
 \(V_D\) & \(\mathcal{A}_6\) & Simple with context         & 2018    & 15       &  0.91  & 0.25 & 0.81 &  \textbf{0.88}    &  0.26   \\
 \(V_D\) & \(\mathcal{A}_7\) & Inverse Simple              & 2018    & 30       &  0.90    & 0.33     & 0.73     &  0.66   &  0.30   \\
 \(V_D\) & \(\mathcal{A}_8\) & Inverse Simple with context & 2018    & 24       & \textbf{0.93}    & \textbf{0.54}  & \textbf{0.83}
                                                                                &  0.81   &  \textbf{0.38}   \\
 \midrule
 \(V_P\) & \(\mathcal{A}_0\) & Baseline                    & 2018    & 20       &  0.71    & 0.26     & 0.20     &  0.44   &  0.39   \\
 \(V_P\) & \(\mathcal{A}_1\) & Kalinowski            & \textbf{2014} & 20       &  0.74    & 0.28     & 0.59     &  0.47   &  0.39   \\
 \(V_P\) & \(\mathcal{A}_2\) & Inverse Kalinowski    & \textbf{2014} & 20       &  0.73    & 0.30     & 0.62     &  0.50   &  0.43   \\
 \(V_P\) & \(\mathcal{A}_3\) & Survey                      & 2018    & 20       &  0.8     & 0.31     & 0.71     &  0.59   &  0.51   \\
 \(V_P\) & \(\mathcal{A}_4\) & Inverse Survey              & 2018    & 20       &  0.81    & 0.49     & 0.75     &  0.63   &  0.55   \\
 \(V_P\) & \(\mathcal{A}_5\) & Simple                      & 2018    & 19       &  0.84    & 0.57     & 0.82     &  0.70   &  0.60   \\
 \(V_P\) & \(\mathcal{A}_6\) & Simple with context\textsuperscript{2}
                                                           & 2018    & 20       &  \textbf{0.89}    & \textbf{0.73}
                                                                            & \textbf{0.84}     &  \textbf{0.73}   &  \textbf{0.71}  \\
 \(V_P\) & \(\mathcal{A}_7\) & Inverse Simple              & 2018    & 19       &  0.77    & 0.41     & 0.69     &  0.61   &  0.53   \\
 \(V_P\) & \(\mathcal{A}_8\) & Inverse Simple with context & 2018    & 19       &  0.82    & 0.47     & 0.80     &  0.66   &  0.63   \\
 \bottomrule
\end{tabular}\\[1ex]

% & time\textsuperscript{2} & loc & job
% &  0.0 min & smarti & 12
% &  5.0 min & local  &  2
% &  6.8 min & local  & 23
% &  0.7 min & local  & 24
% &  0.4 min & smarti & 14
% &  0.6 min & smarti & 15
% &  0.3 min & local  & 22
% &  1.1 min & smarti & 11
%
% &  0.6 min & local  & 21
%
% &  0.0 min & smarti & 13
% &  0.0 min & local  & 25
% &  0.3 min & local  & 27
% &  0.0 min & local  & 29
% &  0.0 min & local  & 30
% &  0.3 min & smarti & 16
% &  0.0 min & local  & 28
% &  0.0 min & local  & 3
% &  3.3 min & smarti & 18
% \textsuperscript{2}Total validation time per 300 inferences. Please note that these numbers are not directly comparable since this includes training times for 10 models and validations have been run with differing parallelism settings due to memory limits.

\begin{minipage}{0.7\textwidth}
  \footnotesize
  \textsuperscript{1}We kept the number of output variables close to 30 (\(V_D\)) and 20 (\(V_P\)) to produce comparable metrics. Divergences are caused by the limited amount of available memory.\\
  \textsuperscript{2}To reduce training time for this particular architecture, we had to limit the number of parents per node to 15.
\end{minipage}
\end{table*}
```

### Architectures

We defined the following eight architectures to be cross-validated. Four of them are inspired by the available literature.

$\mathcal{A}_1$:

:   Kalinowski architecture
    \*\[0.1cm\]

    <embed src="figures/kalinowski.pdf" />

$\mathcal{A}_2$:

:   Inverse Kalinowski architecture
    \*\[0.1cm\]

    <embed src="figures/kalinowski-inverse.pdf" />

$\mathcal{A}_3$:

:   Survey architecture
    \*\[0.1cm\]

    <embed src="figures/survey.pdf" />

$\mathcal{A}_4$:

:   Inverse survey architecture
    \*\[0.1cm\]

    <embed src="figures/survey-inverse.pdf" />

To estimate whether our method achieves better performance than simple guessing does, we also defined a baseline algorithm $\mathcal{A}_0$, predicting for each $v_o \in V_o$ its relative frequency in the input data set.

$\mathcal{A}_1$ and $\mathcal{A}_2$ were evaluated on the 2014 data set since it is the only one providing the necessary cause/effect categories. The remaining architectures were evaluated on the 2018 data set due to the higher number of participants and the better availability of context factors. Filter parameters were determined empirically by repeatedly running the validation on standard hardware, incrementing the filter values until results were obtained within 48 hours and without provoking out-of-memory errors. This results in a maximum duration of $48 \text{h} / 300 \approx 10$ minutes per inference, which we deemed the maximum acceptable inference duration for the case study.

### Metrics

Each tuple of architecture $\mathcal{A}$ and use case $V_o$ was evaluated with the following metrics. Let $s$ be the number of samples in the validation set, $i \in \{1, \ldots, s\}$ the sample index, and $t = \{ 0.1, 0.2, \ldots, 0.9 \}$ a set of probability thresholds. For convenience, $E_i(v_o)$ is the variable's actual value for the sample $i$, ${T_i(v_o, t) = (P(V_o = v_o| E = e_i ) > t)}$ is a boolean indicator function based on the evidence $e_i$ given by sample $i$, and boolean values are equal to 1 or 0 in summation if they are true or false, respectively. $V_o^k = \{ v_1, \ldots, v_k\}$ is the set of the $k$ output variables with the highest predicted probabilities.

-   **binary accuracy**: the number of correct predictions to the number of all predictions. $$\begin{aligned}
        acc(t) = \frac{1}{s \cdot |V_o|} \sum_{i = 1}^s \sum_{v_o \in V_o} \bigg( T_i(v_o, t) = E_i(v_o) \bigg)
    \end{aligned}$$

-   **precision**: the number of correct predictions of true to the number of all predictions of true. $$\begin{aligned}
        pre(t) = \frac{1}{s} \sum_{i = 1}^s \left(  \sum_{v_o \in V_o | T_i(v_o, t)} E_i(v_o) \right) \left( \sum_{v_o \in V_o} T_i(v_o, t) \right)^{-1}
    \end{aligned}$$

-   **recall**: the number of correct predictions of true to the number of all actually true variables. $$\begin{aligned}
        rec(t) = \frac{1}{s}  \sum_{i = 1}^s \left( \sum_{v_o \in V_o | E_i(v_o)} T_i(v_o, t) \right) \left( \sum_{v_o \in V_o} E_i(v_o) \right)^{-1}
    \end{aligned}$$

-   **ranking precision**: the number of actually true variables in the ranking to the ranking length. $$\begin{aligned}
       rre(k) = \frac{1}{s} \sum_{i = 1}^s \left( \sum_{v_o \in V_o^k} E_i(v_o) \right) k^{-1}

    \end{aligned}$$

-   **ranking recall**: the number of actually true variables in the ranking to the number of all actually true variables. $$\begin{aligned}
       rpr(k) = \frac{1}{s} \sum_{i = 1}^s \left( \sum_{v_o \in V_o^k} E_i(v_o) \right) \left( \sum_{v_o \in V_o} E_i(v_o) \right)^{-1}

    \end{aligned}$$

As mentioned before, the dataset is relatively sparse, meaning that our baseline algorithm will correctly predict the absence of most output variables due to their low overall probability, resulting in high binary accuracy for this trivial method (known as the *accuracy paradox*). Thus, this metric is less of an indicator of quality but rather points out if there are fundamental misconceptions in our approach.

Recall should be relatively high for low thresholds $t$, but decrease as fewer variables are considered to be true. Precision should show the inverse behavior, i.e., starting low and increasing as the higher threshold filters out more false positives.

The ranking performance measures produce a good estimate of how accurate a result list similar to the one presented in Figure  is. There is a variety of other metrics measuring the quality of a ranking (most notably MAP, DCG, and NDCG ), but for the external validation with non-experts, a straightforward and easy-to-interpret definition appeared more relevant to us. Recall should increase with the length of the ranking $k$, while the precision value should decrease, which is in accordance with the canonical definitions. A notable difference is that for $k < 5$, 100% recall is unlikely to be achieved since the vast majority of survey participants responded with the expected 5 problem/recall/effect tuples. Because of this effect, we report $rre(5)$, and $rpr(5)$ instead of the respective averages.

### Results

The results of our internal evaluation are presented in Table . All architectures except $\mathcal{A}_1, \mathcal{A}_2$ perform clearly better than the baseline algorithm. This comparison is not entirely valid since both were evaluated on a different data set. However, our early experiments with the other architectures on the 2014 data set, which we do not present here for the sake of brevity, have hinted at a similar disparity in performance. Consequently, the effectiveness of a manual cause/effect categorization is dubious at best and the benefits should be weighed carefully against the effort required during the manual coding process.

Overall, simple models, which resemble Naive Bayesian classifiers and do not use the manifest causality assumptions implied by survey design, perform much better than models with a complex cause-effect chain. Their effectiveness can be improved further by making the $V_o$ depend on relevant context factors, resulting in our best options $\mathcal{A}_8$ for $V_D$ and $\mathcal{A}_6$ for $V_P$, achieving good average recall/precision tuples of 0.54, 0.83 and 0.73, 0.84, respectively. The accuracy of the produced rankings appears reasonable as well, although the ranking precision of the best diagnostic model is still disappointingly low (0.38).

Depending on a practitioner's needs, these metrics can be tuned by trading recall for precision and vice-versa. As visible in Figures  and , a wide range of values is achievable. Choosing the correct trade-off for a given application has been shown to be difficult , and further investigation with a systematic study would be necessary before suggesting a specific point on these curves.

Interestingly, architectures whose edges point towards $V_o$ perform better in many cases than architectures whose edges point away from $V_o$: For $V_D$, an inverse model consistently achieves better metrics than its respective non-inverted model. This is not entirely true for $V_P$, where the Survey and Kalinowski architectures show diverging behavior. We cannot provide a sound reason for this behavior and whether it transfers to other applications remains to be seen.

Furthermore, the baseline algorithm exhibits interesting behavior: unlike expected, recall and precision fall in unison, meaning that no reasonable prediction can be achieved by simply naming the generally most probable problems or causes. Taking into account the specific circumstances of a project is thus of paramount importance for risk management.

### Threats to Validity

The above results rely on a series of assumptions that need to be challenged. Our selection of architectures is by its very nature limited and does not cover all possibilities to connect the variable types. Even our definition of an architecture limits the number of Bayesian Networks which were evaluated because the basic definition does not force variables of the same type to be connected homogeneously. Furthermore, the choice of filter values by experience to reduce the computational load is, to a certain degree, arbitrary. Better or worse results could possibly be achieved by simply using different values.

Due to combinatorial effects, however, it is unlikely that an extensive evaluation will ever be possible and such computational trade-offs are a necessity. By starting with very simplistic architectures ($\mathcal{A}_5$–$\mathcal{A}_8$) whose filters we only tuned as far as necessary to be able to yield actual results, we established at least a lower bound on what should be possible to achieve on the given data sets. Adding the architectures known in the literature ($\mathcal{A}_1$–$\mathcal{A}_4$) helps to validate previously unchallenged assumptions.

The last threat emerges from the data set itself. There are no guarantees that the predefined problems and manually coded effects and causes are an internally coherent representation. Indeed, a simple random inspection of these items yielded a number of overlapping causes such as *Lack of project management*, *Poor project management*, *Missing project management*; clearly, if one of them is present, the other causes should be present as well. In addition, there is not always a clear causality: *Difficulties in project management*, which is basically synonymous to the above causes, is listed as an effect. There are other similar ambiguities in the data set whose effect is difficult to assess.

As for any data-driven method, the quality of the dataset is extremely important (see ). Although the NaPiRE dataset is the largest of its kind that we are aware of, there are still a number of underrepresented cases. It seems that the dataset is skewed towards participants from Europe and South America, and groups with small sizes. Therefore, the performance of our approach might not be well for the minority cases from the dataset.

A minor point is that the discretization of certain context factors naturally resulted in a reduction of the precision of our input data. The break points were chosen to be equiprobable, but there is no strong evidence that these break points are an inherently good choice.

## External Validity

### Research Questions

To evaluate the external validity, i.e., to assess the validity and perceived usefulness of our approach when applied in a realistic context, we are interested in the following research questions:

-   **RQ1:** How do the outputs of our tool compare with the assessment of an RE expert for a specific project?

-   **RQ2:** How do RE experts assess the usefulness of the tool and the style of presentation?

-   **RQ3:** In which contexts and for which ends would RE experts apply the tool?

### Study Design

To answer the research questions, we conducted a case study in the context of a German software consulting company. We contacted an RE expert of the company and asked him whether he is interested in giving feedback on a tool for RE risk prediction. After he agreed, we asked him to think of a current or past RE project and offered him support for one of the two addressed use cases.

The data collection for the case study was conducted as a semi-structured interview. Two authors and the RE expert participated in the interview that lasted around 60 minutes. The whole interview was recorded to support a detailed analysis. The interview comprised the following three parts:

*Part 1:* We asked the RE expert to describe the case project, name the major problems he encountered, and list their causes according to his opinion.

*Part 2:* Only now did we introduce the RE expert to our tool (as depicted in Figure ), backed by the model performing best in the internal validation (Section ). We went through the problems and effects offered by the tool and asked the RE expert whether any of them were present in the project. Afterwards, the tool returned a list of the 5 most likely causes together with their predicted probabilities. As additional information, the tool offered a visualization of all performance metrics and a graphical representation of the underlying graph. We asked the expert to assess the results of the tool in terms of precision, completeness, and level of abstraction.

*Part 3:* Finally, we asked the RE expert to discuss the following questions about the approach in general and the tool's particular result presentation:

1.  Which information presented in the user interface of our tool does the expert consider as important, what information is missing to make informed decisions?

2.  What is more relevant to the RE expert, precision of the presented results or recall? Does the expert favor a bounded list of top-X results or does he prefer a variable-length list of results above a certain probability threshold?

3.  How does the RE expert assess the impact of such data-driven predictions on his personal decisions?

4.  What is the most relevant target group for the approach? Which target groups may not benefit from the approach?

### Case Description

Our study participant is an RE consultant with 20 years of experience in general IT projects. During his early years, he worked on domain and business process modeling before focusing on testing and quality engineering. For three years, he has been working exclusively as an RE consultant.

During the case study, he was interested in analyzing a recently finished project. His role was to coach the client company's product owners for one year. The company had recently decided to move towards more agile practices.

The project itself was concerned with the enhancement of an automation portal for a "digital factory" with mainly automotive products. 250 team members distributed over several locations (Germany, Eastern Europe, India) were involved, a large team compared to the size in the NaPiRE dataset. The applied development process was Scrum and our study participant assessed his relation to the customer as mostly neutral with better and worse moments.

According to him, the project lends itself to a *Post-Mortem-Analysis* (diagnostic reasoning) to identify the causes of several RE problems he experienced in the course of the project.

```latex
\begin{table*}
\centering
 \caption{Problems and causes experienced by the RE expert \label{tbl:cs-participant-answers}}
 \begin{tabular}{ll}
  \toprule
  Problem & Causes\\
  \midrule
   Bad team communication   & Top-down implementation of Scrum\\
                            & No culture of failure \\
                            & Traditional company culture \\[1ex]
   Product owners were not responsible & Agile teams setup according to system components, not features. \\
                                      & Team setup was immutable\\[1ex]
   Poor requirements quality & Poor knowledge about RE \\
                             & No QA for requirements\\[1ex]
   Very technical user stories   & Poor knowledge about agile methodology\\
                                 & Silent rejection of the agile methodology \\
  \bottomrule
  \end{tabular}
\end{table*}
```

<table>
<caption>Evidence provided by the RE expert</caption>
<thead>
<tr>
<th style="text-align: left;">Evidence</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">Problems</td>
</tr>
<tr>
<td style="text-align: left;"><p>Poor communication</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Poor product quality</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Difficulties in project management</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Misunderstandings (overall)</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Poor requirements quality (general)</p></td>
</tr>
<tr>
<td style="text-align: left;"></td>
</tr>
<tr>
<td style="text-align: left;">Effects</td>
</tr>
<tr>
<td style="text-align: left;"><p>Underspecified requirements</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Weak relationship between</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>customer and project team</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Communication flaws within the project team</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Insufficient support by customer</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Weak access to customer needs</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Incomplete or hidden requirements</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Stakeholders with difficulties</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>in separating requirements from solutions</p></td>
</tr>
<tr>
<td style="text-align: left;"><p>Unclear/unmeasurable non-functional requirements</p></td>
</tr>
</tbody>
</table>

<table>
<caption>Tool predictions and expert confirmation </caption>
<thead>
<tr>
<th style="text-align: right;">Rank</th>
<th style="text-align: left;">Cause</th>
<th style="text-align: center;">Conf</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: right;">1 (53%)</td>
<td style="text-align: left;">Missing domain knowledge</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">2 (52%)</td>
<td style="text-align: left;">Missing customer involvement</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">3 (52%)</td>
<td style="text-align: left;">Lack of a well-defined RE process</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">4 (52%)</td>
<td style="text-align: left;">Poor project management</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">5 (52%)</td>
<td style="text-align: left;">Lack of time</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">6 (51%)</td>
<td style="text-align: left;">Lack of requirements management</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">7 (48%)</td>
<td style="text-align: left;">Lack of experience of RE team members</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">8 (44%)</td>
<td style="text-align: left;">Communication flaws</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;"></td>
<td style="text-align: left;"><p>between team and customer</p></td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">9 (31%)</td>
<td style="text-align: left;">Poor requirements elicitation techniques</td>
<td style="text-align: center;"></td>
</tr>
<tr>
<td style="text-align: right;">10 (29%)</td>
<td style="text-align: left;">Lack of communication channels</td>
<td style="text-align: center;"></td>
</tr>
</tbody>
</table>

### Study Results

In the following, we present the results and relate them to our research questions.

**RQ1 – performance:** Table  lists the causes and the related problems named by the interviewee. We entered the evidence reported in Table  into the tool. Of the predicted top-10 causes (Table ), our expert confirmed 7 and rejected 3. These false positives also include the cause with the highest probability (*missing domain knowledge*), which was not an issue at all in the project. Besides the false positives, the expert assessed the mentioned causes as good matches. Especially, *Lack of a well-defined RE process* was a top match. He also confirmed the cause *Poor project management*. However, he considered this cause to be too coarse-grained without a more detailed definition, which is not provided by the data set.

**RQ2 – usefulness:** The general usage of the tool was considered straightforward and the presentation of the results as a ranking was perceived to be very appropriate. The expert judges the presentation of five items on the list as a good choice because it is long enough to allow a variety of possible causes to be presented while still being tractable in group discussions. For this reason, the precision of these five predictions is of high importance, much more than achieving high recall and covering the majority of all causes that might be present.

The presentation of probabilities along the ranking was perceived as a good means to communicate the approximate risk of following the tool's suggestions, although we should have stated more clearly that such a device cannot replace a fully-fledged RE process assessment and only provides very general hints at what might be going wrong in a project. When presenting the option to show the predicted probabilities for all causes, we observed an interesting effect: every item on this long list was interpreted as a prediction of the tool by the expert, regardless of the accompanying probability.

We concluded that, while a ranking of a given length is the preferable way to display the inference results, a hybrid presentation approach is more suitable: the list should be cut off at a given probability threshold so users are not tempted to consider items with an evidently low probability.

**RQ3 – applicability:** The RE expert proposed three primary target groups using the tool for the following purposes.

-   Classic projects with a project lead: Discussion input to improve the development process,

-   Agile teams: Discussion input to improve the development process, e.g., during the Scrum Retrospective,

-   Teams without or with inexperienced requirements engineer: Highlighting of low-hanging fruits to develop a more sophisticated RE process.

These scenarios align with his perception that the tool's results were most likely to influence group decisions, and are less likely to be able to have a tangible effect on an individual's judgments.

### Threats to Validity

The above paragraphs must be seen in the light of the chosen study design: the results of case studies are inherently difficult to generalize. Despite carefully choosing an experienced participant bringing insights from a variety of projects in Software and Requirements Engineering, we conclude from the case description that the findings are probably biased towards agile methodology in larger companies and that experiences in other contexts may differ.

Another issue of case study designs is psychological bias, whose abundance forces us to focus on a selected few. Courtesy bias is hard to exclude during an interview, so most probably the above results judge our tool more positively than justifiable. Consistency bias cannot be ruled out, either. We separated the manual analysis of the situation (*part 1* of the interview) from the tool introduction (*part 2*), but participants would still strive to interpret the tool's predictions in a way that would produce a consistent description of the situation. In particular, generic items like *Poor project management* lend themselves to such adjustments. We estimate that groupthink is less of an issue due to the participant's relatively independent position as a consultant in the project.

In the future, these concerns should be addressed by (1) interviewing RE experts from other backgrounds to increase our coverage of different contexts, and (2) making the tool generally available online and combining it with a questionnaire to allow anonymous feedback without a human interviewer to alleviate psychological biases.

# Conclusions

Based on the NaPiRE data set, we trained a series of Bayesian Networks to model cause-effect relationships in RE projects with different contextual characteristics. These models were firstly used to conduct a post-mortem analysis, deriving probable causes of sub-optimal RE performance, and secondly to conduct a preventive analysis, predicting probable issues a young project might encounter. The method was subject to a rigorous cross-validation procedure for both use cases before assessing its applicability to real-world scenarios with a case study.

Generally, the results are promising. For both use cases, we achieve good recall and precision values with simple network architectures neglecting the causal structure implied by the underlying data set. The same is true for the quality of probability-based rankings of predicted items produced by the networks, except for the precision of the rankings for the post-mortem analysis.

The case study involving a user-friendly interface to these models is equally supportive. The predicted causes generally matched the causes predicted by the interviewed RE expert and the presentation as a ranking was perceived as useful, although minor improvements remain. Precision was determined to be the driving performance metric in this context; unfortunately, this is the one metric in which our models perform sub-optimally. We identified a number of applications for our tool: In both, classic and agile projects its predictions can serve as valuable discussion input to improve the RE process. Moreover, it can help inexperienced teams to focus on the most worthwhile RE process and technique enhancements.

The internal validation questions the causality assumptions behind the design of the problems/causes/effects section in the NaPiRE survey: the fact that neglecting them yields considerably better performance puts into question whether what survey participants qualify as a cause actually is a cause and whether what they qualify as an effect actually is an effect. Given that distinguishing these is a notoriously difficult task even for scientists (with a large number of theoretical approaches and a variety of pitfalls such as spurious correlation), it might be too much to ask survey participants to always identify the causal chain correctly.

Loosening these assumptions in the survey, i.e., asking for correlations only, possibly supplying a predefined list of causes and effects instead of the manual coding effort, and using undirected graphical models with a less strict structure should help improve prediction quality. On the other hand, the fact that the causality assumption seems to play only a minor role opens the field for applying other learning approaches that do not rely on this kind of domain knowledge (e.g., neural networks  or k-nearest neighbors).

[^1]: The stochastic formulation of the problem enables the inference of confidence intervals for all parameters and the predictions, which might be an interesting extension of this work if practitioners show interest.

[^2]: 2014: 5 cause, 5 problem, 5 effect, 3 context factor, 10 category nodes.

[^3]: 2018: 5 cause, 5 problem, 5 effect, 5 context factor, 0 category nodes.

[^4]: <https://github.com/NaPiRE/project_riskdrivenRE>

[^5]: <https://doi.org/10.6084/m9.figshare.9692963>


author: Brian D. Josephson
Trinity College, Cambridge CB2 1TQ, UK
date: October 2011
title: Biological Observer-Participation and Wheeler's 'Law without Law' [^1]

# Abstract

It is argued that at a sufficiently deep level the conventional quantitative approach to the study of nature faces difficult problems, and that biological processes should be seen as more fundamental, in a way that can be elaborated on the basis of Peircean semiotics and Yardley's Circular Theory. In such a world-view, Wheeler's *observer-participation* and emergent law arise naturally, rather than having to be imposed artificially. This points the way to a deeper understanding of nature, where meaning has a fundamental role to play that is invisible to quantitative science.

## Keywords

Observer-participation, Peirce, semiotics, signs, interpretation, emergence, complexity, cognitive development, space-time generation, wholeness, symmetry, Circular Theory.

# Introduction

It is commonly assumed that nature can be described in terms of fixed mathematical laws. However, the discovery that the Standard Model cannot be reconciled with general relativity in a straightforward way has created problems for this point of view. An alternative is Wheeler's proposal to the effect that participation by observers, as postulated in some formulations of quantum mechanics, is the mechanism whereby physical laws emerge. According to Wheeler, that principle might suffice to build everything.

In Wheeler's article the gap between acts of observer-participancy and physical reality was not filled in, an insufficiency that we attribute to the absence of an appropriate theory of observation. In the following we discuss a biologically oriented scheme where observation plays a central role, and show how it can lead to the emergence of physical laws.

The structure of this scheme can be summarised as
$primordial\ reality \rightarrow circular\ mechanics \rightarrow semiotics\ and\ structure \rightarrow technological\ development \rightarrow regulatory\ mechanisms \rightarrow emergent\ laws.$ Here 'circular mechanics' is a reference to a generic scheme of biological organisation proposed by Yardley, encompassing among its aspects sign processes in accord with the semiosis concepts of Peirce, which in turn underlie processes of a technological character, among which we hypothesise are the capacity to form systems such as our universe, to which laws of a mathematical kind are applicable. In this way, we are able to link life, viewed from a generic point of view, to the origin of universes.

We discuss first of all the relationship between idealised situations in physics which can be characterised precisely in mathematical terms on the one hand and on the other, biology, which it will be argued is primarily concerned with patterns and only secondarily with quantities. The characteristics of biosystems are then related to the forward-looking role of signs, and to circular theory approach, thus paving the way to a more detailed analysis of universe generation.

# Physics vs. biology; mathematics vs. semiosis

Theoretical physics is mathematics-based, typically involving differential equations with respect to time. Such a mathematical approach carries the presumption that systems found in nature can be represented adequately by explicit formulae. Experimental biology gives the appearance of demonstrating the derivability of life from conventional physics, such investigations uncovering a great variety of processes that accord with known physics as well as having biological functions. However, things are not what they seem. To see this, compare life with a phenomenon of physics such as superconductivity. In the latter case there is a specific model, the BCS model, defined by a specific mathematical expression, which accords well with many experimental observations. Small changes in the model would have small consequences, and would not affect this agreement. Biosystems differ in that fine details may drastically affect behaviour; rather than there being a specific model there is a *landscape* of possibilities, with only the peaks reflecting viable systems. Thus the properties of biosystems cannot be accounted for on the basis of a first-principles computation, which could not apply to such a landscape.

Biosystems must therefore be addressed in a way different from the way systems that are the subject of mathematical physics are normally studied. They can be conceived of as systems that have passed certain tests, a situation similar to that of prime numbers, where in general a number can be shown to be prime only by testing for factors, rather than there being a formula that generates all primes. Despite the absence of such a formula, passing such tests has important implications. The situation addressed by Gödel, whereby there exist true statements that cannot be proved starting from specified axioms, is similar in the way it demonstrates limits of specifiability. In the biosystem case, the test-passing factor is related to viability, and is also responsible for different instances of an organism behaving similarly, which permits their non-quantitative analysis.

# The forward-looking aspect of life; semiosis

One way in which life differs from nature is general is the way it creates its own structures, in a way that does not admit of any very direct mathematical interpretation. Rather, in life we find systems that have come into existence that are able to pass particular tests, as required for the survival of the given system. One aspect of this is the *semiosis* discussed by Peirce, Semiotics empasises the role of information processing and more specifically the importance of the interpretation of *signs*, in the light of the fact that at the cognitive level the appropriate use and interpretation of signs is essential. In Peirce's scheme there is a specific, possibly context dependent, relationship between signs, and objects to which they are linked, with a third element, the *interpretant*, having the role of linking them. Typically, a complicated interpretant mechanism links the simpler sign and object, reliably producing a well defined situation linked to the sign.

The role played by signs in biological situations can be illustrated by the situation of road traffic. The fact that cars collide with each other much less frequently than if they were driven at random can be related to appropriate interpretation of the relevant signs. Large quantitative changes can be made, and the collision-avoidance phenomenon remains. This phenomenon, in a more general context, makes biology 'a different game' to ordinary physics.

Signs play an important role in advanced activities through the way complicated signs open up new possibilities, the power of natural language providing a simple illustration of this fact.

The question now arises how semiotic processes manifest and develop, and whether this can happen in the primoridal context which we imagine to be the source of universes and physical laws. A more global perspective is required, and we now discuss this in the light of Yardley's Circular Theory.

# Application of Circular Theory

Circular Theory is a work in progress, aimed at expressing structure and function in biological systems in its most basic conceptual form, the key elements being units ('circles'), links between units, and the tendency for units to form (unitisation).

We first discuss the terms unit and link. Unit is not defined in rigorous terms, the existence of units being something that is discovered though attempts to characterise systems of interest; a unit is something that it is convenient to treat as a whole. The concept of a unit may usefully be extended to refer to classes that it is convenient to deal with in an analysis, and it may equally well be applied to processes.

Turning to the concept of link, what is crucial in circular theory is the way systems are able to work together, acting effectively as a single system. A simple example is provided by a thermostatically controlled system, where a controller, together with a controlled system whose temperature is subject to variation from external inputs, become a system with approximately fixed temperature, while a more complicated case consists in a function present in a computer as a part of a program, interacting with some other system so as to exercise that function. A server-client situation such as a web browser interacting with a web server illustrates on the other hand a situation of mutual influence. The point is that there is a special kind of situation of 'systems being attuned to each other' that produces highly coordinated behaviour, and this is very relevant to mechanisms and to life generally. Yet another example is the correlation between the two strands of DNA, in which case the correlations are put to work in the service of copying information.

Intuitively (no attempt will be made here to formulate the concepts rigorously), the point is that the coupling between the systems concerned reduces the range of variation available to the joint system, while still making degrees of freedom available. Arguably, this will tend to happen spontaneously under certain circumstances (as when two clocks are coupled by placing them on a common platform). This coordination may also be induced by a third influence, as happens during learning involving the development of coordination between two processes.

## A packing model

The concepts of circular theory, including the 'attunement' concept, can be underpinned by an idea to the effect that what is involved at root is the *packing together* of a set of dynamical systems subject to certain constraints; indeed learning involves the attempt to make systems that are interacting generate activity that conforms to particular constraints. As an implementation mechanism, we suppose that in place of fixed structures we are concerned in each case with a *collection* of structures distinguished from each other by a set of bits, which are adjusted bit by bit until a high degree of conformance to the relevant constraints is achieved. This process is equivalent to that of Ross Ashby's ultrastability.

We can take the idea further by invoking an additional system that can pack other structures together 'intelligently', that is to say by recognising signs and responding appropriately, in the manner of semiotic theory. Such a grouping of three systems can be expected to cohere together more effectively than with situations where there is no such intelligent response to signs. With such a grouping there is no essential difference between the three components, and all three can be considered interpretants, each interpreting signs originating in the other two systems, and also the interactions between these two.

Conversely, the splitting of a unit into three subunits brings into existence a triadic situation of the kind discussed by Peirce. What remains when systems disperse in this way is the potential to bond with systems similar to those with which they have previously formed the capacity to bond. In this way we can understand creative development, where new structures form, with new capacities.

These points can be illustrated with analogies from chemistry: (i) if a molecule A can split into two specific molecules B and C, then in a different environment B and C can combine again to form A; (ii) in an extension of the idea, we consider A splitting into three consitutents B, C and D. In the context of recombination, D can act as a catalyst holding B and C in the correct configuration to enable all three to bond together; (iii) the point about bonding of similar systems is illustrated by the way that if one halogen can bond in a particular place in a specific molecule then a different halogen is likely also to be able to bond in the same place.

# Universality, fractality and 'turtles all the way down'

Two complementary forms of change to be considered in the above picture are (i) systems joining together to form one unit, and (ii) a system splitting into a number of units. This leads to the possibility of a fractal, or scale-free, situation where similar structures exist at all scales. In this context, some signs would have a universal significance at all levels. However, as systems become more complex, differentiation and specialisation start to occur.

If the multiple scale picture is correct, we would have a situation where details are governed by finer details which are governed by finer details and so on *ad infinitum*, in conformity with the 'turtles all the way down' concept .

# Cognitive and cultural development

We first recall what the purpose of the discussion of semiosis and the circular theory has been. The idea was to be able to treat universe generation as, in essence, a kind of technological development. The familiar technological development is a product of human beings and brains, and clearly cannot be used to account for universe generation, but our discussion of development in terms of semiosis and circular theory indicates that something analogous to cognitive development (including cultural development, assuming that cognitive development, in a social system, provides a basis for cultural emergence) can occur in a wider context, including that of our postulated primordial system.

The hypothesis then is that primordial constructs of various levels of complexity can form, whose links with other systems including their environment can be equated with 'knowing'. What might such systems come to know? If their culture acts on the basis of perceived benefit only (as is tending to become the norm in our modern society), then such developments may have limited outcomes. If wider explorations are not excluded, then developments such as mathematics are possible, which might then be applied to such scientific knowledge as might be discoverable, and subsequently in technological applications including, it is hypothesised, mathematically governed universes that could be beneficial to life.

## Outliers

In this connection, Yardley (private communication) notes that an important role in determining the general direction of development is played by *outliers*, that is to say situations encountered that have not yet made effective links with existing structures. Chance contacts may cause new structures to be built, which structures may on occasion be applicable in a wide range of situations, leading to more extended developments.

## Mathematical precision

One important issue is how mathematical precision emerges from a system that is initially very imprecise. We can usefully consider in this connection Euclidean geometry, a mathematically precise system that emerged through the consideration of properties of the world that were not known with any great precision. Geometry, like any mathematical enterprise, is a symbolic activity that does not depend in any essential way on interaction with the world. It was, nevertheless, inspired by knowledge of real point-like objects and approximate straight lines. By retreating into symbolism one escapes inconvenient facts about the world and is able to create a system that has a certain resemblance to the world even though there is no exact correspondence. The Euclidean plane, is in essence, a fantasy that one can address through symbols even though the real world does not correspond exactly to it. However, in this case the correspondences between the Euclidean world and the real world are sufficiently close that Euclidean geometry is of value in the real world, but this is something that has to be discovered through observation rather than taken for granted.

## Generation of space and physical universes

In our ordinary world, Euclidean geometry is simply a system that provides a good model for phenomena in space, using specialised techniques to connect the model with the reality. From the perspective of our primordial community, it conversely provides a model for *forming* a universe-system (more generally, physical laws provide a basis for forming the corresponding physical reality). The model is not the technology, any more than understanding the sphere equates to the existence of *physical* spheres. We hypothesise however that some such technology, which in due course we may ourselves be able to understand, was discovered at the primordial level, and forms the basis upon which physical universes are generated. Mathematical precision exists only in the world of discourse, and is realised to whatever degree is possible by technology.

Symmetry and symmetry breaking may play a key role here, in view of the fact that conceptually symmetry is defined in terms of transformations that may have physical correlates, while at the same time symmetry is found to play an important role in actual physics.

In this picture locality is understood as an *emergent* property, analogous to the frequency of a physical process. Just as in some circumstances frequencies of physical processes become well defined, with different frequencies becoming independent of each other as far as linkages are concerned, in this case location becomes a well defined quantity, with different locations becoming independent of each other. Quantum entanglement and wholeness, on the other hand, would be derivative of the units of circular theory. More generally, the high degree of correlation associated with the packing model can be expected to be manifested in phenomena similar to those associated with quantum mechanics.

# Discussion

We have addressed in a natural way Wheeler's question of how observer-participation can lead to the emergence of specific laws of nature in particular systems. The key point is the fact that *the interpretation of signs changes the game*, facilitating the emergence of new kinds of system and process, which are correlates of cognitive and cultural development that, in the present context, lead to emergent laws. In this picture, the responsible system or systems are the determiners of the observed laws, rather than the laws concerned being presumed absolute, or derivable from some mathematical analysis.

One can imagine a scenario whereby conventional science would be forced similarly to renounce the idea of a Final Theory. We already have a situation where some theory X (e.g. the Standard Model) proves inadequate and theory Y (e.g. string theory) is proposed to take its place. Then certain further issues lead to the idea that the real 'fundamental theory' is Z (e.g. M-theory). At each stage, however, the supposed fundamental theory gets farther from what is accessible by experiment, and its connections with reality become more obscure.

The idea that nature at some deeper level has biological aspects is not fundamentally absurd, and has been previously explored by authors such as Smolin and Pattee. The above analysis has explored some aspects of the 'biological logic' applicable to such a scenario, in particular the mechanics of development, which could lead to what might be termed 'extended mind'. Faculties such as mathematical intuition, difficult to account for in conventional ways, might be manifestations of the extended mind, which might also be related to experiences of meaning in art.

To what extent can these proposals be considered scientific in character? While the absence of a fixed, universal mathematical law may seem at first sight to be a radical departure from scientific tradition, the idea that the laws manifested in the laboratory are emergent rather than fundamental is already a feature of string theory. And, as practiced, biology is a science that makes extensive use of phenomenology (e.g. that of chemical reactions), and concepts specific to biology, and typically makes less use of the methods of theoretical physics (i.e. mathematical models).

A typical biological concept is the idea that particular systems (e.g. the immune system) have particular functions. Such concepts have value in interpreting what one finds and in guiding investigations. The ideas expounded here can be expected to be of similar value in constructing models where conventional methods prove inadequate.

Some scientists have accepted the idea that not everything can be characterised in quantitative terms, asserting however that the only real knowledge is that based on scientific measurement; but alternatives , offering a broader understanding of what constitutes knowledge, are possible. The present discussion offers some insight into what is involved in that latter position. Nature is pervaded by patterns (signs) which through practice we have become expert in interpreting, a process that has pragmatic value even if it is not amenable to the traditional quantitative methodology. If the picture developed here is correct, there is much more in the way of meaning to be found in the natural world by such means than can be found through the traditional methodology of science.

## Acknowledgments

I am grateful to Ilexa Yardley for numerous discussions that have helped shape these ideas, and to Dr. Plamen Simeonov for helpful comments on the manuscript.

[^1]: Based on talk given at the ACIB '11 conference, INBIOSA project, to be published in *'Integral Biomathics: Tracing the Road to Reality', Proceedings of iBioMath 2011, Paris and ACIB '11, Stirling UK,* P. L. Simeonov, L. S. Smith, A. C. Ehresmann (Eds.), Springer-Verlag, 2012.


abstract: The efficacy of particle identification is compared using artificial neutral networks and boosted decision trees. The comparison is performed in the context of the MiniBooNE, an experiment at Fermilab searching for neutrino oscillations. Based on studies of Monte Carlo samples of simulated data, particle identification with boosting algorithms has better performance than that with artificial neural networks for the MiniBooNE experiment. Although the tests in this paper were for one experiment, it is expected that boosting algorithms will find wide application in physics.
author: Byron P. Roe, Hai-Jun Yang; Ji Zhu; Yong Liu, Ion Stancu; Gordon McGregor
date: 2024-10-01
title: Boosted Decision Trees as an Alternative to Artificial Neural Networks for Particle Identification

# Introduction

The artificial neural network (ANN) technique has been widely used in data analysis of High Energy Physics experiments in the last decade. The use of the ANN technique usually gives better results than the traditional simple-cut techniques. In this paper, another data classification technique, *boosting*, is introduced for data analysis in the MiniBooNE experiment at Fermi National Accelerator Laboratory. The MiniBooNE experiment is designed to confirm or refute the evidence for $\nu_\mu \rightarrow \nu_e$ oscillations at $\Delta m^2 \simeq 1 ~eV^2/c^4$ found by the LSND experiment. It is a crucial experiment which will imply new physics beyond the standard model if the LSND signal is confirmed. Based on our studies, particle identification (PID) with the boosting algorithm is 20 to 80% better than that with our standard ANN PID technique, the boosting performance relative to that of ANN depends on the Monte Carlo samples and PID variables. Although the boosting algorithm was tested in only one experiment, it's anticipated to have wide application in physics, especially in data analysis of particle physics experiments for signal and background events separation.

The boosting algorithm is one of the most powerful learning techniques introduced during the past decade. The boosting algorithm is a procedure that combines many "weak" classifiers to achieve a final powerful classifier. Boosting can be applied to any classification method. In this paper, it is applied to decision trees. Two boosting algorithms, AdaBoost and $\epsilon$-Boost, are considered. A brief description of boosting algorithms is given in the next section. Our results are presented in Section III, while we summarize our conclusions in Section IV.

# Brief Description of Boosting

## Decision Tree

Suppose one is trying to divide events into signal and background and suppose Monte Carlo samples of each are available. Divide each Monte Carlo sample into two parts. The first part, the training sample, will be used to train the decision tree, and the second part, the test sample, to test the final classifier after training.

For each event, suppose there are a number of PID variables useful for distinguishing between signal and background. Firstly, for each PID variable, order the events by the value of the variable. Then pick variable one and for each event value see what happens if the training sample is split into two parts, left and right, depending on the value of that variable. Pick the splitting value which gives the best separation into one side having mostly signal and the other mostly background. Then repeat this for each variable in turn. Select the variable and splitting value which gives the best separation. Initially there was a sample of events at a "node". Now there are two samples called "branches". For each branch, repeat the process, i.e., again try each value of each variable for the events within that branch to find the best variable and splitting point for that branch. One keeps splitting until a given number of final branches, called leaves, are obtained, or until each leaf is pure signal or pure background, or has too few events to continue. This description is a little oversimplified. In fact at each stage one picks as the next branch to split, the branch which will give the best increase in the quality of the separation. A schematic of a decision tree is shown in Fig.1, in which 3 variables are used for signal/background separation: event hit multiplicity, energy, and reconstructed radial position.

What criterion is used to define the quality of separation between signal and background in the split? Imagine the events are weighted with each event having weight $W_i$. Define the purity of the sample in a branch by $$P = {\sum_sW_s\over \sum_sW_s +\sum_bW_b},$$ where $\sum_s$ is the sum over signal events and $\sum_b$ is the sum over background events. Note that $P(1-P)$ is 0 if the sample is pure signal or pure background. For a given branch let $$Gini = (\sum_{i=1}^n W_i)P(1-P),$$ where $n$ is the number of events on that branch. The criterion chosen is to minimize $$Gini_{left\ son} + Gini_{right\ son}.$$

To determine the increase in quality when a node is split into two branches, one maximizes $$Criterion = Gini_{father} - Gini_{left\ son} - Gini_{right\ son}
.$$

At the end, if a leaf has purity greater than 1/2 (or whatever is set), then it is called a signal leaf and if the purity is less than 1/2, it is a background leaf. Events are classified signal if they land on a signal leaf and background if they land on a background leaf. The resulting tree is a *decision tree*.

Decision trees have been available for some time. They are known to be powerful but unstable, i.e., a small change in the training sample can give a large change in the tree and the results.

There are three major measures of node impurity used in practice: misclassification error, the gini index and the cross-entropy. If we define p as the proportion of the signal in a node, then the three measures are: 1 - max(p, 1-p) for the misclassification error, 2p(1-p) for the gini index and -plog(p) - (1-p)log(1-p) for the cross-entropy. The three measures are similar, but the gini index and the cross-entropy are differentiable, and hence more amenable to numerical optimization. In addition, the gini index and the cross-entropy are more sensitive to change in the node probabilities than the misclassification error. The gini index and the cross-entropy are similar.

## Boosting

Within the last few years a great improvement has been made. Start with unweighted events and build a tree as above. If a training event is misclassified, i.e, a signal event lands on a background leaf or a background event lands on a signal leaf, then the weight of that event is increased (boosted).

A second tree is built using the new weights, no longer equal. Again misclassified events have their weights boosted and the procedure is repeated. Typically, one may build 1000 or 2000 trees this way.

A score is now assigned to an event as follows. The event is followed through each tree in turn. If it lands on a signal leaf it is given a score of 1 and if it lands on a background leaf it is given a score of -1. The renormalized sum of all the scores, possibly weighted, is the final score of the event. High scores mean the event is most likely signal and low scores that it is most likely background. By choosing a particular value of the score on which to cut, one can select a desired fraction of the signal or a desired ratio of signal to background. For those familiar with ANNs, the use of this score is the same as the use of the ANN value for a given event. For the MiniBooNE experiment, boosting has been found to be superior to ANNs. Statisticians and computer scientists have found that this method of classification is very efficient and robust. Furthermore, the amount of tuning needed is rather modest compared with ANNs. It works well with many PID variables. If one makes a monotonic transformation of a variable, so that if $x_1>x_2$ then $f(x_1)>f(x_2)$, the boosting method gives exactly the same results. It depends only on the ordering according to the variable, not on the value of the variable.

In articles on boosting within the statistics and computer science communities, it is often recommended that short trees with eight leaves or so be used. For the MiniBooNE Monte Carlo samples it was found that large trees with 45 leaves worked significantly better.

## Some Boosting Algorithms

If there are $N$ total events in the sample, the weight of each event is initially taken as $1/N$. Suppose that there are $N_{tree}$ trees and $m$ is the index of an individual tree. Let

-   $x_i=$ the set of PID variables for the $i$th event.

-   $y_i =1$ if the $i$th event is a signal event and $y_i=-1$ if the event is a background event.

-   $w_i=$ the weight of the $i$th event.

-   $T_m(x_i)=1$ if the set of variables for the $i$th event lands that event on a signal leaf and $T_m(x_i)=-1$ if the set of variables for that event lands it on a background leaf.

-   $I(y_i \ne T_m(x_i)) = 1$ if $y_i \ne T_m(x_i)$ and 0 if $y_i = T_m(x_i)$.

There are at least two commonly used methods for boosting the weights of the misclassified events in the training sample.

The first boosting method is called AdaBoost. Define for the $m$th tree: $$err_m ={\sum_{i=1}^Nw_iI(y_i \ne T_m(x_i))\over \sum_{i=1}^Nw_i}.$$ $$\alpha_m = \beta\times\ln((1-err_m)/err_m).$$ $\beta=1$ is the value used in the standard AdaBoost method. For the MiniBooNE Monte Carlo samples, $\beta = 0.5$ has been found to give better results. Change the weight of each event $i$, $i = 1,...,N$: $$w_i\rightarrow w_i\times e^{\alpha_mI(y_i\ne T_m(x_i))}.$$ Each classifier $T_m$ is required to be better than random guessing with respect to the weighted distribution upon which the classifier is trained. Thus, $err_m$ is required to be less than 0.5, since, otherwise, the weights would be updated in the wrong direction. Next, renormalize the weights, $w_i\rightarrow {w_i/ \sum_{i=1}^Nw_i}.$ The score for a given event is $$T(x) = \sum_{m=1}^{N_{tree}}\alpha_mT_m(x),$$ which is just the weighted sum of the scores over the individual trees, see Fig.2.

The second boosting method is called $\epsilon$-Boost, or sometimes "shrinkage". After the $m$th tree, change the weight of each event $i$, $i = 1,...,N$: $$w_i\rightarrow w_ie^{2\epsilon I(y_i\ne T_m(x_i))},$$ where $\epsilon$ is a constant of the order of 0.01. Renormalize the weights, $w_i\rightarrow {w_i/ \sum_{i=1}^Nw_i}.$ The score for a given event is $$T(x)  = \sum_{m=1}^{N_{tree}}\epsilon T_m(x),$$ which is the renormalized, but unweighted, sum of the scores over individual trees.

The AdaBoost and $\epsilon-$Boost algorithms used in this paper try to minimize the expectation value: $E(e^{-yF(x)})$, where y = 1 for signal, y = -1 for background, $F(x)=\sum_{i=1}^{N_{trees}} f_i(x)$, where the classifier $f_i(x)=1$ if an event lands on signal leaf, and $f_i(x)=-1$ if an event lands on background leaf. This minimization is closely related to minimizing the binomial log-likelihood. It can be shown that $E(e^{-yF(x)})$ is minimized at $$F(x)=\frac{1}{2}ln\frac{P(y=1|x)}{P(y=-1|x)}=\frac{1}{2}ln\frac{p(x)}{1-p(x)}$$ Let $y^* = (y+1)/2$. It is then easy to show that $$e^{-yF(x)} = \frac{|y^*-p(x)|}{\sqrt{p(x)(1-p(x))}}$$ The right-hand side is known as the $\chi$ statistic. $\chi^2$ is a quadrative approximation to the log-likelihood, so $\chi$ can be considered a gentler alternative. It turns out that fitting using $\chi$ is monotone and smooth; the criteria will continually drive the estimates towards purer solutions. An ANN tries to minimize the squared-error $E(y-F(x))^2$, where y = 1 for signal events, y = 0 for background events, and $F(x)$ is the network prediction for training events.

# Results

For the $\nu_\mu \rightarrow \nu_e$ oscillation search in the MiniBooNE experiment, the main backgrounds come from intrinsic $\nu_e$ contamination in the beam, mis-identified $\nu_\mu$ quasi-elastic scattering and mis-identified neutral current $\pi^0$ production. Since intrinsic $\nu_e$ events are real $\nu_e$ events, the PID variables cannot distinguish them from oscillation $\nu_e$ events. This report concentrates on separating the non-$\nu_e$ events from the $\nu_e$ events. Good sensitivity for the $\nu_e$ appearance search requires low background contamination from all kinds of backgrounds. Here, the ANN and the two boosting algorithms are used to separate $\nu_e$ charged current quasi-elastic (CCQE) events from non-$\nu_e$ background events.

500000 Monte Carlo $\nu_\mu$ events distributed among the many possible final states and 200000 intrinsic $\nu_e$ CCQE events were fed into the reconstruction package R-fitter. Among these events, 88233 intrinsic $\nu_e$ CCQE and 162657 background events passed reconstruction and pre-selection cuts.

The signature of each event is given by 52 variables for the R-fitter. All variables are used in the boosting algorithms for training and testing. It is a challenge to have agreement between data and Monte Carlo for all of the PID variables and for the boosting outputs. The MiniBooNE Collaboration is devoting considerable effort to achieve it. Monte Carlo samples using 18 different parameter sets have been generated and run through the same reconstruction programs. The results for both the PID variables and the boosting outputs are consistent. When the present Monte Carlo is compared with the real data samples, the shapes of the various PID variables and the boosting outputs match well. Since the recontruction and PID algorithms are still undergoing continuous modifications, relative results rather than absolute percentages are presented in the following plots.

For the AdaBoost algorithm, the parameter $\beta = 0.5$, the number of leaves $N_{leaves} = 45$ and the number of tree iterations $N_{tree} = 1000$ were used. The relative ratio(defined as the number of background events kept divided by the number kept for 50% intrinsic $\nu_e$ selection efficiency and $N_{tree} = 1000$) as a function of $\nu_e$ selection efficiency for various tree iterations is shown in the top plot of Fig.3 and the AdaBoost output distributions are shown in the bottom plot. 20000 intrinsic $\nu_e$ CCQE signal and 30000 background events were used for training, 68233 $\nu_e$ and 132657 background events were used for testing. All results shown in the paper are for testing samples.

In order to quantify the performance of the boosting algorithm, the AdaBoost results for a particular set of PID variables were compared with ANN results. The results, compared as a function of the intrinsic $\nu_e$ CCQE selection efficiency, are shown in Fig.4. For the intrinsic $\nu_e$ signal efficiency ranging from 40% to 60%, the performances of AdaBoost were improved by a factor of approximately 1.5 and 1.8 over the ANN if trained by the signal and all kinds of backgrounds with 21 (red dots) and 52 (black boxes) input variables respectively, shown in Fig.4.a. If AdaBoost and ANN were trained by the signal and neutral current $\pi^0$ background, the performances of AdaBoost were improved by a factor of approximately 1.3 and 1.6 over the ANN for 22 (red dots) and 52 (black boxes) training variables respectively, shown in Fig.4.b. The best results for the ANN were found with 22 variables, while the best results for boosting were found with 52 variables. Comparison of the best ANN results and the best boosting results indicates that, when trained by the signal and neutral current $\pi^0$ background, the ANN results kept approximately 1.5 times more background events than were kept by the boosting algorithms for about 50% $\nu_e$ CCQE efficiencies.

In Fig.4.c, the ratio of the background kept for a 52 variable AdaBoost to that for a 21(red dots - results for AdaBoost trained by the signal and all kinds of backgrounds) / 22(black boxes - results for AdaBoost trained by the signal and neutral current $\pi^0$ background) variables is shown as a function of $\nu_e$ efficiency. It can be seen that the AdaBoost performance is improved by the use of more training variables.

The above ANN and AdaBoost performance comparison with different input variables indicates that AdaBoost can improve the PID performance significantly by using more input variables, even though many of them have weak discriminant power; ANN, however, seems unlikely to make full use of all input variables because it is more difficult to optimize all the weights between ANN nodes, given more nodes in both the input and the hidden layers. For the MiniBooNE Monte Carlo samples, the ANN are optimum for approximately 20 PID variables. The authors have found a similar number to be true for several other applications. In general, the optimum number for ANN may vary depending on the strength of the PID variables and the correlations between them.

Further evidence of this effect comes from the S-fitter, a second reconstruction–PID program set for the MiniBooNE. A systematic attempt was made to find the optimum sets of variables for ANN and for boosting classifiers by using $\nu_e$ CCQE signal and $\pi^0$ background (which includes 25 NUANCE reaction channels). It is found that, for S-fitter, the optimum ANN result is achieved by a selected set of 22 variables, while for boosting, no obvious improvement is seen after a selected optimum set of 50 variables are used. Comparison of the best ANN results and the best boosting results indicates that, for a given fraction of $\nu_e$ CCQE events kept, the ANN results kept about 1.2 times more $\pi^0$ background events than were kept by the boosting algorithms within target range of keeping close to 50% of the $\nu_e$ CCQE events.

As noted in the introduction, two boosting algorithms are considered in the present paper. The comparison of AdaBoost and $\epsilon$-Boost performance is shown in Fig.5, where parameters $\beta=0.5$ and $\epsilon=0.01$ were selected for AdaBoost and $\epsilon$-Boost training, respectively. The comparison between small tree size (8 leaves) and large tree size (45 leaves) with a comparable overall number of decision leaves, indicates that large tree size with 45 leaves yields 10 $\sim$ 20 % better performance for the MiniBooNE Monte Carlo samples shown in Fig.5.a. Increasing the tree size past 45 leaves did not produce appreciable improvement

Comparison of AdaBoost and $\epsilon$-Boost performance for the background contamination versus the intrinsic $\nu_e$ CCQE selection efficiency as a function of the number of decision tree iterations is shown in Fig.5.b. A smaller relative ratio implies a better performance for AdaBoost. The performance of AdaBoost is better than that of $\epsilon$-Boost if the relative ratio is less than 1. Boosting performance in the high signal efficiency region is continuously improved for more tree iterations. AdaBoost has better performance than $\epsilon$-Boost for less than about 200 tree iterations, but becomes slightly worse than $\epsilon$-Boost for a large number of tree iterations, especially for $\nu_e$ signal efficiency below $\sim$ 60%. For higher $\nu_e$ signal efficiency($>$ 70%), AdaBoost works slightly better than $\epsilon$-Boost.

# Conclusions

PID variables obtained using the R-fitter and the S-fitter event reconstruction programs for the MiniBooNE experiment were used to separate signal events from background events. The ANN and the boosting algorithms were compared for PID. Based on these studies with the MiniBooNE Monte Carlo samples, the boosting algorithms, AdaBoost and $\epsilon$-Boost, improved PID performance significantly compared with the artificial neural network technique. This improvement manifested itself when a large number of PID variables was used. For a small number of variables, the ANN classification was competitive, but as the number of variables was increased, the boosting results proved more efficient and superior to the ANN technique. If more variables are needed, boosting will use them as necessary.

It was also found that boosting with a large tree size of 45 leaves worked significantly better than boosting with a small tree size, 8 leaves, as recommended in some statistics literature.

The boosting technique proved to be quite robust. If a transformation of variables from $x$ to $y=f(x)$ is made, then as long as the ordering is preserved, that is if $x_2>x_1$, then $y_2>y_1$, the boosting results are unchanged. ANNs must be tuned for temperature, learning rate and other variables, while for boosting, there is much less to vary and it is quite straightforward.

There are certainly applications where ANNs prove better than boosting. However, for this application boosting appears superior and seems to be exceptionally robust and simple to use. It is anticipated that boosting techniques will have wide application in physics.

# Acknowledgments

We wish to express our gratitude to the MiniBooNE Collaboration for their excellent work on the Monte Carlo simulation and the software package for physics analysis.

This work is supported by the Department of Energy and by the National Science Foundation of the United States.