abstract: #### Background:
  .
  A myriad of methods to reverse-engineer transcriptional regulatory networks have been developed in recent years. Direct methods directly reconstruct a network of pairwise regulatory interactions while module-based methods predict a set of regulators for modules of coexpressed genes treated as a single unit. To date, there has been no systematic comparison of the relative strengths and weaknesses of both types of methods.
  .
  #### Results:
  .
  We have compared a recently developed module-based algorithm, LeMoNe (Learning Module Networks), to a mutual information based direct algorithm, CLR (Context Likelihood of Relatedness), using benchmark expression data and databases of known transcriptional regulatory interactions for *Escherichia coli* and *Saccharomyces cerevisiae*. A global comparison using recall versus precision curves hides the topologically distinct nature of the inferred networks and is not informative about the specific subtasks for which each method is most suited. Analysis of the degree distributions and a regulator specific comparison show that CLR is 'regulator-centric', making true predictions for a higher number of regulators, while LeMoNe is 'target-centric', recovering a higher number of known targets for fewer regulators, with limited overlap in the predicted interactions between both methods. Detailed biological examples in *E. coli* and *S. cerevisiae* are used to illustrate these differences and to prove that each method is able to infer parts of the network where the other fails. Biological validation of the inferred networks cautions against over-interpreting recall and precision values computed using incomplete reference networks.
  .
  #### Conclusions:
  .
  Our results indicate that module-based and direct methods retrieve largely distinct parts of the underlying transcriptional regulatory networks. The choice of algorithm should therefore be based on the particular biological problem of interest and not on global metrics which cannot be transferred between organisms. The development of sound statistical methods for integrating the predictions of different reverse-engineering strategies emerges as an important challenge for future research.
address:  (1)Department of Plant Systems Biology, VIB, Technologiepark 927, B-9052 Gent, Belgium (2)Department of Molecular Genetics, Ghent University, Technologiepark 927, B-9052 Gent, Belgium (3)CMPG, Department Microbial and Molecular Systems, KU Leuven, Kasteelpark Arenberg 20, B-3001 Leuven, Belgium
author: Tom Michoel$^{1,2}$ ; Riet De Smet$^3$ ; Anagha Joshi$^{1,2}$ ; Yves Van de Peer$^{1,2}$ and Kathleen Marchal$^{3}$
title: Comparative analysis of module-based versus direct methods for reverse-engineering transcriptional regulatory networks

# Background

Due to the success of microarray technology, the available data on the transcriptional regulatory networks of different organisms has grown exponentially. In order to explore these data to the maximum, a myriad of methods to reverse-engineer or reconstruct transcriptional regulatory networks from microarray data have been developed in the past few years. In general, the scientific community has mainly focused on the overall performance of newly developed methods in reconstructing the known network of certain model organisms as compared to a reference network, measuring algorithmic performance with standard measures such as recall and precision. Less attention has been paid to what extent conceptually different approaches differ in the networks they infer. Nonetheless, in order to get a better understanding of the systems studied it is also important to understand which specific problems can be tackled using a certain method, irrespective of the overall performance of the different methods.

Broadly speaking we can distinguish between two classes of methods for reverse-engineering transcriptional regulatory networks from gene expression data which differ vastly in how they approach the network inference problem. Direct methods infer individual regulator-target interactions using a pairwise correlation measure between the expression profiles of a transcription factor and its putative targets . Module-based methods assume a modular structure of the transcriptional regulatory network , with genes subject to the same regulatory input being organized in coexpression modules.

While different direct methods have been compared to each other in the past , no systematic comparison between direct and module-based methods has been undertaken so far. In this study we perform such a comparison using a representative method from each class. The CLR (Context Likelihood of Relatedness) algorithm considers all possible pairwise regulator-target interactions and scores these interactions based on the mutual information of their expression profiles as compared to an interaction specific background distribution. It has been shown to outperform other direct methods . The LeMoNe (Learning Module Networks) algorithm uses probabilistic, ensemble-based optimization techniques to infer high-quality module networks , where genes are first partitioned into coexpression modules and regulators are assigned to modules based on how well they explain the condition-dependent expression behavior of the module. It has been shown to outperform the original module network algorithm .

We have compared both methods at increasing levels of detail using public expression compendia for *Escherichia coli* and *Saccharomyces cerevisiae* , two organisms for which relatively large databases of known transcriptional regulatory interactions exist . We first use recall versus precision curves to give a comparison of the global performance of both methods. We then show that due to the different assumptions underlying both methodologies, they infer topologically distinct networks with limited overlap, even at equal performance thresholds. To understand these distinctions more completely, we examined in detail example subsystems of the network which are well characterized, namely the chemotaxis and flagellar system in *E. coli* and a respiratory module and a membrane lipid and fatty acid metabolism module in *S. cerevisiae*. Biological validation of the inferred networks cautions against over-interpreting recall and precision values computed using incomplete reference networks.

# Results and Discussion

## Global comparison using recall and precision

The output of LeMoNe and CLR consists of a list of respectively ranked regulator-module and ranked regulator-target interactions, scored according to their statistical significance. As a first, global, comparison, we can therefore compute recall and precision with respect to the given reference networks at different score cutoffs. For CLR we can directly compare the inferred network with the true network; for LeMoNe we draw an edge between each regulator assigned to a module and all genes in the module, thereby ignoring at this stage the extra information present in the module structure. We computed recall and precision as in : if an edge is predicted between two genes present but unconnected in the reference network it is counted as a false positive. Figure shows the recall versus precision curves for both algorithms and both organisms.

Both algorithms succesfully prioritize true positive interactions, especially in *E. coli*: all curves go from a high precision, low recall region to a low precision, high recall region. For CLR the curves show a smooth course while for LeMoNe they are more staircase-like. CLR scores individual interactions and as a result, in the recall-precision curve interactions will be added one by one, but interactions corresponding to a certain regulator will be dispersed continuously throughout the recall-precision curve. LeMoNe on the other hand assigns a regulator to a module as a whole and all targets belonging to the same module are added at the same time in the recall-precision curve. For a stringent threshold and subsequently a low number of interactions inferred, the CLR network will cover few interactions for many regulators while the LeMoNe network will retrieve many interactions for few regulators.

At similar levels of precision, the recall in *S. cerevisiae* is nearly an order of magnitude smaller than in *E. coli*, in line with previous studies . This is likely due to the higher complexity of transcriptional regulation in *S. cerevisiae* with a higher degree of combinatorial regulation and posttranscriptional control, and consequently a lower degree of correlation in expression between transcription factors and their targets.

A simple 'area under the curve' measurement would suggest that CLR performs slightly better in the prokaryote *E. coli* and LeMoNe in the eukaryote *S. cerevisiae*. However, as we will show below, both algorithms infer complementary information in both organisms.

## Topological distinctions between inferred networks

As explained in the previous section, due to how interactions are scored, direct and module-based methods will infer different kinds of networks at stringent precision thresholds. For *E. coli*, we compared the LeMoNe and CLR networks at a 30% precision threshold where both networks have nearly equal recall and precision (see Figure ). The LeMoNe network consists of 53 regulators assigned to 62 modules for a total of 1079 predicted interactions; 594 of these interactions are between genes in RegulonDB, with a precision of 29%. The corresponding CLR network contains 1422 predicted interactions for 242 regulators; 597 of these interactions are between genes in RegulonDB, with a precision of 30%. 51 out of 53 LeMoNe regulators are also present in the CLR network, but only 277 interactions are predicted in both networks. For *S. cerevisiae*, there is no 'natural' point on the recall versus precision curve to compare both networks. We therefore compared CLR and LeMoNe at the first 1070 predicted interactions. This number is chosen to give comparably sized networks as in *E. coli* and ensure that the ranked list of LeMoNe interactions is not cut off in the middle of one module. The cutoff of the first 1070 interactions corresponds to precision values of respectively 16% and 10% for LeMoNe and CLR (cfr. Figure ). The LeMoNe network consists of 34 regulators assigned to 39 modules containing 867 genes, while the CLR network contains 214 regulators; 28 regulators are present in both networks, yet only 75 interactions are common.

The networks inferred by LeMoNe and CLR are topologically very distinct (see Supplementary Figures S1 to S4). This distinction can be quantified by their in- and out-degree distributions (Figure ). The in-degree is the number of regulators assigned to a certain target gene and the in-degree distribution counts for each value $k$ the number of targets with in-degree $k$. Likewise, the out-degree is the number of targets assigned to a certain regulator and the out-degree distribution counts for each value $k$ the number of regulators with out-degree $k$. CLR infers for each regulator only the most significant targets. As a result, the out-degree distribution is skewed to the left, with the majority of regulators having only few targets. The in-degree distribution on the other hand has a long tail of genes assigned to many different regulators. LeMoNe infers for each module the most significant regulators, resulting in opposite characteristics of the degree distributions. The in-degree distribution has no tail since for most modules at most 2 significant regulators are identified. The out-degree distribution on the other hand has a long tail since each regulator assignment involves a whole module of genes. For these reasons, we say that CLR is 'regulator-centric' and LeMoNe is 'target-centric'.

## Regulator specific comparison

We make a further comparison of the two methods, focusing on how they differ in the type of regulators they assign. We compared again the 30% precision networks for *E. coli* and the networks of first 1070 interactions for *S. cerevisiae*.

For both methods, a large fraction of the regulators for which known targets are inferred are autoregulators. For *E. coli*, LeMoNe and CLR have respectively 19 and 32 regulators with at least one true positive; 15/19 (79%) and 27/32 (84%) are known autoregulators, while the fraction of autoregulators in the total reference network is 95/150 (63%). For *S. cerevisiae*, LeMoNe and CLR have respectively 6 and 10 regulators with at least one true positive; 5/6 (83%) and 5/10 (50%) are known autoregulators, while the fraction of autoregulators in the total reference network is 79/171 (46%). The abundance of autoregulators is not surprising since autoregulation is a simple mechanism by which the expression profile of a regulator and its targets can be correlated.

In LeMoNe, we get as additional information whether a predicted regulator is positively or negatively correlated with its target module and RegulonDB, the reference network for *E. coli*, contains the activation or repression sign for many interactions. However, although theoretically possible, we could not detect biologically relevant patterns of anticorrelation, in line with previous studies . Even though the assumption of anticorrelation seems intuitively plausible in case of repressors, it is a too simplistic representation of reality. Indeed LeMoNe and CLR both find many targets of mainly autorepressors (e.g. LexA, PurR, LldR and GalS), but they all were positively instead of negatively correlated with their targets. This can be explained by the fact that the activity of such autorepressors is dependent upon the presence of corepressing signals. In the absence of the corepressing signal the repressor is active, limiting its own production as well as that of its target genes. In presence of the corepressing signal the repressors are inactive, which enables the production of both inactive repressor gene and its targets .

In *E. coli*, regulators for which the module-based and direct methods differ in performance are in line with the topological distinctions. CLR is better at inferring interactions for regulators that are known to regulate just one or a few operons (e.g. BetI, CsgD, DnaA, MarA, Yhhg, see Figure ). These operons are found with a relatively high rank in the CLR network since their regulators often belong themselves to the operons and are thus by definition tightly coexpressed with their targets. The clustering method employed by LeMoNe appears to be too coarse grained to identify these operons individually, since they are mostly part of larger clusters. LeMoNe on the other hand is superior at inferring interactions for regulators that are known to regulate larger regulons, such as Fis, LexA, PurR, and RpoS, for which the level of coexpression is not as high as the one observed within a single operon (see Figure ). In *S. cerevisiae*, there is no operonic structure and hence the 'operon regulators' acurately identified by CLR are absent. Figure show however that the regulators for which LeMoNe and CLR infer known targets are still very distinct, but there appears to be no general biological reason underlying these differences.

## Biological validation of inferred networks

Due to the lack of a negative gold standard, we have denoted in the previous analysis an edge as being false positive if both regulator and target are present but not connected in the reference network (the positive gold standard). Since the coverage of these reference networks is still very incomplete, it is likely that the number of false positives is overestimated. Moreover, about half of the regulators in *E. coli* and *S. cerevisiae* are not present in the reference network and their predicted interactions are thus never evaluated.

In , it was already shown that new predictions made by CLR in *E. coli* could be validated experimentally. Here we have performed an in-depth biological validation of the 30% precision module network inferred by LeMoNe. To biologically validate the obtained regulator-module assignments, we calculated for all modules functional enrichment scores and enrichment in targets of previously annotated regulators . Table shows that in nearly all cases the module is enriched in known targets of the predicted regulator (column 4) or at least involved in the same biological function (column 6). In several cases the predicted regulator is the one which has the best target enrichment $p$-value. Nearly half of the regulators are putative regulators without any currently known targets, and these assignments cannot be validated. However, many of the correctly predicted regulators involve neighbor regulators (Table , column 7), i.e. regulators colocalized with their targets on the genome. It has been suggested that many of the putative regulators in *E. coli* constitute such neighbor regulators . Hence this feature of gene neighborhood can be used to attach additional significance to the high-scoring predictions for uncharacterized regulators. One of the advantages of a module-based approach is the fact that if a certain module contains several known targets of the assigned regulators, the rest of the unknown targets in this module can be considered high confidence predictions for that regulator. This is illustrated in Supplementary Table S1, where we list several predictions for 10 different modules which could be confirmed by a thorough literature search.

Module network predictions in *S. cerevisiae* have been experimentally validated in and functionally analysed in . For further validation we compared the CLR and LeMoNe networks to the YEASTRACT database . This database contains most of the interactions in the reference network we use here . In addition it also contains targets inferred by transcription factor deletion microarray experiments. The number of true positives for the LeMoNe network cut off at the first 1070 predictions increases from 40 (precision 16%) in the reference network to 55 (precision 24%) with respect to YEASTRACT. For the CLR network cut off at the first 1070 predictions, the number of true positives increases from 31 (precision 10%) in the reference network to 48 (precision 12%) with respect to YEASTRACT.

Biological validation of inferred networks is tedious and does not provide an easy alternative to the automatic estimation of true and false positives using an established reference network. The results of this section do show however that many 'false positives' with respect to an incomplete network are actually true positives when additional information is taken into account and that recall versus precision plots such as in Figure have to be interpreted with caution.

## The chemotaxis and flagellar system in *Escherichia coli*

Our analysis has shown that at equal levels of recall and precision, LeMoNe predicts interactions for fewer regulators but with higher coverage per regulator while CLR predicts fewer interactions per regulator but for more regulators. It is instructive to analyse in detail the implications of these differences for subsystems of the transcriptional regulatory network which are particularly well perturbed in the data set. For *E. coli*, we have taken a closer look at the chemotaxis and flagellar system which forms a complex and tightly regulated system. It consists of the class 1 master operon *flhDC*, 8 class 2 operons activated by the complex FlhDC, and at least 6 class 3 operons activated by the sigma factor FliA (Figure (a)). The *fliA* operon belongs to class 2, positively regulates its own production and can activate other class 2 operons as well .

Four modules (12, 18, 24 and 45) in the module network are enriched in flagellar functions. Together they contain 60 genes of which 55 are known flagellar genes. The separation of flagellar genes in different modules is strongly supported by the LeMoNe clustering (Figure (b)), suggesting the presence of condition-specific regulation in the flagellar gene network, and corresponds to the difference in regulatory input between different classes of flagellar genes (Figure , see also Supplementary Figure S5). In the 30% precision LeMoNe network, FliA is assigned to all four modules and FlhC is correctly assigned to the class 2 modules 18 and 24 only. FlhD is not assigned with a score high enough to make the threshold.

At the 30% precision cutoff, LeMoNe and CLR agree for the majority of predicted interactions for FliA and FlhC. In addition, CLR infers several correct targets for FlhD. The coexpression of FlhD with its predicted targets is significantly lower than for FliA or FlhC. This is evidenced for instance from the LeMoNe clustering (Figure (b)) or CLR mutual information values (data not shown). However, due to the regulator-centric viewpoint and the 'local' background correction method of CLR, these relatively weakly coexpressed targets still get a significant mutual information $z$-score and are thus part of the predicted network. In the target-centric LeMoNe network, the potential assignment of FlhD to the flagella modules is compared to the much better scoring assignments of FliA and/or FlhC and therefore not deemed significant enough. Hence the regulator-centric CLR approach has the advantage to identify significant targets for all three flagellar regulators, but does not distinguish well between regulation by FlhDC and FliA due to the large overlap in predicted targets. The target-centric LeMoNe approach on the other hand has the advantage to infer detailed condition-specific regulatory information through the division in distinct modules of the flagellar genes, but only infers targets for FliA and FlhC.

## The respiratory module and membrane lipid and fatty acid metabolism module in *Saccharomyces cerevisiae*

Despite the overall low performance on *S. cerevisiae*, LeMoNe and CLR both achieve good results on particular subsystems. The advantage of a target-centric approach is well exhibited by the respiratory system. This system is well perturbed in the data set and clusters of respiratory genes are found repeatedly in it using various approaches . LeMoNe module 7 contains 30 genes of which 23 are known respiratory genes. Hap4, a global regulator of respiratory genes, is the most significant regulator for this module and indeed 25 of its genes are known Hap4 targets. The pairwise correlation between Hap4 and its targets varies, and since CLR scores all interactions individually, they are dispersed throughout the ranked list of interactions. As a result, there are only 12 predicted Hap4 targets (7 TP) in the first 1070 CLR interactions (see also Figure ). Clearly, the preliminary step of clustering genes into target modules was necessary here to infer the complete Hap4 regulated module.

Another interesting example is given by LeMoNe module 11, a module of 47 genes involved in membrane lipid and fatty acid metabolism. The four highest-ranked regulators by LeMoNe for this module (Gat1, Met28, Met32 and Dal80) all have known targets in it. However, due to how regulators are scored in LeMoNe, there are rarely more than two significant regulators per module (see Figure (a) and (c)), and only the assignments of Gat1 (3 TP) and Met28 (4 TP) are present in the network of the first 1070 LeMoNe interactions. CLR on the other hand finds the most significant targets for each regulator individually and thus identifies correct targets from module 11 for the other regulators as well: Met28 (1 TP), Met32 (6 TP) and Dal80 (6 TP). For Gat1, CLR does not find true positives, however it finds 5 TP in module 11 for a fifth regulator Gln3. Hence for this module, the most complete information is retrieved by combining the output of LeMoNe and CLR. The genes and predicted regulators of module 11 are mostly involved in 2 pathways, the methionine pathway (regulated by Met28 and Met32) and the nitrogen catabolite repression (NCR) system (regulated by Gat1, Dal80 and Gln3). Module 11 is overexpressed in nitrogen depletion and amino acid starvation conditions. For NCR-sensitive genes it is known that they are not activated when rich nitrogen sources are available, but get expressed when only poor sources are left. A link between the methionine pathway and nitrogen depletion, as predicted by LeMoNe through the clustering and by CLR through the assignment of common targets to these regulators, is not evident but appears to be confirmed by an ongoing study .

## Conclusion

In recent years, a wide variety of methods to reverse-engineer transcriptional regulatory networks from microarray data have been developed. Whereas the development of a new method mostly coincides with a comparison in overall performance to all existing methods, so far no in-depth study on how conceptual differences relate to differences in the inferred networks have been made. Here we distinguished between two main approaches for reverse-engineering transcriptional regulatory networks: the module-based approach and the direct approach. We compared a representative algorithm of each approach (module based LeMoNe versus direct CLR) at several levels of detail for two different organisms, the prokaryote *E. coli* and the eukaryote *S. cerevisiae*. We have found that CLR is 'regulator-centric', making few but highly significant predictions for a large number of regulators. LeMoNe on the other hand is 'target-centric', identifying few but highly significant regulators for a large number of genes grouped in coexpression modules. Through a regulator specific comparison and analysis of specific biological subsystems, we have shown that at stringent significance cutoffs, the conceptual differences in statistically scoring potential regulatory interactions lead to topologically distinct inferred networks containing different kinds of regulators and biological information. Our results show that the choice of algorithm should be made primarily based on whether the biological question under study falls within the target-centric or regulator-centric viewpoint, and not on global metrics which cannot be transferred between organisms. Ideally, several network inference strategies should be combined for the best overall performance. It is an important challenge for future research to develop sound statistical methods for optimally combining the output of multiple, existing reverse-engineering algorithms.

# Methods

The *E. coli* microarray data compendium contains expression profiles for 4345 genes under 189 different stress conditions and genetic perturbations. We selected a subset of 1882 differentially expressed genes (standard deviation larger than 0.5) and used a list of 316 known or putative transcription factors to reconstruct regulatory networks. LeMoNe (software available at <http://bioinformatics.psb.ugent.be/software/details/LeMoNe>) identified 108 ensemble-averaged modules from 12 independent Gibbs sampler runs, containing 1761 genes in total. It inferred a ranked list of regulator-module edges from an ensemble of 10 regulatory programs per module with 100 regulator samples per regulatory program node (see for more details on the meaning of these parameters). We applied CLR (software available at <http://gardnerlab.bu.edu/clr.html>) on the data for the 2084 selected genes (the union of the 1882 differentially expressed genes and 316 candidate regulators) and kept all mutual information $z$-scores between the 316 transcription factors and 1882 target genes. As a reference network we used RegulonDB version 5.7 , a database of 4840 known transcriptional interactions in *E. coli* between 167 transcription factors and 1693 genes. Recall values are computed with respect to RegulonDB restricted to the subset of 2084 genes. This subnetwork contains 3110 edges between 150 transcription factors and 1053 genes. We used EcoCyc to compute functional enrichment of modules. Target and functional enrichment in Table were computed using a cumulative hypergeometric distribution, Bonferroni corrected for multiple testing, with confidence level 95%.

The *S. cerevisiae* microarray data compendium contains expression profiles for 6153 genes in 173 different stress conditions. We used the same subset of 2355 differentially expressed genes, including a list of 321 potential regulators, as used in previous studies of this data set . LeMoNe was run with the same settings as for *E. coli* and inferred 55 ensemble-averaged modules containing 1075 genes. As reference network we used a network recently compiled from the results of genetic, biochemical and ChIP-chip experiments . It contains 11785 interactions between 154 transcription factors and 4047 genes. After restriction to the subset of 2355 differentially expressed genes, it contains 4513 interactions between 133 transcription factors and 1628 genes. The YEASTRACT database contains 30979 transcriptional interactions in *S. cerevisiae* between 171 transcription factors and 5727 genes. After restriction to the subset of 2355 differentially expressed genes, it contains 12021 interactions between 137 transcription factors and 2182 genes.

# Authors contributions

TM developed software, analyzed data and wrote the manuscript. RDS analyzed data and wrote the manuscript. AJ developed software and analyzed data. YVdP supervised the study. KM wrote the manuscript and supervised the study. All authors read and approved the final manuscript.

# Acknowledgements

RDS is a research assistant of the IWT. AJ is supported by an Early-Stage Marie Curie Fellowship. This work is supported by 1) Research Council KUL: GOA AMBioRICS,GOA/08/011, CoE EF/05/007 SymBioSys, 2) FWO: projects G.0318.05, 3) IWT: SBO-BioFrame, 4) IUAP P6/25 (BioMaGNet).

| Regulator | Module ID | Score | Target enrich. | Autoreg. | Pathway | Local | Function |
|:---|:--:|:--:|:--:|:--:|:--:|:--:|:---|
|  |  |  |  |  |  |  |  |
| gatR_2 | 73 | 1912.98 | $**$ |  | $*$ | $**$ | carbon utilization $>$ carbon compounds |
| gadE | 48 | 1844.50 | $**$ | $*$ | $*$ | $**$ | adaptations $>$ pH |
| gutM | 38 | 1807.24 | $**$ | $*$ | $*$ | $*$ | carbon utilization $>$ carbon compounds |
| **ymfN** | **58** | 1749.11 |  |  |  | $*$ |  |
| **ymfN** | **33** | 1711.17 |  |  |  | $*$ |  |
| fliA | 12 | 1510.48 | $**$ | $*$ | $*$ | $**$ | motility, chemotaxis, energytaxis; flagella; biosynthesis of flagellum |
| rcsB | **62** | 1261.72 |  |  | $*$ | $*$ | biosynthesis of colanic acid (M antigen) |
| fecI | 57 | 1200.77 |  | $*$ | $*$ |  | adaptations $>$ Fe aquisition |
| gatR_2 | 42 | 1176.55 | $**$ |  | $*$ | $**$ | carbon utilization $>$ carbon compounds |
| **yahA** | 82 | 1171.92 |  |  |  |  |  |
| rcsA | 87 | 1151.97 | $**$ | $*$ | $*$ |  | biosynthesis of colanic acid (M antigen) |
| lexA | 20 | 996.62 | $**$ | $*$ | $*$ | $*$ | SOS response; DNA repair; protection $>$ radiation |
| lldR | 65 | 976.84 | $**$ | $*$ | $*$ | $*$ | energy metabolism; aerobic respiration |
| fliA | 45 | 956.70 | $**$ | $*$ | $*$ |  | motility, chemotaxis, energytaxis |
| fliA | 18 | 903.46 | $*$ | $*$ | $*$ |  | biosynthesis of flagellum; motility, chemotaxis, energytaxis; flagella |
| nac | 85 | 827.17 |  | $*$ | $*$ |  | nitrogen metabolism |
| **yiaG** | 15 | 816.55 |  |  |  |  |  |
| **ydaK** | 23 | 815.75 |  |  |  | $**$ |  |
| **ydaK** | 154 | 805.22 |  |  |  |  |  |
| fnr | 23 | 798.27 | $*$ | $*$ | $*$ | $**$ | energy metabolism; anaerobic respiration; membrane |
| lrp | 5 | 777.80 |  | $*$ | $*$ |  | biosynthesis of building blocks $>$ amino acids |
| araC | 46 | 760.44 | $**$ | $*$ | $*$ | $**$ | carbon utilization $>$ carbon compounds |
| appY | 50 | 748.75 |  |  |  |  |  |
| **yfiE** | **67** | 736.50 |  |  |  |  |  |
| **osmE** | 15 | 734.87 |  |  |  |  |  |
| lexA | 78 | 726.67 | $**$ | $*$ | $*$ |  | SOS response |
| purR | 144 | 708.63 |  | $*$ | $*$ |  |  |
| uidR | 81 | 708.36 |  | $*$ |  |  |  |
| araC | 21 | 678.10 | $*$ | $*$ | $*$ |  | carbon utilization $>$ carbon compounds |
| **yfeG** | 29 | 663.94 |  |  |  |  |  |
| **b1450** | **53** | 662.16 |  |  |  |  |  |
| flhC | 18 | 650.64 | $**$ |  | $*$ |  | biosynthesis of flagellum; motility, chemotaxis, energytaxis; flagella |
| **ogrK** | 83 | 645.35 |  |  |  |  |  |
| fliA | 17 | 637.28 |  | $*$ |  |  |  |
| rpoS | 14 | 637.13 | $**$ |  | $*$ | $*$ | adaptations $>$ osmotic pressure |
| pdhR | **55** | 633.52 |  | $*$ | $*$ |  | energy metabolism; anaerobic respiration |
| tdcA | 31 | 619.06 | $*$ | $*$ | $*$ | $*$ | threonine catabolism; carbon utilization $>$ amino acids |
| **yebK** | **106** | 617.44 |  |  |  |  |  |
| araC | 56 | 608.17 | $**$ | $*$ | $*$ |  | carbon utilization $>$ carbon compounds |
| csgD | 26 | 599.30 |  | $*$ |  |  |  |
| hycA | 66 | 596.27 |  |  |  |  |  |
| tdcR | 11 | 593.75 |  |  | $*$ |  | carbon utilization $>$ amino acids |
| fliA | 24 | 593.05 | $*$ | $*$ | $*$ |  | flagella; motility, chemotaxis, energytaxis; biosynthesis of flagellum |
| chbR | 24 | 590.31 |  | $*$ |  |  |  |
| hycA | 29 | 563.45 |  |  |  | $*$ |  |
| galS | 76 | 561.25 | $**$ | $*$ | $*$ | $**$ | carbon utilization $>$ carbon compounds |
| **nlp** | 77 | 559.41 |  |  |  |  |  |
| **yfeC** | 119 | 549.33 |  |  |  |  |  |
| **b1506** | 36 | 548.33 |  |  |  |  |  |
| lrp | 10 | 528.90 | $*$ | $*$ | $*$ |  | biosynthesis of building blocks $>$ amino acids |
| **cspB** | 37 | 527.86 |  |  |  |  |  |
| cusR | 68 | 515.56 | $**$ | $*$ | $*$ | $**$ | extrachromosomal $>$ transposon related |
| **b1284** | 51 | 514.78 |  |  |  |  |  |
| nanR | 9 | 508.87 |  |  |  |  |  |
| **yohL** | 90 | 496.21 |  |  |  |  |  |
| lrp | 126 | 493.60 |  | $*$ | $*$ |  | biosynthesis of building blocks $>$ amino acids |
| **yjjQ** | **179** | 491.02 |  |  |  |  |  |
| **yehV** | **63** | 483.29 |  |  |  |  |  |
| **ogrK** | 27 | 481.75 |  |  |  |  |  |
| slyA | 3 | 474.43 |  |  |  |  |  |
| **ydcN** | 16 | 467.66 |  |  |  |  |  |
| cpxR | 9 | 465.39 | $*$ | $*$ | $*$ |  | adaptations $>$ other (mechanical, nutritional, oxidative stress) |
| **yehV** | 34 | 451.77 |  |  |  |  |  |
| fruR | **63** | 449.25 |  |  |  |  |  |
| araC | 64 | 441.57 | $*$ | $*$ | $*$ |  | carbon utilization $>$ carbon compounds |
| fis | 19 | 436.12 | $**$ | $*$ | $*$ | $*$ | information transfer $>$ RNA related $>$ tRNA |
| fadR | 16 | 435.98 | $*$ |  |  |  |  |
| purR | 10 | 431.78 | $**$ | $*$ | $*$ |  | biosynthesis of building blocks $>$ nucleotides |
| cadC | 37 | 429.32 |  | $*$ |  |  |  |
| fecI | 54 | 429.28 |  | $*$ |  |  |  |
| **rstA** | **102** | 428.94 |  |  |  |  |  |
| tdcR | **61** | 428.84 |  |  |  |  |  |
| flhC | 24 | 426.88 | $**$ |  | $*$ | $*$ | flagella; motility, chemotaxis, energytaxis; biosynthesis of flagellum |

Biological validation of the LeMoNe 30% precision network for *E. coli*. Target enrichment: ($*$) module is enriched in known targets of the predicted regulator, ($**$) module is most enriched for predicted regulator. Autoregulator: ($*$) regulator is an autoregulator. Pathway: ($*$) module is enriched in the same function(s) as the regulator. Local: ($*$) regulator is in the same operon as the module genes, ($**$) Transcription unit of regulator is adjacent to transcription units of the module genes. Function: enriched functions of the module. Regulators in bold face are putative regulators without known targets; module IDs in bold face consist only of uncharacterized genes.

# Supplementary information (1 table + 5 figures)

<embed src="comparison_lemone_clr_table_s1.pdf" style="width:82.0%" />


author: Luc Jaulmes, Miquel Moretó, Mateo Valero,  and Marc Casas
bibliography: IEEEabrv.bib; references.bib
title: Memory Vulnerability: A Case for Delaying Error Reporting

# Introduction

is a major roadblock to successfully design ever smaller and more power efficient high performance computers . Fault rates in transistors and DRAM cells keep increasing with their miniaturization, and a shift in paradigm to gain in energy efficiency, such as using near-threshold voltage , could decuplate the effects of soft errors.

In order to assess the reliability implications of system design decisions, Mukherjee et al. devised a metric: the Architectural Vulnerability Factor (AVF) , which allows to accurately model the error rate of a chip. This metric quantifies whether a given bit matters for reliability, using the probability that a fault at this bit may cause an error in the final outcome of a program. Averaging this value per component for various workloads indicates which components require error mitigating techniques. For example, a speculative component such as a branch predictor will never affect reliability, while an instruction decode unit has a major impact on any program.

In memory subsystems however, all storage bits are uniformly protected with the same level of Error Correcting Code (ECC). This means that bits in memory that will have no impact on the program are protected at the same expense than bits that will have a major impact on the program's outcome. This is because it is impossible to statically assess which storage bits are more likely to affect program reliability. Indeed, as the placement of data in memory can change at every program execution, all bits in DRAM are considered interchangeable. Thus, to generalize the notion of AVF to memory, it is necessary to use a metric dynamically, depending on the data stored at a given bit, and on the way the program uses this data, at the scale of a single program execution.

In this paper, we formalize the *Memory Vulnerability Factor (MVF)*, which is targeted at approximating the probability of error in a program and has been used under different names in the literature. We then extend this metric to take into account false errors, which are reported but would otherwise have no impact on the program being run. The MVF metric is dynamic, adapting to program behaviour, yet is not application-specific as it can be used for any program. We measure it using a cycle-accurate simulator, as well as all previous generalisations of AVF to memory . We compare these metrics to the probability of an architecturally incorrect program outcome due to a fault in its data, obtained from real-world fault injections. These experiments demonstrate that the false error aware memory vulnerability metric correlates best with the probability of an architecturally incorrect program outcome, and gives a consistent upper bound on this probability.

The paper is organised as follows: in Section  we define the vulnerability metrics for memory, while in Section  we introduce the metrics used in the state of the art. In Section  we present our experimental setup and in Section  our results, before presenting our concluding remarks in Section .

# Metric Definition

The goal of quantifying memory vulnerability is to assess which bits matter to the outcome of a program. In order to do this, we first present a classification of the possible outcomes of a program, then introduce the MVF metric, whose scope is limited to memory, while taking into account program behaviour. We then present the *False Error Aware (FEA) MVF*, improving the MVF by analysing how false errors can in fact be ignored.

## Linking Program Outcome and Vulnerability

We categorise the outcome of a program as an Architecturally Correct Execution (ACE) when its execution is indistinguishable from an execution without errors. This includes executions with errors that are corrected, ignored, or benign (thus have no measurable impact on the final state). Architecturally incorrect executions (un-ACE) may be due to a program that finishes running improperly (e.g. crash), stops making progress, finishes but returns an incorrect result, or finishes and returns a correct result but having performed more work than a non-faulty execution. The goal of assessing vulnerability in memory is to know, dynamically, which bits have a higher likelihood of causing an un-ACE outcome.

When encoding data with ECC, data bits are stored together with redundant bits, as an ECC codeword. This makes computing a per-bit metric such as vulnerability slightly more complex. For example a Single-Error-Correct Double-Error-Detect (SECDED) code means any single bit error in the codeword is correctable . However, the bits in the codeword still affect the state of the program. To reflect this, we attribute to an ECC codeword stored in memory the average vulnerability of all the bits in the unencoded data. This allows to quantify the importance of the data that is encoded beyond the simple ECC strength considerations. The properties of the AVF are also maintained: the structure's value is still the average of that of all its bits. Thus, in order to assess the real impact of faults in data stored in memory, we need to measure the program outcomes when injecting faults in the unencoded data bits.

## Memory is Vulnerable Before Being Loaded

As storage bits are interchangeable from one program execution to another, the vulnerability of a memory location is linked to the data that is stored at that location, rather than to the physical DRAM cells themselves. Therefore, we choose to use a definition of memory vulnerability that depends dynamically on the use of a given memory location by a program.

To do this, we classify data residing in memory as either vulnerable or safe, based on the next memory access to this data. Data may be read or overwritten. We classify data as vulnerable when the next access to this data is a load, as any faults in this data will be propagated in the memory request. Conversely, data is safe when the next access to it is a store. We then define the *Memory Vulnerability Factor (MVF)* for each memory location as *the fraction of time a memory location contains data that will be read*. That is, ${MVF = vulnerable~time / (safe~time + vulnerable~time)}$.

The core idea of this definition of the MVF is that only data that is loaded from memory can impact the program state, while ignoring any effects outside memory. Such effects can mask errors in data that is loaded in memory, for example if the data is not used by the program, has a negligible impact, or has been loaded speculatively.

## Accounting for False Detected Uncorrected Errors

The MVF is overestimating the probability of a bit affecting the program outcome, by considering data that is fetched only to be overwritten as vulnerable. Let us look at a write or a set of writes that spans a full ECC word (thus 8B for SECDED or 16B for ChipKill-level ECC ). In a cache with a write-allocate policy, if these writes cause a cache miss, data will be fetched from memory. Any miscorrected errors in this ECC word are masked by the new data being written. Similarly, any uncorrectable errors would be masked as well, if they did not trigger an exception: these are called false errors. Contrarily to benign errors, which affect the program in a negligible way, false errors are caused by faults in data that is not consumed by the program. Thus, in a system without ECC, or with an ECC scheme that has no Detected Uncorrected Errors (DUEs) , the MVF incorrectly categorises that data as vulnerable, as it is based only on fetching the data.

A system with DUE can be enhanced to avoid causing exceptions on false errors, by simply delaying the reporting of any error until it is actually consumed. This is a common pattern in resilience, used for example to handle DUEs in memory discovered during scrubbing. An application likely to access that erroneous data is not terminated preventively, but only whenever it attempts to access the location of the error . Similarly at the architectural level, delaying a machine check exception due to an incorrect instruction allows to avoid exceptions for instructions that do not affect correctness (no-ops, prefetches, etc.), or whose results are either not committed or ultimately do not affect the program .

Thus, we can write off false errors as not affecting the program state. Accordingly, we differentiate between MVF, whose definition is given in Section  and which limited to what happens in memory, and *False Error Aware MVF (FEA)*. To compute the latter, we consider a memory location as safe not only when it is next accessed by a store, but also when it is next accessed by a fill request whose contents will be overwritten. This slight difference in what classifies as a vulnerable memory access does not affect the metrics' mathematical properties.

## Vulnerability under Transient Fault Models

Both the MVF and FEA metrics relate naturally to the probability of consuming a fault, under the common hypotheses for transient fault models.

If we suppose that faults happen randomly and independently of one another, we can model them using an exponential model, with $\lambda$ the average fault arrival rate. Formally, we define for any location in memory $S$ and $U$, the sets of safe and unsafe accesses to that location (i.e. respectively stores and loads for the MVF), and $t_a$ and $f_a$, the time and the number of faults consumed by any access $a$ respectively. Since safe accesses overwrite faults, we have ${a \in S \Rightarrow f_a = 0}$. For unsafe accesses, we consider the period $p_a$ before an access $a$. This lasts ${p_a = t_a - t_{prev(a)}}$, where ${prev(a) = \max\{b \in S \cup U | t_b < t_a\}}$. Thus for unsafe accesses, we have ${P(f_a > 0) = 1 - P(f_a = 0) = 1 - e^{-\lambda p_a}}$, hence the overall probability of consuming a fault is ${P(F) = \sum_{u \in U} (1 - e^{-\lambda p_u})}$. If we reasonably assume that faults are rare, i.e. that the program execution time $T$ is such that ${\lambda T \ll 1}$, then we can approximate ${P(F) \approx \lambda \sum_{u \in U} p_u}$, which is the total time spent before unsafe accesses multiplied by ${\lambda}$.

The vulnerability $V$ is the fraction of time a memory location contains data that is unsafe, thus with the same notations, ${V =
\frac{1}{T}\sum_{u \in U} p_u}$. We then have ${P(F) \approx \lambda TV}$, which confirms the intuition of the vulnerability $V$ being proportional to the probability of consuming a fault in memory.

# Related Work Gauging Memory Vulnerability

Previous work has studied the variable vulnerability of data in memory using metrics similarly inspired by the AVF.

Yu et al. define the *Data Vulnerability Factor (DVF)*  per data structure $d$, defined as the multiplication of the structure's size $S_d$, the program execution time $T$, the number of accesses to this structure in memory $N_{ha}$, and the overall fault rate $FIT$: ${DVF_d = FIT \cdot T \cdot S_d \cdot N_{ha}}$. They then use mathematical models to compute the DVF based on memory access patterns.

Luo et al. use the *safe ratio*, which is the fraction of time that data resides in memory before being overwritten . This is the same as the MVF, except that it chooses to quantify the opposite of vulnerability. The MVF and the safe ratio $sr$ are related by ${MVF + sr = 1}$. Luo et al. do not inject faults in native runs, instead using a debugger.

Gupta et al.  use two metrics, initially measuring the MVF, defined as "the average duration that data is stored in memory before being loaded". They then use a proxy metric, which is the ratio of stores ($ST$) to loads ($LD$), thus $ST/LD$, for the purpose of their runtime page-placement algorithm. The analysis of the impact of faults is never done with the real program outcome, as the experiments are run inside FaultSim, a Monte-Carlo model-based simulator, where loading an uncorrectable or undetected error is equated with program failure.

# Methodology

To compare the vulnerability metrics with the probability of un-ACE outcomes in the event of faults, we examine outcomes when injecting faults in native parallel runs of the Conjugate Gradient (CG), and measure vulnerability ratings precisely using a cycle-accurate simulation infrastructure. We first explain the simulator setup, and then detail the fault injection experiments before presenting the benchmark, CG.

## Measuring the Memory Vulnerability

To be able to gather enough information about when data reaches or is fetched from memory, we use a cycle-accurate simulator. We extend TaskSim , a task-trace based multicore simulator, to compute the exact memory vulnerability ratings of data. Its infrastructure relies on task-based execution models to generate detailed traces for each task, including the basic blocks that are executed and memory addresses that are accessed. TaskSim's multicore architecture simulator then simulates parallel runs in detail by fetching and simulating all instructions, using a simple core model and a full memory hierarchy. The simulator also relies on a real runtime system, to schedule the tasks across the simulated hardware.

To compute the various vulnerability metrics, we capture all loads and stores and the time at which they reach main memory. We then update at each access the necessary counters per memory location: time before stores, time before loads, FEA safe time (before fill requests whose contents will be overwritten), and number of loads and stores. From this data, we compute the fraction of time that each location is vulnerable, the DVF and the load-to-store ratio. We only update these counters during the Region Of Interest (ROI), thus while the solver is running. We compute all metrics at a 64 bit granularity, which is the granularity used for SECDED and a subset of the granularity commonly used in ChipKill-level ECC.

We trace applications on an Intel x86_64 Xeon E5-2670 and simulate a multicore architecture whose configuration mirrors the Xeon E5-2670's characteristics. It consists of 8 cores running at a frequency of 2.6GHz, each with a reorder buffer of 168 entries, and one thread per core. The memory hierarchy's parameters are summed up in Table . All cache levels have 64B lines, write-back and write-allocate policies, are non inclusive, and track outstanding misses in Miss Status Handling Registers (MSHRs). In the simple model used for main memory, every request has the same latency and returns a full cache line.

|  cache | shared        | assoc. | size  | latency    | MSHRs |
|-------:|:--------------|:-------|:------|:-----------|:------|
|    L1D | private       | 8-way  | 32kB  | 4 cycles   | 32    |
|     L2 | private       | 8-way  | 256kB | 12 cycles  | 32    |
|     L3 | shared        | 16-way | 20MB  | 28 cycles  | 128   |
| memory | max bandwidth |        | size  | latency    |       |
|        | 16GB/s        |        | 32GB  | 155 cycles |       |

TaskSim cache and memory parameters

## Measuring the Program Outcome

To analyse and validate the memory vulnerability metrics, they need to be compared against the outcomes of injecting faults in the memory of the program. Thus, we inject faults in native parallel runs on a real system, using the same Xeon E5-2670 as used for the simulation infrastructure.

Faults are injected using a separate thread, at a uniformly random point in the targeted application-level data, and at a uniformly random time during the ROI. The injector thread sleeps for the selected amount of time, then injects the single event by flipping the selected bit. Each experiment consists of a single fault injection, and each experimental campaign consists of at least 6500 parallel CG runs per data structure.

The prologue, which consists of generating the program's input, and the epilogue, which consists of verifying the program's output, are not considered in this analysis. The program runs until it finishes abnormally or until completion, in which case it verifies the validity of the solution it found. This verification is always done against the unmodified input data. If the program performs more work (in this case, more iterations) than the ACE baseline, we classify it as un-ACE regardless of the validity of the solution. If the program does not finish within $10\times$ the baseline (ACE) execution time, we classify its execution as un-ACE and interrupt it.

## The Conjugate Gradient Benchmark

To perform our experiments, we use the Conjugate Gradient (CG) as a benchmark, whose pseudo-code is presented in Listing  . CG solves $Ax = b$ for $x$, where $A$ is a sparse symmetric positive definite matrix. $x$, $g$, $d$, $d'$, and $q$ are vectors, and $\epsilon$, $\epsilon_{old}$, $\alpha$ and $\beta$ are scalars. The matrix A is stored in memory in compressed sparse row format. Thus, we refer to it as 3 separate data structures: the rows $Ar$, columns $Ac$, and values $Av$. We solve the 3D Poisson's equation discretised with a 27 point stencil, and a size of $64^3$ rows.

We implement CG using OmpSs, a task-based dataflow programming model, with tasks each generating one block of each of the vectors or sums (in the case of reductions) . We use two copies of $d$ and swap their pointers (line 15) to allow delaying tasks that depend on one copy, such as ${x
\Leftarrow x + \alpha d}$. This allows the runtime to ignore the false dependency due to overwriting $d$, and to overlap $x$'s update with operations that incur load imbalance such as $\langle d, q\rangle$, thus improving scalability.

# Evaluation

In this section, we compare the probability of un-ACE outcomes when injecting faults against the MVF, FEA, and other metrics used by related work.

## Memory Vulnerability Metric Validation

The results of fault injections in real runs, categorising the outcomes as architecturally correct (ACE) or not (un-ACE), are presented as bars in Figure , while the various vulnerability ratings obtained from cycle-accurate simulations are presented as lines. The bars represent the probability of an un-ACE outcome, and are obtained from fault injections. Data structures are sorted in increasing un-ACE probability, and confidence intervals are not displayed, as they are too small. The largest interval is for $g$, whose probability of suffering an un-ACE outcome is in $[22.0\%, 24.6\%]$ with 99% confidence. The vulnerability metrics, displayed as lines, are the MVF, defined in Section  and similar to the *safe ratio* , the FEA, defined in Section , the load-to-store ratio , and finally the DVF  in the separate graph above.

The probabilities of un-ACE outcomes are consistent with the memory access pattern of the CG algorithm for each data structure, taking into account that the solver converges in 27 iterations for this matrix and that most of the time of one iteration is spent computing the matrix-vector multiplication. The $b$ vector is only used for the initial computation of the residual, and the data is never read afterwards, hence with a vulnerability of $2.9\%$. Since $Ac$ and $Ar$ are arrays of indexes, it is expected that faults in these data structures may cause crashes, in the form of segmentation faults. All other data consists of floating point values. When injecting faults in these structures, un-ACE program outcomes are almost never crashes, but instead incorrect or slower executions. Furthermore, we can note that these data structures generally have lower un-ACE outcomes than the integer ones. While this is due in part to the way data is accessed by the program, it is also inherent to floating point data, as shown by the difference between $Ac$ and $Av$, which both have the exact same access pattern. Errors in lower-order bits of floating point data likely have a negligible effect on the solver, as their relative significance is small.

The various vulnerability metrics are presented as lines in Figure . Comparing the MVF (unbroken line) and FEA (dashed line) to the un-ACE frequency, we can see that both metrics provide an upper bound on this probability. The FEA gives a much tighter bound for the $d$, $d'$ and $q$ vectors. This is due to these vectors being overwritten and not updated, as can be seen in Listing . Thus, false errors would be overwritten approximately 50% of the time: after the vector is read and before it is computed again. On the other hand, $g$ and $x$ are updated, thus with a read-modified-write sequence for each element. Accordingly, both MVF and FEA evaluate these vectors in the same way. Finally, $Av$, $Ac$, and $Ar$ are only ever read, and $b$ is only read once at the beginning of the execution, thus mostly contains inconsequential data. Overall, FEA correlates better with the un-ACE probability.

As the load-to-store ratio $LD/ST$  is unbounded ($b$'s rating is $+\infty$ for example) we normalize it as ${LD/(LD + ST)}$, and present this value as the dotted line in Figure . This transformation maintains the relative order of the ratings' values while bringing them back in the interval $[0, 1]$. We see several problems with this load-to-store ratio: $b$, which is mostly inconsequential, is rated equally vulnerable as $Ar$, and both $x$ and $g$ have the lowest vulnerability ratings while being the third and fourth most vulnerable vectors, where faults cause un-ACE results with probabilities of $25\%$ and $46\%$ respectively. Furthermore, the load-to-store ratio over-estimates but also under-estimates the vulnerability of considerable chunks of data, and can thus not be safely used as an upper bound on failure probability. For example, the first parts of vector $q$ to be computed are very vulnerable, and the last ones very safe, as computing this vector is an operation that takes a long time. However, the load-to-store ratio rates the whole vector with the same vulnerability, by considering only the number of accesses.

The DVF metric  is not bounded either and its values are high and very spread out, hence we display them on a log scale at the top of Figure . The main factor impacting the DVF of a data structure is its size, which causes $Ac$ and $Av$ to have the highest values. The most critical data, $Ar$, is rated as the second safest data structure by the DVF. The other values are very close together, in increasing order: $d$ and $d'$, followed by $g$ and $x$, and then $q$. Overall this correlates poorly with the probability of un-ACE outcomes, and the fact that the metric is not bounded makes it harder to use at runtime. Indeed, the metric only has meaning when comparing values relative to each other, making it impossible to set thresholds on DVF values for example.

# Conclusion

A number of metrics aim at quantifying the risk associated with encountering an error in data in memory. Comparing these metrics with the likelihood of architecturally incorrect executions due to a fault in memory indicates that the *False Error Aware Memory Vulnerability Factor (FEA)*, which is introduced in this paper, is the most accurate one. This can be explained by the fact that it takes into account timing effects, as opposed to DVF or store-to-load ratios, and the fact that overwriting data that has not been accessed in a while may cause false errors, which can be ignored. This work also highlights that hardware support to track false errors in the cache hierarchy would likely significantly decrease error rates due to DUE in memory. Furthermore, this work opens the door to runtime-level optimizations that can now accurately model the risk associated with any given data.

# Acknowledgements

# Acknowledgement

This work has been supported by the RoMoL ERC Advanced Grant (GA 321253), by the European HiPEAC Network of Excellence, by the Spanish Ministry of Economy and Competitiveness (contract TIN2015-65316-P), by the Generalitat de Catalunya (contracts 2014-SGR-1051 and 2014-SGR-1272) and by the European Union's Horizon 2020 research and innovation programme (grant agreements 671697 and 779877). L. Jaulmes has been partially supported by the Spanish Ministry of Education, Culture and Sports under grant FPU2013/06982. M. Moretó has been partially supported by the Spanish Ministry of Economy, Industry and Competitiveness under Ramón y Cajal fellowship number RYC-2016-21104.

The authors would like to thank Francesc Martínez Palau for his precious help and support with the TaskSim infrastructure.


abstract: -1As Artificial Intelligence (AI) becomes pervasive in most fields, from healthcare to autonomous driving, it is essential that we find successful ways of building morality into our machines, especially for decision-making. However, the question of what it means to be moral is still debated, particularly in the context of AI. In this paper, we highlight the different aspects that should be considered when building moral agents, including the most relevant moral paradigms and challenges. We also discuss the top-down and bottom-up approaches to design and the role of emotion and sentience in morality. We then propose solutions including a hybrid approach to design and a hierarchical approach to combining moral paradigms. We emphasize how governance and policy are becoming ever more critical in AI Ethics and in ensuring that the tasks we set for moral agents are attainable, that ethical behavior is achieved, and that we obtain good AI.
author: Reneira Seeamber; Cosmin Badea
date: 2024-09-30
title: If our aim is to build morality into an artificial agent, how might we begin to go about doing so?

If our aim is to build morality into an artificial agent, or machine, how might we begin to go about doing so? This is a crucial question that is of particular relevance at present, in the context of the flurry of recent activity in the field, including recent but already famous applications such as ChatGPT and Bard.

Let us begin by defining key terms in this question, namely 'morality' and 'artificial agent'. Morality can be defined as the differentiation of intentions, decisions, and actions into those that are good (or right) and those that are bad (or wrong) ^^. An artificial agent, or machine, we will consider to be a system that can intelligently perform certain tasks by making decisions.

Deciding whether or not we should build morality into artificial intelligence (AI) is determined by many factors, including whether we want the AI to act as a moral agent or adviser and whether we want to build autonomy into it. Going down the latter route brings with it an obvious need for morality to be implemented, but the fundamentals of what it is to be 'good' have long been debated and no clear consensus has been reached^^.

To begin addressing this dilemma, we can look at the three main philosophical theories of morality in AI Ethics: virtue ethics, utilitarianism, and deontology^^. Deontology emphasizes the imperative moral value of certain actions or rules, rather than their consequences. Virtue ethics, on the other hand, focuses on the character of the individual who is acting, rather than on specific rules or actions. It emphasizes the importance of developing virtuous traits, such as honesty and courage and developing a virtuous character, which allows individuals to make morally good choices. Utilitarianism is a consequentialist theory that focuses on actions that maximize utility, commonly measured as happiness or pleasure, for the greatest number of people ^^.

Existing research has investigated different approaches to developing good AI and important obstacles and potential solutions have been identified. For instance, the *Interpretation Problem*, introduced by Badea and Artus (2022)^^ is the idea that any symbolic representation we might use for AI necessarily gives rise to a plethora of possible interpretations, and thus ambiguity and mistakes by the AI. There are ways to mitigate its effects, such as building character into the agents, thus creating virtuous agents by using values as a tether between what the AI understands and what we mean to communicate to it ^^. There are also more practical questions, such as those around what role agents would play in a clinical context, with proposals including them acting alongside the human experts, perhaps relying on such experts as moral exemplars. ^^. Additionally, there are arguments around the use of particular moral paradigms (such as utilitarianism, deontology and virtue ethics) in building ethical AI and resolving contemporary moral dilemmas like breaking bad news to a patient ^^ or antimicrobial resistance and use ^^. Furthermore, the variety of potential moral paradigms and lack of agreement has also led to a perceived necessity for a moral paradigm-agnostic decision-making framework, which Badea (2022) proposes as *MARS* ^^. However, building decision-making agents is itself a difficult process, and deciding on how to do it, or Meta-decision-making for AI ^^, is a new and very promising field of research, which we aim to contribute to here. An ontology for this process has been proposed before, based on three steps, in Badea and Gilpin (2022). In this paper, we continue previous attempts to identify methods and challenges for building good AI, and to propose solutions that overcome these challenges. With this work, we examine canonical approaches to AI to provide a broader perspective on implementing morality. Having identified many different challenges and potential ways of dealing with them, we found ourselves in the position of wanting to take a step back and focus on the techniques of building ethical AI. Thus, we examined existing methodology from the literature and identified top-down and bottom-up approaches from classical AI ^^. Having identified these canonical approaches, we aimed to apply them to the task of mitigating the issues discussed above. Hence we set out with the goal of coming up with and categorizing such methods, and continuing this line of thought in attempting to examine the potential technical approaches to implementing the suggestions above and constructing working ethical AI.

# THE TOP-DOWN APPROACH ALONE MIGHT BE INSUFFICIENT

An important aspect to consider in designing AI is the influence of culture and how it affects morals. Massachusetts Institute of Technology (MIT) created the Moral Machine, an online simulation where the public can decide for themselves which is the 'right' action to take given variations of the famous 'Trolley dilemma' ^^. The dilemma originally involved choosing between allowing a group of five people to die or choosing to pull a lever and sacrificing one person instead, while the Moral Machine creates variations on this with different demographics in each of the two groups. Global data from the MIT study revealed variations in how people respond to such moral dilemmas, given their demographic. For example, people in Western countries were more likely to favor saving younger people than the elderly compared with people in Eastern countries. This casts doubt on the possibility of using a top-down approach alone because implementing such approaches (utilitarianism, deontology) might be nigh impossible, given that there may never be a global consensus between humans on developing a universal set of 'rules' or 'duties'.

Noothigattu et al. (2018) proposed a voting-based system, whereby data from the moral machine is collected and used to train multiple models to learn societal preference for alternative outcomes across multiple ethical dilemmas. These models are then aggregated to form a model that considers the collective outcome of all voters, thus satisfying the utilitarian approach. This method is limited in that self-reported preferences have been shown to vary from real-life decisions ^^. Additionally, it may be argued that few AI developers would have to deal with ethical dilemmas, such as the Trolley problem, in their AI application and the majority of humans have not or will not have to make such a decision. Instead, we should be looking towards more realistic dilemmas, such as social media advertising and whether it could lead to accessibility to violence or compulsive behavior formation. It is up for debate whether a set of deducible principles could be created and whether a model needs more than simply a set of internal coherence constraints, as it could easily satisfy constraints but still produce undesirable outcomes. In reality, multiple goals and rules come into conflict with each other, producing complexity. Developing rules that a machine will understand in the way we desire is another issue in itself, which may stem from us not knowing explicitly what it is we want them to understand ^^.

In Plato's dialogue Euthyphro, Socrates asked Euthyphro, "Is the pious loved by the Gods because it is pious or is it pious because it is loved by the Gods?". This is proving very relevant to AI morality as we, humans, may begin to see ourselves as the Gods, or creators, of AI. Thus, is it what we (subjectively) believe morality to be that AI should consider as moral, or should we leave it to the AI to determine what is moral for itself, potentially then leading us to reassess what we see as moral? For instance, the aggregation of societal preferences has the potential to produce a morally better system than that of any individual alone, and may even enable us to then identify general principles that underlie our own decision-making ^^.

# EMOTION, SENTIENCE AND MORALITY

It has been debated, especially following Jeremy Bentham's and John Stuart Mill's formulation of 'the greater good for the greatest number', whether the stoic characteristic of machines limits their ability to carry out moral decision-making because of their incapacity to care. The dual-process theory of moral judgment, which proposes the idea of dichotomous thinking, with a fast, instinctive and emotional process driving us alongside a slower and more logical, reason-guided process ^^, has led to the question: Does this emotional capacity of humans lead us to moral subjectivity? Or does it allow us to decide between what is wrong or right? Thinking of humans as having two different systems may play an important role in developing AI machines that have both reason and the capacity to care. 'Cognitive emotion' could be built into machines, whereby one allows for weights to be added to, for example, 'human life', thereby producing somewhat similar caring systems for human life ^^.

However, the innate stoicism of machines may be advantageous in reducing the moral biases we have as humans, one of the limiting factors as to why it is hard to come to a consensus ^^. The idea of 'embodiment' and whether artificial moral decision-making should be extended with emotional and sensory components is interesting. However, various issues come to light when extensive sensor systems are involved. For example, these are more vulnerable to attack, and people may become worried about privacy issues.

Whether or not a sentient agent is achievable remains to be explored. The symbiosis between technology and humans will become ever more important, and similar to a jury determining someone's innocence, or a group of doctors discussing how to proceed with the most complex medical problems, there are simply certain decisions that require several inputs before coming to a conclusion ^^. It is therefore imperative that any system taking on responsibilities that would otherwise be discussed by a group of people should have a notification state. This state would ensure that a board of human overseers intervene and is involved in the final decision when conflicts arise.

# PROPOSING A HYBRID APPROACH

How do we, as humans, learn our morals and values? Why do they differ so much globally as we have seen with the Moral Machine experiment? Children grow to imitate and follow what they have been bought up to see. It is the complex interaction between nature and nurture that ultimately shapes our values, as is becoming more notable in the field of behavioral epigenetics.

With the advent of apprenticeship learning, i.e. inverse reinforcement learning, it would be reasonable to try and develop similar artificial systems that would learn by viewing the behaviors of humans ^^. As opposed to reinforcement learning which uses rewards and punishments to learn certain behaviors, inverse reinforcement learning would involve an AI system observing human behaviors and figuring out the goal of the behavior itself. This paves a way for coding the complexity of human ethical values without having to explicitly code every rule for every possible scenario. However, this highlights the need once again for stringent thought into the data AI should learn from. A child growing up with abusive parents would possibly learn that it is right to behave in such a way. An excellent example of how this type of exploitation could affect AI is Tay bot, which learned racial remarks from Twitter trolls ^^.

Regarding the data an AI should learn from, we could look to the actions of the moral exemplars ^^, and thus to the agent-based ethics approach proposed by Aristotle: virtue ethics. This emphasizes an agent's moral character in their actions, rather than emphasizing duties (deontology) or consequences (utilitarianism). Aristotle himself stresses that virtue is gained through habit, thus a virtuous person consistently embodies the right actions because they have built the right character.

For instance, in a similar manner to how we have courts and ordered decision-making processes in place to judge one's character with a rigorous review panel, it may be possible to have an equivalent scrutiny panel for AI. We could use this as part of a hierarchical structure for building moral AI, in which we could combine choice aspects of different moral paradigms, such as virtue ethics, deontology and utilitarianism, an example of which we provide in (Figure 1).

Furthermore, tying back to one of the first points regarding the differences in moral ideologies across different cultures, it is important to consider the principles that can be taken from other philosophers, for example, Confucian or Buddhist virtues. The four pillars of Confucian ethics (Yen, love and empathy; Yi, righteousness; Li, veneration and comity; Zhi, wisdom) could also be considered when choosing virtuous people for data collection as the 'teachers', or moral exemplars, of the AI.

Hagendorff (2022) performed a reductionist clustering approach to meta-studies on AI ethics guidelines to distill four fundamental AI virtues: Justice, honesty, responsibility and care ^^. The author suggests that these four virtues are behind the majority of practical AI ethical principles discussed in the literature today, such as fairness, transparency, accountability and social cohesion. Notably, prudence and fortitude were identified as second-order virtues that aim to counteract factors, such as innate bias and peer influence. This provides an important starting point for addressing how we might define and identify moral exemplars for developing ethical AI.

As shown in Figure 1, following the selection of 'teachers', or moral exemplars, a refinement process will take place: data will be chosen through the selection of actions that comply with very specific rules imposed that are decided upon a priori, as constituting duties in a deontological fashion. This could be followed by a selection of valid actions which exhibit consequences desired according to utilitarianism or other frameworks, such as being desired by the majority as suggested by Noothigattu et al. (2018) ^^.

It is necessary to highlight an important hypothetical scenario described by Wallach and Allen, regarding AI-based trading systems that lead to overly high oil prices ^^. This would lead to other automated systems moving to coal as an alternative source of energy to reduce costs – a massive influx in coal demand and the need for full-time production could lead to an explosion and power outage. As one can see, the channels of multiple automated decision systems in interaction and under the influence of one another could lead to catastrophic human and economic consequences. It is with that, that there has to be a higher-order framework in place to assess all potential interactions between AI systems, for example, the AI influencing the trade of oil with an AI deciding which power source to use.

This ties into the self-awareness embodiment of AI and awareness of other systems as well as humans. Hence, the cognitive simulation theory seems another way to help avoid such scenarios where AI could be programmed through internal simulations of actions with predictions of their consequences, instead of logic statements ^^. In this paper, a multi-layer architecture involving a controller that generates several actions to be performed to reach a certain goal is proposed. These actions are then sent to an ethical layer that simulates each action using the current state of the environment, the human involved and the robot. Internal states of the human and robot are retrieved and used in conjunction to evaluate the best action to take.

With this, it is important to determine the tests and simulations that must be carried out even before AI systems are given higher moral authority. Devin Gonier (2018) introduced the notion of 'tricking' AI models based on the Hawthorne effect - the observer affects the outcome of an experiment ^^. Based on this, if we were to constantly observe a model, the outcome should be in our favor. We could have two scenarios. The computer thinks it moves from a simulation to the real world, but in reality, it stays in simulation mode. Conversely, the model could think it's in a simulation but actually in the real world. Ultimately, keeping the model in a 'doubt' mode enables control.

Could we then have parallel systems running that would each take a vote on the best action? If the AI working on trading systems could predict that high oil prices would result in decisions to move away from oil by other logically-programmed machines, it would know to avoid such a scenario. Such a framework could be combined with the inverse reinforcement learning approach where the AI learns from observing actions from exemplars, and figuring out the goals and internal states of impacted agents. In doing so, an extensive learned base could be created to enable internal simulations of actions and accurate predictions of consequences without the need for explicit rules. Such a process would take time and experience to learn, hence the introduction of regulatory sandboxes, which will be discussed in the next section, are a key way to ensure a controlled application of ethical AI.

# AI GOVERNANCE PRINCIPLES

Ultimately, the development of moral machines heavily relies on stringent rules in place for deploying such systems. At the very least, it should be proven that accountability has been covered, the system is transparent for inspection, cannot be easily manipulated (i.e. strong security measures have been employed to reduce the chance of hackers intercepting) and that the AI has a high level of predictability that can match the majority of humans. Regarding opacity and accountability, it may be possible to have parallel functioning systems, the black box - referring to complex models that are not highly interpretable to humans - and it's more transparent shadow that aims to conduce step-by-step thought processes. Prospector is one such example, whereby researchers have developed an interactive visual representation of predictive models that aims to help us understand how and why data points are predicted, and the predictive value of different features ^^. Notably, the authors of this paper stress that we need to move away from solely evaluating performance metrics, such as accuracy.

Research is being done to ensure AI is more transparent, for example, gradient-weighted class activation mapping is one method that attempts to explain convoluted neural networks used in histology or magnetic resonance imaging labeling. Maps are created to demonstrate the most relevant areas of images used in the classification. Such techniques enable histopathologists or radiologists to gain some insight into the decision-making process. Whilst this provides information on how a model made a decision, current explanatory methods typically do not justify whether a decision was appropriate or not. Notably, Ghassemi, Oakden-Rayner and Beam discussed the idea that humans unknowingly use heat maps to understand whether the areas identified are justifiable based on intuition, and thus, bias can be introduced. Humans do tend to over-trust computers, and explainability methods could reduce the ability to identify errors and/or unreasonably increase our confidence in algorithmic decisions ^^.

Indeed, detailed logs must be made available by the AI to help understand any decision and to further guide its development. As suggested by Mitchell et al. (2019), documentation detailing the performance of models in different contexts (e.g. age, race, sex) should be provided ^^. Increasing numbers of publications relating to frameworks for dataset generation and AI development are becoming available and a compilation of these academic works must be acknowledged in any regulatory documents going forward.

Incorporating these techniques and tools into the AI development process can help us work towards creating AI systems that reflect the world we want to live in. By fostering transparency, accountability, and fairness, we can ensure that AI systems adhere to the principles of virtue ethics in the age of AI.

Various governments, including the UK, US and Japan, have leaned towards the use of 'regulatory sandboxes', defined as a testbed for innovation in a controlled real-life environment prior to launch. The 2023 UK policy paper, "A pro-innovation approach to AI regulation", highlights the use of sandboxes to encourage innovation and provide an adaptable response to regulating AI ^^. This contrasts with proposals for EU legislation (the EU AI Act or the Liability Directive) which take a different approach with more comprehensive regulations, including the General Data Protection Regulation (GDPR) that applies to AI development and data sharing across EU member states ^^. GDPR emphasizes data subject rights, data minimization, and transparency. The UK's Data Protection Act 2018 incorporates the GDPR into UK law, ensuring similar data protection standards for AI development and data sharing. Nevertheless, Spain has piloted the first regulatory sandbox on AI in the EU to aid in generating future guidelines ^^. Many countries have also adopted a sector-specific approach to regulations, impacting AI development and data sharing in various industries, such as finance, energy and healthcare.

As technology constantly evolves, guidelines will also have to evolve with it. There is no set of rules that can foresee all future problems that arise. This is also why it is important to have interdisciplinary research involving engineers, philosophers, neuroscientists, psychologists and policymakers to ensure the best possible outcomes. It may be that regulations require companies to involve a multidisciplinary team during development. University education may also need to introduce varied modules (i.e. ML courses include a philosophy and ethics module, and medical degrees include ML models) to prepare future generations for such collaborative tasks. Regulatory workshops should be made more regular and accessible to inform and guide companies during the development process. Either way, there is still much yet to debate regarding human ethics before machines can become the moral entities we want them to become. If we are to take the bottom-up approach to train AI, we must ensure the data is a reflection of the world we want to live in.

Encouraging open discussions among individuals and communities with different views will be important in decision-making. This could be done by creating national and international forums, where society can share experiences and develop recommendations for AI governance. Such forums can provide a platform for dialogue, consensus-building, and the exchange of best practices. By fostering a culture of dialogue and debate, society can collectively engage with moral dilemmas to understand and overcome the nuances and complexities. Additionally, individuals can further develop the ability to critically analyze moral dilemmas through the promotion of moral education and critical thinking from an early age, enabling them to understand different viewpoints and appreciate the diversity of moral values across cultures and traditions. We must also ensure that decision-making processes are inclusive and represent the diversity of society. This will help to avoid the domination of a single moral perspective and to embrace multiple moral views.

# TOWARDS MORAL AI

In conclusion, we have looked at the matter of building moral machines from different angles, such as those of technical approaches, culture, emotion, sentience, and governance. Our main contribution is the proposal of a hybrid approach using a combination of top-down and bottom-up and a further combination of different moral paradigms to get around the different obstacles and limitations discussed. We have also discussed some of the practical issues and techniques relevant to implementing our approach.

Ultimately, determining what constitutes good morals is an ongoing and collaborative effort that requires input from various stakeholders and an openness to learn, grow, and adapt. By engaging in these processes, society can work towards establishing shared moral values that contribute to the greater good. It is in doing so, that we may be able to start to answer some of the questions we, as humans, hold regarding morality and what it is to act with moral grounding. This collaborative approach can help create a more responsible, inclusive, and beneficial AI landscape for all.

# ACKNOWLEDGMENTS

Reneira Seeamber was supported by the UKRI CDT in AI for Healthcare http://ai4health.io (Grant No. EP/S023283/1). For the purpose of open access, the authors have applied a Creative Commons Attribution (CC BY) license to the Accepted Manuscript.