TripleAgent: Monitoring, Perturbation and Failure-obliviousness for Automated Resilience Improvement in Java Applications
Long Zhang Martin Monperrus
















1]Long Zhang and Martin Monperrus
[1]KTH Royal Institute of Technology, Sweden
 



: Monitoring, Perturbation and Failure-obliviousness for Automated Resilience Improvement in Java Applications
    [
    
==============================================================================================================



fancy

In this paper, we present a novel resilience improvement system for Java applications. The unique feature of this system is to combine automated monitoring, automated perturbation injection, and automated resilience improvement. The latter is achieved thanks to the failure-oblivious computing, a concept introduced in 2004 by Rinard and colleagues. We design and implement the system as agents for the Java virtual machine. We evaluate the system on two real-world applications: a file transfer client and an email server. Our results show that it is possible to automatically improve the resilience of Java applications with respect to uncaught or mishandled exceptions.



fault injection, dynamic analysis, exception-handling, software resilience




§ INTRODUCTION




In modern software, resilience capabilities are engineered through error-handling code, in particular exception handling code in managed languages such as Java and C#.
This resilience capability is manually engineered by developers, who write the error-handling code. For example, part of their coding activity is to write try-catch blocks to handle exceptions. The problem is that exception-handling code is notably hard to write and to test <cit.>. As a result, there often exists corner-cases where resilience is not provided by developer-written code. In production, when those corner cases are activated, the software system may simply stop providing its function because it crashes after an unhandled exception <cit.>.


In order to improve error handling, two kinds of techniques are being researched: fault injection <cit.> and failure-oblivious computing <cit.>. Fault injection is about injecting failures to trigger a system's error handling code and to analyze the abnormal behaviour<cit.>. Failure-oblivious computing is about adding fully generic error-handling code with automated code transformation <cit.>. In the context of exceptions, failure-oblivious computing means automatically adding catch blocks with a default exception-handling strategy <cit.>



In this paper, our goal is to automatically improve the exception-handling code of software applications. This is made by first finding weaknesses in resilience and then instrumenting the application with automated exception handling.
To achieve our goal of automatically improving resilience, we design a novel system, called , made of three components, called “agent” in this paper[Here, an agent refers to the Java terminology, where it is a component that is attached to the Java Virtual Machine <cit.>.].
Those three agents, automated monitoring, automated perturbation injection, and automated failure-oblivious method validation <cit.> are orchestrated by an agent controller. The controller analyzes all the monitored data and reveals both weaknesses and suggested improvements in the resilience capabilities.




To the best of our knowledge, is the first system which actively injects exceptions during execution in order to, after analysis, automatically detect failure-oblivious methods.



We evaluate on two real-world Java applications. One is TTorrent, a file transfer client which implements the BitTorrent protocol. The other one is HedWig, an email server. In both cases, we consider a production workload: respectively downloading a large file from the Internet, and sending and receiving emails from the server.





By applying , we observe that exceptions thrown from 257(21%) perturbation points do not lead to failures anymore, which shows an automatic resilience improvement.


To sum up, our contributions are the following.



  * The concept of joint usage of fault injection and failure-oblivious code instrumentation to evaluate and improve resilience against uncaught or mishandled exceptions. We propose a corresponding novel algorithm for automatic improvement of software resilience.


  * A system called that combines monitoring, perturbation injection and failure-oblivious computing in Java, implemented with agents for the Java Virtual Machine.
The system is publicly-available for future research in this area at <http://bit.ly/tripleagent-repo>.


  * An empirical evaluation of on two real-world applications of 20.3K lines of code in total. By performing 9968 fault injection experiments under a realistic, production-like workload, it shows 's effectiveness for improving software resilience. 



The rest of the paper is organized as follows: <ref> introduces the background. <ref> and <ref> present the design and evaluation of . <ref> discusses the related work, and <ref> summarizes the paper.



§ BACKGROUND

is founded on techniques from the fault injection and failure-oblivious computing <cit.>. This section presents a basic introduction to the core concepts.



 §.§ Fault Injection

Fault injection is a popular research topic in software testing and dependability evaluation. Fault injection techniques actively inject different kinds of errors into a target system in order to assess its dependability<cit.>. This can happen in several phases:
1) during unit testing, fault injection generates more test cases so that corner cases are detected, and the coverage of testing is improved.
2) during integration, fault injection can trigger different failure scenarios so that developers gain more confidence in their system's error-handling design.
3) when done in production, fault injection is usually called “chaos engineering” <cit.>.

The kinds of failures that can be injected vary depending on the considered dependability aspect. For example, injecting processor errors or hardware-based errors is often done for evaluating the dependability of operating systems <cit.>. Injecting connection errors between different micro-services supports to test a service's retry logic and its robustness of interacting with other services<cit.>. Injecting an exception in a certain method is useful for validating an application's error-handling capability<cit.>.

In this paper, we focus on fault injection in the context of Java applications, which means rather high-level, application-level fault injection.



 §.§ Failure-oblivious Computing

In order to improve the resilience of an application, different techniques can be applied to prevent the application from crashing when an error occurs<cit.>. Failure-oblivious computing <cit.> is one of these approaches to overcome software failures at runtime. The main idea of failure-oblivious computing is to discard certain failures in a principled way. For example, if a method tries to write data into an invalid memory address, with failure-oblivious computing, the writing operation would be ignored. It has been shown that failure-oblivious computing is able to increase availability <cit.>, eg to serve requests to more users despite errors.

In this paper, we use the concept of failure-oblivious computing for the Java programming language. In Java, there is no invalid memory addresses, but the biggest reason for crashing are exceptions thrown at runtime. Thus, we do failure-oblivious computing for uncaught and mishandled exceptions.










§ DESIGN OF 

This section presents the design of , including relevant definitions, algorithms and its architecture.



 §.§ Definitions




  
Exception All major programming languages provide a way to signal problems through so-called exceptions. In some statically typed languages such as  C# and Java, exceptions are typed, and some of them are statically verified at compile-time (e.g., checked exceptions in Java). For these checked exceptions, developers need to either handle them at the call site or explicitly declare them in the method signature (with the keyword throws in Java<cit.>)



  
Perturbation point
In this paper, a “perturbation point” is a unique location in code where a fault can be injected. In , the considered perturbations are injected exceptions, which means that a perturbation point is defined as a statement that potentially throws an exception. A perturbation point is noted <m, l, e>. m describes the method where this point is located. l is the line number before which the exception is thrown. e is the type of this exception.



  
Fault model In this paper, we consider two fault models: 1) injecting only one exception, when the perturbation point is reached for the first time and 2) always injecting exceptions when the perturbation point is reached.



  
Perturbation search space We define the “perturbation search space” as the Cartesian product of all possible perturbation points and all fault models with respect to a workload <cit.>. The size of the search space is the number of workload executions required to have an exhaustive picture of the behavior under perturbation.



  
Exception handling method
When an exception e is handled in a method, this method is called the “exception handling method” for e. In a tuple <exception source, exception type, method>, the source refers to the location where the exception was thrown.
In this paper, we make a distinction between “default exception handling methods” and “failure-oblivious method”:
the former refers to methods with manually written catch blocks while the latter refers to method with automatically instrumented catch blocks.



  
Acceptability oracle
An acceptability oracle is a mechanism for determining whether an application's behaviour remains acceptable under perturbation.

In order to evaluate and improve an application's resilience, we use oracles to describe acceptable behaviour, hence we call them “acceptability oracle”. In this paper, an acceptability oracle is a combination of generic oracles (like the absence of crash) and domain-specific ones. For example, in the context of a file downloading client, an acceptability oracle could be that 1) the client does not crash and exits normally 2) the client successfully downloads the file with a correct checksum.



  
Failure-oblivious method
A method fo is said to be failure-oblivious with respect to a perturbation point p if and only if: 
1) when an exception is thrown from p, it is possible to catch and stop its propagation in fo that is upper in the call stack;
2) the behaviour of the application verifies the acceptability oracle if the exception thrown from p is caught in fo. This is noted <m, l, e> ↦ fo (here ↦ denotes “thrown exception e in method m before location l and caught at fo”).



  
Fault injection experiment Given an application a and a workload w, injecting an exception during the execution of a under w is called an “experiment”.
is designed to conduct experiments in order to evaluate and improve an application's error-handling capability.

Example. Let us assume an invocation chain across three methods: m2 → m1 → m0. Method m0 can throw an IOException before line number l, and the developers write a try-catch block to handle it in m2 (the method upper in the stack). 
Consequently, m2 is the default exception handling method for this exception.
If the exception is caught and silenced in method m1 and the application behaviour is still acceptable according to the oracle, method m1 is considered as a failure-oblivious method for <m0, l, IOException>.



 §.§ Goals of 



aims at improving the exception-handling capabilities of Java applications. The main goals of are:
1) to give developers feedback about the effectiveness of their exception-handling design;
2) to automatically identify improvements of exception handling.

The former is about detecting the weakness points of the system under consideration and the latter is about finding new failure-oblivious methods that improve the application's resilience. 

Input to :
takes arbitrary software written in Java as input and a workload. No manual change is required from the developer. Neither source code nor test suite are required for improving an application's resilience.
also takes as input an acceptability oracle, which will be explained below in <ref>.

Output for the developer:
The output of is a report for developers. The report gives three pieces of information:
1) the perturbation points and their classification as defined next;
2) the verified failure-oblivious methods, i.e. the resilience improvements;
3) a log file which contains all the monitored information for the purpose of further analysis.

classifies the perturbation points into three categories as follows:

Fragile points:
A fragile point is a statement in a method before which injecting one exception results in the application crashing or freezing.


Sensitive points:
A sensitive point is a statement in a method before which injecting one single exception does not influence the application in the workload under consideration. But, continuously injecting exceptions results in the application to crash or freeze.


Immunized points:
An immunized point is a statement in a method before which no matter how many exceptions are injected, the application still behaves acceptably.




 §.§ Core Algorithm








The whole procedure of to identify failure-oblivious methods could be split into 3 steps:

1) Define acceptability oracles and detect perturbation points. executes the application normally, in order to monitor and record the application's normal behaviour. With this execution, the perturbation agent goes through all classes loaded into JVM and locates every  perturbation point, based on method signature, which is described in <ref>.

2) Classify all the perturbation points into fragile, sensitive or immunized ones (as defined in <ref>). For each perturbation point, conducts two experiments: only injecting one exception when the point is reached for the first time, and always injecting exceptions when the point is reached. Based on the observation of the application behaviour under perturbation, the perturbation point is classified using <ref>.

3) Identify candidate failure-oblivious methods and evaluate each of them as described in <ref>. detects candidate failure-oblivious methods with call stack analysis: every method in the stack before the default handling method is identified as a candidate failure-oblivious method. Then two fault injection experiments are conducted (only inject one exception, inject several exceptions). The difference is that all the thrown exceptions are caught in a catch block instrumented in candidate failure-oblivious methods. By analyzing the behavior once the exception is caught in this catch block, confirms whether the method under evaluation is indeed failure-oblivious or not.



 §.§ Architecture of 






<ref> shows the general architecture of . 
considers a Java application in a JVM, such as a backend web application or a Java micro-service.

When an application is loaded into the JVM, attaches to it three different agents: a monitoring agent, a perturbation agent and a failure-oblivious agent. The monitoring agent is responsible for collecting the information needed by to evaluate the system's resilience capabilities. The perturbation agent injects exceptions into the application in order to trigger its error-handling logic. The failure-oblivious agent tries to improve the application's resilience by catching and silencing exceptions before they are handled by default exception handling methods.

All the agents are controlled by a controller which makes two kinds of decisions:
1) given an application under some specific workload, which perturbation point should be activated,
2) whether the point's corresponding failure-oblivious method should be switched on.
Finally, the controller generates a report for the developer based on data gathered from a series of fault injection experiments.



  §.§.§ Monitoring agent

In order to study the influence of perturbations and evaluate all possible failure-oblivious methods in a software system, it is necessary to collect different kinds of monitoring information. For this, we propose to use a monitoring agent that is attached to the runtime process.

Our monitoring agent works as follows.
For each method in the code loaded in the JVM, the agent collects static and dynamic information.

The static information is:
1) its position in the code,
2) whether it declares checked exceptions to be thrown.

The collected dynamic information is:
3) the number of method executions over an fault injection experiment,
4) each time an exception is caught, the agent collects the stack information, including the stack distance between the method raising the exception and the method catching it. This includes both exceptions caught in default exception handlers and in failure-oblivious methods (as defined in <ref>).

The monitoring agent also collects the following information:


  * The set of classes that have been loaded into the JVM.

  * Whether the application has exited normally or crashed due to an unhandled exception.




  §.§.§ Perturbation agent

The perturbation agent injects specific perturbations at a specific point in time. The perturbation commands come from the agent controller.

The perturbation agent detects every method with a throws keyword and attaches itself into this method by rewriting the bytecode. In order to explore the entire perturbation search space, the agent injects different perturbation points before each statement in the method. In this way, the agent is able to throw such an exception anywhere in the method and compare the difference.

[caption=The Perturbation Strategy in ,label=lst:perturbationAgent]
// a perturbation point in Class1
void exampleMethod() throws ExceptionA, ExceptionB 
  // code injected with code transformation
  PAgent.throwExceptionPerturbation(key1);
  PAgent.throwExceptionPerturbation(key2);
  // a statement
  PAgent.throwExceptionPerturbation(key3);
  PAgent.throwExceptionPerturbation(key4);



<ref> gives an example of how this perturbation agent works. When a method like exampleMethod() throws multiple exceptions, corresponding perturbation points are automatically injected with code transformation. The perturbation agent controls every perturbation point separately. When a specific point is activated, it throws an exception at the beginning of the method.



  §.§.§ Failure-oblivious agent

The failure-oblivious agent instruments the code with try-catch blocks during a fault injection experiment.
For reasoning about resilience with respect to uncaught exceptions, the failure-oblivious agent injects a try-catch wrapper in all methods. Basically, the whole method body is wrapped with a try-catch block which handles all types of exceptions ( in Java).
By default, the catch block simply throws again the exception which makes it semantically equivalent to the original code.
When the failure-oblivious method is activated, the injected catch block silences the exception and prevents it from propagating (note that the exception may come from this method or from other methods transitively called from this method).

When an exception is caught by the injected catch block, there are three possible outcomes:
1) the application runs normally;
2) the application runs in a gracefully degraded mode;
3) the application crashes.

[caption=Automated Code Instrumentation for Identifying Failure-oblivious Methods in ,label=lst:foAgent]
// a candidate failure-oblivious method in Class2
void callExampleMethod() throws ExceptionA, ExceptionB 
  try 
    new Class1().exampleMethod();
   catch (Exception a) 
    if (!FOAgent.modeIsOn(key2)) 
      throw a;
     else  // nothing, the exception is silenced 
  





<ref> illustrates how this is done. In method callExampleMethod,  exampleMethod is invoked. The failure-oblivious agent detects it as a possible failure-oblivious method. So the whole method body of callExampleMethod is wrapped with a try catch block. When the agent controller activates this failure-oblivious method, it silences all exceptions coming from exampleMethod. Otherwise it throws the caught exception so that it is propagated as usual.



  §.§.§ Agents controller

The agent controller is responsible for conducting a series of experiments (see <ref>). It controls every agent and gathers all the information to analyze the system resilience. Additionally, the controller is configurable. For example, developers can define a filter to focus on resilience improvement for a specific package.



 §.§ Implementation

There are different kinds of agents in the JVM.
The monitoring agent is implemented on top of the JVM Tool Interface (JVMTI) [See <https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html>]. The perturbation agent and failure-oblivious agent are implemented as JVM agents, using the ASM library for binary code transformation [See <http://asm.ow2.org>]. The agents controller is a standalone service, it communicates with the JVM and the agents through local files.



For sake of open-science, the code is made publicly available at <http://bit.ly/tripleagent-repo>.










§ EVALUATION


For evaluating this contribution, we apply a case-based evaluation methodology: this methodology consists of an in-depth analysis of relevant cases selected in a principled way <cit.>. In our research domain, it has been shown appropriate in Rinard et al's original paper on failure-oblivious computing <cit.>. 

We select two case studies according to the following three criteria:
1) the case should be a real-world application (i.e., not a toy example)
2) it should be medium-sized in order to be appropriate for the computing power available in the laboratory
3) it is possible to define a production-like workload.
Those criteria yield two cases: TTorrent and HedWig. TTorrent is a file transferring tool which implements the BitTorrent protocol. HedWig is an email server for the IMAP, SMTP and POP3 protocols. They are also exemplary of applications with high resilience requirements: an email server must not crash, a file download on the dynamic internet must succeed regardless of unexpected network events, peer failures, or local machine issues.

The analysis of requires several executions. For each perturbation point, its classification requires 2 executions under the workload (as discussed in <ref>).
For each candidate failure-oblivious method, its evaluation also needs 2 executions  under the workload  (see <ref>). 
For both cases, an execution takes no more than 1 minute in our testing environment. In total, the cost of the  experiments presented in this section is upper-bounded by 2 × 1 × (1046 + 2844 + 372 + 722) = 9968 minutes. Note that some experiments lead applications to a crash. It actually takes around 3 days to finish all the experiments.




 §.§ Evaluation on TTorrent




  §.§.§ Experiment Protocol


We apply <ref> to TTorrent, version https://github.com/mpetazzoni/ttorrent/releases/tag/ttorrent-2.02.0.

The workload W for TTorrent consists of downloading a large file (https://cdimage.debian.org/debian-cd/current/amd64/bt-cd/debian-9.9.0-amd64-netinst.iso, a Debian distribution installer of 292.0MB).
Since BitTorrent is a peer-to-peer protocol, this workload involves other machines on the internet which serve (aka "seed") the file. To that extent, the workload is a production one.
We perform a series of fault injection experiments, as described in <ref>.

For TTorrent, we consider the following definition of acceptable behaviour to evaluate candidate failure-oblivious methods:
the behaviour is considered acceptable if an end-user can successfully download a file with a correct checksum.



  §.§.§ Experimental Results


Per <ref>, the first step of is to execute TTorrent normally and to monitor all possible perturbation points.
It detects 1046 points in total within the package .

Then, performs two series of experiments:
1) it injects one exception per perturbation point and compares the behaviour between these experiments and the normal execution,
2) it always injects exceptions when a perturbation point is reached and also compares the behaviour against the reference one.

As a result, all perturbation points get classified in the 3 categories defined in <ref>. In total, identifies  642 fragile points, 296 sensitive points and 108 immunized points. <ref> shows the distribution of these perturbation points, which are used as a base line for the following experiments.



The next step of 's main algorithm is to compute and assess the possible failure-oblivious methods.
As explained in <ref>, for a perturbation point, a set of failure-oblivious methods is identified. In our experiment, detects 2844 possible failure-oblivious methods, summed over all the perturbation points. The minimum, median and maximum number of candidate failure-oblivious methods per perturbation point is respectively 0, 2, 10.

Then, 2844 × 2 = 5688 executions are made to assess the failure-obliviousness of the candidate points (one per injection mode). Hopefully, the added catch blocks inserted by will increase the number of immunized points. 


Let us now consider <ref>.
The fragile, sensitive and immunized perturbation points are respectively shown in blue, orange and green. The area of bubbles corresponds to the numbers of perturbation points under consideration. For example, the bubble  represents the 155 sensitive points transformed into immunized ones with failure-oblivious computing.
Overall, successfully transforms 13 fragile points into sensitive ones, 70 fragile points into immunized ones and 155 sensitive points into immunized ones. The original 108 immunized points remain immunized. This means that resilience of the TTorrent has been automatically improved.



<ref> presents a sample of perturbation points. Every row describes 
1) a perturbation point (the class name, method name and its line number), the thrown exception type, and the corresponding default exception handler written by developers;
2) the failure-oblivious improvement (failure-oblivious method and concrete change of the perturbation point's category).
For example, row 1 and row 2 show that verifies failure-oblivious methods which improve the original fragile perturbation points into sensitive ones. Row 7 and row 8 also describe the case that is able to detect multiple failure-oblivious methods for the same perturbation point. For original immunized perturbation points, alternative failure-oblivious methods which provide the same resilience are verified as well, which is shown in the last two rows.





  §.§.§ Case Studies


In the following, we detail 3 case studies where the resilience is improved.





[caption=IllegalStateException in SharingPeer/send,label=lst:ttorrentCase1]
void send(PeerMsg m) throws IllegalStateException 
  if (this.isConnected()) 
    ...
   else 
    // perturbation point here
    unbind(true);
  
  ...





  
Failure-oblivious Method as Alternative to Normal Resilience

First, <ref> shows a failure-oblivious method with respect to exception . This method is executed only 1 time during normal download of the file. Under perturbation, identifies that if one single exception is thrown from this method, the application is still able to download the file correctly. By analyzing the stack, detects another two methods as candidate failure-oblivious methods:  and .

By activating a failure-oblivious try-catch block in these two methods (i.e. the method body is wrapped with a try-catch block which blocks the exception), TTorrent still succeeds in downloading the file. It means that successfully detects 2 alternative methods in the stack that provide the same resilience as the original manually-written catch block.


[caption=IOException in WorkingReceiver/processAndGetNext,label=lst:ttorrentCase2]
DataProcessor processAndGetNext(ByteChannel sc) throws IOException 
  ...
  // perturbation point here
  if (this.pstrLength > MAX_MESSAGE_SIZE) 
    return new ShutdownAndRemovePeerProcessor(...).processAndGetNext(socketChannel);
  





  
Improving Resilience under a High Number of Exceptions
<ref> shows method  in class . This method is executed 34304 times during the reference execution. If injects one single exception in this method when it is called for the first time, the application still downloads the file correctly. However, when the perturbation agent keeping injecting exceptions when downloading the file, the application gets stalled. 

After analyzing the call stack, detects 4 candidate failure-oblivious methods, namely , ,  and .
After two fault injection experiments per candidate failure-oblivious method, observes that the last three methods are failure-oblivious, the application downloads the file successfully, no matter how many exceptions are thrown in . 

In this case, succeeds in detecting 3 failure-oblivious methods that provide better resilience compared to the normal error-handling code written by the developer.


[caption=InvalidBEncodingException in TorrentParser/getStringOrNull,label=lst:ttorrentCase3]
String getStringOrNull(Map<...> m, String k) throws InvalidBEncodingException 
  // perturbation point here
  BEValue v = dictionaryMetadata.get(key);
  if (v == null) return null;
  return v.getString();





  
Improving Resilience from Crashing to Resilient
Let now us consider <ref>. With a fault injection experiment in method  before line 3, identifies that an exception  thrown at this location crashes the whole process. Hence, the perturbation point is a fragile one. After analyzing the stack information,  is the default handling method. There are 7 methods including  itself before this default handling method, which are all considered as candidate failure-oblivious methods by .

Then, performs 2 fault injection experiments for each method according to <ref>, that is 2 × 7 = 14 experiments in total. The first experiment assesses whether the candidate failure-oblivious methods could handle only one injected exception. The second assesses whether they could handle as many as injected exceptions.
Indeed, observes that when a catch block is automatically injected in , the application does not crash anymore, and even better, the resulting behaviour is correct (the file is correctly downloaded, its content is the expected one, bit-per-bit).
In this case, has automatically transformed a crashing exception into acceptable behaviour.

[style=mpdframe,frametitle=Insights from the TTorrent experiment]
Under a realistic workload of downloading a 200MB+ file from the internet, performs 7780 experiments to evaluate 1046 perturbation points spread over 6.5kLOC. 
identifies 642 fragile points, 296 sensitive points and 108 immunized points.
After analyzing all 2844 candidate failure-oblivious methods, confirms that there are 238 failure-oblivious methods in the application.
This shows that it is feasible to automatically improve resilience by combining perturbation injection and failure-obliviousness analysis.




 §.§ Evaluation on HedWig



  §.§.§ Experiment Protocol

HedWig is an email server written in Java, a typical server side application. The main process of HedWig is a perpetual loop, which creates sub-threads to handle different user requests. HedWig relies on a MySql database to store email metadata and saves the email contents as files on the disk. In this experiment, we consider the latest version of HedWig (https://sourceforge.net/projects/hwmail/files/0.7/v0.7).

The considered workload is as follows.
First, HedWig is deployed on a server. Then sends a specific email with a unique content to a testing email address using the SMTP protocol. Finally, logs in with the corresponding account and fetches the same email to do the comparison. The acceptable behaviour is that both successfully sends and fetches the email, and that the content of this email after final fetching is totally correct.

The experiments are performed sequentially. We note that some of the perturbation experiments crash the email server. In this case, the server is automatically restarted. 
Some perturbation experiments put it in a corrupted state: to detect this, adds a checking point after each experiment. All the perturbation agents are switched off and an email is sent and fetched as usual. If the server works correctly the next experiment goes on, otherwise runs a restart script to bring the server back to normal state.



  §.§.§ Experimental Results















Within the package  detects 372 perturbation points. Each perturbation point is evaluated by two fault injection experiments: 1) only one exception is injected during the email sending and fetching process, when the point is reached for the first time and 2) exceptions are always injected when the point is reached. Based on these 744 experiments classifies all the perturbation points using the classification algorithm described in <ref>. Overall, finds in Hedwig 264 fragile points, 14 sensitive points and 94 immunized points, which are shown in <ref>



The next step for is to identify the candidate failure-oblivious methods. By summing over all perturbation points, detects 722 candidate methods. The minimum, median, maximum number of candidate failure-oblivious methods per perturbation point is respectively 0, 2, 10.

Similar to classifying perturbation points, each candidate failure-oblivious method also needs two fault injection experiments to be evaluated. Finally, 1444 executions are made to evaluate all the candidate failure-oblivious methods based on <ref>. By silencing exceptions in the candidate failure-oblivious methods, shows that 23 fragile perturbation points can be improved into sensitive ones. 31 fragile points are transformed to immunized ones. It upgrades 1 sensitive perturbation point to an immunized one as well. All those improvements are shown in <ref>.



<ref> shows a sample of interesting perturbation points. It shows different levels of automatic resilience improvement. Similar to <ref>, every row describes one perturbation point and one of its corresponding failure-oblivious methods. For example, the first row gives details about perturbation point  in class , line number 100. When a  is thrown from this point, by default it is handled by a try-catch block written by developers in . But this catch block does not prevent the exception from failing user requests. If the same exception is caught earlier in the stack, in , the server is able to bear at least one exception. Note that it is possible to have multiple failure-oblivious methods for the same perturbation point. Such as row 3 and row 4, row 8 and row 9 in the table.





  §.§.§ Case Studies


We now discuss two interesting case studies.

[caption=SQLException in AnsiMailboxDao$1/mapRow,label=lst:hedwigCase1]
mailboxRowMapper = new RowMapper<Mailbox>() 
  public Mailbox mapRow(ResultSet r, int n) throws SQLException 
    // perturbation point here
    Mailbox mb = new Mailbox();
    mb.setMailboxID(r.getLong("mailboxid"));
    ...
    return mb;
  
;




  
A Failed Failure-oblivious Experiment
<ref> shows a perturbation point found by in Class , line 3. First, detects that when one  is thrown from this location, the application fails to receive and send the test email. Hence, the perturbation point is a fragile one. By analyzing the call stack, method  is considered as a candidate failure-oblivious method. 

automatically wraps the method with a try-catch block. This specific failure-oblivious operation results in inserting an incorrect record into the database, which influences the upcoming experiments. Thanks to running the checkpoint procedure described above, detects this problem, restarts the server, definitely labels this method as non failure-oblivious and excludes this perturbation point for later experiments.

[caption=IOException in CountingInputStream/read,label=lst:hedwigCase2]
public int read() throws IOException 
  // perturbation point here
  int next = in.read();
  ...
  return next;





  
A Perturbation Point with Multiple Failure-oblivious Methods
In <ref>, line 2 is a fragile perturbation point in Class . If an  is thrown from this location, the user is not able to fetch any emails. By default the exception is handled by , 5 methods upper in the stack. It means that methods before the default exception handler are all candidate failure-oblivious methods. evaluates them one by one and verifies 3 out of them. In the call stack, if the exception is silenced in ,  or , the server works properly no matter how many exceptions are thrown. This is a strong improvement to the resilience.

[style=mpdframe,frametitle=Insights from the HedWig experiment]
Under a production-like email task, performs 2188 experiments to evaluate 372 perturbation points spread over 13.8 kLOC. 261 fragile points, 68 sensitive points and 43 immunized points are identified in the original code. 
assesses that 60 out of 722 methods can be transformed into failure-oblivious methods.
This further confirms that can improve  the resilience of a server application in an automated manner.




 §.§ Overhead of 


The overhead of varies a lot among different perturbation points, failure-oblivious methods. Considering that the ultimate goal of is to automatically improve resilience, we manually evaluate the overhead of failure-oblivious experiments. The overhead caused by is evaluated in 3 aspects:
1) at the application level, the execution time is compared. In TTorrent this metric means the downloading time. In HedWig experiments this means the time spent on sending and receiving the email.
2) at the operating system level, the CPU and memory usage, peak thread count are taken into consideration.
3) at the binary code level, the code bloat due to instrumentation is evaluated.
For statistical purposes, we conduct the same measurement 30 times and calculate the average <cit.>.

For TTorrent experiments, <ref> records the overhead of verifying failure-oblivious method  in <ref>, row 5. For HedWig experiments, failure-oblivious method  in <ref>, row 3 is taken as an example. The overhead of execution time, CPU time, memory usage, peak thread count, relevant class files size are respectively 0%, 6.0%, 0%, 5.4%, 3.0%.















The reason why has such a low overhead is that the instrumentation is small. The perturbation agent and failure-oblivious agent only instrumented one or two class files. Meanwhile, the monitoring agent does not cause high overhead thanks to the JVMTI framework. By evaluating the overhead, developers are more confident about the resilience improvement suggested by .



§ DISCUSSION

Fault model. Currently considers two fault models. In both models, exceptions are injected at a single location. But there also exists common mode failures which involve a series of different exceptions. An exception could also be mixed with data errors. Devising and implementing fault models that stimulate common mode failures or data errors is an interesting direction for future work.

Workload impact. A threat to the validity of our experiments comes from the workload. takes a production-like workload to exercise the application code. When identifies a failure-oblivious method, it guarantees that it works under the tested workload.
But it may break some behavior with a more comprehensive workload.
Overall, the more diverse the workload is, the more confidence has in the found failure-oblivious methods.

Scalability. During our experiments, the deepest call stack considered was composed of 39 methods. has not been tested with larger applications with deeper stacks.
As such, the full scalability of is not yet verified. 
We note that the number of candidate failure-oblivious methods to assess is linear in the depth of the stack, which means that, in theory, would be scalable.



§ RELATED WORK

Now we discuss the related work along three aspects.



  §.§.§ Fault injection

Fault injection is a widely-researched topic in software dependability. In the 1990s, the research was mostly about hardware implemented fault injection tools. For example, Madeira et al. <cit.> invented RIFLE, a pin-level fault injector to generate processor errors. Next, more software-based fault injection tools were invented. 
Kanawati et al. <cit.> proposed FERRARI, a tool for the emulation of hardware faults and control flow errors. Han et al. <cit.> designed DOCTOR, a tool for injecting hardware failures and network communication failures.
Wei et al. <cit.> built a software-based hardware faults injector called LLFI, and quantitatively compared the accuracy of fault-injection with assembly code level injector PINFI.
Lee et al. <cit.> presented SFIDA, a tool to test the dependability of distributed applications on the Linux platform.
Kao et al.<cit.> invented “FINE”, a fault injection and monitoring tool to inject both hardware-induced software errors and software faults. Kouwe and Tanenbaum<cit.> presented HSFI, a fault injection tool that injects faults with context information from  source code and applies fault injection decisions efficiently on the binary.

Fu et al.<cit.> presented an approach to measure the coverage of recovery code with respect to operating system and I/O hardware faults. The common idea with is to inject exceptions to trigger error handling code. Yet, our and their goal are notably different. Fu el al. use fault injection to increase recovery code coverage. combines fault injection with failure-oblivious computing to improve resilience.

The novelty of is that it is designed to inject application-level exceptions (and not hardware faults) in Java applications. gives developers concrete insights at the source code level about their exception-handling implementation.



  §.§.§ Self-healing software

Self-healing software follows the idea that it is possible to automatically make software recover from failures. Different techniques have been applied to achieve this goal, such as automatic reboot, checkpoint-restart, and failure-oblivious transformation.

Reboot techniques <cit.> require the system to be able to restart, which may bring some down-time.
Checkpoint-restart techniques significantly reduce the recovery time by saving and reloading runtime states saved at checkpoints. Qin et al. <cit.> invented Rx, which enables the program to rollback to a recent checkpoint upon a software failure, and then to re-execute in a modified environment.
Sidiroglou et al. <cit.> proposed ASSURE, a system that introduces rescue points to recover from unknown faults.

Regarding failure-oblivious computing,
Rinard et al. <cit.> invented a safe compiler for C to enable servers to execute despite memory errors.
Perkins et al. <cit.> proposed ClearView, a system for automatically patching errors in deployed software. It observes values of registers and memory locations and tries to detect violations of invariants at this level.
Rigger et al. <cit.> presented an approach that allows C programmers to perform explicit sanity checks and to react according to invalid arguments or states. They also designed a C dialect called Lenient C <cit.> that checks undefined behaviours in the C standard including memory management, pointer operations and arithmetic operations.

None of these tools combine fault injection and failure-oblivious computing together as we do in . They do not actively inject failures into the system, nor do they conduct application-level analysis to detect valuable failure-oblivious positions.



  §.§.§ Exception analysis

Byeong-Mo et al. <cit.> gave a comprehensive review on exception analysis.
Magiel Bruntink et al. <cit.> proposed a characterization and evaluation method to statically discover faults in  exception handling.
Fu and Ryder<cit.> described a static analysis method for exception chains in Java.
Martins et al. <cit.> presented VerifyEx to test Java exceptions by inserting exceptions at the beginning of try blocks.
Zhang and Elbaum<cit.> presented an approach that amplifies tests to validate exception handling.
Cornu et al.<cit.> proposed a classification of try-catch blocks at testing time.

Those tools rely on test suites to analyze resilience with respect to error-handling. On the contrary, analyzes the system behaviour based on user-level traffic and usages.



§ CONCLUSION

In this paper, we have presented , a system which combines automated monitoring, automated perturbation injection and automated resilience improvement. By evaluating on two real-world Java applications, we have shown that it is able to detect weaknesses in exception-handling of Java code and to improve resilience.

In the future, we will further explore the design space of perturbation and failure-obliviousness strategies. For instance, we would like to inject timeout on requests and interactions in asynchronous software.
Our long-term goal is to use in production, and consequently, we will also keep reducing the overhead of at runtime.



§ ACKNOWLEDGEMENTS

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation and SSF project TrustFull.

plain


