FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks
Kai Zhao Sheng Di Sihuan Li Xin Liang Yujia Zhai Jieyang Chen Kaiming Ouyang Franck Cappello Zizhong Chen
cs.DC cs.LG
































































































Convolutional neural networks (CNNs) are becoming more and more important for solving challenging and critical problems in many fields. CNN inference applications have been deployed in safety-critical systems, which may suffer from soft errors caused by high-energy particles, high temperature, or abnormal voltage. 
Of critical importance is ensuring the stability of the CNN inference process against soft errors. Traditional fault tolerance methods are not suitable for CNN inference because error-correcting code is unable to protect computational components, instruction duplication  techniques incur high overhead, and existing algorithm-based fault tolerance (ABFT) techniques cannot protect all convolution implementations. 
In this paper, we focus on how to protect the CNN inference process against soft errors as efficiently as possible, with the following three contributions. (1) We propose several systematic ABFT schemes based on checksum techniques and  analyze their fault protection ability and runtime thoroughly.
Unlike traditional ABFT based on matrix-matrix multiplication, our schemes support any convolution implementations. (2) We design a novel workflow integrating all the proposed schemes to obtain a high detection/correction ability with limited total runtime overhead. (3) We perform our evaluation using ImageNet with well-known CNN models including AlexNet, VGG-19, ResNet-18, and YOLOv2. Experimental results demonstrate that our implementation can handle soft errors with very limited runtime overhead 
(4%∼8% in both error-free and error-injected situations).  



Algorithm-Based Fault Tolerance, Deep Learning, Silent Data Corruption, Reliability, High-Performance Computing




FT-CNN: Algorithm-Based Fault Tolerance for Convolutional Neural Networks
    Kai Zhao,
        Sheng Di, Senior, IEEE,
        Sihuan Li,
        Xin Liang,
        Yujia Zhai,
        Jieyang Chen,
        Kaiming Ouyang,
        Franck Cappello, Fellow, IEEE,
        and Zizhong Chen, Senior, IEEE

Kai Zhao, Sihuan Li, Yujia Zhai,and Zizhong Chen are with the Department of Computer Science and Engineering at University of California, Riverside, CA 92521.
Sheng Di and Franck Cappello are with the Mathematics and Computer Science division at Argonne National Laboratory, Lemont, IL 60439.
Xin Liang and Jieyang Chen are with the Computer Science and Mathematics Division at Oak Ridge National Laboratory, Oak Ridge, TN 37831.

    December 30, 2023
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================























































§ INTRODUCTION

Deep learning using convolutional neural networks (CNNs) is becoming the key state-of-the-art technique in science and technology fields such as image classification<cit.>, object detection<cit.>, natural language processing<cit.>, medical image analysis<cit.>, and drug design<cit.>. More and more scientific research (such as cosmological simulation and materials analysis) also is addressing the great potential of leveraging CNN techniques to analyze extremely large amounts of data in a supercomputer environment, achieving unprecedented discoveries in their domains <cit.>.

The reliability of the CNN inference is becoming a critical concern <cit.> because CNN inference applications are being widely utilized in different scenarios, including high-performance scientific simulations and safety-critical systems <cit.> such as aerospace and autonomous vehicles. CNN inference applications usually run for a long time or continuously to process many inference tasks. For example, the inference engine in autonomous vehicles is running continuously to predict road conditions. As a result, even a single inference task for one input finishes in seconds, the reliability of CNN inference is still critically important given the long execution time of the inference applications. 

In the domain of CNN inference, machine learning applications could be very error prone because of two reasons. 
On the one hand, recent literature indicates that soft errors are inevitable in modern  systems, from edge computing devices to supercomputers<cit.>, because of multiple factors <cit.> such as high-energy cosmic radiation <cit.>, aging, and wear of devices <cit.>. On the other hand, CNN inference applications often call for power-efficient and cost-efficient machine learning accelerators, which <cit.> may adopt overclocking with voltage underscaling, incurring more soft errors than common hardware incurs.




Soft errors may cause serious consequences to CNN inference systems.
Recent studies <cit.> indicate that resilient convolutional neural networks are essential for guaranteeing the correctness of inference applications. 
Researchers <cit.> demonstrate that a single bit flip happened during CNN image classification could result in as much as 40% and 70% SDC rate in datapath and memory, respectively. Such high SDC rates would downgrade the CNN prediction accuracy dramatically.  
Furthermore, the neutron beam test <cit.> shows that when running YOLO<cit.> classification, the Failure In Time (FIT) caused by SDCs could be as much as 38 for Nvidia K40 and 96 for Nvidia Tegra X1, which fail to meet the ISO 26262 standard for functional safety of road vehicles <cit.>. 









Existing resilient solutions are insufficient for protecting  CNN inference applications against these soft errors. Error-correcting code (ECC), for example, suffers from memory area cost and relatively high latency and power consumption. According to <cit.>, ECC with chip-kill applied to all data, compared with no ECC protection, has an average of 40% overhead in memory energy, 20% overhead in system energy and 20% overhead in performance for computation-bounded applications. Moreover, ECC cannot handle multiple bit flips or computational errors. 

Techniques based on instruction duplication (ID)  <cit.> incur high overhead and require both application-specific and hardware-specific optimization; and optimizing and deploying  ID techniques on all CNN accelerators is difficult.



Considering all the drawbacks and limitations of ECC and ID, algorithm-based fault tolerance (ABFT) <cit.> is an attractive solution to realize resilient CNN. It has much lower overhead than other techniques have; and it is architecture independent, meaning that it supports any hardware accelerator. The idea of ABFT is to detect and/or correct soft errors based on the known invariants that the algorithm has. Over the past thirty years, ABFT schemes have been successful in  detecting errors for matrix operations <cit.>, iterative methods <cit.>, data transformation kernels<cit.> and sorting algorithm <cit.>. 
However, the existing ABFT schemes for matrix operations focus mainly on large and square matrices. Moreover, they incur more than 50% overhead when applied for CNN soft error protection (shown  by our experiments in Section <ref> ).




In this paper, we propose a strategy comprising a series of ABFT schemes for protecting the CNN inference stage against soft errors. We focus on the convolutional layers in CNN because they consume the major portion of the computation time <cit.>.

The main contributions of this paper are summarized as follows.

  
  * We design several ABFT schemes that can be applied to any convolution implementation on any hardware. They can detect and correct errors at runtime. We provide an in-depth analysis of the ABFT schemes in terms of fault protection ability and runtime.
  
  * We design a multischeme workflow for soft error protection with layerwise optimization to obtain a high detection/correction ability with limited runtime overhead. Additionally, our solution can protect the bias operation, grouped convolution, and back propagation.
  
  * We implement an efficient soft error detection library for CNN, called FT-Caffe, and evaluate FT-Caffe on ImageNet <cit.> using four popular CNN models: Alexnet <cit.>, VGG-19 <cit.>, ResNet-18 <cit.>, and YOLOv2 <cit.>. Experimental results on the Bebop supercomputer <cit.> using up to 128 nodes demonstrate that  FT-Caffe can keep the correctness of the inferences with 4%∼8% overhead in both error-free and erroneous cases. 
 

In the rest of the paper, we first introduce background about convolutional layers and existing ABFT techniques applicable to matrix-matrix multiplication (MM)-based convolution implementation. In Section <ref>, we propose four novel ABFT schemes that can be applied to any convolution implementations. In Section <ref>, we analyze the fault protection ability and runtime of the four schemes and propose an efficient multischeme workflow integrating all the four schemes. In Section <ref>, we discuss how to support bias, grouped convolution, and back propagation. In Section <ref>, we evaluate our solutions for both error-free case and erroneous case. In Section <ref>, we discuss related work on fault tolerance in convolutional neural networks. We present our concluding remarks in Section  <ref>.


§ BACKGROUND

This section introduces some high-level ideas of convolutional layers and the existing ABFT techniques to MM-based convolution algorithms. The notations and symbols used in this paper are summarized in Table <ref>.



 §.§ Definition of Convolutional Layer

The convolutional layer can be represented as the following convolution operation.



    [                                            O[n][m][x][y]=B[m]+; ∑^Ch-1_k=0∑^R-1_i=0∑^R-1_j=0D[n][k][Ux+i][Uy+j] ×W[m][k][i][j];                              0≤ n< N,0≤ m<M,0≤ x,y<E,E=H-R+U/U ]


The convolution operation involves two significant inputs: the feature map (fmap) D, D∈ℝ^N × Ch× H × H, and the convolutional kernels W, W∈ℝ^M× Ch × R × R. Note that all the matrices and vectors in this paper are highlighted in bold in order to differentiate from the scalar numbers, according to the naming convention. The bias, denoted as B, is applied to the output after convolution, and the final result is denoted as O, O∈ℝ^N× M × E × E. Since the bias operation is independent of the convolution computation, in the rest of this section we describe only the protection for convolution computation. In Section <ref>, we will discuss the protection for bias.







 §.§ Implementation of Convolutional Layer


 Convolution can be implemented efficiently in several ways <cit.>. The first option is MM-based convolution <cit.>, which reshapes the kernel and feature map to two temporary matrices and then applies matrix-matrix multiplication (MM) on them.
 

 Another way to implement convolution is called direct convolution, which performs the convolution operation directly. It is widely used in AI accelerators including Eyeriss<cit.>, DianNao<cit.> and NVIDIA Deep Learning Accelerator<cit.>.

Fast Fourier transform–based convolution <cit.> leverages FFT to compute the convolution. It is particularly suitable for the relatively large feature map and kernel. However, it is inferior to the Winograd convolution <cit.> when the sizes of the feature map and kernel are relatively small.





















Modern CNN frameworks and accelerators generally automatically choose the best implementations of convolution based on hardware resources and model structure, because various implementations have different constraints on memory, architecture, and CNN model. 




 §.§ ABFT for Matrix-Matrix Multiplication



Traditional ABFT designed for matrix-matrix multiplication can be applied to the MM calculation of the MM-based convolution <cit.>, but it has at least three limitations. (1) It  supports only MM-based convolution implementation, which is not always the best-fit implementation selected by the CNN framework and accelerator. (2) It incurs high overhead (more than 50%, as shown in Section <ref>), due to the small and irregular shape of the matrices used by MM-based convolution. (3) Moreover, it cannot cover the reorganization operations of feature before the MM calculation. Therefore,  new ABFT schemes are needed in order to protect the convolutional layer more effectively. 
























§ NOVEL ABFT SCHEMES FOR CONVOLUTION




In this section, we present four novel ABFT schemes, each supporting any convolution implementation and being able to protect the whole convolution process. In Section <ref>, we propose a multischeme workflow using all the schemes in different stages to maximize the soft error protection ability with minimized performance overhead. 




 §.§ Preliminary Analysis – Convolution

For clear description, we interpret convolution at the block level. 
Specifically, in Equation (<ref>), D, W, and O are all 4D matrices. They can be represented as being composed of multiple blocks as shown in Figure <ref>.


For any n and m (0≤ n<N,0≤ m<M), D_n, W_m, and O_nm are blocks. The dimension of D_n, W_m, and O_nm are Ch× H × H, Ch× R × R and E× E, respectively.
The notation ⊗ is used to represent the convolution computation between blocks D_n and W_m.

The convolution operation defined by Equation (<ref>) can be simplified at the block level as follows. 











    [  O_nm=D_n⊗W_m; 0≤ n<N,0≤ m<M ]






Equation (<ref>) can be interpreted by using the blue part in Figure <ref>. Since each of the 3D substructures of D and W is treated as a block, D and W can be thought of as two 1D vectors of blocks. At the block level, the convolution operation is similar to matrix-matrix multiplication. The element (i,j) of O is calculated by using the ith element of D and the jth element of W. As illustrated in Figure <ref>, the elements involved in the convolutional layers can be split into two levels, which are covered by our protection solution, respectively.   


We can derive that the convolution operation (denoted by ⊗) has a distributive property as follows.

    [                                                D_1 ⊗W_1 + D_2 ⊗W_1=;         ∑^Ch-1_k=0∑^R-1_i=0∑^R-1_j=0D_1[k][Ux+i][Uy+j]×W_1[k][i][j];       + ∑^Ch-1_k=0∑^R-1_i=0∑^R-1_j=0D_2[k][Ux+i][Uy+j]×W_1[k][i][j]; =∑^Ch-1_k=0∑^R-1_i=0∑^R-1_j=0(D_1+D_2)[k][Ux+i][Uy+j] ×W_1[k][i][j];                                                     =(D_1+D_2) ⊗W_1 ]

Similarly, we can get the following equation.

    D_1 ⊗W_1 + D_1 ⊗W_2 =D_1 ⊗ (W_1 + W_2)

The distributive property, Formula (<ref>), is the key to proving the equivalence between the sum of the output and the output checksum. This property will be used later to prove the correctness of our design. 



 §.§ Preliminary Analysis – CNN Checksums


In general, we compute checksums for D and W and then use them to derive the checksums for O. Soft errors can be detected and corrected by comparing O with its checksums. 

We introduce all the checksums (as shown in Table <ref> and as
yellow blocks in Figure <ref>)  that are necessary for our ABFT schemes.



We define the checksums of D and W as follows.

    [   C_d1=∑^N-1_n=0D_n; C_d2=∑^N-1_n=0n D_n;   C_w1=∑^M-1_m=0W_m; C_w2=∑^M-1_m=0m W_m;                     ]


The four checksums (denoted as input checksums) can be treated as four blocks of D and W. The checksums of O (denoted as output checksums) are defined as the convolution result of input checksums and/or inputs.

    [    C_o1 = C_d1⊗W;    C_o2 = D⊗C_w1;    C_o3 = C_d2⊗W;    C_o4 = D⊗C_w2; C_o5 = C_d1⊗C_w1; C_o6 = C_d1⊗C_w2; C_o7 = C_d2⊗C_w1;                  ]


The output O is represented in the form of blocks (i.e., Level 1  in Figure <ref>). Elements inside the same block are independent with respect to checksums ( Level 2 in Figure <ref>). That is, we perform the checksum comparison independently for each element across blocks. Therefore, multiple soft errors in the same block can be detected and corrected independently.

In what follows, we describe the four schemes we proposed, each involving one or more input and output checksums. The required checksums used by each scheme are summarized in Table <ref>. 



 §.§ Full Checksum  Scheme (FC)



The first scheme we designed is called full checksum scheme, or FC, because it is based on checksums from both D and W, as shown in Figure <ref> and Table <ref>.  












C_d1 and C_w1 are calculated before the convolution operation, so any memory error striking D or W during the convolution would not affect C_d1 or C_w1. As for the output checksums, we can get the following equations by applying the distributive property of ⊗.

    [  C_o1[m]=  (∑^N-1_n=0D_n) ⊗W_m=∑^N-1_n=0 (D_n⊗W_m)=∑^N-1_n=0O_nm; C_o2[n]  = D_n⊗ (∑^M-1_m=0W_m)=∑^M-1_m=0 (D_n⊗W_m)=∑^M-1_m=0O_nm ]


These  equations show the equality between the sum of output and the output checksums. Let S_o1 and S_o2 be the summation of the output, where S_o1[m]=∑^N-1_n=0O_nm , S_o2[n]=∑^M-1_m=0O_nm. We can compare C_o1, C_o2 with S_o1, S_o2 to detect, locate, and correct soft errors if they exist. 



 §.§ Row Checksum  Scheme (RC)


Compared with the full checksum scheme, the second ABFT scheme we designed involves only the row checksums of output O, so we call it row checksum scheme.

The row checksums used in this scheme are C_o1 and C_o3. C_o3 is computed from convolution operation between C_d2 and W, and the related output summation is defined by S_o3[m]=∑^N-1_n=0 n×O_nm. 

For the detection of soft errors, we need to compare C_o1 with S_o1. If they are not equal to each other at location j, 

the error can be located by i=C_o3[j]-S_o3[j]/C_o1[j]-S_o1[j] and j, and it can be corrected by adding C_o1[j]-S_o1[j] to the block (i,j).





 §.§ Column Checksum  Scheme (ClC) 

The third scheme we proposed is called column checksum scheme because it involves only the column checksums of output O.
The column checksums used in this scheme are C_o2 and C_o4. C_o4 is defined by performing convolution operation between D and C_w2,  and the related output summation is defined as S_o4[n]=∑^M-1_m=0 m×O_nm. 
To detect soft errors, we compare C_o2 with S_o2 first. If they are not equal to each other at location i, the error can be located by i and j (=C_o4[i]-S_o4[i]/C_o2[i]-S_o2[i]), and it can be recovered by adding C_o2[i]-S_o2[i] to the block (i,j).



 §.§ Checksum-of-Checksum Scheme (CoC/CoC-D)



Unlike the three schemes that all need D and/or W to calculate output checksums, the last scheme we proposed involves neither D nor W but only their checksums, so it is named  checksum-of-checksum scheme (or CoC scheme for short). Specifically, C_o5, C_o6, and C_o7 are the output checksums we will use in this scheme.
Similar to C_o1, using the  distributive property can get three equations between the output checksums and output as follows.

    C_o5 =∑_n=0^N-1∑_m=0^M-1O_nm =S_o5


    C_o6 =∑_n=0^N-1∑_m=0^M-1 m ×O_nm =S_o6
  

    C_o7 =∑_n=0^N-1∑_m=0^M-1 n ×O_nm =S_o7

  

S_o5, S_o6, and S_o7 are defined as output summations corresponding to C_o5, C_o6, and C_o7. Let O(i,j) be the corrupted output block, O' be the correct output, and let δ = O'_ij - O_ij be the difference. Using the output checksums, we can get the following.




    C_o5 -S_o5 =∑_n=0^N-1∑_m=0^M-1O'_nm-O_nm = δ


    C_o6 -S_o6 =∑_n=0^N-1∑_m=0^M-1 m× (O'_nm-O_nm) = j×δ


    C_o7 -S_o7 =∑_n=0^N-1∑_m=0^M-1 n×(O'_nm-O_nm) = i×δ


The location i, j can be obtained by i=(C_o7 -S_o7)/ δ and j=(C_o6 -S_o6) / δ. Then the soft error can be fixed by adding δ to O_ij.

If only soft error detection is required, we do not need to compute C_o6 and C_o7, thus reducing the number of computations.
Input checksums     regarding C_d1, C_d2 and C_w1, C_w2, however,
are still required for soft error detection. We denote such a detection scheme by CoC-D.



§ MULTISCHEME WORKFLOW


In this section, we first discuss the fault protection abilities and runtime of the four schemes we proposed in Section <ref>. Then, we propose a multischeme workflow, powered by calibrated arrangement of the four schemes and layerwise optimization.



 §.§ Analysis of Protection Ability for Convolution Checksum Schemes

In this section, we analyze the fault protection ability of all the schemes. 




  §.§.§ Fault Model

The fault model for soft errors that we discuss in this paper includes transient faults in computational units and data corruption faults (both transient and persistent) in memory (including cache). In the following text, we use fault to represent a malfunction event, and we denote its corresponding symptom  as soft error. 

Soft error protection  includes error detection and error correction.  Error detection means that the scheme can detect soft errors without knowing the exact location. Error correction means that the scheme can locate the soft error locations and recover the incorrect result.

Without loss of generality, in the following analysis we consider at most one fault per convolution. One convolutional neural network contains several or even tens of convolutional layers, and the total forward execution time of a CNN model is usually within seconds. Thus, we can reasonably assume that at most one fault may strike to one convolutional layer, considering the short executing time of a single layer. Multiple faults per convolution can also be detected by our schemes and recovered by recomputing the corrupted convolutional layer.



  §.§.§ Analysis of Soft Error in D and W




One fault occurring during the convolution execution can result in multiple soft errors in W and D. The soft errors in W can be detected by comparing the checksum of W with C_w1 and corrected by reloading weights from the CNN model. The soft errors in D do not need correction because D will be discarded after convolution computation; the resulting errors in the output can be detected and corrected by the checksums of the output, as demonstrated below.








  §.§.§ Analysis of Soft Error in O

One fault during the convolution execution can result in corruption of one block row or column of O. 
By definition, the row i of O is computed by the ith block of D with W. Thus, one fault in D would result in at most one corrupted row. The column j of O is computed by D with the jth block of W. Thus, one fault in W would result in at most one corrupted column. Moreover, the intermediate result will be reused only by the same row or column, such that one fault in the computational units would  corrupt only values in the same row or column. Accordingly, in the following sections we discuss the soft error protection ability in the context of at most one corrupted row or column of O.



  §.§.§ Soft Error Protection Ability of CoC Scheme


Figure <ref> demonstrates the protection ability of the CoC scheme when soft errors strike the input or output data.
As shown in Figure <ref>(a), multiple soft errors can be detected by using only C_o5. A single soft error in O can be corrected by CoC using all checksums including C_o5, C_o6, and C_o7, as shown in Figure <ref> (b). However, CoC cannot correct soft errors across multiple blocks in O. 




















Figure <ref> illustrates the protection ability of the CoC scheme when soft errors happen inside the checksums. Such soft errors can cause inconsistency among the output checksums of CoC, which can be used for error detection. For example, in Figure <ref>(a), C_d1 is corrupted, leading to corrupted C_o5 and C_o6 with correct C_o7. We can detect this abnormal pattern when comparing checksums with the summation of O to detect the input checksum corruption. The input D, W, and output O are clean and without soft errors since fault frequency is at most once per convolution. Thus, we can safely discard all the checksums and finish this convolution computation.






  §.§.§ Soft Error Protection Ability of Row Checksum Scheme and Column Checksum Scheme




Since the row checksum scheme and column checksum scheme are symmetric with each other, we discuss them together in this section. As shown in Figure <ref>(a), the row checksum scheme can detect and correct soft errors if they are in the same row. If the soft errors are in the same column, as shown in Figure <ref>(b), the row checksum scheme can only detect soft errors; it has no correction ability. The column checksum scheme, on the contrary, can detect and correct errors located in the same column but fail to correct those appearing in the same row.






  §.§.§ Soft Error Protection Ability of Full Checksum Scheme



The full checksum scheme has the highest ability to correct soft errors. The scheme uses both the row checksum C_o1 and column checksum C_o2 so that it can correct soft errors in both directions, as shown in Figure <ref>(a)(b). 
If soft errors exist in C_o1 (Figure <ref>(d)), however, C_o1 can no longer be used to locate or correct soft errors. To support error correction in this situation, we use checksum C_o5 and C_o6 from the CoC scheme to locate the corrupted column, and we then use C_o2 to correct the soft errors. 
If soft errors exist in C_o2 (Figure <ref>(c)), C_o5 and C_o7 are used to locate the corrupted row, and C_o1 is used to correct the soft errors.









  §.§.§ Conclusion
 In this section, we define our fault model and analyze the soft error protection ability of four schemes. We conclude that the CoC scheme has the lowest error correction ability and that the full checksum scheme has the best error correction ability. The abilities of the row checksum scheme and column checksum scheme are higher than that of the CoC scheme but lower than that of the full checksum scheme. CoC-D (discussed in Section <ref>) can detect multiple soft errors but without correction ability. The analysis here serves as the fundamental basis of our low-overhead high-protection design, which will be  presented in Section <ref>.



 §.§ Runtime Analysis


In this section, we analyze the time complexity theoretically and present runtimes of all schemes based on experiments.

Table <ref> shows the time complexity of some basic checksum operations, where α is the coefficient of CPU-intensive operations and β represents the coefficient for memory-intensive operations. 





Table <ref> shows the theoretical time complexity of all the schemes. The full checksum scheme has the best soft error correction ability; however, its runtime is relatively long. Although the CoC scheme has lower ability than the  other three schemes in correcting soft errors, it has the shortest runtime. Note that the kernel checksum C_w1 and C_w2 can be precalculated before the application; there is no cost in generating kernel checksum in the row, column, and CoC schemes.


To verify the correctness of the derived time complexity of the four schemes, we execute them on a supercomputer using four CNN models. We show the normalized worst-case runtime of the four schemes in the separate column of Figure <ref>. Other columns of this Figure represent the worst-case runtime of multischeme workflows and will be discussed in the next section.
Experiments confirm our conclusion that CoC and CoC-D have the shortest runtime and that the runtime of the full checksum scheme is relatively long. We also see that the column checksum scheme has a much longer runtime than the row checksum scheme does. The reason is twofold. On the one hand, W blocks have smaller sizes than D blocks have, leading to longer time to compute D⊗C_w2  by the column checksum scheme  than to compute C_d2⊗W  by the row checksum scheme. 

On the other hand, computing row checksums (C_o1 and C_o3) is more efficient than computing column checksums (C_o2 and C_o4), because the row checksum calculation can be reduced to efficient column-summation operations.








 §.§ Multischeme Workflow for Soft Error Protection


The four schemes we proposed have pros and cons in terms of their soft error correction ability and runtime overhead. To achieve the highest protection ability and lowest overhead, we propose a multischeme workflow by integrating the four schemes, as shown in Figure <ref>. The workflow is made up of two modules: error detection and error correction. In our designed workflow, we use CoC-D to detect errors because it has the lowest overhead. For the error correction, we put CoC in the beginning because it is the most lightweight method. By comparison, FC has highest correction ability but also highest time overhead, so we put it at the end of the workflow.



The error detection modules will be executed for every execution whether there is a soft error or not. Thus, any unnecessary computations should be avoided in order to reduce the overall overhead. For instance, both CoC-D and FC are able to detect all the soft errors, but we adopt only CoC-D in the workflow for error detection because FC has a much higher overhead. RC and ClC cannot detect soft errors correctly if the checksum is corrupted.

The error correction module will not be executed until some soft errors are detected. The schemes in this module will be invoked to fix soft errors according to the workflow. If it fails to correct the errors due to inconsistency of checksum blocks or illegal error locations, the next-level scheme will be invoked. 

Since the checksums can be reused among different CNN schemes in the workflow, the runtime of the workflow is actually lower than the sum of all schemes' runtimes. For example, both CoC-D and CoC use C_o5; if CoC-D detects soft errors and CoC is invoked to correct soft errors, CoC can save the time of computing C_o5 and its corresponding summation S_o5, since they have been computed by CoC-D. This analysis can also be confirmed by our experiments. As shown in Figure <ref>, the relative runtime of CoC in the second column is reduced compared with that of CoC in the first column. The relative runtime of RC in the third column is reduced compared with that of RC in the first column.  

The decision to put RC and ClC in the workflow between CoC and FC is controlled by each layer.  
The reason to control RC/ClC in each layer is that their relative runtimes differ across layers.
Since RC and ClC are symmetric based, in the following we present our analysis based mainly on RC, without loss of generality.

We denote the runtime of the workflow CoC+FC as t_0, the runtime of workflow CoC+RC as t_1, and the runtime of workflow CoC+RC+FC as t_2. Enabling RC can fix some soft errors before FC, thus changing the runtime from t_0 to t_1.  When RC fails to correct soft errors, however, FC still needs to be invoked; and the runtime will increase from t_0 to t_2. Denoting the probability of row soft errors by p_r and the probability of column soft errors by p_c, we can derive the average time saved by RC  as t_y=p_r(t_0-t_1) and the average time increase by RC  as t_n=p_c(t_2-t_0). In order to minimize the total runtime, RC should be enabled when t_y > t_n. 

We give an example to further illustrate when RC should be enabled. Figure <ref>(b) shows the average runtime among all the convolutional layers in YOLOv2. In this figure, the runtime of CoC+RC is much lower than that of CoC+FC, and the runtime of CoC+RC+FC is slightly higher than that of  CoC+FC. Therefore,  enabling RC  can save significant runtime when the soft errors are able to be corrected by RC. On the other hand,  a bit runtime penalty is incurred if RC fails to correct the soft errors. However, for the conv8 layer in YOLOv2 (shown in Figure <ref>(c)), CoC+RC's runtime is close to that of CoC+FC. Thus, enabling RC in this layer would barely reduce the overall runtime even though the soft errors can be corrected by RC. Moreover, CoC+RC+FC's runtime is much higher than CoC+RC's. As a result, the total runtime will increase significantly if the soft errors cannot be corrected by RC. Hence, for this layer, it is better to use CoC+FC for error correction with RC disabled. 

In practice, the runtime t_0, t_1 and t_2 can be computed by offline profiling. The probability values p_c and p_r can be estimated based on the size of D and size of W. For instance, the soft error often strikes each element in the input under the independent and identical distribution. In this situation, it is easy to drive that the probability of soft errors occurring in D is proportional to that of W (i.e., p_r/p_c = number of elements inD/number of elements inW. 




§  RESOLVING BIAS, GROUPED CONVOLUTION, AND BACK PROPAGATION



In this section, we extend our solution to support bias, grouped convolution, and the back propagation of convolutional layers.



 §.§ Bias
 
Bias is a 1D vector that needs be added to the output of the convolutional layers.  FT-Caffe provides protection for the bias operation. 

Many CNN frameworks add bias on the fly with the convolution calculation. As a result, the output O already contains bias, whereas the output checksums do not contain bias since they are calculated by inputs and input checksums without bias. In order to compare the output checksums and the output O, bias has to be subtracted from output summation before comparison. Subtracting bias from output O directly before verification and then adding bias to O after verification is not feasible, however, because of the overhead of modifying every element in O. Table <ref> shows the output checksums and adjusted output summation for comparison in order to detect errors. The bias part of the formulations can be precomputed.  





 §.§ ABFT for Grouped Convolution

Grouped convolution is a special kind of convolution. Our schemes need to be modified to support this convolution.
Define the number of groups as G. Each fmap basic block has Ch/G instead of Ch channels. All the M kernel basic blocks are divided into G groups, each having M/G 3D basic blocks. The kernel block in the gth group does convolution only with the gth channel group of every fmap block. Figure <ref> shows this process for N=2, M=4, and G=2.



The checksums for fmap C_d1 and C_d2 stay the same. The checksum for kernel are redefined as

    C_w1=[∑^M/G-1_m=0W_m,∑^2M/G-1_m=M/GW_m, ..., ∑^M-1_m=(G-1)M/GW_m ]


    C_w2 =[∑^M/G-1_m=0m×W_m, ∑^2M/G-1_m=M/Gm×W_m, ...,∑^M-1_m=(G-1)M/G m×W_m ]

where C_w1 and C_w2 are the combination of G checksums from each kernel group. Each checksum has Ch/G channels, so Ch_w1 and Ch_w1 each have GCh/G=Ch channels, which are the same with every fmap block. 

The definition of output checksums C_o1, C_o2, ..., C_o7 stays the same. Let X[l..r] represent the channels from l to r in matrix X.  We can prove the following property for any D_n and W_m according to Equation (<ref>).

    [    D_n ⊗W_m= D_n[1..k-1]⊗W_m[1..k-1]; + D_n[k..Ch] ⊗W_m[k..Ch], 0 ≤ k < Ch ]

Using this equation, we can prove the relation between C_o2 and O  as follows. 

    [ C_o2[n]=
    D_n ⊗[∑^M/G-1_m=0W_m, ∑^2M/G-1_m=M/GW_m, ⋯, ∑^M-1_m=(G-1)M/Gw_m ];     = D_n[0..C/G-1] ⊗∑^M/G-1_m=0w_m
      +D_n[C/G..2C/G-1] ⊗∑^2M/G-1_m=M/Gw_m;                           + ⋯ 
      + D_n[(G-1)C/G..C-1] ⊗∑^M-1_m=(G-1)M/Gw_m;                                      = ∑^M-1_m=0D_n ⊗w_m
      = ∑^M-1_m=0O_nm ]


Similar equations can be proved for C_o1,C_o3,C_o4, C_o5, C_o6, and C_o7. Therefore, all the ABFT schemes we proposed can be applied to grouped convolution.  



 §.§ ABFT for Convolution Back Propagation

Our schemes can also be applied to back propagation together with forward pass so that the convolutional layers can be fully protected in the training phase. 

During back propagation, the gradient of kernel ∇W is used by methods such as gradient descent in order to update W. The gradient of fmap ∇D is used to get ∇O of the previous layer. As shown in Figure <ref>, the gradients are calculated as D⊗∇O= ∇W and W^T ⊗∇O= ∇D. Checksums for ∇O are used in this situation to protect the two convolution operations.



Since CNN models are usually trained in a more stable environment than the inference stage and since the training stage can tolerate some soft errors because of their iterative-convergent nature, we focus our experiments on the inference stage. 








§ EXPERIMENTAL EVALUATION

In this section, we evaluate our multischeme workflow using our FT-Caffe fault tolerance CNN framework.





 §.§ Experimental Setup


FT-Caffe. Our FT-Caffe framework is based on Intel-Caffe. MKL-DNN is enabled to support dynamic selection of convolution execution. MKL-DNN contains all the convolution implementations we discussed in Section <ref>. It automatically chooses the most suitable implementation to use for each convolutional layer. To compare the runtime overhead of our solution with that of the ABFT designed for matrix-matrix multiplication, we also perform the experiments based on the  MM-based convolution implementation.

CNN models and dataset. We tested our FT-Caffe with four widely used networks: AlexNet, VGG-19, ResNet-18, and YOLOv2. Pretrained Caffe models are used together with model prototypes for deployment. We adopt the ImageNet validation set, which contains 50k images. The images are preprocessed to smaller size in order to save picture processing time when the program starts. The batch size is set to 64. 

Experimental platforms. We conducted our experiments on the Bebop supercomputer<cit.> at Argonne National Laboratory using up to 128 nodes. Each node is equipped with 128 GB memory and two Intel Xeon E5-2695 v4 processors (each with 16 cores)


Error injection. 

To demonstrate the overhead of our fault tolerant solutions, we inject soft errors at the source code level as most ABFT works did <cit.>. The consequences of one computational fault or memory fault are simulated by randomly corrupting selected row or column of output. We denote the total number of convolutional layers of a CNN model as L. To assess the overhead accurately, we run the experiments for L epochs corresponding to the numbers of convolutional layers of each network (L= 5, 9, 16, 21 for AlexNet, YOLOv2, VGG-19, and ResNet-18, respectively). For the ith epoch, we inject errors to ith convolutional layer. The final overhead is the arithmetic mean of all the inference executions and the standard deviation in our experiments is within 5%. 



 §.§ Experimental Results with MKL-DNN



In this section, we present our evaluation results with MKL-DNN . We analyze the results from the perspective of execution time overhead for both error-free cases and erroneous cases.



Error-free cases. The experimental results in the error-free cases are presented in Figure <ref>(a). 
We can see from the figure that our FT-caffe can protect the inference stage with less than 4%, 4.5%, 8%, and 5% overhead for AlexNet, VGG-19, YOLOv2, and ResNet-18, respectively, regardless of the convolution implementation. These results show that our CoC-D error detection scheme has relatively short runtime compared  with the convolution execution, which is attributed to the design of avoiding unnecessary computations in our solution (see Section <ref>). The reason ResNet-18 has higher overhead than the other models have is that the ResNet-18 has small convolution kernels (W size is M×C×3×3) in all the convolutional layers, which have relatively short computing time; thus, the checksum computation and verification time percentage would be relatively large. 

Erroneous cases – RC/ClC disabled. To show the effectiveness of our layerwise optimization for RC/ClC, we first test our multischeme workflow with RC/ClC disabled in erroneous cases. Figure <ref>(b) demonstrates that the runtime overheads (including both error detection and error correction) of the four CNN models are all below 9%. The error detection overhead is higher than the error correction overhead because the error detection scheme is executed for every convolution operation whereas the error correction schemes are invoked only when errors are detected. The full checksum scheme dominates the error correction overhead, thus confirming our analysis in Section <ref> that FC has high protection ability and relatively longer runtime.


Erroneous cases –  layerwise RC/ClC optimization. 
Figure <ref>(c) demonstrates the runtime overhead with layerwise optimization enabled. Every layer decides whether to use RC/ClC independently, as described in Section <ref>. Compared with Figure <ref>(b), the error correction overhead decreases by 40%∼60% (e.g., 1.55% → 0.72% for YOLOv2 as shown in Figure <ref>(b) vs. (c)) in all CNN models because of the effectiveness of RC. 
Figure <ref>(a) shows the distribution of varies workflows that is the result of layerwise RC/ClC optimization. We can see that RC is enabled in all layers of AlexNet and VGG-19, while it is disabled in 30% to 40% of layers in ResNet-18 and YOLOv2. The results demonstrate the need for layerwise RC optimization since RC is not suitable for all layers in the same CNN model. Figure <ref>(b) shows the distribution of soft errors by the schemes that correct them. Less than 5% of soft errors are corrected by CoC because of the low correction ability of CoC. RC corrects nearly 90% of the soft errors in AlexNet and VGG-19 because RC is enabled in all layers of the two CNN models and the probability of soft errors striking a row in O is higher than the probability of soft errors striking a column.







Erroneous cases – breakdown of error correction overhead by layer. To better illustrate the overhead of our solution for each model, we present in Figure <ref> the breakdown of the overhead by layer. The figure demonstrates that layers have diverse overheads of error protection due to the different shapes of D and W. We also notice that the overhead of RC differs among layers in the same model, thus confirming the functionality of our layerwise RC/ClC optimization.








 §.§ Experimental Results with MM-Based Convolution

 


In this section, we evaluate the runtime overhead of our multischeme workflow and the traditional MM-based ABFT. 
Since the MM-based ABFT supports only the MM-based convolution implementation, we set the convolution implementation to the MM-based mode in MKL-DNN. We implemented MM-based ABFT rigorously based on <cit.>, 
which has ≤1% overhead for large and square matrices as claimed by the authors of that work.
The overhead of the MM-based ABFT in convolution execution is shown in Table <ref>. The MM-based ABFT incurs up to 60% overhead even without error injection for the four CNN models. This result is consistent with our analysis in Section <ref>. Considering that the MM-based ABFT cannot protect the whole process of MM-based convolution and cannot protect other convolution implementations, we  conclude that the MM-based ABFT is unsuitable for soft error protection of CNN applications. 

Figure <ref> shows the overhead of our multischeme workflow for MM-based convolution. The overhead of our solution is below 6% in the error-free cases and below 6.5% in the cases with injected errors for all CNN models. The layerwise RC/ClC optimization reduces the overhead for error correction by as much as 77%. Figure <ref>(a) shows the fractions of different workflows chosen by convolutional layers. Figure <ref>(b) shows the distribution of soft errors that are corrected by different schemes. Compared with the MKL-DNN implementation, more layers adopt RC for error correction in the MM-based convolution (see  Figure <ref> versus  Figure <ref>). The reason is that the relative runtime of RC compared with FC is lower in the MM-based convolution implementation than other implementations. 











 §.§ Parallel Performance Evaluation

In this section, we present the parallel performance evaluation results of AlexNet, YOLOv2, VGG-19, and ResNet-18. 
Original images of the ImageNet validation dataset are used without preprocessing in order to better demonstrate the process of parallel CNN inference application. In the beginning of the parallel process, images are distributed to the local disk of each node; then each node starts to do the data processing step first to convert the images to suitable size required by CNN models, and then execute the inference step under the protection of our multischeme workflow.

We conducted the parallel evaluation in both error-free and erroneous cases. However, because of space limits, we present only the parallel performance evaluation results in the situation with injected errors (as shown in Figure <ref>). In fact, the evaluation results in the error-free situation are  similar. Specifically, experiments show that our multischeme workflow has a very good scalability:  that is, the soft error protection overhead does not increase with the number of nodes at all. In absolute terms, the overhead stays around 2%∼6% in the erroneous cases and is only 1%∼4% in the error-free cases.











§ RELATED WORK


The importance of fault tolerance for convolution has been emerging in recent years. Guaranteeing the correctness of inference is  vital in a safety-critical use case <cit.>. To achieve  better resiliency for CNN networks, researchers have been exploring solutions from different perspectives including hardware, system, and software. For hardware, Kim et al. <cit.> proposed a hardened 3D die-stacked memory based on the fault characteristics in convolutional DNNs. Li et al. <cit.> proposed to add redundant circuits selectively to harden the latches based on analysis of data resiliency. Compared with traditional full-hardware redundancy techniques, those partial-hardware redundancy techniques may not double the power usage. However, hardware modification incurs significant effort considering the varied CNN models and their accelerators. At the system level, other than the DMR/TMR protection, checkpoint/restart (C/R) is also applied to  large-scale machine learning systems. Subsequently,  Qiao et al. proposed a more efficient C/R scheme based on their derived upper bound on extra iteration cost with perturbations <cit.>. While those C/R techniques are promising to protect model training from soft errors, they are not good fits for inference since one inference execution could be very fast and applying C/R incurs significant overhead. Researchers have therefore pursued lightweight software-level solutions. By applying ABFT techniques for MM-based convolution, Santos et al. <cit.> reported that 50%∼60% of radiation-induced corruptions could be corrected. Unfortunately, the traditional ABFT works only for MM-based convolution, which is inefficient in most cases. 
In contrast, our solutions can work for any convolution implementations.




§ CONCLUSION AND FUTURE WORK


This work focus on extending ABFT to convolution operations in convolutional neural networks. We propose four ABFT schemes and a multischeme workflow to protect the convolutional layer. We further extend our schemes to support bias, grouped convolution, and convolution back propagation. We implement an efficient CNN framework, FT-Caffe, that is resilient to silent data corruption. 

Experiments demonstrate that our proposed fault-tolerant solutions incur negligible overhead. In absolute terms, FT-Caffe can acheive less than 8% overhead for the most widely used CNN models, including AlexNet, YOLO, VGG-19, and ResNet-18, in both error-free and erroneous cases. 

We plan to extend the implementation to more CNN frameworks and to design architecture-specific optimizations for different hardware including GPU, FPGA, and AI accelerators.





§ ACKNOWLEDGMENTS

This research was supported by the Exascale Computing Project (ECP), Project Number: 17-SC-20-SC, a collaborative effort of two DOE organizations - the Office of Science and the National Nuclear Security Administration, responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering and early testbed platforms, to support the nation’s exascale computing imperative. The material was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357. This work was also supported by the National Science Foundation under Grants CCF-1513201, CCF-1619253, and OAC-2034169. We acknowledge the computing resources provided on Bebop, which is operated by the Laboratory Computing Resource Center at Argonne National Laboratory.














































































































































IEEEtran
















[
    < g r a p h i c s >
]Kai Zhao
received his bachelor's degree from Peking University in 2014 and will receive his Ph.D. degree from University of California, Riverside in 2022. He is a long-term intern at Argonne National Laboratory.
His research interests include high-performance computing, scientific data management and reduction, and resilient machine learning. Email: kzhao016@ucr.edu.


-2plus -1fil

[
    < g r a p h i c s >
]Sheng Di
(Senior Member, IEEE) received his master's degree from Huazhong University of Science and Technology in 2007 and Ph.D. degree from the University of Hong Kong in 2011. He is currently a computer scientist at Argonne National Laboratory. 
Dr. Di's research interest involves resilience on high-performance computing (such as silent data corruption, optimization checkpoint model, and in-situ data compression) and broad research topics on cloud computing (including optimization of resource allocation, cloud network topology, and prediction of cloud workload/hostload).
He is working on multiple HPC projects, such as detection of silent data corruption, characterization of failures and faults for HPC systems, and optimization of multilevel checkpoint models. Email: sdi1@anl.gov.


-2plus -1fil

[
    < g r a p h i c s >
]Sihuan Li
is a Ph.D. student in computer science at University of California, Riverside. He obtained his bachelor's degree in math from Huazhong University of Science and Technology, China. He did a long-term internship at Argonne National Laboratory. Broadly speaking, his research interests fall into High Performance Computing. Specifically, he mainly studies Algorithm Based Fault Tolerance (ABFT), lossy compression and their applications in large scale scientific simulations. He is an IEEE student member. Email: sli049@ucr.edu.


-2plus -1fil

[
    < g r a p h i c s >
]Xin Liang is a Computer/Data Scientist at Oak Ridge National Laboratory. He received his Ph.D. degree from University of California, Riverside in 2019 and his bachelor's degree from Peking University in 2014. His research interests include high-performance computing, parallel and distributed systems, scientific data management and reduction, big data analytic, scientific visualization, and cloud computing. He has interned in multiple national laboratories and worked on several exascale computing projects. He is a member of the IEEE. Email: liangx@ornl.gov.


-2plus -1fil

[
    < g r a p h i c s >
]Yujia Zhai
received his bachelor's degree from University of Science and Technology of China in 2016, a master's degree from Duke University in 2018, and will receive his Ph.D. degree from University of California, Riverside in 2023. His research interests include high-performance computing, parallel and distributed systems, and numerical linear algebra software. Email: yzhai015@ucr.edu.


-2plus -1fil

[
    < g r a p h i c s >
]Jieyang Chen (Member, IEEE) is a Computer Scientist at Computer Science and Mathematics Division at Oak Ridge National Laboratory.
He received his master and Ph.D. degrees in Computer Science from University of California, Riverside in 2014 and 2019. He received a bachelor’s degree in Computer Science and Engineering from Beijing University of Technology in 2012. His research interests include high-performance computing, parallel and distributed systems, and big data analytics. Email: chenj3@ornl.gov.


-2plus -1fil

[
    < g r a p h i c s >
]Kaiming Ouyang
received his bachelor's degree from University of Electronic Science and Technology of China and joined the University of California, Riverside SuperLab in Fall 2016. 
He will receive his Ph.D. degree from University of California, Riverside in 2021. He is a long-term intern at Argonne National Laboratory PMRS group led by Dr. Balaji and supervised by Dr. Si. His research interest is parallel runtime system. Email: kouya001@ucr.edu.
 



-2plus -1fil

[
    < g r a p h i c s >
]Franck Cappello

(Fellow, IEEE) is the director of the Joint-Laboratory on Extreme Scale Computing gathering six of the leading high-performance computing institutions in the world: Argonne National Laboratory, National Center for Scientific Applications, Inria, Barcelona Supercomputing Center, Julich Supercomputing Center, and Riken AICS. He is a senior computer scientist at Argonne National Laboratory and an adjunct associate professor in the Department of Computer Science at the University of Illinois at Urbana-Champaign. He is an expert in resilience and fault tolerance for scientific computing and data analytics. Recently he started investigating lossy compression for scientific data sets to respond to the pressing needs of scientist performing large-scale simulations and experiments. His contribution to this domain is one of the best lossy compressors for scientific data set respecting user-set error bounds. He is a member of the editorial board of the IEEE Transactions on Parallel and Distributed Computing and of the ACM HPDC and IEEE CCGRID steering committees. He is a fellow of the IEEE. Email: cappello@mcs.anl.gov.



-2plus -1fil

[
    < g r a p h i c s >
]Zizhong Chen

(Senior Member, IEEE) received a bachelor's degree in mathematics from Beijing Normal University, a master's degree degree in economics from the Renmin University of China, and a Ph.D. degree in computer science from the University of Tennessee, Knoxville. He is a professor of computer science at the University of California, Riverside. 
His research interests include high-performance computing, parallel and distributed systems, big data analytics, cluster and cloud computing, algorithm-based fault tolerance, power and energy efficient computing, numerical algorithms and software, and large-scale computer simulations. His research has been supported by National Science Foundation, Department of Energy, CMG Reservoir Simulation Foundation, Abu Dhabi National Oil Company, Nvidia, and Microsoft Corporation. 
He received a CAREER Award from the US National Science Foundation and a Best Paper Award from the International Supercomputing Conference. 
He is a Senior Member of the IEEE and a Life Member of the ACM. He currently serves as a subject area editor for Elsevier Parallel Computing journal and an associate editor for the IEEE Transactions on Parallel and Distributed Systems. Email: chen@cs.ucr.edu.



















