Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Mixed-precision arithmetic combining both single- and half-precision operands in the same operation have been successfully applied to train deep neural networks. Despite the advantages of mixed-precision arithmetic in terms of reducing the need for key resources like memory bandwidth or register file size, it has a limited capacity for diminishing computing costs and requires 32 bits to represent its output operands. This paper proposes two approaches to replace mixed-precision for half-precision arithmetic during a large portion of the training. The first approach achieves accuracy ratios slightly slower than the state-of-the-art by using half-precision arithmetic during more than 99% of training. The second approach reaches the same accuracy as the state-of-the-art by dynamically switching between half- and mixed-precision arithmetic during training. It uses half-precision during more than 94% of the training process. This paper is the first in demonstrating that half-precision can be used for a very large portion of DNNs training and still reach state-of-the-art accuracy. The use of Deep Neural Networks (DNNs) is becoming ubiquitous in areas like computer vision , speech recognition, or language translation . DNNs display very remarkable pattern detection capacities and, more specifically, Convolutional Neural Networks (CNNs) are able to accurately detect and classify objects over very large image sets . Despite this success, a large amount of samples must be exposed to the model for tens or even hundreds of times during training until an acceptable accuracy threshold is reached, which drives up training costs in terms of resources like memory storage or computing time. To mitigate these very large training costs, approaches based on data representation formats simpler than the Floating Point 32-bit (FP32) standard have been proposed . These approaches successfully mitigate the enormous training costs of DNNs by using data representation formats that either reduce computing costs or diminish the requirements in terms of memory storage and bandwidth. In particular, some of these proposals have shown the benefits of combining half-precision and single-precision compute during training in terms of keeping model accuracy and reducing compute and memory costs . These approaches accelerate linear algebra operations by accumulating half-precision input operands to generate 32-bit outputs. While this mixed-precision (MP) arithmetic can successfully reduce the use of resources like memory bandwidth or hardware components like register file size, it has a very limited capacity for diminishing computing costs and it is unable to reduce output data size. In this paper we propose new training methodologies able to exclusively use half-precision for a large part of the training process, which constitutes a very significant improvement over mixedprecision approaches in terms of compute and memory bandwidth requirements. We propose two different approaches, the first one statically assigns either the Brain Floating Point 16-bit (BF16) or the FP32 format to the model parameters involved in the training process, while the second dynamically switches between BF16 and MP during training depending on its progress. Our approaches do not require mixed-precision arithmetic while computing linear algebra operations for a large portion of the training process, which enables them to deliver the same performance as if they were operating with half-precision arithmetic during the whole training while providing the same model accuracy as if FP32 was used. This paper is the first in demonstrating that half-precision can be extensively used during DNNs training without the need for mixed-precision arithmetic. We made our code available 1. Mixed-Precision training has been extensively explored in recent years. Approaches mixing Floating Point 16-bit (FP16) and FP32 datatypes have been proposed . In these approaches, multiplications of FP16 parameters are accumulated in FP32 registers to minimize data representation range and precision issues. Importantly, relevant phases of the training process like computing weight updates (WU) or dealing with batch normalization (BN) layers entirely use FP32, which implies that a FP32 representation of network weights and biases is kept during the whole training. This approach requires some additional computations to enforce that FP32 values are converted to FP16 without data representation range issues. This approach is used by Nvidia Tesla V100 GPUs via mixed-precision computing units called tensor cores, which are able to multiply FP16 parameters and store the in FP32. Figure 1a displays the most fundamental operation of this approach combining FP16 and FP32, the mixed-precision Fused Multiply-Add (FMA) instruction, which computes D = A · B + C. Input parameters A and B are represented in the FP16 format. The of the A · B operation is kept in FP32 and added to the C parameter, which is represented in FP32 as well. The final output D is also represented in FP32. FMA instructions constitute around 60% of the whole training workload for several relevant CNN models, as Section 3 shows. A more recent approach proposes mixed-precision arithmetic combining BF16 and FP32 . It is very close to its FP16 counterpart with the exception of the full to half precision conversion. Since BF16 has the same data representation range as FP32, conversion from full to half precision is very simple in this case since it just requires applying the Round to Nearest Even (RNE) technique. This approach also processes WU and BN layers with FP32. Figure 1b shows a representation of a mixed-precision FMA combining BF16 and FP32. It is very close to the previously described FP16-FP32 FMA with the only difference being the data representation format of input parameters A and B. While mixed-precision FMA instructions bring significant benefits since they require less memory bandwidth and register storage than FP32 FMAs, there is still a large margin for improvement if an entirely BF16 FMA like the one represented in Figure 1c could be extensively used for training purposes. First, since BF16 FMA requires exactly one half of the register storage of a FP32 FMA, it doubles its Single Instruction Multiple Data (SIMD) vectorization capacity and, therefore, it may significantly increase its FMA instructions per cycle ratio. Extensions of Instruction Set Architectures (ISA) to allow SIMD parallelism are becoming a key element for floating-point performance, which has motivated major hardware vendors to include them in their products . Finally, BF16 FMA instructions also bring significant reductions in terms of memory bandwidth since they involve 50% and 25% less data than FP32 and MP FMAs, respectively. While half-precision arithmetic has not been used to train DNNs due to its lack of training convergence, this paper describes two techniques to fully use it while keeping the same convergence properties as FP32. This paper analyzes in detail 3 relevant training workloads in Section 3, and applies these findings to build its two main contributions in Section 4. We consider three CNN models: AlexNet, Inception V2, and ResNet-50. Section 5 describes the exact way we use these models and the methodology we follow to analyze their training workload. Figure 2a shows an instruction breakdown of processing one batch on these networks. This figure shows how floating-point instructions constitute a large portion of these workloads. For example, they represent 58.44% of the total in the case of AlexNet. A very large portion of these floatingpoint instructions, 57.42% of the total, are FMA instructions. In the cases of Inception and ResNet-.95% of the total, respectively. Therefore, FMA instructions constitute a large portion of the whole training workload, while other FP32 instructions represent a small instruction count that remain below 1.1% for these three CNNs. This justifies to focus on FMA instructions, as executing them in half-precision has a large potential for performance improvement. Prior research describes the need for using 32-bit arithmetic in Weight Updates (WU) and Batch Normalization (BN) layers when using training approaches based on mixed-precision arithmetic. We run an experimental campaign to confirm this observation and to measure the number of instructions devoted to WU and BN. For the case of ResNet-50, this instruction count is around 30 million instructions per batch, that is, just 0.04% of the FP instructions. AlexNet and Inception V2 produce similar . In , reducing the cost of FMA instructions has a high potential for very significant performance improvements even if WU and BN layers are computed using full precision arithmetic. Processing one training batch for the cases of AlexNet, Inception and ResNet-50 requires running 53.3, 37.2, and 70.0 billion dynamic instructions per batch, respectively. The number of model parameters drives the size of these workloads. AlexNet was trained with a batch size of 256, while Inception and ResNet use a batch size of 64. We propose two training methodologies that rely exclusively on half-precision BF16 for a large portion of the training process, i.e., a large portion of FMA instructions. Prior mixed-precision approaches preclude large gains in computing costs as some of the data elements remain in FP32. However, an FMA entirely relying on BF16 can potentially double the SIMD vectorization throughput of current processors and alleviate memory bandwidth requirements. We first propose a scheme that performs all FMA instructions in BF16 (see Figure 1c) except those involved in computing WU and processing BN layers, which are entirely performed in FP32. While this method might not deliver the desired level of accuracy for all CNNs, Section 6 shows how it behaves remarkably well for the Inception V2 model, since it obtains the same level of accuracy as state-of-the training using MP and FP32. However, some CNNs cannot entirely rely on half-precision arithmetic during training. For example, Figure 2b shows top1 accuracy achieved by three training techniques during 15 epochs for ResNet-50. The first technique (referred as FP32 in Figure 2b) entirely relies in FP32 arithmetic, the second approach (referred as MP in Figure 2b) represents the state-of-the art Mixed-Precision training , and the third approach (referred as BF16 in Figure 2b) performs all FMA instructions in BF16 except for WU and BN. While the BF16 approach behaves relatively well, it displays lower accuracy than MP and FP32 for all the epochs, which indicates the need for an approach able to take advantage of BF16 arithmetic while delivering the same accuracy as mixed-or full-precision approaches. The methodology we use to generate Figure 2b is described in Section 5. Our second contribution dynamically switches between MP and BF16 to deliver the same accuracy as MP while relying in BF16 FMAs during a large portion of the training process. Algorithm 1 displays a high level pseudo-code of our proposal. It starts the training process using the state-of-the art mixed-precision approach for several batches, defined by numBatchesM P parameter. Then, it computes the Exponential Moving Average (EMA) of the training loss and, if its reduction is larger than a certain threshold (emaT hreshold parameter), it computes the next numBatchesBF 16 using BF16 FMAs, except for WU and BN. Once training has gone through these numBatchesBF 16 batches, our algorithm checks EMA and compares its reduction with the emaT hreshold parameter. If this reduction is not large enough, the algorithm switches back to MP arithmetic. Otherwise, it keeps using BF16 arithmetic for numBatchesBF 16 batches before checking EMA again. Our experiments are performed on Intel Xeon Platinum 8160 processors, which include the AVX512 ISA. We use the Intel-Caffe (Intel, a) framework (version 1.1.6a). We use the Intel MKLDNN (Intel, c) (version 0.18.0) Deep Neural Network library and the Intel MKL library (Intel, b) (version 2019.0. 3) to run numerical kernels since both libraries are optimized to perform well on our testing infrastructure. Finally, to define and run the experiments we use the pyCaffe python interface, which takes care of loading the data and orchestrating the execution. Due to the lack of available hardware implementing the BF16 numerical format, we rely on an emulation technique to perform our experiments. Several approaches have been used in the past to emulate the behaviour of reduced floating-point representations, most notably via libraries that perform transformations like truncation and rounding (; Dawson & Düben, 2017;). We develop a binary analysis tool based on PIN 3.7 . Our tool captures and instruments dynamic instructions, which enables adaptating numerical operands to the targeted numerical data format. Our approach seamlessly works on complex frameworks like PyTorch, Tensorflow, or Caffe, with interpreted languages, and is able to instrument instructions triggered by dynamically linked libraries. Our binary analysis tool performs the following steps: • It checks the current operation mode, which can be FP32, MP, or BF16 (see Figure 1). • It checks the current execution routine to determine if we are executing routines that belong to WU or BN layers. If that is the case, computation proceeds with FP32. • The tool intercepts the dynamic instructions of the workload and detects all floating-point operations, including FMAs. For each FMA instruction, operands that need to be rounded to BF16, depending on the current operation mode, are rounded using the RNE algorithm. countBatchesBF 16 ← 0 • The tool can dynamically change its operation mode anytime via a simple inter-process communication method that can be invoked from the python high-level interface. To mitigate the overhead of our binary analysis tool, we implement two optimizations: First, we vectorize the truncation and rounding routines via AVX512 instructions. Second, we avoid redundant rounding and truncation operations by identifying instructions belonging to the same basic block sharing some input operands already stored in the register file. These two optimizations reduce the overhead of the tool from 100× to 25× with respect to native runs of the binary on real hardware. This paper considers two different types of training techniques: static schemes and dynamic schemes. When using static schemes, the training procedure uses the same data representation form for a given parameter during its complete execution. For example, the three techniques displayed in Figure 2b are static. We define the following schemes: • MP: FMA instructions belonging to WU and BN layers always use FP32 precision. The remaining FMA instructions use the mixed-precision approach represented in Figure 1b ). This scheme replicates prior work on mixed-precision . • BF16: FMA instructions belonging to WU and BN layers always use FP32 precision. The remaining FMA instructions use BF16 operands to multiply and to accumulate (Figure 1c). The BF16 method is the first contribution of this paper. It extensively uses half-precision arithmetic while displaying good convergence properties. The Dynamic scheme we propose in this paper switches between the MP and BF16 static techniques during training, as explained in Section 4 and detailed in Algorithm 1. This dynamic method im-proves the training convergence properties of BF16 while still relying in half-precision FMAs for a very large portion of the execution. The EMA threshold (emaTreshold) is set at 4%. This value is computed as the average EMA reduction when using FP32 computations. The minimum number of batches to be performed in BF16, defined by the numBatchesBF 16 parameter is set to 1,000, which precludes frequent unnecessary transitions between the two schemes. We set the numBatchesM P parameter to 10, which keeps the number of batches using the MP regime low while keeping its benefits in terms of convergence. To evaluate our proposals we consider the AlexNet , Inception V2 and ResNet50 (b) models. They are representative CNN state-of-the-art. We use the ImageNet database as training input. To keep execution times manageable when using our binary instrumentation tool, we run the experiments using a reduced ImageNet Database, similar to the Tiny ImageNet Visual Recognition challenge data set (Fei-Fei). Therefore, we use 256,000 images divided into 200 categories for training, and 10,000 images for validation. The images have no modifications in terms of the size. All the evaluated CNN models remain unmodified, the only change is loading a reduced dataset. AlexNet is selected due to its simplicity in terms of structure and amount of required computations. To train AlexNet we consider a batch size of 256 and the base learning rate is 0.01, which is adjusted every 20 epochs taking into account a weight decay of 0.0005 and a momentum of 0.9. This model is trained for 32 epochs. We use Inception because it is a model conceived to reduce computational costs via cheap 1x1 convolutions. To train it we use a batch size of 64 and a base learning rate of 0.045, which is updated every 427 steps (0.11 epochs). The gamma, momentum and weight decay are set to 0.96, 0.9, and 0.0002, respectively. The training process is executed for 16 epochs. Finally we use ResNet-50. It is a network that delivers good accuracy and avoids the vanishing gradients issue by using residual blocks and the MSRA initializer (a). We train it using a multi-step approach. The batch size is 64 and the base learning rate is 0.05, which is updated every 30 epochs. The gamma hyperparameter, momentum value, and weight decay are set to 0.1, 0.9, and 0.0001, respectively. The training process runs for a total of 32 epochs. Figure 3 and Table 1 show from our evaluation campaign. The x-axis of the three plots belonging to Figure 3 represent the epochs of the training process while the y-axis represents the accuracy reached by the model over the validation set. Table 1 shows the test accuracy we reach for the three network models when using the FP32 and MP baselines and our two contributions: BF16 and Dynamic. The AlexNet model, due to its structure, shows a good response when lower precision numerical data types are used. As can be seen in Figure 3a all techniques converge, although the BF16 approach shows the worse accuracy when compared to the Dynamic or the MP techniques. Table 1 shows that FP32, MP, Dynamic, and BF16 reach top-5 accuracies of 84.50%, 84.43%, 84.02% and 82.56% for AlexNet after 32 epochs. Importantly, Dynamic reaches the same accuracy as FP32 and MP while using the BF16 approach for 94.60% of the FMAs. In contrast, the BF16 static technique does 99.93% of the FMAs in full BF16 precision (0.07% are in WU and BN layers), but the accuracy drops by almost 3% in Top-1 and 2% in Top-5. This drop in accuracy happens just by doing an additional 5% BF16 FMAs. This give us some space to improve the Dynamic approach by reducing the percentage of BF16 FMAs with the objective to increase the accuracy of the model. Figure 3b shows the validation accuracy during 16 epochs for the Inception V2 model. It shows fluctuation on the accuracy evaluation during training due to its structure and hyperparameters tuning. Dynamic responds in a robust way to these changes, which highlights its general applicability. Table 1 shows that FP32, MP, Dynamic, and BF16 reach top-5 accuracies of 93.36%, 92.67%, 92.02%, and 92.05% for Inception V2 after 16 epochs. Finally, the evaluation on ResNet-50 demonstrates that the Dynamic approach is effective when applied to deeper CNNs. In this case, the precision of the model reaches state-of-the-art levels while using half-precision for 96.4% of the FMA instructions. Figure 3c and Table 1 display the exact accuracy numbers we get from our evaluation after 32 epochs. In this experiment the Top-1 accuracy drops just 1.2% comparing the BF16 and dynamic approaches, however we could improve the dynamic technique relaxing the quantity of BF16 FMA executed to gain more accuracy. We provide a sensitivity analysis for the parameters employed in Algorithm 1. The objective is to show that for a range of reasonable parameters the algorithm behaves as expected. To do this analysis we set one of the parameters to the currently used value (numBatchesMP to 10) to have a manageable number of combinations. We then test all the possible combinations using numBatchesBF16 = {500, 1000, 2000} and emaThreshold = {0.02, 0.04, 0.08}, that is, a total of 9 different combinations. As stated in Section 5.3, during our evaluation we used the configuration {numBatchesMP, numBatchesBF16, emaThreshold} = {10, 1000, 0.04} for all the evaluated networks. Figure 4 shows, for a number of ResNet-50 epochs, the accuracy obtained for each of the 9 tested configurations as part of the sensitivity analysis. The name convention for these configurations is Dyn-<emaThreshold> <numBatchesBF16>. In addition, we include accuracy for BF16, MP, and FP32 executions. As shown in the figure, the accuracies obtained at each epoch are always above that of the BF16 technique. For early epochs (i.e., 2 and 4) the dynamic configurations remain between BF16 and FP32 accuracy, or even slightly above FP32, due to initial noise. As training advances all dynamic techniques behave similarly and present accuracies that are above BF16 and similar to those obtained with MP and FP32, as we would expect. The most important parameter is the emaThreshold, as it decides when a precision change occurs. As long as this parameter is reasonably set to detect training loss improvement or degradation the algorithm is bound to behave as expected. Prior work indicates that dynamic fixed-point is effective to train deep neural networks with low precision multipliers . This approach obtains state-of-the-art by uniformly applying the dynamic fixed point format with different scaling factors, which are driven by the overflow rate displayed by the fixed-point numbers. Our proposals target deeper neural networks than this approach and do not uniformly apply the same format to all network parameters. Instead, we differentiate between computations requiring FP32 during the whole training, like weight updates, from the ones that are well-suited for dynamic data representation schemes. Previous approaches show the benefits of applying stochastic rounding to 16-bit fixed-point multiply and add operators. This previous work rely on FPGA emulation to show the benefits of stochastic rounding when applied to a custom fully connected neural network dealing with the MNIST dataset. The authors also consider a CNN similar to LeNet-5 to enlarge their experimental campaign. Previous approaches propose a training process of DNN using 8-bit floating point numbers . They rely on a combination of 8-bit and 16-bit and additionally using stochastic rounding to obtain state-of-the-art . The neural networks used in this previous approach are much simpler than the ones we consider in this paper, which do not allow 8-bit arithmetic. The BF16 numerical format has been applied to specific-purpose hardware targeting deep neural networks . This specific hardware used an approach very similar to the mixed-precision techniques described in the paper by and, consequently, our Dynamic approach can be applied on top of them to reduce computing costs. This paper analyzes the instruction breakdown of workloads focused on deep neural network training that rely on mixed-precision training. We show that mixed-precision FMAs constitute around 60% of these workloads and propose two approaches based on half-precision FMAs to accelerate the training process without hurting accuracy. The first approach uses BF16 FMAs for most of the training workload, except routines involved in weight updates or batch normalization layers. This approach uses BF16 for more than 99% of the FMAs, which has a very strong potential for performance improvement, while reaching slightly smaller accuracy than the state-of-the-art. We propose a second approach that dynamically switches between different data representation formats. This dynamic approach uses BF16 for around 96% of the FMAs while reaching the same precision levels as the standard single-precision and mixedprecision approaches. Our two proposals are evaluated considering three state-of-the-art deep neural networks and a binary analysis tool that applies the required precision for each instruction. To the best of our knowledge, this is the first paper that demonstrates that half-precision can be used extensively on ≥94% of all FMAs during the training of very deep models without the need for mixed-precision arithmetic.

Dynamic precision technique to train deep neural networks

We present eligibility propagation an alternative to BPTT that is compatible with experimental data on synaptic plasticity and competes with BPTT on machine learning benchmarks.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We introduce the “inverse square root linear unit” (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noise- robust deactivation state, lessening the over fitting risk. The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs. This work also suggests a computationally efficient variant called the “inverse square root unit” (ISRU) which can be used for RNNs. Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions. ISRU has less computational complexity but still has a similar curve to tanh and sigmoid. Two popular activation functions for neural networks are the rectified linear unit (ReLU) BID6 and the exponential linear unit (ELU) BID5. The ReLU activation function is the identity for positive arguments and zero otherwise. The ELU activation function is the identity for positive arguments and has an exponential asymptotic approach to -1 for negative values. From previous analysis of the Fisher optimal learning, i.e., the natural gradient BID1 BID5, we can reduce the undesired bias shift effect without the natural gradient, either by centering the activation of incoming units at zero or by using activation functions with negative values. We introduce the inverse square root linear unit (ISRLU), an activation function like ELU, that has smoothly saturating negative values for negative arguments, and the identity for positive arguments. In addition this activation function can be more efficiently implemented than ELU in a variety of software or purpose-built hardware. The inverse square root linear unit (ISRLU) with α is DISPLAYFORM0 The ISRLU hyperparameter α controls the value to which an ISRLU saturates for negative inputs (see FIG0). ISRLUs and ELUs have very similar curves so at a high level one would expect to see the same general characteristics in most cases. ISRLUs have smooth and continuous first and second derivatives. ELUs are only continuous in the first derivative (see FIG0). In contrast, ReLU is non-differentiable at zero. Since ISRLUs and ELUs share most of the same characteristics we use the same weight initialization guidelines as are used for ELUs BID5 ). The primary advantage of ISRLU is in its reduced computational complexity compared to ELU. Inverse square roots are faster to calculate than exponentials. When calculating ISRLU for negative inputs, first one calculates 1/ √ 1 + αx 2. Multiplying this function by x provides the value for the forward calculation. Multiplying this function by itself twice (i.e. cubing) provides the value for back-propagation. With α = 1, ISRLU saturation approaches -1. With α = 3, the negative saturation is reduced, so a smaller portion of the back-propagated error signal will pass to the next layer. This allows the network to output sparse activations while preserving its ability to reactivate dead neurons. Note that under variations of the α parameter, the ISRLU curve and its derivative remain smooth and continuous. Future work will establish what deeper saturation (α < 1) is appropriate when applying ISRLU to self-normalizing neural networks BID11.In the same manner as parametric ReLUs (PReLUs) only one additional hyperparameter is required and methods can be used to directly learn its value during back-propagation BID8. Similarly, ISRLU's α can be learned during the training phase along with the weights and biases. Indeed for PReLUs, BID8 have empirically shown that learning the slope parameter "a" gives better performance than manually setting it to a pre-defined value.3 ACTIVATION FUNCTION PERFORMANCE BID14 showed that ELU was faster than the combination of ReLU and Batch Normalization for deep neural network (DNN) ResNet architectures. On CIFAR-10 and CIFAR-100 they showed that ELU not only speeds up learning but also improves the accuracy as the depth of the convolutional neural network (CNN) increases. More than learning rate needs to be considered when evaluating the overall performance of CNNs. The amount of time and computational resources required to perform both the convolutions and activation functions combined should be considered. The trend in CNNs is that less time is being spent calculating convolutions. There are three factors that we are seeing. First is that small convolution filters such as 5x5 or 3x3 filters are the basis of many architectures. Second, architectures as Inception-v3 and Inception-v4 now decompose 2d filters such as a 3x3 into a 3x1 filter and a 1x3 filter BID15. Third, more efficient calculations of convolution that rely on techniques such as Winograd's minimal filtering algorithm BID12 BID16 are being used for 3x3 and smaller filters as are FFTs to reduce calculation time in 5x5 or larger filters. All of these techniques reduce the amount of calculations for each element in the convolution output. TAB0 shows "cycles per output element" for an Intel Xeon Platinum 8160 (Skylake). Due to all of these reductions in convolution computational complexity, activation function performance is now a greater part of overall learning performance. Another characteristic that is changing with the use of smaller filters is the decrease in the compute intensity BID2 b), which raises the importance of memory systems performance for CNNs. The compute intensity of an algorithm is the ratio of the number of operations divided by number of words accessed. For a given algorithm it is straightforward to calculate the upper bound of the computation rate that can be supported on a given memory bandwidth. The main advantage of ISRLU over ELU is that it is based on the inverse square root, which has been faster to evaluate than the exponential for many generations of systems. In the past, whenever it has not been faster, optimization potentials for inverse square root implementation improvement have been found. It is instructive to understand the current CPU performance of the inverse square root intrinsic performance compared to exponentials and tanh. Intel x86 CPUs with SIMD instructions have vector intrinsic functions to accelerate performance. Intel publishes CPE (Clocks per Element) for various vector functions on their "Vector Mathematics (VM) Performance and Accuracy Data" website, see TAB1 . For example, on a 3x1 filter using ELU in the negative region, approximately the same CPE is required to evaluate the convolution as is required for the exponential (cf. TAB0). Improvements in activation function performance will impact overall time spent in each learning step. We measured the vector performance of AVX2 implementations for the various activation functions. The dataset used was 50% negative and 50% positive. Results are shown in TAB2. These show that ISRLU (α = 1.0) is 2.6× faster than ELU. The fast approximation of ISRLU is within 1% of the evaluation speed of ReLU while still retaining all of the desired learning curve properties mentioned in this paper. This fast approximation for ISRLU on this processor has only 3 × 10 −4 maximum relative error (∼11.6 accurate bits). One Newton-Raphson iteration doubles that to ∼23.4 accurate bits out of the 24 bits of mantissa, and two iterations achieves full precision. We plan to evaluate if the fast approximation has similar learning rates of the full precision ISRLU. It is instructive to look at a practical trick for the computation of the inverse square root as it may serve as inspiration for those implementing ISRLU in hardware. Software implementations on CPUs can take advantage of floating-point formats for faster evaluation of the inverse square root. John Carmack and Terje Mathisen are often associated with implementing fast inverse square root in 2002 BID13. In 1986, one of the authors of this paper originally invented this method, which was called "The K Method," to implement vector square root for the production FPS T Series Hypercube Supercomputer BID7. William Kahan and K.C. Ng at Berkeley also independently discovered this around 1986.Carmack & Mathisen only used one iteration of the Newton method after their fast approximation. One iteration had an error of approximately 0.175%, which was suitable for their graphics applications. Since various piecewise functions have been used to approximate activation functions for CNNs and RNNs, part of our future research will look into if fast approximations to ISRLUs are suitable for DNNs. Another avenue to look at for hardware implementations of the inverse square root is table-lookup hardware. Our expectation is that an efficient hardware approximation for the inverse square root should take about the same execution time as a fused multiply and add (FMA). We used TensorFlow BID0 to train a CNN on the (Lecun) MNIST dataset. We tested the MNIST gray images in 10 classes, 60k train and 10k test. The first CNN architecture (see TAB3) in our experiments used 28x28 input, a convolutional layer with 6x6 with 6 feature maps, a convolutional layer with 5x5 with 12 feature maps, a convolutional layer with 4x4 with 24 feature maps, a fully connected layer of 1176 hidden units, and a softmax output layer with 10 units. Only a full-precision ISRLU was used in these initial tests due to time constraints. Convolutional neural networks with ISRLUs (α = 1.0, α = 3.0), ELUs (α = 1.0), and ReLUs were trained on the MNIST digit classification dataset while each hidden units activation was tracked. Each network was trained for 17 epochs by using ADAM optimizer with learning rate 0.003 exponentially decreasing to 0.0001 and mini-batches of size 100. The weights have been initialized to truncated normal with standard deviation 0.1. The training error of ISRLU networks decreases much more rapidly than for the other networks. We also calculated the final cross-entropy loss function for each test. The second CNN architecture (see TAB4) in our experiments used 28x28 input, a convolutional layer with 3x3 with 64 feature maps, a convolutional layer with 3x3 with 64 feature maps, 2x2 Maxpooling, DropOut, a convolutional layer with 3x3 with 64 feature maps, a convolutional layer with 3x3 with 64 feature maps, 2x2 Maxpooling, DropOut, a fully connected (FC) layer of 512 hidden units, and a softmax output layer with 10 units. Full-precision ISRLU was used. Convolutional neural networks with ISRLUs (α = 1.0, α = 3.0) and ELUs (α = 1.0) were trained on the MNIST digit classification dataset while each hidden units activation was tracked. The network was trained for 20 epochs by using ADAM optimizer with learning rate 0.003 exponentially decreasing to 0.0001 and mini-batches of size 100. The weights have been initialized to truncated normal with standard deviation 0.1. We did not expect significant differences in accuracy in ISRLU and ELU in this test of shallow networks due to the similar nature of the curves. The cross-entropy loss was reasonable, at between 2 and 3.2 for all activation functions. Future testing will be done on deeper networks where we expect larger advantages that are similar to ELU BID5 BID14. The work with ISRLU in this paper suggests that the inverse square root unit (ISRU) may be useful for a variety of neural networks. ISRUs are defined as: DISPLAYFORM0 In RNNs that use LSTM BID9 and GRU BID4, the most common activation functions are sigmoid and tanh. We assert that ISRUs can be more efficient calculation than tanh and be more efficient than sigmoid when properly shifted and scaled. As shown above in TAB1, the inverse square root is 3x to 6x faster than tanh (depending on x86 architecture). ISRUs will be an area of our future research. Activation function performance is becoming more important overall in convolutional neural networks (CNNs) because of the trending reductions in the computational complexity of the convolutions used in CNNs. We have introduced a new activation function, the inverse square root linear unit (ISRLU) for faster and precise learning in deep convolutional neural networks. ISRLUs have similar activation curves to ELUs, including the negative values. This decreases the forward propagated variation and brings the mean activations to zero. Mean activations close to zero decreases the bias shift for units in the next layer which speeds up learning by bringing the natural gradient closer to the unit natural gradient. Future work may prove the effectiveness of applying ISRLUs and the related ISRUs to other network architectures, such as recurrent neural networks, and to other tasks, such as object detection. ISRLUs have lower computational complexity than ELUs. Even greater savings on computation can be realized by implementing ISRLUs in custom hardware implementations. We expect ISRLU activations to increase the training efficiency of convolutional networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We introduce the ISRLU activation function which is continuously differentiable and faster than ELU. The related ISRU replaces tanh & sigmoid.

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.