An Investigation into the Stochasticity of Batch Whitening
Lei Huang Lei Zhao Yi Zhou Fan Zhu Li Liu Ling Shao
cs.CV cs.LG

	
	


	





	
	An Investigation into the Stochasticity of Batch Whitening
    Lei Huang^†   Lei Zhao   Yi Zhou^†   Fan Zhu^†   Li Liu^†   Ling Shao^†

	
	^†Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi, UAE

	
	
	
	
	
		{lei.huang, yi.zhou,  fan.zhu, li.liu, ling.shao}@inceptioniai.org   bhneo@126.com
	

    
==========================================================================================================================================================================================================================================================




    Batch Normalization (BN) is extensively employed in various network architectures by performing standardization within mini-batches.
     A full understanding of the process has been a central target in the deep learning communities. 
    Unlike existing works, which usually only analyze the standardization operation, this paper investigates the more general Batch Whitening (BW). Our work originates from the observation that while various whitening transformations equivalently improve the conditioning, they show significantly different behaviors in discriminative scenarios and training Generative Adversarial Networks (GANs). 
    We attribute this phenomenon to the stochasticity that BW introduces.
    We quantitatively  investigate the stochasticity of  different whitening transformations and show that it correlates well with the optimization behaviors during training. 
    We also investigate how stochasticity relates to the estimation of population statistics during inference.
    Based on our analysis, we provide a framework for designing and comparing  BW algorithms in different scenarios. 
    Our proposed BW algorithm improves the residual networks by a significant margin on ImageNet classification.
     Besides, we show that the stochasticity of  BW  can improve the GAN's performance with, however, the sacrifice of the training stability. 








§ INTRODUCTION



 Normalization techniques have been extensively used for learning algorithms during data preprocessing <cit.>.
 It has been shown  that centering, scaling and decorrelating the inputs  speeds up the training <cit.>.
 Furthermore, whitening the input that combines all above operations, improves the conditioning of the Hessian, making the gradient descent updates similar to the Newton updates <cit.>.

 Batch Normalization (BN) <cit.> extends the idea of normalizing the input into the activations of intermediate layers of Deep Neural Networks (DNNs), and represents a milestone technique in the deep learning community <cit.>.
 BN standardizes the activations by executing centering and scaling within a mini-batch of data, facilitating the optimization <cit.> and  generalization <cit.>. Batch Whitening (BW) further extends the scope of BN by removing the correlation of standardized activations <cit.>. It has been shown to improve the performance in discriminative scenarios <cit.> and Generative Adversarial Networks (GAN) <cit.>. 
 
 
 
 
 Despite BW's theoretical support in improving conditioning, there remains some intriguing observations relating to BW that are not well explored. Firstly, while various whitening transformations can equivalently improve the conditioning <cit.>, they show significant differences in performance: 1)  Principal Component
 Analysis (PCA) whitening hardly converges while Zero-phase Component
 Analysis (ZCA) whitening works well in discriminative scenarios <cit.>; 2) Cholesky Decomposition (CD) whitening achieves significantly better performance than ZCA in GAN training, while it has slightly worse performance in discriminative cases <cit.>. Secondly, while group based whitening—where features are divided into groups and whitening is performed within each one—is essential in discriminative scenarios <cit.>, full feature whitening has been shown to achieve better performance in training GANs <cit.>. 
 

 
 This paper focuses on explaining the above observations of BW. 
 We find that the stochasticity introduced by normalization over batch data (Figure <ref>) can be key to uncovering the  intriguing phenomena of BW. We quantitatively investigate the magnitude of stochasticity in different whitening transformations, using the evaluation method called Stochastic Normalization Disturbance (SND) <cit.>.
  By doing so, we demonstrate that PCA whitening has significantly larger stochasticity, which is difficult to be controlled by either increasing batch size or using group based whitening. On the other side, ZCA whitening has the smallest stochasticity, while CD whitening has a moderate value, and more importantly, their stochasticity can be well controlled. This suggests ZCA whitening should have better optimization behaviors in training, while PCA whitening has problems in converging, due to the increased stochasticity which slows down   the progress of the optimization <cit.>. 
 
  We also investigate the stochasticity during inference, which is caused by the estimation of population statistics averaged over the mini-batch statistics during training. We show that in terms of estimating the population statistics of the whitening matrix, it is more stable to use the mini-batch covariance matrix indirectly (We calculate the whitening matrix after training) than the mini-batch whitening matrix directly. We further provide an empirical investigation to understand the reasons behind this observation, and find that the stochastic sequences of the mini-batch whitening matrix have a large diversity than the covariance matrix.
  
  
  
   Based on the above analyses, we provide a general framework for designing and comparing BW algorithms in different scenarios. We design new BW algorithm and apply them to the residual networks <cit.> for ImageNet dataset   <cit.>, significantly improving the performance over the original one. 
 We further conduct thorough experiments on training GANs. We show that full feature whitening, when combined with coloring,  can improve the final score of evaluation. However, it reduces the training stability and is more sensitive to the hyper-parameters configurations. We attribute this phenomenon to two main effects caused by the introduced stochasticity of BW: 1) Strong stochasticity can increase the diversity of generated images and thus improve the GAN's performance; 2) At the same time, high stochasticity harms optimization and thus is more sensitive to the hyper-parameters. We argue that controlling the magnitude of whitening (stochasticity) is also important in training GANs and we validate this argument with our experiments. 
 
 

 
 



§ RELATED WORK





The previous analyses on BN mainly focus on the optimization. One main argument is that BN can improve the conditioning of the optimization problem. This argument was initially introduced in the BN paper <cit.> and further refined in <cit.>,  showing that BN leads to  
 a smoother landscape of the optimization problem under certain assumptions. Ghorbani <cit.> investigated this explanation by computing the spectrum of the Hessian for a large-scale dataset. It is believed that the improved conditioning enables large learning rates, thus improving the generalization, as shown in <cit.>.
 Another argument is that BN can adaptively adjust the learning rate <cit.> due to its scale invariance <cit.>. This effect  has been  further discussed in combination with weight decay <cit.>. Other works have included an investigation into  the signal propagation and gradient back-propagation <cit.>. Different from these approaches, our work focuses on  analyzing the stochasticity of  whitening over batch data. 
 
 The stochasticity introduced by normalization over batch data was first mentioned in the BN paper <cit.>,  and further explored in <cit.> from the perspective of Bayesian optimization. This stochasticity results in differences between the training distribution (using mini-batch statistics) and the test distribution (using estimated population statistics) <cit.>, which is believed to be the main cause of the small-batch-problem of BN <cit.>. To address this issue, a number of  approaches have been proposed <cit.>.
 Furthermore, it has been observed that BN also encounters difficulties in
 optimization during training <cit.>. This phenomenon is explored by the stochastic analysis shown in <cit.>.  
 Different from the above research which focuses on standardization, we analyze, for the first time, the stochasticity on batch whitening. We propose that analyzing whitening rather than standardization, has several advantages in understanding the behaviors of normalization over batch data: 1) There are an infinite number of whitening transformations and the main ones show significant differences as discussed in Section <ref>; 2) The extent of the whitening (stochasticity) can be well controlled by the batch and group size, which provides more information in designing experiments. 
 



Our work is related to the previously proposed whitening methods regarding the activation of DNNs. One approach is to consider the whitening matrix as model parameters to be estimated using full data <cit.>. This kind of whitening has also been exploited in image style transformation tasks <cit.>. 
Another line of research is batch whitening, which is what this paper  discusses. This approach treats the normalization as a function over a mini-batch input. The main works include PCA whitening, ZCA whitening <cit.> and its approximation ItN <cit.>,  and CD whitening <cit.>. Pan <cit.> propose switchable whitening to learn different batch/instance whitening/standardization operations in DNNs. However, they used only the ZCA whitening transformation. Our work aims to understand different whitening transformation behaviors based on stochastic analysis.  





§ STOCHASTICITY ANALYSIS OF BATCH WHITENING




Let 𝐗∈𝐑^d × m be a data matrix denoting the mini-batch input  of size m in a certain layer. For simplifying the discussion, we assume that the data is centered, by performing : = 	𝐗 - μ·1^T where μ = 1/m𝐗·1 is the mean of 𝐗, and  1 is a column vector of all ones.

 Whitening performs normalization
 over the mini-batch input as follows:
 
 	 
 	
 
	
    = ,

 
where  is the mini-batch whitening matrix that is derived from the corresponding covariance matrix Σ = 1/m𝐗𝐗^T.
 
 The population statistics of the  whitening matrix  used for inference,  is usually calculated by running average over the mini-batches as follows:
 
 
  	 
  	
 	
    =    (1-λ)  + λ.

 
It is clear that both the whitened output  (Eqn. <ref>) and the population statistics  (Eqn. <ref>) can be viewed as stochastic variables,  because they  depend on the mini-batch inputs which are sampled over datasets. For illustration, we defer the analysis of the stochasticity to Sections <ref> and <ref>, and first provide a review of the whitening transformations.






 §.§ Whitening Transformations



There are an infinite number of possible whitening matrices, as shown in <cit.>, since any whitened data with a rotation is still whitened. This paper focuses on the whitening transformations based on PCA, ZCA and CD, since these three transformations have shown significant differences in performance when used in training DNNs <cit.>.
Note that BN <cit.> can be viewed as a special case of batch whitening, since it performs standardization without removing correlations, where BN= (())^-1/2 with (·) setting the off-diagonal elements of a matrix to zeros. 
To simplify the description, this paper regards BN as a (reduced) whitening transformation, unless otherwise stated.





  
PCA Whitening
 uses PCA=Λ^-1/2𝐃^T, where  Λ=(σ_1, …,σ_d) and 𝐃=[𝐝_1, ...,
𝐝_d] are the eigenvalues and associated eigenvectors of Σ, Σ = 𝐃Λ𝐃^T. Under this transformation, the variables are first rotated by the eigen-matrix (𝐃) of the covariance,  then  scaled by the square root inverse of the eigenvalues (Λ^-1/2). PCA whitening over batch data suffers significant instability in training DNNs, and hardly converges, due to the so called Stochastic Axis Swapping (SAS), as explained in <cit.>.





  
ZCA Whitening
 uses ZCA=𝐃Λ^-1/2𝐃^T, where the  PCA whitened input is rotated back by the corresponding rotation matrix 𝐃. ZCA whitening works by stretching/squeezing  the dimensions along the eigenvectors. It has been shown that ZCA whitening avoids the SAS and achieves better performance over standardization (used in BN) on discriminative classification tasks  <cit.>.






  
CD Whitening
 uses CD=𝐋^-1 where 𝐋 is a lower triangular matrix from the Cholesky decomposition, with 𝐋𝐋^T=. This kind of whitening works by recursively decorrelating the current dimension over the previous decorrelated ones, resulting in a triangular form of its whitening matrix. CD whitening has been shown to achieve the state-of-the-art performance in training GANs, while ZCA whitening has degenerated performance. 

The primary motivation of this paper is to investigate the following problem: while all whitening methods equivalently improve the conditioning of the layer input, why do they show significantly different behaviors in training DNNs?
In the following sections, we provide a unified analysis and demonstrate that the key is the stochasticity introduced by normalization over batch data.






 §.§ Stochasticity During Training










Given a sample 𝐱∈ℝ^d from a distribution P_χ, we take a sample set 𝐗^B={𝐱_1,...,𝐱_B, 𝐱_i ∼ P_χ} with a size of B. The whitened output 𝐱̂ can be formulated as 𝐱̂=( 𝐗^B ;𝐱).
For a certain 𝐱,  𝐗^B can be viewed as a random variable <cit.>. 𝐱̂ is thus another random variable showing stochasticity. Here, we investigate the stochasticity effects of different whitening transformations.

To provide a more intuitive illustration, we conduct a toy experiment to show how the normalized output of one sample  changes when combined with a different sample set 𝐗^B, by performing different whitening methods. Figure <ref> (a), (b), (c) and (d) show the results when performing BN, PCA, ZCA and CD whitening, respectively.
 It is clear that the distribution of the PCA whitened output of one sample is very sparse, which means that 𝐱̂ has significant diversity.
 This suggests that PCA whitening provides large stochasticity. On the other side, the BN standardized output shows a tight Gaussian-style distribution, which suggests that BN has smaller stochasticity.
 Note that BN can not guarantee that the normalized output has an identity covariance matrix, while other whitening methods can.
 Similarly, we also observe that ZCA whitening provides reduced stochasticity, compared to CD.  In fact, ZCA whitening has been shown to minimize the total squared distance between the original and whitened variables <cit.>.
 We conjecture that such a property results in ZCA whitening having smaller stochasticity than CD.





  §.§.§ Quantitative Analysis



To provide a qualitative comparison, we exploit the evaluation called Stochastic Normalization Disturbance (SND),  introduced in <cit.>.
 The empirical estimation of SND for sample 𝐱 over the normalization (·) is defined as:
 
 	 	 
 	 	
 	
    Δ_(𝐱)=1/s∑_i=1^s( 𝐗^B_i ; 𝐱) - 1/s∑_j=1^s ( 𝐗^B_j ; 𝐱)     ,

 
where s denotes the number of mini-batches {𝐗^B_j}_j=1^s that are randomly sampled from the dataset. SND can be used  to evaluate the stochasticity of  a sample after the normalization operation <cit.>.
Further, a normalization operation G(·)'s empirical SND is defined as  Δ_G=1/N∑_i=1 ^NΔ(𝐱_i) given N samples.  Δ_G describes the magnitudes  of stochasticity for the corresponding normalization operations.


Here, we conduct experiments to quantitatively evaluate the effects of different normalization methods. Noticeably, the stochasticity is related to the batch size m and the dimension d.
Figure <ref> (a) shows the SND of different normalization methods with respect to the dimensions, when fixing the batch size to 1024. We find that PCA whitening shows the largest SND while BN the smallest, over all the dimensions, which is consistent with the observation shown in Figure <ref>. We notice that all whitening methods have an increased SND when the dimension increases. Besides, ZCA has a smaller SND than CD, over all the dimensions, which is also consistent with the data shown in Figure <ref>.
 Figure <ref> (b) shows the SND of different normalization methods with respect to the batch size, when fixing the dimension to 128.
An interesting observation is that PCA whitening has nearly the same large SND among different batch sizes. This suggests that the PCA whitening is extremely unstable, no matter how accurate the estimation of the mini-batch covariance matrix is. This effect is in accordance with the explanation of Stochastic Axis Swapping (SAS) shown in <cit.>, where a small change over the examples (when performing PCA whitening) results in a large change of representation.

To further investigate how this stochasticity affects DNN training, we perform experiments on a four-layer Multilayer Perceptron (MLP) with 256 neurons in each layer. We evaluate the training loss with respect to the epochs, and show the results in Figure <ref> (a). We find that, among all the whitening methods, ZCA works the best, while PCA is the worst. We argue that this correlates closely with the SND they produce. Apparently, the increased stochasticity can slow
down training, even though all the whitening methods have equivalently improved conditioning.
 An interesting observation is that, in this case, BN works better than ZCA whitening. This is surprising since ZCA has improved conditioning over BN by removing the correlation <cit.>, and it should theoretically have a better optimization behavior. However, the amplified stochasticity of ZCA whitening mitigates this advantage in optimization, thus resulting in a degenerated performance. Therefore,   from an  optimization perspective, we should control the extent of the stochasticity.

 
 
 
 



  §.§.§ Controlling the Stochasticity by Groups




Huang <cit.> proposed  to use groups to control the extent of whitening.

They argue that this method reduces the inaccuracy in estimating the full covariance matrix when the batch size is not sufficiently large. Here, we empirically show how group based whitening affects the SND, providing a good trade-off between introduced stochasticity and improved conditioning. This is essential for achieving a better optimization behavior.

We evaluate the SND of different whitening transformations by varying the group size ranging from 2 to 512, as shown in Figure <ref> (a). We also display the spectrum of the covariance matrix of the whitened  output (based on groups) in Figure <ref> (b). We find that the group size effectively controls the SND of the ZCA/CD whitening.
With decreasing group size, ZCA and CD show reduced stochasticity  (Figure <ref> (a)), while having also degenerated conditioning (Figure <ref> (b)), since the output is only partially whitened.  Besides, we observe that  PCA whitening still has a  large SND over all group sizes, and with no significant differences. This observation further corroborates the explanation of SAS given in <cit.>, , that the PCA whitening is extremely unstable.

We also show the SND of the approximate ZCA whitening method (called ItN <cit.>) in Figure <ref> (a), which uses Newton's iteration to approximately calculate the whitening matrix. We denote `ItN5' as the ItN method with an iteration number of 5.
An interesting observation is that ItN has smaller SND than BN, when using a large group size (, 256) with a smaller iteration (, T=5). This suggests that we can further combine group size and iteration number to control the stochasticity for ItN, providing an efficient and stable solution to approximate ZCA whitening <cit.>.
The above observations furnish substantial insights into the application of whitening  in neural networks (especially in scenarios that requires decorating the activation in a certain layer), and we will further elaborate on this in Section <ref>.

We also use group-based ZCA/CD whitening methods on the four-layer MLP experiments. The results are shown in Figure <ref> (b).  We observe that ZCA and CD whitening, with a group size of 16 to control the stochasticity, achieve better training behaviors than BN.




	



 §.§ Stochasticity During Inference

	


In the previous section we have shown that performing different whitening transformations,  by introducing different magnitudes of stochasticity, results in significantly different training behaviors from an optimization perspective. It's clear that such a stochasticity will also affect the final test performance during inference, because the population statistics  is estimated by running average (Eqn. <ref>) over the stochastic sequences {t}_t=1^T, where T is the total number of training iterations.
The more diverse of the stochastic sequence  {t}_t=1^T, the more difficult to accurately estimate . Rather than estimating  directly, Siarohin <cit.> proposed to first estimate the population statistic of the covariance matrix , then compute  based on  after training.  However, no further analysis was provided to explain why they do like this.

Here, we provide an empirical investigation on how the estimation object (/) in Eqn. <ref> affects the test performance. Let  and  denote the method to estimate  by  and , respectively.
 We conduct experiments on MLP with variations in width (the number of neurons in each layer) and batch size, for MNIST classification.
 

Figure <ref> (a) and (b) show the results of ZCA and CD whitening, respectively.

We find that  has better performance than  , especially under the scenarios with large width and small batch size (Intuitively, estimating in high-dimensional space with a small batch size will make the estimation noisier).  This suggests that using  to estimate   indirectly  is more stable than using  directly. Another interesting observation is that, by comparing Figure <ref> (a) and (b), the differences between the estimation methods using CD whitening is smaller than the ZCA whitening.







We further analyze how the diversity of stochastic sequences {𝐌_t}_t=1^T  affect the  estimation of , where 𝐌∈{,  }.
Intuitively, if the stochastic sequence has high diversity, the effects of estimation will be worse.
 We view each element of  as an independent stochastic variable and calculate the standard deviation of each element during training, as follows:
  
  	 	 
  	 	
  	
    δ(𝐌^ij) = √(1/T∑_t=1^T (𝐌_t^ij - 1/T∑_t'=1^T(𝐌_t'^ij) )^2 ),

  
where ^ij (𝐌_t^ij)  indicates the (i,j)-th element of  (𝐌_t).
Furthermore, we calculate the normalized standard deviation of each element δ̃ (𝐌^ij), which, as defined in Eqn. <ref>, is calculated over  𝐌_t^ij=𝐌_t^ij/ √(∑_t=1^T (𝐌_t^ij)^2). Figure <ref> (a) and (b) show the histogram of δ(𝐌)  and δ̃ (𝐌), respectively, when using ZCA whitening. We clearly find that  has a larger standard deviation on average, thus a large diversity in general, compared to . This reveals why using  is more stable for estimating   than using .









§ EVALUATION ON VISION TASKS

	



Based on the previous analysis, we can design new BW algorithms, and construct more effective DNNs by using BW. 
We investigate this in classification and training GANs. The code to reproduce the experiments is available at https://github.com/huangleiBuaa/StochasticityBW.
	



 §.§ The Landscape of Batch Whitening Algorithms


	

We provide a general view of batch whitening algorithms in Algorithm <ref> for vectorial inputs 𝐗∈ℝ^ d × m. Back-propagation is necessary to pass through the whitening transformation, and we provide the details in Appendix <ref> for completeness.   For feature map inputs 𝐗_F ∈ℝ^h × w × d
	× m, where h and w indicate the height and width, the whitening transformation is performed  over the unrolled 𝐗∈ℝ^ d × (m h w),

 since each spatial position of the
feature map can be viewed as a sample <cit.>.

Note that an extra step to recover the representation capacity of normalization is given in Line 11 of Algorithm <ref>, which is shown to be empirically effective in <cit.>.
There are two alternatives for ϕ_3: One is a dimension-wise scale and shift <cit.>: 𝐘_k = γ_k k + β_k, (k=1,...,d); The other is a  coloring transformation: 𝐘=𝐖 + 𝐛, which was proposed in <cit.> to achieve better performance in training GANs.
By combining whitening transformation ϕ_1, estimation object ϕ_2 and recovery operation ϕ_3, we can design different BW algorithms. Table <ref> shows the scope of value for different components this paper discusses. Note that ItN <cit.> is the efficient and numerical stable approximation of ZCA whitening. We use the `Scale & Shift' operation in ϕ_3 for all algorithm, unless otherwise stated.









	



 §.§ Investigation on Discriminative Scenarios



In the section, we first investigate how different whitening transformations affect the training of VGG network  <cit.> for CIFAR-10 datasets   <cit.>, from the  perspective of optimization.
We then show the performance on large-scale ImageNet classification   <cit.>, by applying the designed BW algorithm on residual networks <cit.>.







	



  §.§.§ VGG on CIFAR-10

	



We use the VGG networks <cit.> tailored for 32× 32 inputs (16 convolutional layers and 1 fully-connected layer).




We add  the normalization methods after each convolutional layer in the VGGs.
We compare several methods including: 1) The full whitening methods `PCA', `ZCA' and `CD'; 2) The approximate whitening `ItN' and the standardization `BN'; 3) Group-based whitening methods with the group size ranging in {512, 256, 128, 64, 32, 16}. We denote `ZCA-16' as ZCA whitening with group size 16. We focus on comparing the training performance from an optimization perspective, and we use mini-batch covariance matrix `' as the estimation object for all methods during inference. 
We use SGD with a batch size of 256 to optimize the model. We set the initial learning rate to 0.1, then divide it by 5 after  60 epochs and finish the training at 120 epochs.






The main experimental observations include: 1) `PCA' fails training under all the configurations, which means that either the training loss does not decrease  or numerical instability appears. This observation is consistent with the previous MLP models. 2) `ZCA-16' can train well, while other `ZCA' related configurations fail training due to numerical instability. This is caused by the back-propagation through the eigen decomposition, which requires different eigenvalues for the mini-batch covariance matrix <cit.>. 3) `CD' has no numerical instability and can ensure full feature whitening of the models. Fully whitening features by `CD' have significantly worse performance than the group-based ones shown in Figure <ref>. This again suggests that it is essential to control the extent of the whitening for discriminative models. 4) We find that `ItN' (the approximates of the ZCA whitening) works the best.















	



  §.§.§ Residual Network on ImageNet

	




We investigate the effectiveness of all kinds of whitening algorithms on residual networks for ImageNet classification with 1000 classes  <cit.>.
We use the given official 1.28M training images as a training set, and evaluate the top-1 accuracy on the validation
set with 50k images.




  
Ablation Study on ResNet-18 We first execute an ablation study on the 18-layer residual network (ResNet-18) to explore multiple positions for replacing BN with BW. We consider three architectures:
 1) ARC_A:  where we only replace the first BN module of ResNet-18  proposed in <cit.>;
 2) ARC_B: where we further plug-in the BW layer after the last average pooling (before the last linear layer) to learn the decorrelated feature representations based on ARC_A, as proposed in <cit.>;
 3) ARC_C: where we also replace the {2n, n=1,2,...}th BN modules (the {3n}th for ResNet-50) based on ARC_B.
 
 We compare all the whitening transformations and estimation objects in Table <ref>.  `ZCA' and `ZCA_Σ' denote the ZCA whitening that estimate the population statistics using  and , respectively.
 For `PCA', `ZCA' and `CD', we use a group size in {16, 64 } and report the best performance from these two configurations.



We follow the same experimental setup as described in  <cit.>, except that we use one GPU and train over 100 epochs. We apply SGD with a mini-batch size of 256, momentum of 0.9 and weight decay of 0.0001. The initial learning rate is set to 0.1 and divided by 10 at 30, 60  and 90 epochs.

The results are shown in Table <ref>. For ARC_A, we find that all whitening methods, except PCA related methods, improve the performance over the baselines. We observe that ZCA (ZCA_Σ) and its approximate ItN (ItN_Σ), achieve slightly better performance than CD (CD_Σ), and this observation is consistent with the results in <cit.>. This suggests ZCA  whitening, which minimizes the distortion introduced by whitening under the L2 distance <cit.>, usually works better than other whitening methods for discriminate classification tasks.
Under ARC_B and ARC_C, ZCA/PCA related methods suffer from numerical instability. We also observe that CD-related methods have significantly degenerated performance under ARC_C. This implies that the  stochasticity introduced by decorrelating multiple layers with CD whitening harms the learning. We find that ItN-related methods can effectively control the stochasticity and achieve further performance improvement on ARC_C.  We try a ResNet-18 where all BN layers are replaced with ItN. However, the network has no performance improvement over ARC_A, while the  computations introduced are significant, as already observed in <cit.>.  These results demonstrate that controlling the extent of whitening (stochasticity) is important for achieving performance improvements over the standardization.

From all the architectures and whitening methods, we observe that using Σ to estimate the population statistics is better than using , especially on ARC_C.





  
Results on ResNet-50 Based on the above observation, we further apply the ItN_Σ to ResNet-50. In addition to the standard step learning rate decay <cit.> used in the previous setup, we also consider the cosine learning rate decay <cit.> that is also a basic setting <cit.> when training on ImageNet, and we want to address that the improvement of the proposed method can be obtained under different setups. For cosine decay, we start with a learning rate of 0.1 and decay it to 0.00001 over 100 epochs. 
We find that the proposed models significantly improves the performance over the original one, under all configurations. The additional time cost of `ItN_Σ-ARC_B' and `ItN_Σ-ARC_C' is 7.03% and 30.04% over the original. 
Note that ItN_Σ consistently provides a slight improvement (around 0.1 to 0.4 for all configurations in this experiment) over the original ItN proposed in <cit.>, and please see Appendix <ref> for details. Moreover, ItN_Σ can bear a larger group size and smaller batch size than ItN, which is particularly important in the scenario of training GANs, as we will discuss. 





























	



 §.§ Investigation on Training GANs

	

In this section, we provide an empirical analysis on BW algorithms for training GANs. We focus on investigating the effects of different BW algorithms in stabilizing the training and improving  the performance. We evaluate on unconditional image generation for the CIFAR-10 dataset. In our qualitative evaluation of the generated samples, we use the two most common metrics: Fréchet Inception Distance (FID) <cit.> (the lower the better) and Inception Score (IS) <cit.> (the higher the better). Following the setup in <cit.>, we only apply the BW on the generator. Unless otherwise noted, the batch whitening methods used in this section use the covariance matrix Σ to estimate the whitening matrix .







	



  §.§.§ Stability Experiments

	


Following the analysis in <cit.>, we conduct experiments to demonstrate the effects of BW algorithms in stabilizing the training of GANs. We use  DCGAN <cit.> architectures and use the hinge loss <cit.>.   We provide the implementation details in the Appendix <ref>.

 We use the Adam optimizer <cit.> and train for 100 epochs.
We consider 15 hyper-parameter settings (See Appendix <ref> for details) by varying the learning rate α, first and second momentum   (β_1, β_2) of Adam, and number of discriminator updates per generator update n_dis.
We use both the `Scale&Shift' (denoted as `-S' ) and `Coloring' (denoted as `-C' ) to recover the representation of BW.
 We calculate the FID distribution of the trained models under 15 configurations, and the results are shown in Figure <ref>. The lower the variance, the more stable the model from an optimization perspective.
 In Table <ref>,
 we also provide the best FID and the corresponding IS from all the configurations  for different whitening methods. Note that full ZCA whitening also suffers from the numerical instability in this experiment, and we thus use the group-based ZCA whitening.

 We observe that `CD' whitening combined with coloring transformation (CD-C) achieves the best performances. However, it has the worst stability over all whitening methods.  We argue that full feature whitening  introduces strong stochasticity, which benefits the diversity of the generated examples. However, the strong stochasticity also harms the training and makes the model sensitive to  the hyperparameter. We find that the coloring operation can improve the performances, especially for the whitening transformations (, BN benefits less). Nevertheless, it also makes the training unstable and sensitive to  the hyperparameter. Besides, we observe that `ZCA-64' achieves better performance than 'CD-64`. This suggests that the advantage of ZCA whitening over CD in discriminative scenarios still exists when training GANS.
 
  We argue that controlling the magnitude of whitening (stochasticity) is also still important in training GANs. 
  
   For example, for ItN, the approximation of ZCA whitening has nearly the same performance as CD (Table <ref>), and shows better stability (Figure <ref>) due to its effective control of the stochasticity, as explained in <cit.>.


 We also have obtained similar results when using the non-saturated loss <cit.>, as shown in the Appendix <ref>.
 


 

  
Comparison of Estimation Object 
  Considering that the CD whitening in <cit.> uses the covariance matrix to indirectly estimate the whitening matrix  when training a GAN,  we also compare the whitening methods with different estimation objects. The results are shown in Figure <ref>. We observe that the methods using the covariance matrix as the estimation object,  achieve consistently better FID scores and are generally more stable than using  the whitening matrix. 
 
  




	



  §.§.§ Validation on Larger Architectures


	


Based on our previous analysis, we observe that ItN and ZCA whitening (with an appropriate group size)  have similar performance to CD in training GANs, when using the covariance matrix as the estimation object. We further apply the ZCA whitening and ItN to the DCGAN and Resnet models, where CD whitening achieves nearly the state-of-the-art performance on CIFAR-10. 
We use the code provided in <cit.> and use the same setup as in <cit.>. We replace all the CD whitening in the models with ItN/ZCA whitening. 
The results are shown in Table <ref>. 
We observe that the CD whitening has no significant advantages over ItN/ZCA-G64. Ultimately, ItN achieves the best performances in terms of the FID evaluation, while CD whitening is the best in terms of IS.   This suggests that there is still room to improve the performance of batch whitening in training GANs by further finely controlling the stochasticity.  





§ CONCLUSIONS

	

In this paper, we provided a stochasticity analysis of batch whitening that thoroughly explains why different whitening transformations show significant differences in performance when training DNNs, despite their equivalent improvement of the conditioning.  Our analysis provides insights for designing new normalization algorithms and constructing new network architectures. We believe that our analysis will open new avenues of research in better understanding normalization over batch data.    





Acknowledgement
We  thank Anna Hennig and Ying Hu for their help with proofreading.



ieee_fullname
	























§ BACK-PROPAGATION


Given the Batch Whitening (BW) algorithms described in Algorithm <ref> of Section <ref>, we provide the corresponding back-propagation during training in Algorithm <ref>. Note that it is easy to calculate  in Line. 3 of Algorithm <ref>, given the specific recovery operation shown in the  paper. Here, we mainly elaborate on the nontrivial solution of back-propagation through the whitening transformation, , how to calculate  given , shown in Line. 5 of  Algorithm <ref>.

We consider the following whitening transformations discussed in the paper: Principal Component
Analysis (PCA) whitening <cit.>, Zero-phase Component
Analysis (ZCA) whitening <cit.>, Cholesky Decomposition (CD) whitening <cit.> and ItN <cit.> that approximates ZCA whitening. The back-propagations of all whitening transformations are derived in their original papers <cit.>. Here, we provide the results for completeness.









 §.§ PCA Whitening

PCA Whitening uses PCA=Λ^-1/2𝐃^T, where  Λ=(σ_1, …,σ_d) and 𝐃=[𝐝_1, ...,
𝐝_d] are the eigenvalues and associated eigenvectors of Σ, Σ = 𝐃Λ𝐃^T. 















The backward pass of PCA whitening is:

    Λ   =(PCA) 𝐃 (-1/2Λ^-3/2 )
    𝐃   =  ( PCA)^T Λ^-1/2
    Σ   =𝐃{ (𝐊^T ⊙ (𝐃^T 𝐃 )) + (Λ)_diag}𝐃^T,

where  𝐊∈ℝ^d× d is 0-diagonal with
𝐊_ij= 1/σ_i  -σ_j [i≠ j],
the ⊙ operator is an element-wise matrix multiplication, and (Λ)_diag sets the off-diagonal elements of Λ to zero.



 §.§ ZCA Whitening

ZCA whitening uses ZCA=𝐃Λ^-1/2𝐃^T, where the  PCA whitened input is rotated back by the corresponding rotation matrix 𝐃.

The backward pass of ZCA whitening is:

    Λ   =𝐃^T (ZCA) 𝐃 (-1/2Λ^-3/2 )
    𝐃   =  (ZCA+  (ZCA)^T)  𝐃Λ^-1/2
    Σ   =𝐃{ (𝐊^T ⊙ (𝐃^T 𝐃 )) + (Λ)_diag}𝐃^T.









 §.§ CD Whitening

CD whitening uses CD=𝐋^-1, where 𝐋 is a lower triangular matrix from the Cholesky decomposition, with 𝐋𝐋^T=. 
The backward pass of CD whitening is:


    𝐋   = -  CD^T CDCD^T 
       =1/2𝐋^-T(P ⊙𝐋^T𝐋 +(P ⊙𝐋^T𝐋)^T ) 𝐋^-1.



 §.§ ItN Whitening

ItN, which approximates the ZCA whitening using Newton's iteration, is shown in Algorithm <ref>, where T is the iteration number and tr(·) denotes the trace of a matrix. 


Given ItN, the back-propagation is: 

	
    𝐏_T=   1/√(tr(Σ))ItN
    Σ_N=    -1/2∑_k=1^T (𝐏_k-1^3)^T 𝐏_k
    Σ =   1/tr(Σ)Σ_N
    	-1/(tr(Σ))^2 tr(Σ_N^T Σ)   𝐈
          -1/2(tr(Σ))^3/2 tr((ItN)^T 𝐏_T) 𝐈.


Here, 𝐏_k can be calculated by the following iterations:

	
	
    𝐏_k-1    =3/2∂L/∂𝐏_k
    	-1/2𝐏_k  (𝐏_k-1^2 Σ_N)^T
    	-1/2  (𝐏_k-1^2)^T  𝐏_kΣ_N^T
    	
       -  1/2(𝐏_k-1)^T 𝐏_k (𝐏_k-1Σ_N)^T
    	,    k=T,...,1.








§ STOCHASTICITY DURING INFERENCE


In Section <ref> of the  paper, we mentioned that we try other learning rates in the experiments to compare the estimation methods during inference. Here, we provide the results under a learning rate of 0.5, shown in Figure <ref>.










§ MORE DETAILS FOR EXPERIMENTS










 §.§ Classification on ImageNet


In Section <ref> of the  paper, we mentioned that ItN_Σ consistently provides a slight improvement over the original ItN proposed in <cit.> (using  as an estimation object). Here, we provide the results, including the original ItN, in Table <ref>. We use an iteration number of 5 for all ItN-related methods. 
For ItN, we use a group size of 64, as recommended in <cit.>. 
For ItN_Σ, we use a larger group size of 256, since ItN_Σ can still provide a good estimation of population statistics in high dimensions during inference, as shown in the  paper. 















 §.§ Training GANs





  §.§.§ Stability Experiments




  
Experimental Setup
The DCGAN architecture in these experiments is shown in Table <ref>. Note that spectral normalization <cit.> is applied on the discriminator, following the setup shown in <cit.>.
We evaluate the FID between the 10k test data and 10K randomly generated examples. 

We use the Adam optimizer <cit.> and train for 100 epochs.
We consider 15 hyper-parameter settings (Table <ref>) by varying the learning rate α, first and second momentum   (β_1, β_2) of Adam, and number of discriminator updates per generator update n_dis.
The learning rate is linearly decreased until it reaches 0. 



  
Results
In Figures <ref> and <ref> of the paper, we only show the results with FID ranging in (20, 100) for better representation. Here, we provide the corresponding FIDs with full range in Figure <ref> and <ref>. 

We perform experiments on DCGAN with the non-saturating loss <cit.>. Figure <ref> shows the results. Here, we obtain similar observations as for the DCGAN with the hinge loss, shown in Section <ref> of the paper. 












  §.§.§ Validation on Larger Architecture




  
Experimental Setup
The architectures in these experiments are shown in Table <ref> (DCGAN) and <ref> (ResNet). The experimental setup is as follows <cit.>:


We evaluate the FID between the 10k test data and 10K randomly generated examples. 
We use the Adam optimizer <cit.> with a  learning rate α=2e^-4. The learning rate is linearly decreased until it reaches 0.  We use  first momentum β_1=0
and second momentum β_2=0.9. 
For DCGAN, we use the number of discriminator updates per generator update n_dis=1 and train the model for 100 epochs, while for ResNet, we use n_dis=5 and train for 50 epochs. 








 §.§ Comparison in Running Time


The main disadvantage of whitening methods, compared to  standardization, is their expensive computational costs. 
Here, we provide the time costs of the trained models discussed in the paper. 

We note that the computational cost of the Singular Value Decomposition (SVD) used in ZCA/PCA whitening and the Cholesky Decomposition highly depends on the specific deep learning platform (, PyTorch <cit.>  or Tensorflow <cit.>). We thus conduct experiments both on PyTorch (version number: 1.0.1) and Tensorflow (version number: 1.13), with CUDA (version number: 10.0.130). 
All implementations of BW are based on the API provided by PyTorch and Tensorflow. 
We hope that our experiments can provide  good guidelines for applying BW in practice. 





  §.§.§ Classification Models Using PyTorch

Table <ref> shows the time costs of the VGG models for CIFAR-10 classification, described in Section <ref> of the  paper. Figure <ref> and Table <ref> show the time costs of the ResNet-18 and ResNet-50 described in Section <ref> of the paper, respectively. Note that we run ResNet-18 on one GPU, while ResNet-50 on two GPUs. 

Our main observations include: (1) ItN  is more computationally efficient. (2) CD whitening is slower than ZCA whitening, which is contrary to the observation in <cit.> where CD whitening was significantly more efficient than ZCA. The main reason for this is that CD whitening requires the Cholesky decomposition and matrix inverse, which are slower than SVD under the PyTorch platform. 
We note that under Tensorflow, which was used to implement CD whitening in <cit.>, CD whitening is more efficient than ZCA, as shown in Section <ref>. 
(3) Group based whitening with small groups requires more training time than larger groups. The main reason is our unoptimized implementation where we whiten each group sequentially instead of in parallel, because the corresponding  platform does not provide an easy way to use linear algebra library of CUDA in parallel, which is also mentioned in <cit.>. We believe BW would be more widely used if the group based methods could be easily implemented in parallel.  




























  §.§.§ GANs Using Tensorflow


Figure <ref> shows the time costs of the GAN models  described in Section <ref> of the paper. We perform  experiments with the BW operation running on both a GPU and a CPU, shown in Figure <ref> (a) and <ref> (b), respectively. We observe that ZCA whitening running on GPUs has a significantly more expensive computational cost than other methods, which is consistent with the observation in <cit.>. We find this can be alleviated  when running ZCA whitening on CPUs. 


Table <ref> shows the time cost of the GAN models  described in Section <ref> of the paper. 









































































