Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)? So, recently there's a Layer Normalization paper. There's also an implementation of it on Keras.

But I remember there are papers titled Recurrent Batch Normalization (Cooijmans, 2016) and Batch Normalized Recurrent Neural Networks (Laurent, 2015). What's the difference between those three?

There is this Related Work section I don't understand:


  Batch normalization has been previously extended to recurrent neural networks [Laurent et al., 2015, Amodei et al., 2015, Cooijmans et al., 2016]. The previous work [Cooijmans et al., 2016] suggests the best performance of recurrent batch normalization is obtained by keeping independent normalization statistics for each time-step. The authors show that initializing the gain parameter in the recurrent batch normalization layer to 0.1 makes significant difference in the final performance of the model. Our work is also related to weight normalization [Salimans and Kingma, 2016]. In weight normalization, instead of the variance, the L2 norm of the incoming weights is used to normalize the summed inputs to a neuron. Applying either weight normalization or batch normalization using expected statistics is equivalent to have a different parameterization of the original feed-forward neural network. Re-parameterization in the ReLU network was studied in the Pathnormalized SGD [Neyshabur et al., 2015]. Our proposed layer normalization method, however, is not a re-parameterization of the original neural network. The layer normalized model, thus, has different invariance properties than the other methods, that we will study in the following section
Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.
Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao &amp; Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.
Batch Normalized Recurrent Neural Networks (Laurent, 2015): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.
Streaming Normalization (Liao et al. 2016) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time): 
Weight Normalization (Salimans and Kingma 2016): whenever a weight is used, it is divided by its $L2$ norm first, such that the resulting weight has $L2$ norm $1$. That is, output $y = x*(w/|w|)$, where $x$ and $w$ denote the input and weight respectively. A scalar scaling factor $g$ is then multiplied to the output $y = y*g$. But in my experience $g$ seems not essential for performance (also downstream learnable layers can learn this anyway). 
Cosine Normalization (Luo et al. 2017): weight normalization is very similar to cosine normalization, where the same $L2$ normalization is applied to both weight and input: $y = (x/|x|)*(w/|w|)$. Again, manual or automatic differentiation can compute appropriate gradients of $x$ and $w$. 


Note that both Weight and Cosine Normalization have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.

Ref: The HMAX Model Reference

Ref: Cortical Network Simulator Reference

Ref: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,  Luo Chunjie, Zhan jianfeng, Wang lei, Yang Qiang