Paper: What's the difference between Layer Normalization, Recurrent Batch Normalization (2016), and Batch Normalized RNN (2015)?
Layer normalization (Ba 2016): Does not use batch statistics. Normalize using the statistics collected from all units within a layer of the current sample. Does not work well with ConvNets.
Recurrent Batch Normalization (BN) (Cooijmans, 2016; also proposed concurrently by Qianli Liao &amp; Tomaso Poggio, but tested on Recurrent ConvNets, instead of RNN/LSTM): Same as batch normalization. Use different normalization statistics for each time step. You need to store a set of mean and standard deviation for each time step.
Batch Normalized Recurrent Neural Networks (Laurent, 2015): batch normalization is only applied between the input and hidden state, but not between hidden states. i.e., normalization is not applied over time.
Streaming Normalization (Liao et al. 2016) : it summarizes existing normalizations and overcomes most issues mentioned above. It works well with ConvNets, recurrent learning and online learning (i.e., small mini-batch or one sample at a time): 
Weight Normalization (Salimans and Kingma 2016): whenever a weight is used, it is divided by its $L2$ norm first, such that the resulting weight has $L2$ norm $1$. That is, output $y = x*(w/|w|)$, where $x$ and $w$ denote the input and weight respectively. A scalar scaling factor $g$ is then multiplied to the output $y = y*g$. But in my experience $g$ seems not essential for performance (also downstream learnable layers can learn this anyway). 
Cosine Normalization (Luo et al. 2017): weight normalization is very similar to cosine normalization, where the same $L2$ normalization is applied to both weight and input: $y = (x/|x|)*(w/|w|)$. Again, manual or automatic differentiation can compute appropriate gradients of $x$ and $w$. 


Note that both Weight and Cosine Normalization have been extensively used (called normalized dot product) in the 2000s in a class of ConvNets called HMAX (Riesenhuber 1999) to model biological vision. You may find them interesting.

Ref: The HMAX Model Reference

Ref: Cortical Network Simulator Reference

Ref: Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks,  Luo Chunjie, Zhan jianfeng, Wang lei, Yang Qiang