Batch Normalization will disrupt multi-threading?
My experience with batch and layer normalisation is that it is around 3 to 5 times slower per step than without it. Although the model may converge in less number of steps (which is what the authors report in the article) it takes much longer to calculate each step, and hence the whole training takes longer.

This has been reported in a few places like: 

https://www.reddit.com/r/MachineLearning/comments/4ufmxy/layer_normalization_implemented_in_tensorflow/

https://github.com/cooijmanstim/recurrent-batch-normalization/issues/2

https://github.com/keras-team/keras/issues/1309