How to perform gradient checking in a neural network with batch normalization?
You should be able to do gradient checking as long as you fix the randomness by fixing the random seed, on python you might want to look at numpy.random.seed.

From http://cs231n.github.io/neural-networks-3/#ensemble :


  When performing gradient check, remember to turn off any non-deterministic effects in the network, such as dropout, random data augmentations, etc. Otherwise these can clearly introduce huge errors when estimating the numerical gradient. The downside of turning off these effects is that you wouldn’t be gradient checking them (e.g. it might be that dropout isn’t backpropagated correctly). Therefore, a better solution might be to force a particular random seed before evaluating both (f(x+h)) and (f(x-h)), and when evaluating the analytic gradient.