Batch Normalization with CUDNN
In general, you perform batch normalization before the activation.  The entire point of the scaling/bias parameters ($\beta$ and $\gamma$) in the original paper is to scale the normalized value ($\hat{x}$ in the paper) so that it fully captures the dynamic range of the activation operator. 

For example (and this is the example used in the paper), suppose the normalized output of the layer in question (convolution or dense) had unit variance (pointwise within the matrix). Then if you applied the sigmoid function pointwise as activation, you would be constrained to be within the linear region of sigmoid function, hence negating any non-linear properties that are supposedly crucial to making the network work.  

However, I'm sure you can find an architecture where the order is switched. In the end, people will do what gets the best results empirically for their use-case.

As far as the updating the parameters go (your fourth question), he does update the parameters. I took a look at the source. Parameter updates occur in the "update" functions associated with the convolutional layer. I.e. he only packages batch norm with other layers, which is a reasonable optimization to make.

For questions 2&amp;3: You are correct except for the following caveat: if you are only using dense layers in the head of a CNN, you don't need to suddenly switch strategies. You would use the MODE_PER_ACTIVATION when you expect each dimension of the feature map to have a different distribution. Think non-vision use cases where the input map might be different types of features. If you are using dense layers in a CNN, you would still still assume the normalization is the same across the "spatial" scale. After you flatten in the final layer, you essentially flatten into $C$ channels with 1x1 spatial dimension. So your "spatial" Batch Norm will have $C$ parameters anyways.  

For question 5: No. Think of the BatchNorm as a learned re-scaling and a bias operation together. You wouldn't expect the output of your neural network to be consistent if you trained it with bias then removed the biases at inference time.