How to handle the BatchNorm layer when training fully convolutional networks by finetuning?
Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?

Some options:


delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)
Freeze the parameters and statistics of the BN layers
....


which is better and any demo for implementation in pytorch/tf/caffe?

Having only one element will make the batch normalization zero if epsilon is non-zero (variance is zero, mean will be same as input).
Its better to delete the BN layers from the network and try the activation function SELU (scaled exponential linear units). This is from the paper 'Self normalizing neural networks' (SNNs).

Quote from the paper:


  While batch normalization requires explicit normalization, neuron
  activations of SNNs automatically  converge  towards  zero  mean  and 
  unit  variance.   The  activation function of SNNs are “scaled
  exponential linear units” (SELUs), which induce self-normalizing
  properties.


The SELU is defined as:  

def selu(x, name="selu"):
  alpha = 1.6732632423543772848170429916717
  scale = 1.0507009873554804934193349852946
  return scale * tf.where(x &gt;= 0.0, x, alpha * tf.nn.elu(x))

