The Batch Size of batch normalisation and gradient descent
We need to choose a batch size for GD as well as the normalization, they both called batch size, but in actual implementation, do they need to be the same? 
Or otherwise how the framework handle them? 

In Pytorch for example, one batch size is defined in dataloader, e.g. 

torch.utils.data.DataLoader(image_datasets[x],
batch_size=16,
shuffle=True,
num_workers=4)


and if use BN

self.bn1 = nn.BatchNorm2d(16)


Does it has to be the same (16) or it can be different ? If different, is there any preferred relationship between the two 'batch size' 
Thanks 

No, batch_size is only defined in the data loader, not in the model.

The BatchNorm2d has a num_features parameter, and it depends on the number of channels and not the batch size, as you can see in the docs.

They are completely unrelated.


  BatchNorm2d
  
  torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

  
  Parameters
  
  
  num_features – C from an expected input of size (N,C,H,W)(N, C, H, W)(N,C,H,W)
  eps – a value added to the denominator for numerical stability. Default: 1e-5
  momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1
  affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True
  track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True
  

