Questions about Batch Normalization in Pytorch
Recently when I use the BN in the PyTorch, I have several questions.

Based on the BN2d documentation in PyTorch, when inferencing(evaluation), it will automatically use the mean and variance (running estimate when training) for BN layer. However, my first question is that when we save out the model after training, does it contain the running mean and variance? I originally thought the model will only save the learnable parameters. But the running mean and variance are not really learnable?

By default, when we use eval() in PyTorch, the BN layer will use some running mean and variance from a specific variable (maybe the results from training), right? But not to calculate the mean and variance of mini-batch? (Because I see some answers mention that the bad performance of BN when inferencing is cased by the 1-size batch. But it should use the running mean and variance of training, why the batch size of testing will influence?
The third question is kind of related to the second question. Will the BN performs the same when inferencing. For example, when I iterate the test set twice. Will the results behave differently? (The maybe more direct question is that will the mean and variance change when inferencing?)
I also want to know if it is possible to use mini-batch mean &amp; variance or running mean &amp; variance for inferencing. If I only set BN layer to train, its learnable parameter will also update. But maybe I just want them to get new mean and variance. Is there a possible way to do that?
The last question is related to the fourth question. So is it fair to use mean &amp; variance of all test data, or batch of test data to calculate the mean &amp; variance? I mean fair here is that is it improper (tricky?) to use some attributes of the test set?

Look forward to your answers. I'm a kind of new bird, willing to learn and discuss with you!!
Thanks in advance!!!

I get the answers from my senior classmates, and I think it's useful for others. (If you have different points, feel free to comment)

When we save out the whole model, it will contain the running mean and variance for the BN layers. These two parameters are not learnable (not updated in backward process, but updated in the forward process)

If use .eval(), the BN layer will automatically use the running mean and variance stored in that layer, and will not update in forward process again. This means that when inferencing, BN layers use the running mean and variance calculated in the training process.

Yes, if just simply use .eval(), the BN layer will use the same mean and variance at all times.

It's possible to calculate the running mean and variance based on the test set. Just make the BN layer in train() mode. This will not influence the learnable parameters of BN layer. Because when inferencing, we only have a forward process (update the mean and variance) without a backward process. Maybe if we have to reset the mean and variance at the beginning of the evaluation or inference period to make mean and variance totally unrelated to the training process.

It's kind of some tricks. I hear that some GAN papers adopt such strategies for BN layers.


