Unstable results in test mode with fractional max pooling in PyTorch
The answer (maybe one of) is FractionalMaxPool2d layer I use. If I use say ResNet-34, train it a bit (say 1/6 of epoch), everything is ok. If I add one FractionalMaxPool2d layer:

...
self.layer1 = self._make_layer(block, 2, 64, layers[0])
self.maxpool2 = nn.FractionalMaxPool2d(kernel_size=3, output_size=38) 
self.layer2 = self._make_layer(block, 3, 128, layers[1], stride=2,
...


(and run it in the forward method of course)

and repeat, the difference appears.