The result is different when I apply torch.manual_seed before loading cuda() after loading the model
I tried to make sure my code to be reproducible (always get the same results)

So I applied below settings before my codes.

os.environ['PYTHONHASHSEED'] = str(args.seed)
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
torch.cuda.manual_seed_all(args.seed) # if you are using multi-GPU.
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True


With these settings, I always achieved the same results with the same environment and GPU.

Howerver, when I applied torch.manual_seed() after loading the model.

torch.manual_seed(args.seed)
model = Net()
Net.cuda()


torch.manual_seed(args.seed)
model = Net()
torch.manual_seed(args.seed)
Net.cuda()


The above two results were different.
How should I understand this situation?

Does seed reinitialize after loading the model?

The Net.cuda() has no effect on the random number generator. Under the hood it just calls cuda() for all of the model parameters. So basically it's multiple calls to Tensor.cuda().

https://github.com/pytorch/pytorch/blob/ecd3c252b4da3056797f8a505c9ebe8d68db55c4/torch/nn/modules/module.py#L293

We can test this by doing the following:

torch.random.manual_seed(42)
x = torch.rand(1)
x.cuda()
y = torch.rand(1)
y.cuda()
print(x, y)

# the above prints the same as below
torch.random.manual_seed(42)
print(torch.rand(1), torch.rand(1))


So that means Net() is using the number generator to initialize random weights within the layers.

torch.manual_seed(args.seed)
model = Net()
print(torch.rand(1))

# the below will print a different result
torch.manual_seed(args.seed)
model = Net()
torch.manual_seed(args.seed)
print(torch.rand(1))


I would recommend narrowing the scope of how random numbers are managed within your Python source code. So that a global block of code outside of the Model isn't responsible for how internal values are generated.

Simply said, pass the seed as a parameter to the __init__ of the model.

model = Net(args.seed)
print(torch.rand(1))


This will force developers to always provide a seed for consistency when using the model, and you can make the parameter optional if seeding isn't always necessary. 

I'd avoid using the same seed all the time, because you're going to learn to use parameters that work best with that seed.
