How to reproduce RNN results on several runs?
I call same model on same input twice in a row and I don't get the same result, this model have nn.GRU layers so I suspect that it have some internal state that should be release before second run?

How to reset RNN hidden state to make it the same as if model was initially loaded?

UPDATE:

Some context:

I'm trying to run model from here:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L93

I'm calling generate:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

Here it's actually have some code using random generator in pytorch:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L110

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py#L129

I have placed (I'm running code on CPU):

torch.manual_seed(0)
torch.cuda.manual_seed_all(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)


in 

https://github.com/erogol/WaveRNN/blob/master/utils/distribution.py

after all imports.

I have checked GRU weights between runs and they are the same:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L153

Also I have checked logits and sample between runs and logits are the same but sample are not, so @Andrew Naguib seems were right about random seeding, but I'm not sure where the code that fixes random seed should be placed?

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L200

UPDATE 2:

I have placed seed init inside generate and now results are consistent:

https://github.com/erogol/WaveRNN/blob/master/models/wavernn.py#L148

I believe this may be highly related to Random Seeding. To ensure reproducible results (as stated by them) you have to seed torch as in this:

import torch
torch.manual_seed(0)


And also, the CuDNN module.

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False


If you're using numpy, you could also do:

import numpy as np
np.random.seed(0)


However, they warn you:


  Deterministic mode can have a performance impact, depending on your model.




A suggested script I regularly use which has been working very good to reproduce results is:

# imports
import numpy as np
import random
import torch
# ...
""" Set Random Seed """
if args.random_seed is not None:
    """Following seeding lines of code are to ensure reproducible results 
       Seeding the two pseudorandom number generators involved in PyTorch"""
    random.seed(args.random_seed)
    np.random.seed(args.random_seed)
    torch.manual_seed(args.random_seed)
    # https://pytorch.org/docs/master/notes/randomness.html#cudnn
    if not args.cpu_only:
        torch.cuda.manual_seed(args.random_seed)
        cudnn.deterministic = True
        cudnn.benchmark = False

