title: Reproducable Pytorch Results & Random Seeds
body: <p>I have a simple toy NN with Pytorch. I am setting all the seeds I can find in the docs as well as numpy random. </p>
  
  <p>If I run the code below from top to bottom, the results appear to be reproducible. </p>
  
  <p>BUT, if I run block 1 only once and then each time run block 2, the result changes (sometimes dramatically). I am unsure why this happens since the network is being re-initialized and optimizer reset each time.</p>
  
  <p>I am using version 0.4.0</p>
  
  <h3>BLOCK #1</h3>
  
  <pre><code>from __future__ import division
  import numpy as np
  import matplotlib.pyplot as plt
  import pandas as pd
  
  import torch
  import torch.utils.data as utils_data
  from torch.autograd import Variable
  from torch import optim, nn
  from torch.utils.data import Dataset 
  import torch.nn.functional as F
  from torch.nn.init import xavier_uniform_, xavier_normal_,uniform_
  
  torch.manual_seed(123)
  
  import random
  random.seed(123)
  
  
  from sklearn.datasets import load_boston
  from sklearn.metrics import mean_squared_error
  from sklearn.model_selection import train_test_split
  
  %matplotlib inline 
  
  cuda=True #set to true uses GPU
  
  if cuda:
      torch.cuda.manual_seed(123)
  
  #load boston data from scikit
  boston = load_boston()
  x=boston.data
  y=boston.target
  y=y.reshape(y.shape[0],1)
  
  #train and test
  x_train, x_test, y_train, y_test = train_test_split(x,y,test_size=0.3, random_state=123, shuffle=False)
  
  
  #change to tensors
  x_train = torch.from_numpy(x_train)
  y_train = torch.from_numpy(y_train)
  
  #create dataset and use data loader
  training_samples = utils_data.TensorDataset(x_train, y_train)
  data_loader_trn = utils_data.DataLoader(training_samples, batch_size=64,drop_last=False)
  
  #change to tensors
  x_test = torch.from_numpy(x_test)
  y_test = torch.from_numpy(y_test)
  
  #create dataset and use data loader
  testing_samples = utils_data.TensorDataset(x_test, y_test)
  data_loader_test = utils_data.DataLoader(testing_samples, batch_size=64,drop_last=False)
  
  #simple model
  class Net(nn.Module):
      def __init__(self):
          super(Net, self).__init__()
  
          #all the layers
          self.fc1   = nn.Linear(x.shape[1], 20)
          xavier_uniform_(self.fc1.weight.data) #this is how you can change the weight init
          self.drop = nn.Dropout(p=0.5)
          self.fc2   = nn.Linear(20, 1)
  
  
      def forward(self, x):
          x = F.relu(self.fc1(x))
          x=  self.drop(x)
          x = self.fc2(x)
          return x
  </code></pre>
  
  <hr>
  
  <h3>BLOCK #2</h3>
  
  <pre><code>net=Net()
  
  if cuda:
      net.cuda()
  
  # create a stochastic gradient descent optimizer
  optimizer = optim.Adam(net.parameters())
  # create a loss function (mse)
  loss = nn.MSELoss(size_average=False)
  
  # run the main training loop
  epochs =20
  hold_loss=[]
  
  for epoch in range(epochs):
      cum_loss=0.
      cum_records_epoch =0
  
      for batch_idx, (data, target) in enumerate(data_loader_trn):
          tr_x, tr_y = data.float(), target.float()
          if cuda:
              tr_x, tr_y = tr_x.cuda(), tr_y.cuda() 
  
          # Reset gradient
          optimizer.zero_grad()
  
          # Forward pass
          fx = net(tr_x)
          output = loss(fx, tr_y) #loss for this batch
  
          cum_loss += output.item() #accumulate the loss
  
          # Backward 
          output.backward()
  
          # Update parameters based on backprop
          optimizer.step()
  
          cum_records_epoch +=len(tr_x)
          if batch_idx % 1 == 0:
              print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
              epoch, cum_records_epoch, len(data_loader_trn.dataset),
              100. * (batch_idx+1) / len(data_loader_trn), output.item()))
      print('Epoch average loss: {:.6f}'.format(cum_loss/cum_records_epoch))
  
      hold_loss.append(cum_loss/cum_records_epoch)  
  
  #training loss
  plt.plot(np.array(hold_loss))
  plt.show()
  </code></pre>