title: What is the correct way of using hogwild! on multi-gpus
body: <p>The <a href="https://github.com/pytorch/examples/tree/master/mnist_hogwild" rel="nofollow noreferrer">example</a> on Hogwild! gives 99% accuracy, but when I upgrade to multi-gpu versions, it gives 11% accuracy.</p>
  <p>The difference is as follows:</p>
  <pre><code># main.py
  
  model = Net() # The original version move it to device before multi-processing begins.
  model.share_memory()
  
  processes = []
  for rank in range(args.num_processes):
      local_device = torch.device(rank%2)
      p = mp.Process(target=train, args=(rank, args, model, local_device,
                                         dataset1, kwargs))
      # We first train the model across `num_processes` processes
      p.start()
      processes.append(p)
  </code></pre>
  <p>And I move the model to device in the subprocesses.</p>
  <pre><code># train.py
  def train(rank, args, model, device, dataset, dataloader_kwargs):
      model = model.to(device)  # move to specific device in the sub-process
      torch.manual_seed(args.seed + rank)
  
      train_loader = torch.utils.data.DataLoader(dataset, **dataloader_kwargs)
  
      optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)
      for epoch in range(1, args.epochs + 1):
          train_epoch(epoch, args, model, device, train_loader, optimizer)
  </code></pre>
  <p>It seems the model is not shared any more.</p>
  <ol>
  <li>Where are the mistakes?</li>
  <li>What is the correct steps?</li>
  </ol>