class_docstr: Testing DistributedHistory with multiprocessing
  
  This is not a proper test for how DistributedHistory should be used in the
  wild, which would be in a multi-GPU setting using DDP (possibly through
  accelerate). However, since we cannot test this setting in CI, this is the
  next best approximation.
  
  Since DDP creates forks of the main process, each one would have its own
  history instance, leading in them diverging. This can be bad, e.g. if the
  history is used to steer the training (think early stopping). This test
  intends to show that the histories do not diverge.
  
  In a sense, this test actually tests incorrect behavior because the
  different processes all write the exact same content to the history.
  However, if DDP was used, each process would see different batches and thus
  write different records to their histories. So when reading this test,
  imagine that this would be what's truly happening.
  
  This test is very high level, unlike TestHistory. For instance, there is no
  attempt at testing "weird" read and write patterns, like creating multiple
  empty batches. It is very much assumed that the history is used "correctly",
  as in a normal skorch fit loop.
class_docstr_tok: ["Testing","DistributedHistory","with","multiprocessing","This","is","not","a","proper","test","for","how","DistributedHistory","should","be","used","in","the","wild",",","which","would","be","in","a","multi-GPU","setting","using","DDP","(","possibly","through","accelerate",")",".","However",",","since","we","can","not","test","this","setting","in","CI",",","this","is","the","next","best","approximation",".","Since","DDP","creates","forks","of","the","main","process",",","each","one","would","have","its","own","history","instance",",","leading","in","them","diverging",".","This","can","be","bad",",","e.g",".","if","the","history","is","used","to","steer","the","training","(","think","early","stopping",")",".","This","test","intends","to","show","that","the","histories","do","not","diverge",".","In","a","sense",",","this","test","actually","tests","incorrect","behavior","because","the","different","processes","all","write","the","exact","same","content","to","the","history",".","However",",","if","DDP","was","used",",","each","process","would","see","different","batches","and","thus","write","different","records","to","their","histories",".","So","when","reading","this","test",",","imagine","that","this","would","be","what","'s","truly","happening",".","This","test","is","very","high","level",",","unlike","TestHistory",".","For","instance",",","there","is","no","attempt","at","testing","``","weird","''","read","and","write","patterns",",","like","creating","multiple","empty","batches",".","It","is","very","much","assumed","that","the","history","is","used","``","correctly","''",",","as","in","a","normal","skorch","fit","loop","."]
code: def test_distributed_history(self, nprocs):
      from skorch.toy import make_classifier
      from skorch._version import Version
  
      # TODO: remove this once PyTorch 1.11 is no longer supported
      version_torch = Version(torch.__version__)
      if version_torch < Version("1.12.0"):
          # Tests with DistributedHistory are flaky with PyTorch 1.11 and when
          # using pytest on CI. Running directly, i.e. without pytest, does
          # not appear to cause issues.
          pytest.skip(msg="Skipping multiprocessing for PyTorch < 1.12")
  
      X, y = make_classification(
          500, 20, n_informative=10, random_state=0, flip_y=0.1
      )
      X = X.astype(np.float32)
      y = y.astype(np.int64)
  
      torch.set_num_threads(1)
      module = make_classifier(
          input_units=20,
          hidden_units=10,
          num_hidden=2,
          dropout=0.5,
      )
  
      args = zip(
          range(nprocs),
          [nprocs] * nprocs,
          [module] * nprocs,
          [(X, y)] * nprocs,
      )
      # https://github.com/pytorch/pytorch/wiki/Autograd-and-Fork
      ctx = mp.get_context("spawn")
      with ctx.Pool(nprocs) as pool:
          nets = pool.map(self.train, args)
  
      # expected entries for training batches:
      # - 500 samples, 400 used for train, 100 for valid
      # - batch size is 128, i.e. 4 batches per epoch for train
      # - the 100 valid samples fit into 1 batch
      # note that when using DDP correctly, since batches are split across the
      # processes, there would *not* be nprocs times more entries.
      train_batches = 4
      valid_batches = 1
      for net in nets:
          h = net.history
          assert (
              len(h[0, "batches", :, "train_loss"])
              == train_batches * nprocs
          )
          assert (
              len(h[1, "batches", :, "train_loss"])
              == train_batches * nprocs
          )
          assert (
              len(h[0, "batches", :, "valid_loss"])
              == valid_batches * nprocs
          )
          assert (
              len(h[1, "batches", :, "valid_loss"])
              == valid_batches * nprocs
          )
  
      # the batch information is synchronized, so should be identical; the
      # epoch information is *not* synchronized and can diverge, most notably
      # with things like the duration
      for net0, net1 in zip(nets, nets[1:]):
          train_loss0 = net0.history[:, "batches", :, "train_loss"]
          train_loss1 = net1.history[:, "batches", :, "train_loss"]
          assert train_loss0 == train_loss1
  
          valid_loss0 = net0.history[:, "batches", :, "valid_loss"]
          valid_loss1 = net1.history[:, "batches", :, "valid_loss"]
          assert valid_loss0 == valid_loss1
  
          assert net0.history[:, "dur"] != net1.history[:, "dur"]
  
imports: import pickle
  import time
  import multiprocessing
  from functools import partial
  import numpy
  import pytest
  import torch
  from torch.distributed import TCPStore
  from sklearn.datasets import make_classification
label_desc: Use image node_id 3 for calling the TestDistributedHistoryMultiprocessing obj's underlying member method code with example usage: obj.test_distributed_history(nprocs) without return types