class_docstr: Process pool that shares large arrays to avoid memory copy.
  
  This drop-in replacement for `multiprocessing.pool.Pool` makes
  it possible to work efficiently with shared memory in a numpy
  context.
  
  Existing instances of numpy.memmap are preserved: the child
  suprocesses will have access to the same shared memory in the
  original mode except for the 'w+' mode that is automatically
  transformed as 'r+' to avoid zeroing the original data upon
  instantiation.
  
  Furthermore large arrays from the parent process are automatically
  dumped to a temporary folder on the filesystem such as child
  processes to access their content via memmaping (file system
  backed shared memory).
  
  Note: it is important to call the terminate method to collect
  the temporary folder used by the pool.
  
  Parameters
  ----------
  processes: int, optional
      Number of worker processes running concurrently in the pool.
  initializer: callable, optional
      Callable executed on worker process creation.
  initargs: tuple, optional
      Arguments passed to the initializer callable.
  temp_folder: str, optional
      Folder to be used by the pool for memmaping large arrays
      for sharing memory with worker processes. If None, this will try in
      order:
      - a folder pointed by the JOBLIB_TEMP_FOLDER environment variable,
      - /dev/shm if the folder exists and is writable: this is a RAMdisk
        filesystem available by default on modern Linux distributions,
      - the default system temporary folder that can be overridden
        with TMP, TMPDIR or TEMP environment variables, typically /tmp
        under Unix operating systems.
  max_nbytes int or None, optional, 1e6 by default
      Threshold on the size of arrays passed to the workers that
      triggers automated memory mapping in temp_folder.
      Use None to disable memmaping of large arrays.
  mmap_mode: {'r+', 'r', 'w+', 'c'}
      Memmapping mode for numpy arrays passed to workers.
      See 'max_nbytes' parameter documentation for more details.
  forward_reducers: dictionary, optional
      Reducers used to pickle objects passed from master to worker
      processes: see below.
  backward_reducers: dictionary, optional
      Reducers used to pickle return values from workers back to the
      master process.
  verbose: int, optional
      Make it possible to monitor how the communication of numpy arrays
      with the subprocess is handled (pickling or memmaping)
  prewarm: bool or str, optional, "auto" by default.
      If True, force a read on newly memmaped array to make sure that OS pre-
      cache it in memory. This can be useful to avoid concurrent disk access
      when the same data array is passed to different worker processes.
      If "auto" (by default), prewarm is set to True, unless the Linux shared
      memory partition /dev/shm is available and used as temp_folder.
  
  `forward_reducers` and `backward_reducers` are expected to be
  dictionaries with key/values being `(type, callable)` pairs where
  `callable` is a function that give an instance of `type` will return
  a tuple `(constructor, tuple_of_objects)` to rebuild an instance out
  of the pickled `tuple_of_objects` as would return a `__reduce__`
  method. See the standard library documentation on pickling for more
  details.
class_docstr_tok: ["Process","pool","that","shares","large","arrays","to","avoid","memory","copy",".","This","drop-in","replacement","for","`","multiprocessing.pool.Pool","`","makes","it","possible","to","work","efficiently","with","shared","memory","in","a","numpy","context",".","Existing","instances","of","numpy.memmap","are","preserved",":","the","child","suprocesses","will","have","access","to","the","same","shared","memory","in","the","original","mode","except","for","the","'","w+","'","mode","that","is","automatically","transformed","as","'","r+","'","to","avoid","zeroing","the","original","data","upon","instantiation",".","Furthermore","large","arrays","from","the","parent","process","are","automatically","dumped","to","a","temporary","folder","on","the","filesystem","such","as","child","processes","to","access","their","content","via","memmaping","(","file","system","backed","shared","memory",")",".","Note",":","it","is","important","to","call","the","terminate","method","to","collect","the","temporary","folder","used","by","the","pool",".","Parameters","--","--","--","--","--","processes",":","int",",","optional","Number","of","worker","processes","running","concurrently","in","the","pool",".","initializer",":","callable",",","optional","Callable","executed","on","worker","process","creation",".","initargs",":","tuple",",","optional","Arguments","passed","to","the","initializer","callable",".","temp_folder",":","str",",","optional","Folder","to","be","used","by","the","pool","for","memmaping","large","arrays","for","sharing","memory","with","worker","processes",".","If","None",",","this","will","try","in","order",":","-","a","folder","pointed","by","the","JOBLIB_TEMP_FOLDER","environment","variable",",","-","\/dev\/shm","if","the","folder","exists","and","is","writable",":","this","is","a","RAMdisk","filesystem","available","by","default","on","modern","Linux","distributions",",","-","the","default","system","temporary","folder","that","can","be","overridden","with","TMP",",","TMPDIR","or","TEMP","environment","variables",",","typically","\/tmp","under","Unix","operating","systems",".","max_nbytes","int","or","None",",","optional",",","1e6","by","default","Threshold","on","the","size","of","arrays","passed","to","the","workers","that","triggers","automated","memory","mapping","in","temp_folder",".","Use","None","to","disable","memmaping","of","large","arrays",".","mmap_mode",":","{","'","r+","'",",","'","r","'",",","'","w+","'",",","'","c","'","}","Memmapping","mode","for","numpy","arrays","passed","to","workers",".","See","'max_nbytes","'","parameter","documentation","for","more","details",".","forward_reducers",":","dictionary",",","optional","Reducers","used","to","pickle","objects","passed","from","master","to","worker","processes",":","see","below",".","backward_reducers",":","dictionary",",","optional","Reducers","used","to","pickle","return","values","from","workers","back","to","the","master","process",".","verbose",":","int",",","optional","Make","it","possible","to","monitor","how","the","communication","of","numpy","arrays","with","the","subprocess","is","handled","(","pickling","or","memmaping",")","prewarm",":","bool","or","str",",","optional",",","``","auto","''","by","default",".","If","True",",","force","a","read","on","newly","memmaped","array","to","make","sure","that","OS","pre-","cache","it","in","memory",".","This","can","be","useful","to","avoid","concurrent","disk","access","when","the","same","data","array","is","passed","to","different","worker","processes",".","If","``","auto","''","(","by","default",")",",","prewarm","is","set","to","True",",","unless","the","Linux","shared","memory","partition","\/dev\/shm","is","available","and","used","as","temp_folder",".","`","forward_reducers","`","and","`","backward_reducers","`","are","expected","to","be","dictionaries","with","key\/values","being","`","(","type",",","callable",")","`","pairs","where","`","callable","`","is","a","function","that","give","an","instance","of","`","type","`","will","return","a","tuple","`","(","constructor",",","tuple_of_objects",")","`","to","rebuild","an","instance","out","of","the","pickled","`","tuple_of_objects","`","as","would","return","a","`","__reduce__","`","method",".","See","the","standard","library","documentation","on","pickling","for","more","details","."]
code: def __init__(
      self,
      processes=None,
      temp_folder=None,
      max_nbytes=1e6,
      mmap_mode="r",
      forward_reducers=None,
      backward_reducers=None,
      verbose=0,
      context_id=None,
      prewarm=False,
      **kwargs,
  ):
      if forward_reducers is None:
          forward_reducers = dict()
      if backward_reducers is None:
          backward_reducers = dict()
      if context_id is not None:
          warnings.warn(
              "context_id is deprecated and ignored in joblib"
              " 0.9.4 and will be removed in 0.11",
              DeprecationWarning,
          )
  
      # Prepare a sub-folder name for the serialization of this particular
      # pool instance (do not create in advance to spare FS write access if
      # no array is to be dumped):
      use_shared_mem = False
      pool_folder_name = "joblib_memmaping_pool_%d_%d" % (
          os.getpid(),
          id(self),
      )
      if temp_folder is None:
          temp_folder = os.environ.get("JOBLIB_TEMP_FOLDER", None)
      if temp_folder is None:
          if os.path.exists(SYSTEM_SHARED_MEM_FS):
              try:
                  temp_folder = SYSTEM_SHARED_MEM_FS
                  pool_folder = os.path.join(
                      temp_folder, pool_folder_name
                  )
                  if not os.path.exists(pool_folder):
                      os.makedirs(pool_folder)
                  use_shared_mem = True
              except IOError:
                  # Missing rights in the the /dev/shm partition,
                  # fallback to regular temp folder.
                  temp_folder = None
      if temp_folder is None:
          # Fallback to the default tmp folder, typically /tmp
          temp_folder = tempfile.gettempdir()
      temp_folder = os.path.abspath(os.path.expanduser(temp_folder))
      pool_folder = os.path.join(temp_folder, pool_folder_name)
      self._temp_folder = pool_folder
  
      # Register the garbage collector at program exit in case caller forgets
      # to call terminate explicitly: note we do not pass any reference to
      # self to ensure that this callback won't prevent garbage collection of
      # the pool instance and related file handler resources such as POSIX
      # semaphores and pipes
      pool_module_name = whichmodule(delete_folder, "delete_folder")
  
      def _cleanup():
          # In some cases the Python runtime seems to set delete_folder to
          # None just before exiting when accessing the delete_folder
          # function from the closure namespace. So instead we reimport
          # the delete_folder function explicitly.
          # https://github.com/joblib/joblib/issues/328
          # We cannot just use from 'joblib.pool import delete_folder'
          # because joblib should only use relative imports to allow
          # easy vendoring.
          delete_folder = __import__(
              pool_module_name, fromlist=["delete_folder"]
          ).delete_folder
          delete_folder(pool_folder)
  
      atexit.register(_cleanup)
  
      if np is not None:
          # Register smart numpy.ndarray reducers that detects memmap backed
          # arrays and that is alse able to dump to memmap large in-memory
          # arrays over the max_nbytes threshold
          if prewarm == "auto":
              prewarm = not use_shared_mem
          forward_reduce_ndarray = ArrayMemmapReducer(
              max_nbytes,
              pool_folder,
              mmap_mode,
              verbose,
              prewarm=prewarm,
          )
          forward_reducers[np.ndarray] = forward_reduce_ndarray
          forward_reducers[np.memmap] = reduce_memmap
  
          # Communication from child process to the parent process always
          # pickles in-memory numpy.ndarray without dumping them as memmap
          # to avoid confusing the caller and make it tricky to collect the
          # temporary folder
          backward_reduce_ndarray = ArrayMemmapReducer(
              None, pool_folder, mmap_mode, verbose
          )
          backward_reducers[np.ndarray] = backward_reduce_ndarray
          backward_reducers[np.memmap] = reduce_memmap
  
      poolargs = dict(
          processes=processes,
          forward_reducers=forward_reducers,
          backward_reducers=backward_reducers,
      )
      poolargs.update(kwargs)
      super(MemmapingPool, self).__init__(**poolargs)
  
imports: from mmap import mmap
  import errno
  import os
  import stat
  import sys
  import threading
  import atexit
  import tempfile
  import shutil
  import warnings
  from time import sleep
  from pickle import whichmodule
  from pickle import Pickler
  from pickle import HIGHEST_PROTOCOL
  from io import BytesIO
  from ._multiprocessing_helpers import mp, assert_spawning
  from multiprocessing.pool import Pool
  from .numpy_pickle import load
  from .numpy_pickle import dump
  from .hashing import hash
label_desc: Use image node_id 1 to create a new MemmapingPool object from inherited base classes: PicklingPool with example: obj = MemmapingPool(processes, temp_folder, max_nbytes, mmap_mode, forward_reducers, backward_reducers, verbose, context_id, prewarm)