class_docstr: Parallel processing engine.
  
  Engine for running computational graph.
  
  :class:`ParallelProcessing` is a manager for executing a sequence of tasks
  in a given caller, where each task is run sequentially, but assumed to be
  parallelized internally. The main responsibility of
  :class:`ParallelProcessing` is to handle memory-mapping, estimation
  cache updates, input and output array updates and output collection.
  
  Parameters
  ----------
  caller :  obj
      the caller of the job. Either a Layer or a meta layer class
      such as Sequential.
  
  *args : optional
      Optional arguments to :class:`~mlens.parallel.backend.BaseProcessor`
  
  **kwargs : optional
      Optional keyword arguments to
      :class:`~mlens.parallel.backend.BaseProcessor`.
class_docstr_tok: ["Parallel","processing","engine",".","Engine","for","running","computational","graph",".",":","class",":","`","ParallelProcessing","`","is","a","manager","for","executing","a","sequence","of","tasks","in","a","given","caller",",","where","each","task","is","run","sequentially",",","but","assumed","to","be","parallelized","internally",".","The","main","responsibility","of",":","class",":","`","ParallelProcessing","`","is","to","handle","memory-mapping",",","estimation","cache","updates",",","input","and","output","array","updates","and","output","collection",".","Parameters","--","--","--","--","--","caller",":","obj","the","caller","of","the","job",".","Either","a","Layer","or","a","meta","layer","class","such","as","Sequential",".","*","args",":","optional","Optional","arguments","to",":","class",":","`","~mlens.parallel.backend.BaseProcessor","`","*","*","kwargs",":","optional","Optional","keyword","arguments","to",":","class",":","`","~mlens.parallel.backend.BaseProcessor","`","."]
docstr: Parallel task mapping.
  
  Run independent tasks in caller in parallel.
  
  Warning
  -------
  By default, the :~mlens.parallel.backend.ParallelProcessing.map` runs
  on a shallow cache, where all tasks share the same cache. As such, the
  user must ensure that each task has a unique name, or cache retrieval
  will be corrupted. To commit a seperate sub-cache to each task, set
  ``split=True``.
  
  Parameters
  ----------
  caller : iterable
      Iterable that generates accepted task instances. Caller should be
      a child of the :class:`~mlens.parallel.base.BaseBackend` class,
      and tasks need to implement an appropriate call method.
  
  job : str
      type of job to complete with each task. One of ``'fit'``,
      ``'predict'`` and ``'transform'``.
  
  X : array-like of shape [n_samples, n_features]
      Input data
  
  y : array-like of shape [n_samples,], optional.
      targets. Required for fit, should not be passed to predict or
      transform jobs.
  
  path : str or dict, optional
      Custom estimation cache. Pass a string to force use of persistent
      cache on disk. Pass a ``dict`` for in-memory cache (requires
      ``backend != 'multiprocessing'``.
  
  return_preds : bool or list, optional
      whether to return prediction ouput. If ``True``, final prediction
      is returned. Alternatively, pass a list of task names for which
      output should be returned.
  
  warm_start : bool, optional
      whether to re-use previous input data initialization. Useful if
      repeated jobs are made on the same input arrays.
  
  split : bool, default = False
      whether to commit a separate sub-cache to each task.
  
  **kwargs : optional
      optional keyword arguments to pass onto each task.
  
  Returns
  -------
  out: array-like, list, optional
      Prediction array(s).
docstr_tok: ["Parallel","task","mapping",".","Run","independent","tasks","in","caller","in","parallel",".","Warning","--","--","--","-","By","default",",","the",":","~mlens.parallel.backend.ParallelProcessing.map","`","runs","on","a","shallow","cache",",","where","all","tasks","share","the","same","cache",".","As","such",",","the","user","must","ensure","that","each","task","has","a","unique","name",",","or","cache","retrieval","will","be","corrupted",".","To","commit","a","seperate","sub-cache","to","each","task",",","set","``","split=True","``",".","Parameters","--","--","--","--","--","caller",":","iterable","Iterable","that","generates","accepted","task","instances",".","Caller","should","be","a","child","of","the",":","class",":","`","~mlens.parallel.base.BaseBackend","`","class",",","and","tasks","need","to","implement","an","appropriate","call","method",".","job",":","str","type","of","job","to","complete","with","each","task",".","One","of","``","'fit","'","``",",","``","'predict","'","``","and","``","'transform","'","``",".","X",":","array-like","of","shape","[","n_samples",",","n_features","]","Input","data","y",":","array-like","of","shape","[","n_samples",",","]",",","optional",".","targets",".","Required","for","fit",",","should","not","be","passed","to","predict","or","transform","jobs",".","path",":","str","or","dict",",","optional","Custom","estimation","cache",".","Pass","a","string","to","force","use","of","persistent","cache","on","disk",".","Pass","a","``","dict","``","for","in-memory","cache","(","requires","``","backend","!","=","'multiprocessing","'","``",".","return_preds",":","bool","or","list",",","optional","whether","to","return","prediction","ouput",".","If","``","True","``",",","final","prediction","is","returned",".","Alternatively",",","pass","a","list","of","task","names","for","which","output","should","be","returned",".","warm_start",":","bool",",","optional","whether","to","re-use","previous","input","data","initialization",".","Useful","if","repeated","jobs","are","made","on","the","same","input","arrays",".","split",":","bool",",","default","=","False","whether","to","commit","a","separate","sub-cache","to","each","task",".","*","*","kwargs",":","optional","optional","keyword","arguments","to","pass","onto","each","task",".","Returns","--","--","--","-","out",":","array-like",",","list",",","optional","Prediction","array","(","s",")","."]
code: def map(
      self,
      caller,
      job,
      X,
      y=None,
      path=None,
      return_preds=False,
      wart_start=False,
      split=False,
      **kwargs,
  ):
      """Parallel task mapping.
  
      Run independent tasks in caller in parallel.
  
      Warning
      -------
      By default, the :~mlens.parallel.backend.ParallelProcessing.map` runs
      on a shallow cache, where all tasks share the same cache. As such, the
      user must ensure that each task has a unique name, or cache retrieval
      will be corrupted. To commit a seperate sub-cache to each task, set
      ``split=True``.
  
      Parameters
      ----------
      caller : iterable
          Iterable that generates accepted task instances. Caller should be
          a child of the :class:`~mlens.parallel.base.BaseBackend` class,
          and tasks need to implement an appropriate call method.
  
      job : str
          type of job to complete with each task. One of ``'fit'``,
          ``'predict'`` and ``'transform'``.
  
      X : array-like of shape [n_samples, n_features]
          Input data
  
      y : array-like of shape [n_samples,], optional.
          targets. Required for fit, should not be passed to predict or
          transform jobs.
  
      path : str or dict, optional
          Custom estimation cache. Pass a string to force use of persistent
          cache on disk. Pass a ``dict`` for in-memory cache (requires
          ``backend != 'multiprocessing'``.
  
      return_preds : bool or list, optional
          whether to return prediction ouput. If ``True``, final prediction
          is returned. Alternatively, pass a list of task names for which
          output should be returned.
  
      warm_start : bool, optional
          whether to re-use previous input data initialization. Useful if
          repeated jobs are made on the same input arrays.
  
      split : bool, default = False
          whether to commit a separate sub-cache to each task.
  
      **kwargs : optional
          optional keyword arguments to pass onto each task.
  
      Returns
      -------
      out: array-like, list, optional
          Prediction array(s).
      """
      out = self.initialize(
          job=job,
          X=X,
          y=y,
          path=path,
          warm_start=wart_start,
          return_preds=return_preds,
          split=split,
          stack=False,
      )
      return self.process(caller=caller, out=out, **kwargs)
  
imports: from __future__ import with_statement, division
  import gc
  import os
  import shutil
  import subprocess
  import tempfile
  import warnings
  from abc import ABCMeta, abstractmethod
  import numpy
  from scipy.sparse import issparse, hstack
  from ..None import config
  from ..externals.joblib import Parallel, dump, load
  from ..utils import check_initialized
  from ..utils.exceptions import ParallelProcessingError, ParallelProcessingWarning
  from ..externals.sklearn.validation import check_random_state
label_desc: Use image node_id 2 for calling the ParallelProcessing obj's underlying member method code with example usage: obj.map(caller, job, X, y, path, return_preds, wart_start, split) and returns: self