class_docstr: Step is a building block of steppy pipelines.
  
  It is an execution wrapper over the transformer (see :class:`~steppy.base.BaseTransformer`),
  which realizes single operation on data. With Step you can:
  
  1. design multiple input/output data flows and connections between Steps.
  2. handle persistence and caching of transformer and intermediate results.
  
  Step executes `fit_transform` method inspired by the sklearn on every step recursively
  starting from the very last Step and making its way forward through the `input_steps`.
  One can easily debug the data flow by plotting the pipeline graph
  (see: :func:`~steppy.utils.persist_as_png`) or return step in a jupyter notebook cell.
  
  Attributes:
      transformer (obj): object that inherits from BaseTransformer or Step instance.
          When Step instance is passed, transformer from that Step will be copied and used to
          perform transformations. It is useful when both train and valid data are passed in
          one pipeline (common situation in deep learning).
  
      name (str): Step name.
          Each step in a pipeline must have a unique name. It is name of the persisted
          transformer and output of this Step.
          Default is transformer's class name.
  
      experiment_directory (str): path to the directory where all execution artifacts will be
          stored.
          Default is ``~/.steppy``.
          The following directories will be created under ``~/.steppy``, if they were not created by
          preceding Steps:
  
          * transformers: transformer objects are persisted in this folder
          * output:      step output dictionaries are persisted in this folder
            (if ``persist_output=True``)
  
      input_data (list): Elements of this list are keys in the data dictionary that is passed
          to the Step's `fit_transform` and `transform` methods.
          List of str, default is empty list.
  
          Example:
  
              .. code-block:: python
  
                  data_train = {'input': {'images': X_train,
                                          'labels': y_train}
                               }
  
                  my_step = Step(name='random_forest',
                                 transformer=RandomForestTransformer(),
                                 input_data=['input']
                                 )
  
                  my_step.fit_transform(data_train)
  
              `data_train` is dictionary where:
  
               * keys are names of data packets,
               * values are data packets, that is dictionaries that describes dataset.
                 In this example keys in the data packet are `images` and `labels` and values
                 are actual data of any type.
  
              `Step.input_data` takes the key from `data_train` (values must match!) and extracts
              actual data that will be passed to the `fit_transform` and `transform` method of
              the `self.transformer`.
  
      input_steps (list): List of input Steps that the current Step uses as its input.
          list of Step instances, default is empty list.
          Current Step will combine output from `input_steps` and `input_data` using `adapter`.
          Then pass it to the transformer methods `fit_transform` and `transform`.
  
          Example:
  
              .. code-block:: python
  
                  self.input_steps=[cnn_step, rf_step, ensemble_step, guesses_step]
  
              Each element of the list is Step instance.
  
      adapter (obj): It renames and arranges inputs that are passed to the Transformer
          (see :class:`~steppy.base.BaseTransformer`).
          Default is ``None``.
          If ``not None``, then must be an instance of the :class:`~steppy.adapter.Adapter` class.
  
          Example:
              .. code-block:: python
  
                  self.adapter=Adapter({'X': E('input', 'images'),
                                        'y': E('input', 'labels')}
                                       )
  
          Adapter simplifies the renaming and combining of inputs from multiple steps.
          In this example, after the adaptation:
  
          * `X` is key to the data stored under the `images` key
          * `y` is key to the data stored under the `labels` key
  
              where both `images` and `labels` keys comes from `input`
              (see :attr:`~steppy.base.Step.input_data`)
  
      cache_output (bool): If True, Step output dictionary will be cached under
          ``self.output``, when transform method of the Step transformer
          is completed. If the same Step is used multiple times, transform method is invoked
          only once. Further invokes simply use cached output.
          Default ``False``: do not cache output
  
          Warning:
              One should always run `step.clean_cache_upstream()` before executing
              `step.fit_transform(data)` or `step.transform(data)`
              When working with large datasets, cache might be very large.
  
      persist_output (bool): If True, persist Step output to disk under the
          ``<experiment_directory>/output/<name>`` directory.
          Default ``False``: do not persist any files to disk.
          If True then Step output dictionary will be persisted to the
          ``<experiment_directory>/output/<name>`` directory, after transform method of the Step
          transformer is completed. Step persists to disk the output after every run of the
          transformer's transform method. It means that Step overrides files. See also
          `load_persisted_output` parameter.
  
          Warning:
              When working with large datasets, cache might be very large.
  
      load_persisted_output (bool): If True, Step output dictionary already persisted to the
          ``<experiment_directory>/output/<name>`` will be loaded when Step is called.
          Default ``False``: do not load persisted output.
          Useful when debugging and working with ensemble models or time consuming feature
          extraction. One can easily persist already computed pieces of the pipeline and save
          time by loading them instead of calculating.
  
          Warning:
              Re-running the same step on new data with `load_persisted_output` set ``True``
              may lead to errors when output from old data are loaded while user would expect
              the pipeline to use new data instead.
  
      force_fitting (bool): If True, Step transformer will be fitted (via `fit_transform`)
          even if ``<experiment_directory>/transformers/<step_name>`` exists.
          Default ``True``: fit transformer each time `fit_transform()` is called.
          Helpful when one wants to use ``persist_output=True`` and load ``persist_output=True``
          on a previous Step and fit current Step multiple times. This is a typical scenario
          for tuning hyperparameters for an ensemble model trained on the output from first
          level models or a model build on features that are time consuming to compute.
class_docstr_tok: ["Step","is","a","building","block","of","steppy","pipelines",".","It","is","an","execution","wrapper","over","the","transformer","(","see",":","class",":","`","~steppy.base.BaseTransformer","`",")",",","which","realizes","single","operation","on","data",".","With","Step","you","can",":","1.","design","multiple","input\/output","data","flows","and","connections","between","Steps",".","2.","handle","persistence","and","caching","of","transformer","and","intermediate","results",".","Step","executes","`","fit_transform","`","method","inspired","by","the","sklearn","on","every","step","recursively","starting","from","the","very","last","Step","and","making","its","way","forward","through","the","`","input_steps","`",".","One","can","easily","debug","the","data","flow","by","plotting","the","pipeline","graph","(","see",":",":","func",":","`","~steppy.utils.persist_as_png","`",")","or","return","step","in","a","jupyter","notebook","cell",".","Attributes",":","transformer","(","obj",")",":","object","that","inherits","from","BaseTransformer","or","Step","instance",".","When","Step","instance","is","passed",",","transformer","from","that","Step","will","be","copied","and","used","to","perform","transformations",".","It","is","useful","when","both","train","and","valid","data","are","passed","in","one","pipeline","(","common","situation","in","deep","learning",")",".","name","(","str",")",":","Step","name",".","Each","step","in","a","pipeline","must","have","a","unique","name",".","It","is","name","of","the","persisted","transformer","and","output","of","this","Step",".","Default","is","transformer","'s","class","name",".","experiment_directory","(","str",")",":","path","to","the","directory","where","all","execution","artifacts","will","be","stored",".","Default","is","``","~\/.steppy","``",".","The","following","directories","will","be","created","under","``","~\/.steppy","``",",","if","they","were","not","created","by","preceding","Steps",":","*","transformers",":","transformer","objects","are","persisted","in","this","folder","*","output",":","step","output","dictionaries","are","persisted","in","this","folder","(","if","``","persist_output=True","``",")","input_data","(","list",")",":","Elements","of","this","list","are","keys","in","the","data","dictionary","that","is","passed","to","the","Step","'s","`","fit_transform","`","and","`","transform","`","methods",".","List","of","str",",","default","is","empty","list",".","Example",":","..","code-block",":",":","python","data_train","=","{","'input","'",":","{","'images","'",":","X_train",",","'labels","'",":","y_train","}","}","my_step","=","Step","(","name='random_forest","'",",","transformer=RandomForestTransformer","(",")",",","input_data=","[","'input","'","]",")","my_step.fit_transform","(","data_train",")","`","data_train","`","is","dictionary","where",":","*","keys","are","names","of","data","packets",",","*","values","are","data","packets",",","that","is","dictionaries","that","describes","dataset",".","In","this","example","keys","in","the","data","packet","are","`","images","`","and","`","labels","`","and","values","are","actual","data","of","any","type",".","`","Step.input_data","`","takes","the","key","from","`","data_train","`","(","values","must","match","!",")","and","extracts","actual","data","that","will","be","passed","to","the","`","fit_transform","`","and","`","transform","`","method","of","the","`","self.transformer","`",".","input_steps","(","list",")",":","List","of","input","Steps","that","the","current","Step","uses","as","its","input",".","list","of","Step","instances",",","default","is","empty","list",".","Current","Step","will","combine","output","from","`","input_steps","`","and","`","input_data","`","using","`","adapter","`",".","Then","pass","it","to","the","transformer","methods","`","fit_transform","`","and","`","transform","`",".","Example",":","..","code-block",":",":","python","self.input_steps=","[","cnn_step",",","rf_step",",","ensemble_step",",","guesses_step","]","Each","element","of","the","list","is","Step","instance",".","adapter","(","obj",")",":","It","renames","and","arranges","inputs","that","are","passed","to","the","Transformer","(","see",":","class",":","`","~steppy.base.BaseTransformer","`",")",".","Default","is","``","None","``",".","If","``","not","None","``",",","then","must","be","an","instance","of","the",":","class",":","`","~steppy.adapter.Adapter","`","class",".","Example",":","..","code-block",":",":","python","self.adapter=Adapter","(","{","'","X","'",":","E","(","'input","'",",","'images","'",")",",","'","y","'",":","E","(","'input","'",",","'labels","'",")","}",")","Adapter","simplifies","the","renaming","and","combining","of","inputs","from","multiple","steps",".","In","this","example",",","after","the","adaptation",":","*","`","X","`","is","key","to","the","data","stored","under","the","`","images","`","key","*","`","y","`","is","key","to","the","data","stored","under","the","`","labels","`","key","where","both","`","images","`","and","`","labels","`","keys","comes","from","`","input","`","(","see",":","attr",":","`","~steppy.base.Step.input_data","`",")","cache_output","(","bool",")",":","If","True",",","Step","output","dictionary","will","be","cached","under","``","self.output","``",",","when","transform","method","of","the","Step","transformer","is","completed",".","If","the","same","Step","is","used","multiple","times",",","transform","method","is","invoked","only","once",".","Further","invokes","simply","use","cached","output",".","Default","``","False","``",":","do","not","cache","output","Warning",":","One","should","always","run","`","step.clean_cache_upstream","(",")","`","before","executing","`","step.fit_transform","(","data",")","`","or","`","step.transform","(","data",")","`","When","working","with","large","datasets",",","cache","might","be","very","large",".","persist_output","(","bool",")",":","If","True",",","persist","Step","output","to","disk","under","the","``","<","experiment_directory",">","\/output\/","<","name",">","``","directory",".","Default","``","False","``",":","do","not","persist","any","files","to","disk",".","If","True","then","Step","output","dictionary","will","be","persisted","to","the","``","<","experiment_directory",">","\/output\/","<","name",">","``","directory",",","after","transform","method","of","the","Step","transformer","is","completed",".","Step","persists","to","disk","the","output","after","every","run","of","the","transformer","'s","transform","method",".","It","means","that","Step","overrides","files",".","See","also","`","load_persisted_output","`","parameter",".","Warning",":","When","working","with","large","datasets",",","cache","might","be","very","large",".","load_persisted_output","(","bool",")",":","If","True",",","Step","output","dictionary","already","persisted","to","the","``","<","experiment_directory",">","\/output\/","<","name",">","``","will","be","loaded","when","Step","is","called",".","Default","``","False","``",":","do","not","load","persisted","output",".","Useful","when","debugging","and","working","with","ensemble","models","or","time","consuming","feature","extraction",".","One","can","easily","persist","already","computed","pieces","of","the","pipeline","and","save","time","by","loading","them","instead","of","calculating",".","Warning",":","Re-running","the","same","step","on","new","data","with","`","load_persisted_output","`","set","``","True","``","may","lead","to","errors","when","output","from","old","data","are","loaded","while","user","would","expect","the","pipeline","to","use","new","data","instead",".","force_fitting","(","bool",")",":","If","True",",","Step","transformer","will","be","fitted","(","via","`","fit_transform","`",")","even","if","``","<","experiment_directory",">","\/transformers\/","<","step_name",">","``","exists",".","Default","``","True","``",":","fit","transformer","each","time","`","fit_transform","(",")","`","is","called",".","Helpful","when","one","wants","to","use","``","persist_output=True","``","and","load","``","persist_output=True","``","on","a","previous","Step","and","fit","current","Step","multiple","times",".","This","is","a","typical","scenario","for","tuning","hyperparameters","for","an","ensemble","model","trained","on","the","output","from","first","level","models","or","a","model","build","on","features","that","are","time","consuming","to","compute","."]
code: def _validate_step_name(self, name):
      if name is not None:
          assert (
              isinstance(name, str)
              or isinstance(name, float)
              or isinstance(name, int)
          ), "Step name must be str, float or int. Got {} instead.".format(
              type(name)
          )
  
imports: import os
  import pprint
  from collections import defaultdict
  from sklearn.externals import joblib
  from steppy.adapter import Adapter, AdapterError
  from steppy.utils import display_upstream_structure, persist_as_png, get_logger, initialize_logger
label_desc: Use image node_id 29 for calling the Step obj's underlying member method code with example usage: obj._validate_step_name(name) without return types