docstr: Container for individual :class:`FeatureEngineer` step functions
  
  Compartmentalizes functions of singular engineer steps and allows for greater customization
  than a raw engineer step function
  
  Parameters
  ----------
  f: Callable
      Feature engineering step function that requests, modifies, and returns datasets `params`
  
      Step functions should follow these guidelines:
  
          1. Request as input a subset of the 11 data strings listed in `params`
          2. Do whatever you want to the DataFrames given as input
          3. Return new DataFrame values of the input parameters in same order as requested
  
      If performing a task like target transformation, causing predictions to be transformed,
      it is often desirable to inverse-transform the predictions to be of the expected form.
      This can easily be done by returning an extra value from `f` (after the datasets) that
      is either a callable, or a transformer class that was fitted during the execution of `f`
      and implements an `inverse_transform` method. This is the only instance in which it is
      acceptable for `f` to return values that don't mimic its input parameters. See the
      engineer function definition using SKLearn's `QuantileTransformer` in the Examples
      section below for an actual inverse-transformation-compatible implementation
  stage: String in {"pre_cv", "intra_cv"}, or None, default=None
      Feature engineering stage during which the callable `f` will be given the datasets
      `params` to modify and return. If None, will be inferred based on `params`.
  
          * "pre_cv" functions are applied only once in the experiment: when it starts
          * "intra_cv" functions are reapplied for each fold in the cross-validation splits
  
      If `stage` is left to be inferred, "pre_cv" will *usually* be selected. However, if
      any `params` (or parameters in the signature of `f`) are prefixed with "validation..."
      or "non_train...", then `stage` will inferred as "intra_cv". See the Notes section
      below for suggestions on the `stage` to use for different functions
  name: String, or None, default=None
      Identifier for the transformation applied by this engineering step. If None,
      `f.__name__` will be used
  params: Tuple[str], or None, default=None
      Dataset names requested by feature engineering step callable `f`. If None, will be
      inferred by parsing the signature of `f`. Must be a subset of the following 11 strings:
  
      Input Data
  
      1. "train_inputs"
      2. "validation_inputs"
      3. "holdout_inputs"
      4. "test_inputs"
      5. "all_inputs"
          ``("train_inputs" + ["validation_inputs"] + "holdout_inputs" + "test_inputs")``
      6. "non_train_inputs"
          ``(["validation_inputs"] + "holdout_inputs" + "test_inputs")``
  
      Target Data
  
      7. "train_targets"
      8. "validation_targets"
      9. "holdout_targets"
      10. "all_targets"
          ``("train_targets" + ["validation_targets"] + "holdout_targets")``
      11. "non_train_targets"
          ``(["validation_targets"] + "holdout_targets")``
  
      As an alternative to the above list, just remember that the first half of all parameter
      names should be one of {"train", "validation", "holdout", "test", "all", "non_train"},
      and the second half should be either "inputs" or "targets". The only exception to this
      rule is "test_targets", which doesn't exist.
  
      Inference of "validation" `params` is affected by `stage`. During the "pre_cv" stage,
      the validation dataset has not yet been created and is still a part of the train
      dataset. During the "intra_cv" stage, the validation dataset is created by removing a
      portion of the train dataset, and their values passed to `f` reflect this fact. This
      also means that the values of the merged ("all"/"non_train"-prefixed) datasets may or
      may not contain "validation" data depending on the `stage`; however, this is all handled
      internally, so you probably don't need to worry about it.
  
      `params` may not include multiple references to the same dataset, either directly or
      indirectly. This means `("train_inputs", "train_inputs")` is invalid due to duplicate
      direct references. Less obviously, `("train_inputs", "all_inputs")` is invalid because
      "all_inputs" includes "train_inputs"
  do_validate: Boolean, or "strict", default=False
      ... Experimental...
      Whether to validate the datasets resulting from feature engineering steps. If True,
      hashes of the new datasets will be compared to those of the originals to ensure they
      were actually modified. Results will be logged. If `do_validate` = "strict", an
      exception will be raised if any anomalies are found, rather than logging a message. If
      `do_validate` = False, no validation will be performed
  
  See Also
  --------
  :class:`FeatureEngineer`
      The container for `EngineerStep` instances - `EngineerStep`s should always be provided
      to HyperparameterHunter through a `FeatureEngineer`
  :class:`~hyperparameter_hunter.space.dimensions.Categorical`
      Can be used during optimization to search through a group of `EngineerStep`s given as
      `categories`. The `optional` kwarg of `Categorical` designates a `FeatureEngineer` step
      that may be one of the `EngineerStep`s in `categories`, or may be omitted entirely
  :func:`get_engineering_step_stage`
      More information on `stage` inference and situations where overriding it may be prudent
  
  Notes
  -----
  `stage`: Generally, feature engineering conducted in the "pre_cv" stage should regard each
  sample/row as independent entities. For example, steps like converting a string day of the
  week to one-hot encoded columns, or imputing missing values by replacement with -1 might be
  conducted "pre_cv", since they are unlikely to introduce an information leakage. Conversely,
  steps like scaling/normalization, whose results for the data in one row are affected by the
  data in other rows should be performed "intra_cv" in order to recalculate the final values
  of the datasets for each cross validation split and avoid information leakage.
  
  `params`: In the list of the 11 valid `params` strings, "test_inputs" is notably missing the
  "..._targets" counterpart accompanying the other datasets. The "targets" suffix is missing
  because test data targets are never given. Note that although "test_inputs" is still
  included in both "all_inputs" and "non_train_inputs", its lack of a target column means that
  "all_targets" and "non_train_targets" may have different lengths than their
  "inputs"-suffixed counterparts
  
  Examples
  --------
  >>> from sklearn.preprocessing import StandardScaler, QuantileTransformer
  >>> def s_scale(train_inputs, non_train_inputs):
  ...     s = StandardScaler()
  ...     train_inputs[train_inputs.columns] = s.fit_transform(train_inputs.values)
  ...     non_train_inputs[train_inputs.columns] = s.transform(non_train_inputs.values)
  ...     return train_inputs, non_train_inputs
  >>> # Sensible parameter defaults inferred based on `f`
  >>> es_0 = EngineerStep(s_scale)
  >>> es_0.stage
  'intra_cv'
  >>> es_0.name
  's_scale'
  >>> es_0.params
  ('train_inputs', 'non_train_inputs')
  >>> # Override `stage` if you want to fit your scaler on OOF data like a crazy person
  >>> es_1 = EngineerStep(s_scale, stage="pre_cv")
  >>> es_1.stage
  'pre_cv'
  
  *Watch out for multiple requests to the same data*
  
  >>> es_2 = EngineerStep(s_scale, params=("train_inputs", "all_inputs"))
  Traceback (most recent call last):
      File "feature_engineering.py", line ? in validate_dataset_names
  ValueError: Requested params include duplicate references to `train_inputs` by way of:
     - ('all_inputs', 'train_inputs')
     - ('train_inputs',)
  Each dataset may only be requested by a single param for each function
  
  *Error is the same if `(train_inputs, all_inputs)` is in the actual function signature*
  
  *EngineerStep functions aren't just limited to transformations. Make your own features!*
  
  >>> def sqr_sum(all_inputs):
  ...     all_inputs["square_sum"] = all_inputs.agg(
  ...         lambda row: np.sqrt(np.sum([np.square(_) for _ in row])), axis="columns"
  ...     )
  ...     return all_inputs
  >>> es_3 = EngineerStep(sqr_sum)
  >>> es_3.stage
  'pre_cv'
  >>> es_3.name
  'sqr_sum'
  >>> es_3.params
  ('all_inputs',)
  
  *Inverse-transformation Implementation:*
  
  >>> def q_transform(train_targets, non_train_targets):
  ...     t = QuantileTransformer(output_distribution="normal")
  ...     train_targets[train_targets.columns] = t.fit_transform(train_targets.values)
  ...     non_train_targets[train_targets.columns] = t.transform(non_train_targets.values)
  ...     return train_targets, non_train_targets, t
  >>> # Note that `train_targets` and `non_train_targets` must still be returned in order,
  >>> #   but they are followed by `t`, an instance of `QuantileTransformer` we just fitted,
  >>> #   whose `inverse_transform` method will be called on predictions
  >>> es_4 = EngineerStep(q_transform)
  >>> es_4.stage
  'intra_cv'
  >>> es_4.name
  'q_transform'
  >>> es_4.params
  ('train_targets', 'non_train_targets')
  >>> # `params` does not include any returned transformers - Only data requested as input
code: def __init__(
      self,
      f: Callable,
      stage=None,
      name=None,
      params=None,
      do_validate=False,
  ):
      """Container for individual :class:`FeatureEngineer` step functions
  
      Compartmentalizes functions of singular engineer steps and allows for greater customization
      than a raw engineer step function
  
      Parameters
      ----------
      f: Callable
          Feature engineering step function that requests, modifies, and returns datasets `params`
  
          Step functions should follow these guidelines:
  
              1. Request as input a subset of the 11 data strings listed in `params`
              2. Do whatever you want to the DataFrames given as input
              3. Return new DataFrame values of the input parameters in same order as requested
  
          If performing a task like target transformation, causing predictions to be transformed,
          it is often desirable to inverse-transform the predictions to be of the expected form.
          This can easily be done by returning an extra value from `f` (after the datasets) that
          is either a callable, or a transformer class that was fitted during the execution of `f`
          and implements an `inverse_transform` method. This is the only instance in which it is
          acceptable for `f` to return values that don't mimic its input parameters. See the
          engineer function definition using SKLearn's `QuantileTransformer` in the Examples
          section below for an actual inverse-transformation-compatible implementation
      stage: String in {"pre_cv", "intra_cv"}, or None, default=None
          Feature engineering stage during which the callable `f` will be given the datasets
          `params` to modify and return. If None, will be inferred based on `params`.
  
              * "pre_cv" functions are applied only once in the experiment: when it starts
              * "intra_cv" functions are reapplied for each fold in the cross-validation splits
  
          If `stage` is left to be inferred, "pre_cv" will *usually* be selected. However, if
          any `params` (or parameters in the signature of `f`) are prefixed with "validation..."
          or "non_train...", then `stage` will inferred as "intra_cv". See the Notes section
          below for suggestions on the `stage` to use for different functions
      name: String, or None, default=None
          Identifier for the transformation applied by this engineering step. If None,
          `f.__name__` will be used
      params: Tuple[str], or None, default=None
          Dataset names requested by feature engineering step callable `f`. If None, will be
          inferred by parsing the signature of `f`. Must be a subset of the following 11 strings:
  
          Input Data
  
          1. "train_inputs"
          2. "validation_inputs"
          3. "holdout_inputs"
          4. "test_inputs"
          5. "all_inputs"
              ``("train_inputs" + ["validation_inputs"] + "holdout_inputs" + "test_inputs")``
          6. "non_train_inputs"
              ``(["validation_inputs"] + "holdout_inputs" + "test_inputs")``
  
          Target Data
  
          7. "train_targets"
          8. "validation_targets"
          9. "holdout_targets"
          10. "all_targets"
              ``("train_targets" + ["validation_targets"] + "holdout_targets")``
          11. "non_train_targets"
              ``(["validation_targets"] + "holdout_targets")``
  
          As an alternative to the above list, just remember that the first half of all parameter
          names should be one of {"train", "validation", "holdout", "test", "all", "non_train"},
          and the second half should be either "inputs" or "targets". The only exception to this
          rule is "test_targets", which doesn't exist.
  
          Inference of "validation" `params` is affected by `stage`. During the "pre_cv" stage,
          the validation dataset has not yet been created and is still a part of the train
          dataset. During the "intra_cv" stage, the validation dataset is created by removing a
          portion of the train dataset, and their values passed to `f` reflect this fact. This
          also means that the values of the merged ("all"/"non_train"-prefixed) datasets may or
          may not contain "validation" data depending on the `stage`; however, this is all handled
          internally, so you probably don't need to worry about it.
  
          `params` may not include multiple references to the same dataset, either directly or
          indirectly. This means `("train_inputs", "train_inputs")` is invalid due to duplicate
          direct references. Less obviously, `("train_inputs", "all_inputs")` is invalid because
          "all_inputs" includes "train_inputs"
      do_validate: Boolean, or "strict", default=False
          ... Experimental...
          Whether to validate the datasets resulting from feature engineering steps. If True,
          hashes of the new datasets will be compared to those of the originals to ensure they
          were actually modified. Results will be logged. If `do_validate` = "strict", an
          exception will be raised if any anomalies are found, rather than logging a message. If
          `do_validate` = False, no validation will be performed
  
      See Also
      --------
      :class:`FeatureEngineer`
          The container for `EngineerStep` instances - `EngineerStep`s should always be provided
          to HyperparameterHunter through a `FeatureEngineer`
      :class:`~hyperparameter_hunter.space.dimensions.Categorical`
          Can be used during optimization to search through a group of `EngineerStep`s given as
          `categories`. The `optional` kwarg of `Categorical` designates a `FeatureEngineer` step
          that may be one of the `EngineerStep`s in `categories`, or may be omitted entirely
      :func:`get_engineering_step_stage`
          More information on `stage` inference and situations where overriding it may be prudent
  
      Notes
      -----
      `stage`: Generally, feature engineering conducted in the "pre_cv" stage should regard each
      sample/row as independent entities. For example, steps like converting a string day of the
      week to one-hot encoded columns, or imputing missing values by replacement with -1 might be
      conducted "pre_cv", since they are unlikely to introduce an information leakage. Conversely,
      steps like scaling/normalization, whose results for the data in one row are affected by the
      data in other rows should be performed "intra_cv" in order to recalculate the final values
      of the datasets for each cross validation split and avoid information leakage.
  
      `params`: In the list of the 11 valid `params` strings, "test_inputs" is notably missing the
      "..._targets" counterpart accompanying the other datasets. The "targets" suffix is missing
      because test data targets are never given. Note that although "test_inputs" is still
      included in both "all_inputs" and "non_train_inputs", its lack of a target column means that
      "all_targets" and "non_train_targets" may have different lengths than their
      "inputs"-suffixed counterparts
  
      Examples
      --------
      >>> from sklearn.preprocessing import StandardScaler, QuantileTransformer
      >>> def s_scale(train_inputs, non_train_inputs):
      ...     s = StandardScaler()
      ...     train_inputs[train_inputs.columns] = s.fit_transform(train_inputs.values)
      ...     non_train_inputs[train_inputs.columns] = s.transform(non_train_inputs.values)
      ...     return train_inputs, non_train_inputs
      >>> # Sensible parameter defaults inferred based on `f`
      >>> es_0 = EngineerStep(s_scale)
      >>> es_0.stage
      'intra_cv'
      >>> es_0.name
      's_scale'
      >>> es_0.params
      ('train_inputs', 'non_train_inputs')
      >>> # Override `stage` if you want to fit your scaler on OOF data like a crazy person
      >>> es_1 = EngineerStep(s_scale, stage="pre_cv")
      >>> es_1.stage
      'pre_cv'
  
      *Watch out for multiple requests to the same data*
  
      >>> es_2 = EngineerStep(s_scale, params=("train_inputs", "all_inputs"))
      Traceback (most recent call last):
          File "feature_engineering.py", line ? in validate_dataset_names
      ValueError: Requested params include duplicate references to `train_inputs` by way of:
         - ('all_inputs', 'train_inputs')
         - ('train_inputs',)
      Each dataset may only be requested by a single param for each function
  
      *Error is the same if `(train_inputs, all_inputs)` is in the actual function signature*
  
      *EngineerStep functions aren't just limited to transformations. Make your own features!*
  
      >>> def sqr_sum(all_inputs):
      ...     all_inputs["square_sum"] = all_inputs.agg(
      ...         lambda row: np.sqrt(np.sum([np.square(_) for _ in row])), axis="columns"
      ...     )
      ...     return all_inputs
      >>> es_3 = EngineerStep(sqr_sum)
      >>> es_3.stage
      'pre_cv'
      >>> es_3.name
      'sqr_sum'
      >>> es_3.params
      ('all_inputs',)
  
      *Inverse-transformation Implementation:*
  
      >>> def q_transform(train_targets, non_train_targets):
      ...     t = QuantileTransformer(output_distribution="normal")
      ...     train_targets[train_targets.columns] = t.fit_transform(train_targets.values)
      ...     non_train_targets[train_targets.columns] = t.transform(non_train_targets.values)
      ...     return train_targets, non_train_targets, t
      >>> # Note that `train_targets` and `non_train_targets` must still be returned in order,
      >>> #   but they are followed by `t`, an instance of `QuantileTransformer` we just fitted,
      >>> #   whose `inverse_transform` method will be called on predictions
      >>> es_4 = EngineerStep(q_transform)
      >>> es_4.stage
      'intra_cv'
      >>> es_4.name
      'q_transform'
      >>> es_4.params
      ('train_targets', 'non_train_targets')
      >>> # `params` does not include any returned transformers - Only data requested as input
      """
      self._f = f
      self._name = name
      self.params = params
      self._stage = stage
      self.do_validate = do_validate
  
      self.inversion = None
      self.merged_datasets: List[str] = validate_dataset_names(
          self.params, self.stage
      )
      self.original_hashes = dict()
      self.updated_hashes = dict()
  
imports: from hyperparameter_hunter.keys.hashing import make_hash_sha256
  from hyperparameter_hunter.space.dimensions import Categorical, RejectedOptional
  from hyperparameter_hunter.utils.boltons_utils import remap, default_visit, default_enter
  from hyperparameter_hunter.utils.general_utils import subdict
  import ast
  from contextlib import suppress
  from inspect import getsource
  import numpy
  import pandas
  from typing import List, Callable, Dict, Union, Tuple
label_desc: Use image node_id 1 to create a new EngineerStep object  with example: obj = EngineerStep(f, stage, name, params, do_validate)