class_docstr: Creates a kernel dataset. This dataset can perform MICE on itself,
  and impute new data from models obtained during MICE.
  
  Parameters
  ----------
  data : np.ndarray or pandas DataFrame.
  
      .. code-block:: text
  
          The data to be imputed.
  
  variable_schema : None or list or dict, default=None
  
      .. code-block:: text
  
          Specifies the feature - target relationships used to train models.
          This parameter also controls which models are built. Models can be built
          even if a variable contains no missing values, or is not being imputed
          (train_nonmissing must be set to True).
  
              - If None, all columns will be used as features in the training of each model.
              - If list, all columns in data are used to impute the variables in the list
              - If dict the values will be used to impute the keys. Can be either column
                  indices or names (if data is a pd.DataFrame).
  
          No models will be trained for variables not specified by variable_schema
          (either by None, a list, or in dict keys).
  
  imputation_order: str, list[str], list[int], default="ascending"
  
      .. code-block:: text
  
          The order the imputations should occur in. If a string from the
          items below, all variables specified by variable_schema with
          missing data are imputed:
              ascending: variables are imputed from least to most missing
              descending: most to least missing
              roman: from left to right in the dataset
              arabic: from right to left in the dataset.
          If a list is provided:
              - the variables will be imputed in that order.
              - only variables with missing values should be included in the list.
              - must be a subset of variables specified by variable_schema.
          If a variable with missing values is in variable_schema, but not in
          imputation_order, then models to impute that variable will be trained,
          but the actual values will not be imputed. See examples for details.
  
  train_nonmissing: boolean
  
      .. code-block:: text
  
          Should models be trained for variables with no missing values? Useful if you
          expect you will need to impute new data which will have missing values, but
          the training data is fully recognized.
  
          If True, parameters are interpreted like so:
              - models are run for all variables specified by variable_schema
              - if variable_schema is None, models are run for all variables
              - each iteration, models build for fully recognized variables are
                  always trained after the models trained during mice.
              - imputation_order does not have any affect on fully recognized
                  variable model training.
  
          WARNING: Setting this to True without specifying a variable schema will build
          models for all variables in the dataset, whether they have missing values or
          not. This may or may not be what you want.
  
  data_subset: None or int or float or dict.
  
      .. code-block:: text
  
          Subsets the data used in each iteration, which can save a significant amount of time.
          This can also help with memory consumption, as the candidate data must be copied to
          make a feature dataset for lightgbm.
  
          The number of rows used for each variable is (# rows in raw data) - (# missing variable values)
          for each variable. data_subset takes a random sample of this.
  
          If float, must be 0.0 < data_subset <= 1.0. Interpreted as a percentage of available candidates
          If int must be data_subset >= 0. Interpreted as the number of candidates.
          If 0, no subsetting is done.
          If dict, keys must be variable names, and values must follow two above rules.
  
          It is recommended to carefully select this value for each variable if dealing
          with very large data that barely fits into memory.
  
  mean_match_scheme: Dict, default = None
  
      .. code-block:: text
  
          An instance of the miceforest.MeanMatchScheme class.
  
          If None is passed, a sensible default scheme is used. There are multiple helpful
          schemes that can be accessed from miceforest.builtin_mean_match_schemes, or
          you can build your own.
  
          A description of the defaults:
          - mean_match_default (default, if mean_match_scheme is None))
              This scheme is has medium speed and accuracy for most data.
  
              Categorical:
                  If mmc = 0, the class with the highest probability is chosen.
                  If mmc > 0, get N nearest neighbors from class probabilities.
                      Select 1 at random.
              Numeric:
                  If mmc = 0, the predicted value is used
                  If mmc > 0, obtain the mmc closest candidate
                      predictions and collect the associated
                      real candidate values. Choose 1 randomly.
  
          - mean_match_shap
              This scheme is the most accurate, but takes the longest.
              It works the same as mean_match_default, except all nearest
              neighbor searches are performed on the shap values of the
              predictions, instead of the predictions themselves.
  
          - mean_match_scheme_fast_cat:
              This scheme is faster for categorical variables,
              but may be less accurate as well..
  
              Categorical:
                  If mmc = 0, the class with the highest probability is chosen.
                  If mmc > 0, return class based on random draw weighted by
                      class probability for each sample.
              Numeric or binary:
                  If mmc = 0, the predicted value is used
                  If mmc > 0, obtain the mmc closest candidate
                      predictions and collect the associated
                      real candidate values. Choose 1 randomly.
  
  categorical_feature: str or list, default="auto"
  
      .. code-block:: text
  
          The categorical features in the dataset. This handling depends on class of impute_data:
  
              pandas DataFrame:
                  - "auto": categorical information is inferred from any columns with
                      datatype category or object.
                  - list of column names (or indices): Useful if all categorical columns
                      have already been cast to numeric encodings of some type, otherwise you
                      should just use "auto". Will throw an error if a list is provided AND
                      categorical dtypes exist in data. If a list is provided, values in the
                      columns must be consecutive integers starting at 0, as required by lightgbm.
  
              numpy ndarray:
                  - "auto": no categorical information is stored.
                  - list of column indices: Specified columns are treated as categorical. Column
                      values must be consecutive integers starting at 0, as required by lightgbm.
  
  initialization: str
  
      .. code-block:: text
  
          "random" - missing values will be filled in randomly from existing values.
          "empty" - lightgbm will start MICE without initial imputation
  
  save_all_iterations: boolean, optional(default=True)
  
      .. code-block:: text
  
          Save all the imputation values from all iterations, or just
          the latest. Saving all iterations allows for additional
          plotting, but may take more memory
  
  save_models: int
  
      .. code-block:: text
  
          Which models should be saved:
              = 0: no models are saved. Cannot get feature importance or
                  impute new data.
              = 1: only the last model iteration is saved. Can only get
                  feature importance of last iteration. New data is
                  imputed using the last model for all specified iterations.
                  This is only an issue if data is heavily Missing At Random.
              = 2: all model iterations are saved. Can get feature importance
                  for any iteration. When imputing new data, each iteration is
                  imputed using the model obtained at that iteration in mice.
                  This allows for imputations that most closely resemble those
                  that would have been obtained in mice.
  
  copy_data: boolean (default = False)
  
      .. code-block:: text
  
          Should the dataset be referenced directly? If False, this will cause
          the dataset to be altered in place. If a copy is created, it is saved
          in self.working_data. There are different ways in which the dataset
          can be altered:
  
          1) complete_data() will fill in missing values
          2) To save space, mice() references and manipulates self.working_data directly.
              If self.working_data is a reference to the original dataset, the original
              dataset will undergo these manipulations during the mice process.
              At the end of the mice process, missing values will be set back to np.NaN
              where they were originally missing.
  
  save_loggers: boolean (default = False)
  
      .. code-block:: text
  
          A logger is created each time mice() or impute_new_data() is called.
          If True, the loggers are stored in a list ImputationKernel.loggers.
          If you wish to start saving logs, call ImputationKernel.start_logging().
          If you wish to stop saving logs, call ImputationKernel.stop_logging().
  
  random_state: None,int, or numpy.random.RandomState
  
      .. code-block:: text
  
          The random_state ensures script reproducibility. It only ensures reproducible
          results if the same script is called multiple times. It does not guarantee
          reproducible results at the record level, if a record is imputed multiple
          different times. If reproducible record-results are desired, a seed must be
          passed for each record in the random_seed_array parameter.
class_docstr_tok: ["Creates","a","kernel","dataset",".","This","dataset","can","perform","MICE","on","itself",",","and","impute","new","data","from","models","obtained","during","MICE",".","Parameters","--","--","--","--","--","data",":","np.ndarray","or","pandas","DataFrame",".","..","code-block",":",":","text","The","data","to","be","imputed",".","variable_schema",":","None","or","list","or","dict",",","default=None","..","code-block",":",":","text","Specifies","the","feature","-","target","relationships","used","to","train","models",".","This","parameter","also","controls","which","models","are","built",".","Models","can","be","built","even","if","a","variable","contains","no","missing","values",",","or","is","not","being","imputed","(","train_nonmissing","must","be","set","to","True",")",".","-","If","None",",","all","columns","will","be","used","as","features","in","the","training","of","each","model",".","-","If","list",",","all","columns","in","data","are","used","to","impute","the","variables","in","the","list","-","If","dict","the","values","will","be","used","to","impute","the","keys",".","Can","be","either","column","indices","or","names","(","if","data","is","a","pd.DataFrame",")",".","No","models","will","be","trained","for","variables","not","specified","by","variable_schema","(","either","by","None",",","a","list",",","or","in","dict","keys",")",".","imputation_order",":","str",",","list","[","str","]",",","list","[","int","]",",","default=","''","ascending","''","..","code-block",":",":","text","The","order","the","imputations","should","occur","in",".","If","a","string","from","the","items","below",",","all","variables","specified","by","variable_schema","with","missing","data","are","imputed",":","ascending",":","variables","are","imputed","from","least","to","most","missing","descending",":","most","to","least","missing","roman",":","from","left","to","right","in","the","dataset","arabic",":","from","right","to","left","in","the","dataset",".","If","a","list","is","provided",":","-","the","variables","will","be","imputed","in","that","order",".","-","only","variables","with","missing","values","should","be","included","in","the","list",".","-","must","be","a","subset","of","variables","specified","by","variable_schema",".","If","a","variable","with","missing","values","is","in","variable_schema",",","but","not","in","imputation_order",",","then","models","to","impute","that","variable","will","be","trained",",","but","the","actual","values","will","not","be","imputed",".","See","examples","for","details",".","train_nonmissing",":","boolean","..","code-block",":",":","text","Should","models","be","trained","for","variables","with","no","missing","values","?","Useful","if","you","expect","you","will","need","to","impute","new","data","which","will","have","missing","values",",","but","the","training","data","is","fully","recognized",".","If","True",",","parameters","are","interpreted","like","so",":","-","models","are","run","for","all","variables","specified","by","variable_schema","-","if","variable_schema","is","None",",","models","are","run","for","all","variables","-","each","iteration",",","models","build","for","fully","recognized","variables","are","always","trained","after","the","models","trained","during","mice",".","-","imputation_order","does","not","have","any","affect","on","fully","recognized","variable","model","training",".","WARNING",":","Setting","this","to","True","without","specifying","a","variable","schema","will","build","models","for","all","variables","in","the","dataset",",","whether","they","have","missing","values","or","not",".","This","may","or","may","not","be","what","you","want",".","data_subset",":","None","or","int","or","float","or","dict",".","..","code-block",":",":","text","Subsets","the","data","used","in","each","iteration",",","which","can","save","a","significant","amount","of","time",".","This","can","also","help","with","memory","consumption",",","as","the","candidate","data","must","be","copied","to","make","a","feature","dataset","for","lightgbm",".","The","number","of","rows","used","for","each","variable","is","(","#","rows","in","raw","data",")","-","(","#","missing","variable","values",")","for","each","variable",".","data_subset","takes","a","random","sample","of","this",".","If","float",",","must","be","0.0","<","data_subset","<","=","1.0",".","Interpreted","as","a","percentage","of","available","candidates","If","int","must","be","data_subset",">","=","0",".","Interpreted","as","the","number","of","candidates",".","If","0",",","no","subsetting","is","done",".","If","dict",",","keys","must","be","variable","names",",","and","values","must","follow","two","above","rules",".","It","is","recommended","to","carefully","select","this","value","for","each","variable","if","dealing","with","very","large","data","that","barely","fits","into","memory",".","mean_match_scheme",":","Dict",",","default","=","None","..","code-block",":",":","text","An","instance","of","the","miceforest.MeanMatchScheme","class",".","If","None","is","passed",",","a","sensible","default","scheme","is","used",".","There","are","multiple","helpful","schemes","that","can","be","accessed","from","miceforest.builtin_mean_match_schemes",",","or","you","can","build","your","own",".","A","description","of","the","defaults",":","-","mean_match_default","(","default",",","if","mean_match_scheme","is","None",")",")","This","scheme","is","has","medium","speed","and","accuracy","for","most","data",".","Categorical",":","If","mmc","=","0",",","the","class","with","the","highest","probability","is","chosen",".","If","mmc",">","0",",","get","N","nearest","neighbors","from","class","probabilities",".","Select","1","at","random",".","Numeric",":","If","mmc","=","0",",","the","predicted","value","is","used","If","mmc",">","0",",","obtain","the","mmc","closest","candidate","predictions","and","collect","the","associated","real","candidate","values",".","Choose","1","randomly",".","-","mean_match_shap","This","scheme","is","the","most","accurate",",","but","takes","the","longest",".","It","works","the","same","as","mean_match_default",",","except","all","nearest","neighbor","searches","are","performed","on","the","shap","values","of","the","predictions",",","instead","of","the","predictions","themselves",".","-","mean_match_scheme_fast_cat",":","This","scheme","is","faster","for","categorical","variables",",","but","may","be","less","accurate","as","well","..","Categorical",":","If","mmc","=","0",",","the","class","with","the","highest","probability","is","chosen",".","If","mmc",">","0",",","return","class","based","on","random","draw","weighted","by","class","probability","for","each","sample",".","Numeric","or","binary",":","If","mmc","=","0",",","the","predicted","value","is","used","If","mmc",">","0",",","obtain","the","mmc","closest","candidate","predictions","and","collect","the","associated","real","candidate","values",".","Choose","1","randomly",".","categorical_feature",":","str","or","list",",","default=","''","auto","''","..","code-block",":",":","text","The","categorical","features","in","the","dataset",".","This","handling","depends","on","class","of","impute_data",":","pandas","DataFrame",":","-","``","auto","''",":","categorical","information","is","inferred","from","any","columns","with","datatype","category","or","object",".","-","list","of","column","names","(","or","indices",")",":","Useful","if","all","categorical","columns","have","already","been","cast","to","numeric","encodings","of","some","type",",","otherwise","you","should","just","use","``","auto","''",".","Will","throw","an","error","if","a","list","is","provided","AND","categorical","dtypes","exist","in","data",".","If","a","list","is","provided",",","values","in","the","columns","must","be","consecutive","integers","starting","at","0",",","as","required","by","lightgbm",".","numpy","ndarray",":","-","``","auto","''",":","no","categorical","information","is","stored",".","-","list","of","column","indices",":","Specified","columns","are","treated","as","categorical",".","Column","values","must","be","consecutive","integers","starting","at","0",",","as","required","by","lightgbm",".","initialization",":","str","..","code-block",":",":","text","``","random","''","-","missing","values","will","be","filled","in","randomly","from","existing","values",".","``","empty","''","-","lightgbm","will","start","MICE","without","initial","imputation","save_all_iterations",":","boolean",",","optional","(","default=True",")","..","code-block",":",":","text","Save","all","the","imputation","values","from","all","iterations",",","or","just","the","latest",".","Saving","all","iterations","allows","for","additional","plotting",",","but","may","take","more","memory","save_models",":","int","..","code-block",":",":","text","Which","models","should","be","saved",":","=","0",":","no","models","are","saved",".","Can","not","get","feature","importance","or","impute","new","data",".","=","1",":","only","the","last","model","iteration","is","saved",".","Can","only","get","feature","importance","of","last","iteration",".","New","data","is","imputed","using","the","last","model","for","all","specified","iterations",".","This","is","only","an","issue","if","data","is","heavily","Missing","At","Random",".","=","2",":","all","model","iterations","are","saved",".","Can","get","feature","importance","for","any","iteration",".","When","imputing","new","data",",","each","iteration","is","imputed","using","the","model","obtained","at","that","iteration","in","mice",".","This","allows","for","imputations","that","most","closely","resemble","those","that","would","have","been","obtained","in","mice",".","copy_data",":","boolean","(","default","=","False",")","..","code-block",":",":","text","Should","the","dataset","be","referenced","directly","?","If","False",",","this","will","cause","the","dataset","to","be","altered","in","place",".","If","a","copy","is","created",",","it","is","saved","in","self.working_data",".","There","are","different","ways","in","which","the","dataset","can","be","altered",":","1",")","complete_data","(",")","will","fill","in","missing","values","2",")","To","save","space",",","mice","(",")","references","and","manipulates","self.working_data","directly",".","If","self.working_data","is","a","reference","to","the","original","dataset",",","the","original","dataset","will","undergo","these","manipulations","during","the","mice","process",".","At","the","end","of","the","mice","process",",","missing","values","will","be","set","back","to","np.NaN","where","they","were","originally","missing",".","save_loggers",":","boolean","(","default","=","False",")","..","code-block",":",":","text","A","logger","is","created","each","time","mice","(",")","or","impute_new_data","(",")","is","called",".","If","True",",","the","loggers","are","stored","in","a","list","ImputationKernel.loggers",".","If","you","wish","to","start","saving","logs",",","call","ImputationKernel.start_logging","(",")",".","If","you","wish","to","stop","saving","logs",",","call","ImputationKernel.stop_logging","(",")",".","random_state",":","None",",","int",",","or","numpy.random.RandomState","..","code-block",":",":","text","The","random_state","ensures","script","reproducibility",".","It","only","ensures","reproducible","results","if","the","same","script","is","called","multiple","times",".","It","does","not","guarantee","reproducible","results","at","the","record","level",",","if","a","record","is","imputed","multiple","different","times",".","If","reproducible","record-results","are","desired",",","a","seed","must","be","passed","for","each","record","in","the","random_seed_array","parameter","."]
docstr: Impute a new dataset
  
  Uses the models obtained while running MICE to impute new data,
  without fitting new models. Pulls mean matching candidates from
  the original data.
  
  save_models must be > 0. If save_models == 1, the last model
  obtained in mice is used for every iteration. If save_models > 1,
  the model obtained at each iteration is used to impute the new
  data for that iteration. If specified iterations is greater than
  the number of iterations run so far using mice, the last model
  is used for each additional iteration.
  
  Type checking is not done. It is up to the user to ensure that the
  kernel data matches the new data being imputed.
  
  Parameters
  ----------
  new_data: pandas DataFrame or numpy ndarray
      The new data to impute
  
  datasets: int or List[int] (default = None)
      The datasets from the kernel to use to impute the new data.
      If None, all datasets from the kernel are used.
  
  iterations: int
      The number of iterations to run.
      If None, the same number of iterations run so far in mice is used.
  
  save_all_iterations: bool
      Should the imputation values of all iterations be archived?
      If False, only the latest imputation values are saved.
  
  copy_data: boolean
      Should the dataset be referenced directly? This will cause the dataset to be altered
      in place. If a copy is created, it is saved in self.working_data. There are different
      ways in which the dataset can be altered:
  
      1) complete_data() will fill in missing values
      2) mice() references and manipulates self.working_data directly.
  
  random_state: int or np.random.RandomState or None (default=None)
      The random state of the process. Ensures reproducibility. If None, the random state
      of the kernel is used. Beware, this permanently alters the random state of the kernel
      and ensures non-reproduceable results, unless the entire process up to this point
      is re-run.
  
  random_seed_array: None or np.ndarray (int32)
  
      .. code-block:: text
  
          Record-level seeds.
  
          Ensures deterministic imputations at the record level. random_seed_array causes
          deterministic imputations for each record no matter what dataset each record is
          imputed with, assuming the same number of iterations and datasets are used.
          If random_seed_array os passed, random_state must also be passed.
  
          Record-level imputations are deterministic if the following conditions are met:
              1) The associated seed is the same.
              2) The same kernel is used.
              3) The same number of iterations are run.
              4) The same number of datasets are run.
  
          Notes:
              a) This will slightly slow down the imputation process, because random
              number generation in numpy can no longer be vectorized. If you don't have a
              specific need for deterministic imputations at the record level, it is better to
              keep this parameter as None.
  
              b) Using this parameter may change the global numpy seed by calling np.random.seed().
  
              c) Internally, these seeds are hashed each time they are used, in order
              to obtain different results for each dataset / iteration.
  
  
  verbose: boolean
      Should information about the process be printed?
  
  Returns
  -------
  miceforest.ImputedData
docstr_tok: ["Impute","a","new","dataset","Uses","the","models","obtained","while","running","MICE","to","impute","new","data",",","without","fitting","new","models",".","Pulls","mean","matching","candidates","from","the","original","data",".","save_models","must","be",">","0",".","If","save_models","==","1",",","the","last","model","obtained","in","mice","is","used","for","every","iteration",".","If","save_models",">","1",",","the","model","obtained","at","each","iteration","is","used","to","impute","the","new","data","for","that","iteration",".","If","specified","iterations","is","greater","than","the","number","of","iterations","run","so","far","using","mice",",","the","last","model","is","used","for","each","additional","iteration",".","Type","checking","is","not","done",".","It","is","up","to","the","user","to","ensure","that","the","kernel","data","matches","the","new","data","being","imputed",".","Parameters","--","--","--","--","--","new_data",":","pandas","DataFrame","or","numpy","ndarray","The","new","data","to","impute","datasets",":","int","or","List","[","int","]","(","default","=","None",")","The","datasets","from","the","kernel","to","use","to","impute","the","new","data",".","If","None",",","all","datasets","from","the","kernel","are","used",".","iterations",":","int","The","number","of","iterations","to","run",".","If","None",",","the","same","number","of","iterations","run","so","far","in","mice","is","used",".","save_all_iterations",":","bool","Should","the","imputation","values","of","all","iterations","be","archived","?","If","False",",","only","the","latest","imputation","values","are","saved",".","copy_data",":","boolean","Should","the","dataset","be","referenced","directly","?","This","will","cause","the","dataset","to","be","altered","in","place",".","If","a","copy","is","created",",","it","is","saved","in","self.working_data",".","There","are","different","ways","in","which","the","dataset","can","be","altered",":","1",")","complete_data","(",")","will","fill","in","missing","values","2",")","mice","(",")","references","and","manipulates","self.working_data","directly",".","random_state",":","int","or","np.random.RandomState","or","None","(","default=None",")","The","random","state","of","the","process",".","Ensures","reproducibility",".","If","None",",","the","random","state","of","the","kernel","is","used",".","Beware",",","this","permanently","alters","the","random","state","of","the","kernel","and","ensures","non-reproduceable","results",",","unless","the","entire","process","up","to","this","point","is","re-run",".","random_seed_array",":","None","or","np.ndarray","(","int32",")","..","code-block",":",":","text","Record-level","seeds",".","Ensures","deterministic","imputations","at","the","record","level",".","random_seed_array","causes","deterministic","imputations","for","each","record","no","matter","what","dataset","each","record","is","imputed","with",",","assuming","the","same","number","of","iterations","and","datasets","are","used",".","If","random_seed_array","os","passed",",","random_state","must","also","be","passed",".","Record-level","imputations","are","deterministic","if","the","following","conditions","are","met",":","1",")","The","associated","seed","is","the","same",".","2",")","The","same","kernel","is","used",".","3",")","The","same","number","of","iterations","are","run",".","4",")","The","same","number","of","datasets","are","run",".","Notes",":","a",")","This","will","slightly","slow","down","the","imputation","process",",","because","random","number","generation","in","numpy","can","no","longer","be","vectorized",".","If","you","do","n't","have","a","specific","need","for","deterministic","imputations","at","the","record","level",",","it","is","better","to","keep","this","parameter","as","None",".","b",")","Using","this","parameter","may","change","the","global","numpy","seed","by","calling","np.random.seed","(",")",".","c",")","Internally",",","these","seeds","are","hashed","each","time","they","are","used",",","in","order","to","obtain","different","results","for","each","dataset","\/","iteration",".","verbose",":","boolean","Should","information","about","the","process","be","printed","?","Returns","--","--","--","-","miceforest.ImputedData"]
code: def impute_new_data(
      self,
      new_data: _t_dat,
      datasets: Optional[List[int]] = None,
      iterations: Optional[int] = None,
      save_all_iterations: bool = True,
      copy_data: bool = True,
      random_state: _t_random_state = None,
      random_seed_array: Optional[np.ndarray] = None,
      verbose: bool = False,
  ) -> ImputedData:
      """
      Impute a new dataset
  
      Uses the models obtained while running MICE to impute new data,
      without fitting new models. Pulls mean matching candidates from
      the original data.
  
      save_models must be > 0. If save_models == 1, the last model
      obtained in mice is used for every iteration. If save_models > 1,
      the model obtained at each iteration is used to impute the new
      data for that iteration. If specified iterations is greater than
      the number of iterations run so far using mice, the last model
      is used for each additional iteration.
  
      Type checking is not done. It is up to the user to ensure that the
      kernel data matches the new data being imputed.
  
      Parameters
      ----------
      new_data: pandas DataFrame or numpy ndarray
          The new data to impute
  
      datasets: int or List[int] (default = None)
          The datasets from the kernel to use to impute the new data.
          If None, all datasets from the kernel are used.
  
      iterations: int
          The number of iterations to run.
          If None, the same number of iterations run so far in mice is used.
  
      save_all_iterations: bool
          Should the imputation values of all iterations be archived?
          If False, only the latest imputation values are saved.
  
      copy_data: boolean
          Should the dataset be referenced directly? This will cause the dataset to be altered
          in place. If a copy is created, it is saved in self.working_data. There are different
          ways in which the dataset can be altered:
  
          1) complete_data() will fill in missing values
          2) mice() references and manipulates self.working_data directly.
  
      random_state: int or np.random.RandomState or None (default=None)
          The random state of the process. Ensures reproducibility. If None, the random state
          of the kernel is used. Beware, this permanently alters the random state of the kernel
          and ensures non-reproduceable results, unless the entire process up to this point
          is re-run.
  
      random_seed_array: None or np.ndarray (int32)
  
          .. code-block:: text
  
              Record-level seeds.
  
              Ensures deterministic imputations at the record level. random_seed_array causes
              deterministic imputations for each record no matter what dataset each record is
              imputed with, assuming the same number of iterations and datasets are used.
              If random_seed_array os passed, random_state must also be passed.
  
              Record-level imputations are deterministic if the following conditions are met:
                  1) The associated seed is the same.
                  2) The same kernel is used.
                  3) The same number of iterations are run.
                  4) The same number of datasets are run.
  
              Notes:
                  a) This will slightly slow down the imputation process, because random
                  number generation in numpy can no longer be vectorized. If you don't have a
                  specific need for deterministic imputations at the record level, it is better to
                  keep this parameter as None.
  
                  b) Using this parameter may change the global numpy seed by calling np.random.seed().
  
                  c) Internally, these seeds are hashed each time they are used, in order
                  to obtain different results for each dataset / iteration.
  
  
      verbose: boolean
          Should information about the process be printed?
  
      Returns
      -------
      miceforest.ImputedData
  
      """
  
      datasets = (
          list(range(self.dataset_count()))
          if datasets is None
          else datasets
      )
      kernel_iterations = self.iteration_count()
      iterations = (
          kernel_iterations if iterations is None else iterations
      )
      iter_pairs = self._iter_pairs(iterations)
      __IND_TIMED_EVENTS = ["prepare_xy", "predict", "mean_matching"]
      logger = Logger(
          name=f"ind {str(iter_pairs[0][1])}-{str(iter_pairs[-1][1])}",
          verbose=verbose,
      )
  
      if isinstance(self.working_data, pd_DataFrame):
          assert isinstance(new_data, pd_DataFrame)
          assert set(self.working_data.columns) == set(
              new_data.columns
          ), "Different columns from original dataset."
          assert all(
              [
                  self.working_data[col].dtype == new_data[col].dtype
                  for col in self.working_data.columns
              ]
          ), "Column types are not the same as the original data. Check categorical columns."
  
      if self.save_models < 1:
          raise ValueError("No models were saved.")
  
      imputed_data = ImputedData(
          impute_data=new_data,
          datasets=len(datasets),
          variable_schema=self.variable_schema.copy(),
          imputation_order=self.variable_training_order.copy(),
          train_nonmissing=False,
          categorical_feature=self.categorical_feature,
          save_all_iterations=save_all_iterations,
          copy_data=copy_data,
      )
  
      ### Manage Randomness.
      if random_state is None:
          assert (
              random_seed_array is None
          ), "random_state is also required when using random_seed_array"
          random_state = self._random_state
      else:
          random_state = ensure_rng(random_state)
      # use_seed_array = random_seed_array is not None
      random_seed_array = self._initialize_random_seed_array(
          random_seed_array=random_seed_array,
          expected_shape=imputed_data.data_shape[0],
      )
      self._initialize_dataset(
          imputed_data,
          random_state=random_state,
          random_seed_array=random_seed_array,
      )
  
      for ds_kern in datasets:
          logger.log("Dataset " + str(ds_kern))
          self.complete_data(dataset=ds_kern, inplace=True)
          ds_new = datasets.index(ds_kern)
          imputed_data.complete_data(dataset=ds_new, inplace=True)
  
          for iter_abs, iter_rel in iter_pairs:
              logger.log(str(iter_rel) + " ", end="")
  
              # Determine which model iteration to grab
              if self.save_models == 1 or iter_abs > kernel_iterations:
                  iter_model = kernel_iterations
              else:
                  iter_model = iter_abs
  
              for var in imputed_data.imputation_order:
                  var_name = self._get_var_name_from_scalar(var)
                  logger.log(" | " + var_name, end="")
                  log_context = {
                      "dataset": ds_kern,
                      "variable_name": var_name,
                      "iteration": iter_rel,
                  }
                  nawhere = imputed_data.na_where[var]
                  predictor_variables = self.variable_schema[var]
  
                  # Select our model.
                  current_model = self.get_model(
                      variable=var,
                      dataset=ds_kern,
                      iteration=iter_model,
                  )
                  objective = current_model.params["objective"]
                  model_seed = current_model.params["seed"]
  
                  # Start building mean matching kwargs
                  mean_match_args = (
                      self.mean_match_scheme.get_mean_match_args(
                          objective
                      )
                  )
                  mm_kwargs = {}
  
                  if "lgb_booster" in mean_match_args:
                      mm_kwargs["lgb_booster"] = current_model
  
                  # Procure bachelor information
                  if {
                      "bachelor_preds",
                      "bachelor_features",
                  }.intersection(mean_match_args):
                      logger.set_start_time()
                      bachelor_features = _subset_data(
                          imputed_data.working_data,
                          row_ind=imputed_data.na_where[var],
                          col_ind=predictor_variables,
                      )
                      logger.record_time(
                          timed_event="prepare_xy", **log_context
                      )
                      if "bachelor_features" in mean_match_args:
                          mm_kwargs[
                              "bachelor_features"
                          ] = bachelor_features
  
                      if "bachelor_preds" in mean_match_args:
                          logger.set_start_time()
                          bachelor_preds = (
                              self.mean_match_scheme.model_predict(
                                  current_model, bachelor_features
                              )
                          )
                          logger.record_time(
                              timed_event="predict", **log_context
                          )
                          mm_kwargs["bachelor_preds"] = bachelor_preds
  
                  # Procure candidate information
                  if {
                      "candidate_values",
                      "candidate_features",
                      "candidate_preds",
                  }.intersection(mean_match_args):
                      # Need to return candidate features if we need to calculate
                      # candidate_preds or candidate_features is needed by mean matching function
                      calculate_candidate_preds = (
                          ds_kern,
                          var,
                          iter_model,
                      ) not in self.candidate_preds.keys() and "candidate_preds" in mean_match_args
                      return_features = (
                          "candidate_features" in mean_match_args
                      ) or calculate_candidate_preds
                      # Set up like this so we only have to subset once
                      logger.set_start_time()
                      if return_features:
                          (
                              candidate_features,
                              candidate_values,
                              _,
                          ) = self._make_features_label(
                              variable=var,
                              subset_count=self.data_subset[var],
                              random_seed=model_seed,
                          )
                      else:
                          candidate_values = self._make_label(
                              variable=var,
                              subset_count=self.data_subset[var],
                              random_seed=model_seed,
                          )
                      logger.record_time(
                          timed_event="prepare_xy", **log_context
                      )
  
                      if "candidate_values" in mean_match_args:
                          # lightgbm requires integers for label. Categories won't work.
                          if candidate_values.dtype.name == "category":
                              candidate_values = (
                                  candidate_values.cat.codes
                              )
                          mm_kwargs[
                              "candidate_values"
                          ] = candidate_values
  
                      if "candidate_features" in mean_match_args:
                          mm_kwargs[
                              "candidate_features"
                          ] = candidate_features
  
                      # Calculate the candidate predictions if
                      # the mean matching function calls for it
                      if "candidate_preds" in mean_match_args:
                          if calculate_candidate_preds:
                              logger.set_start_time()
                              candidate_preds = (
                                  self.mean_match_scheme.model_predict(
                                      current_model, candidate_features
                                  )
                              )
                              logger.record_time(
                                  timed_event="predict", **log_context
                              )
                          else:
                              candidate_preds = self.candidate_preds[
                                  ds_kern, var, iter_model
                              ]
                          mm_kwargs["candidate_preds"] = candidate_preds
  
                  if "random_state" in mean_match_args:
                      mm_kwargs["random_state"] = random_state
  
                  if "hashed_seeds" in mean_match_args:
                      if isinstance(random_seed_array, np.ndarray):
                          seeds = random_seed_array[nawhere]
                          rehash_seeds = True
  
                      else:
                          seeds = None
                          rehash_seeds = False
  
                      mm_kwargs["hashed_seeds"] = seeds
  
                  else:
                      rehash_seeds = False
  
                  logger.set_start_time()
                  imp_values = self.mean_match_scheme._mean_match(
                      var, objective, **mm_kwargs
                  )
                  logger.record_time(
                      timed_event="mean_matching", **log_context
                  )
                  imputed_data._insert_new_data(
                      dataset=ds_new,
                      variable_index=var,
                      new_data=imp_values,
                  )
                  # Refresh our seeds.
                  if rehash_seeds:
                      assert isinstance(random_seed_array, np.ndarray)
                      random_seed_array[nawhere] = hash_int32(seeds)
  
                  imputed_data.iterations[
                      ds_new, imputed_data.imputation_order.index(var)
                  ] += 1
  
              logger.log("\n", end="")
  
      imputed_data._ampute_original_data()
      if self.save_loggers:
          self.loggers.append(logger)
  
      return imputed_data
  
imports: from .ImputedData import ImputedData
  from .MeanMatchScheme import MeanMatchScheme
  from .utils import _t_dat, _t_var_list, _t_var_dict, _t_var_sub, _t_random_state, _assert_dataset_equivalent, _draw_random_int32, _interpret_ds, _subset_data, ensure_rng, hash_int32, stratified_categorical_folds, stratified_continuous_folds, stratified_subset
  from .compat import pd_DataFrame, pd_Series
  from .default_lightgbm_parameters import default_parameters, make_default_tuning_space
  from .logger import Logger
  import numpy
  from numpy.random import RandomState
  from warnings import warn
  from lightgbm import train, Dataset, cv, log_evaluation, early_stopping, Booster
  from lightgbm.basic import _ConfigAliases
  from io import BytesIO
  import blosc
  import dill
  from copy import copy
  from typing import Union, List, Dict, Any, Optional
label_desc: Use image node_id 25 for calling the ImputationKernel obj's underlying member method code with example usage: obj.impute_new_data(new_data, datasets, iterations, save_all_iterations, copy_data, random_state, random_seed_array, verbose) and returns: imputed_data