class_docstr: Ultra-high, conditional sure independence screening
  
  This is an adjusted version of Fan's sure independence screening.
  
  Parameters
  ----------
  model : instance of penalizing model
      examples: GLMPenalized, PoissonPenalized and LogitPenalized.
      The attributes of the model instance `pen_weight` and `penal` will be
      ignored.
  pen_weight : None or float
      penalization weight use in SCAD penalized MLE
  k_add : int
      number of exog to add during expansion or forward selection
      see Notes section for tie handling
  k_max_add : int
      maximum number of variables to include during variable addition, i.e.
      forward selection. default is 30
  threshold_trim : float
      threshold for trimming parameters to zero, default is 1e-4
  k_max_included : int
      maximum total number of variables to include in model.
  ranking_attr : str
      This determines the result attribute or model method that is used for
      the ranking of exog to include. The availability of attributes depends
      on the model.
      Default is 'resid_pearson', 'model.score_factor' can be used in GLM.
  ranking_project : bool
      If ranking_project is True, then the exog candidates for inclusion are
      first projected on the already included exog before the computation
      of the ranking measure. This brings the ranking measure closer to
      the statistic of a score test for variable addition.
  
  Notes
  -----
  Status: experimental, tested only on a limited set of models and
  with a limited set of model options.
  
  Tie handling: If there are ties at the decision threshold, then all those
  tied exog columns are treated in the same way. During forward selection
  all exog columns with the same boundary value are included. During
  elimination, the tied columns are not dropped. Consequently, if ties are
  present, then the number of included exog can be larger than specified
  by k_add, k_max_add and k_max_included.
  
  The screening algorithm works similar to step wise regression. Each
  iteration of the screening algorithm includes a forward selection step
  where variables are added to the model, and a backwards selection step
  where variables are removed. In contrast to step wise regression, we add
  a fixed number of variables at each forward selection step. The
  backwards selection step is based on SCAD penalized estimation and
  trimming of variables with estimated coefficients below a threshold.
  The tuning parameters can be used to adjust the number of variables to add
  and to include depending on the size of the dataset.
  
  There is currently no automatic tuning parameter selection. Candidate
  explanatory variables should be standardized or should be on a similar
  scale because penalization and trimming are based on the absolute values
  of the parameters.
  
  
  TODOs and current limitations:
  
  freq_weights are not supported in this. Candidate ranking uses
  moment condition with resid_pearson or others without freq_weights.
  pearson_resid: GLM resid_pearson does not include freq_weights.
  
  variable names: do we keep track of those? currently made-up names
  
  currently only supports numpy arrays, no exog type check or conversion
  
  currently only single columns are selected, no terms (multi column exog)
class_docstr_tok: ["Ultra-high",",","conditional","sure","independence","screening","This","is","an","adjusted","version","of","Fan","'s","sure","independence","screening",".","Parameters","--","--","--","--","--","model",":","instance","of","penalizing","model","examples",":","GLMPenalized",",","PoissonPenalized","and","LogitPenalized",".","The","attributes","of","the","model","instance","`","pen_weight","`","and","`","penal","`","will","be","ignored",".","pen_weight",":","None","or","float","penalization","weight","use","in","SCAD","penalized","MLE","k_add",":","int","number","of","exog","to","add","during","expansion","or","forward","selection","see","Notes","section","for","tie","handling","k_max_add",":","int","maximum","number","of","variables","to","include","during","variable","addition",",","i.e",".","forward","selection",".","default","is","30","threshold_trim",":","float","threshold","for","trimming","parameters","to","zero",",","default","is","1e-4","k_max_included",":","int","maximum","total","number","of","variables","to","include","in","model",".","ranking_attr",":","str","This","determines","the","result","attribute","or","model","method","that","is","used","for","the","ranking","of","exog","to","include",".","The","availability","of","attributes","depends","on","the","model",".","Default","is","'resid_pearson","'",",","'model.score_factor","'","can","be","used","in","GLM",".","ranking_project",":","bool","If","ranking_project","is","True",",","then","the","exog","candidates","for","inclusion","are","first","projected","on","the","already","included","exog","before","the","computation","of","the","ranking","measure",".","This","brings","the","ranking","measure","closer","to","the","statistic","of","a","score","test","for","variable","addition",".","Notes","--","--","-","Status",":","experimental",",","tested","only","on","a","limited","set","of","models","and","with","a","limited","set","of","model","options",".","Tie","handling",":","If","there","are","ties","at","the","decision","threshold",",","then","all","those","tied","exog","columns","are","treated","in","the","same","way",".","During","forward","selection","all","exog","columns","with","the","same","boundary","value","are","included",".","During","elimination",",","the","tied","columns","are","not","dropped",".","Consequently",",","if","ties","are","present",",","then","the","number","of","included","exog","can","be","larger","than","specified","by","k_add",",","k_max_add","and","k_max_included",".","The","screening","algorithm","works","similar","to","step","wise","regression",".","Each","iteration","of","the","screening","algorithm","includes","a","forward","selection","step","where","variables","are","added","to","the","model",",","and","a","backwards","selection","step","where","variables","are","removed",".","In","contrast","to","step","wise","regression",",","we","add","a","fixed","number","of","variables","at","each","forward","selection","step",".","The","backwards","selection","step","is","based","on","SCAD","penalized","estimation","and","trimming","of","variables","with","estimated","coefficients","below","a","threshold",".","The","tuning","parameters","can","be","used","to","adjust","the","number","of","variables","to","add","and","to","include","depending","on","the","size","of","the","dataset",".","There","is","currently","no","automatic","tuning","parameter","selection",".","Candidate","explanatory","variables","should","be","standardized","or","should","be","on","a","similar","scale","because","penalization","and","trimming","are","based","on","the","absolute","values","of","the","parameters",".","TODOs","and","current","limitations",":","freq_weights","are","not","supported","in","this",".","Candidate","ranking","uses","moment","condition","with","resid_pearson","or","others","without","freq_weights",".","pearson_resid",":","GLM","resid_pearson","does","not","include","freq_weights",".","variable","names",":","do","we","keep","track","of","those","?","currently","made-up","names","currently","only","supports","numpy","arrays",",","no","exog","type","check","or","conversion","currently","only","single","columns","are","selected",",","no","terms","(","multi","column","exog",")"]
docstr: batched version of screen exog
  
  This screens variables in a two step process:
  
  In the first step screen_exog is used on each element of the
  exog_iterator, and the batch winners are collected.
  
  In the second step all batch winners are combined into a new array
  of exog candidates and `screen_exog` is used to select a final
  model.
  
  Parameters
  ----------
  exog_iterator : iterator over ndarrays
  
  Returns
  -------
  res_screen_final : instance of ScreeningResults
      This is the instance returned by the second round call to
      `screen_exog`. Additional attributes are added to provide
      more information about the batched selection process.
      The index of final nonzero variables is
      `idx_nonzero_batches` which is a 2-dimensional array with batch
      index in the first column and variable index within batch in the
      second column. They can be used jointly as index for the data
      in the exog_iterator.
      see ScreeningResults for a full description
docstr_tok: ["batched","version","of","screen","exog","This","screens","variables","in","a","two","step","process",":","In","the","first","step","screen_exog","is","used","on","each","element","of","the","exog_iterator",",","and","the","batch","winners","are","collected",".","In","the","second","step","all","batch","winners","are","combined","into","a","new","array","of","exog","candidates","and","`","screen_exog","`","is","used","to","select","a","final","model",".","Parameters","--","--","--","--","--","exog_iterator",":","iterator","over","ndarrays","Returns","--","--","--","-","res_screen_final",":","instance","of","ScreeningResults","This","is","the","instance","returned","by","the","second","round","call","to","`","screen_exog","`",".","Additional","attributes","are","added","to","provide","more","information","about","the","batched","selection","process",".","The","index","of","final","nonzero","variables","is","`","idx_nonzero_batches","`","which","is","a","2-dimensional","array","with","batch","index","in","the","first","column","and","variable","index","within","batch","in","the","second","column",".","They","can","be","used","jointly","as","index","for","the","data","in","the","exog_iterator",".","see","ScreeningResults","for","a","full","description"]
code: def screen_exog_iterator(self, exog_iterator):
      """
      batched version of screen exog
  
      This screens variables in a two step process:
  
      In the first step screen_exog is used on each element of the
      exog_iterator, and the batch winners are collected.
  
      In the second step all batch winners are combined into a new array
      of exog candidates and `screen_exog` is used to select a final
      model.
  
      Parameters
      ----------
      exog_iterator : iterator over ndarrays
  
      Returns
      -------
      res_screen_final : instance of ScreeningResults
          This is the instance returned by the second round call to
          `screen_exog`. Additional attributes are added to provide
          more information about the batched selection process.
          The index of final nonzero variables is
          `idx_nonzero_batches` which is a 2-dimensional array with batch
          index in the first column and variable index within batch in the
          second column. They can be used jointly as index for the data
          in the exog_iterator.
          see ScreeningResults for a full description
      """
      k_keep = self.k_keep
      # res_batches = []
      res_idx = []
      exog_winner = []
      exog_idx = []
      for ex in exog_iterator:
          res_screen = self.screen_exog(ex, maxiter=20)
          # avoid storing res_screen, only for debugging
          # res_batches.append(res_screen)
          res_idx.append(res_screen.idx_nonzero)
          exog_winner.append(
              ex[:, res_screen.idx_nonzero[k_keep:] - k_keep]
          )
          exog_idx.append(res_screen.idx_nonzero[k_keep:] - k_keep)
  
      exog_winner = np.column_stack(exog_winner)
      res_screen_final = self.screen_exog(exog_winner, maxiter=20)
  
      exog_winner_names = [
          "var%d_%d" % (bidx, idx)
          for bidx, batch in enumerate(exog_idx)
          for idx in batch
      ]
  
      idx_full = [
          (bidx, idx)
          for bidx, batch in enumerate(exog_idx)
          for idx in batch
      ]
      ex_final_idx = res_screen_final.idx_nonzero[k_keep:] - k_keep
      final_names = np.array(exog_winner_names)[ex_final_idx]
      res_screen_final.idx_nonzero_batches = np.array(idx_full)[
          ex_final_idx
      ]
      res_screen_final.exog_final_names = final_names
      history = {"idx_nonzero": res_idx, "idx_exog": exog_idx}
      res_screen_final.history_batches = history
      return res_screen_final
  
file_path: statsmodels/statsmodels/base/_screening.py
imports: from collections import defaultdict
  import numpy
  from statsmodels.base._penalties import SCADSmoothed
label_desc: Use image node_id 5 for calling the VariableScreening obj's underlying member method code with example usage: obj.screen_exog_iterator(exog_iterator) and returns: res_screen_final