class_docstr: Generates ngram features from text features.
  
  Parameters
  ----------
  vectorizer : :class:`sklearn.feature_extraction.text.CountVectorizer` or :class:`sklearn.feature_extraction.text.TfidfVectorizer`, default CountVectorizer(min_df=30, ngram_range=(1, 3), max_features=10000, dtype=np.uint8)  # noqa
      sklearn CountVectorizer which is used to generate the ngrams given the text data.
      Can also specify a TfidfVectorizer, but note that memory usage will increase by 4-8x relative to CountVectorizer.
  vectorizer_strategy : str, default 'combined'
      If 'combined', all text features are concatenated together to fit the vectorizer.
      Features generated in this way have their names prepended with '__nlp__.'.
      If 'separate', all text features are fit separately with their own copy of the vectorizer.
      Their ngram features are then concatenated together to form the output.
      If 'both', the outputs of 'combined' and 'separate' are concatenated together to form the output.
      It is generally recommended to keep vectorizer_strategy as 'combined' unless the text features are not associated with each-other,
      as fitting separate vectorizers could increase memory usage and model training time.
      Valid values: ['combined', 'separate', 'both']
  max_memory_ratio : float, default 0.15
      Safety measure to avoid out-of-memory errors downstream in model training.
      The number of ngrams generated will be capped to take at most max_memory_ratio proportion of total available memory,
      treating the ngrams as float32 values.
      ngram features will be removed in least frequent to most frequent order.
      Note: For vectorizer_strategy values other than 'combined', the resulting ngrams may use more than this value.
      It is recommended to only increase this value above 0.15 if confident that higher values will not result in out-of-memory errors.
  **kwargs :
      Refer to :class:`AbstractFeatureGenerator` documentation for details on valid key word arguments.
class_docstr_tok: ["Generates","ngram","features","from","text","features",".","Parameters","--","--","--","--","--","vectorizer",":",":","class",":","`","sklearn.feature_extraction.text.CountVectorizer","`","or",":","class",":","`","sklearn.feature_extraction.text.TfidfVectorizer","`",",","default","CountVectorizer","(","min_df=30",",","ngram_range=","(","1",",","3",")",",","max_features=10000",",","dtype=np.uint8",")","#","noqa","sklearn","CountVectorizer","which","is","used","to","generate","the","ngrams","given","the","text","data",".","Can","also","specify","a","TfidfVectorizer",",","but","note","that","memory","usage","will","increase","by","4-8x","relative","to","CountVectorizer",".","vectorizer_strategy",":","str",",","default","'combined'","If","'combined","'",",","all","text","features","are","concatenated","together","to","fit","the","vectorizer",".","Features","generated","in","this","way","have","their","names","prepended","with","'__nlp__.","'",".","If","'separate","'",",","all","text","features","are","fit","separately","with","their","own","copy","of","the","vectorizer",".","Their","ngram","features","are","then","concatenated","together","to","form","the","output",".","If","'both","'",",","the","outputs","of","'combined","'","and","'separate","'","are","concatenated","together","to","form","the","output",".","It","is","generally","recommended","to","keep","vectorizer_strategy","as","'combined","'","unless","the","text","features","are","not","associated","with","each-other",",","as","fitting","separate","vectorizers","could","increase","memory","usage","and","model","training","time",".","Valid","values",":","[","'combined","'",",","'separate","'",",","'both","'","]","max_memory_ratio",":","float",",","default","0.15","Safety","measure","to","avoid","out-of-memory","errors","downstream","in","model","training",".","The","number","of","ngrams","generated","will","be","capped","to","take","at","most","max_memory_ratio","proportion","of","total","available","memory",",","treating","the","ngrams","as","float32","values",".","ngram","features","will","be","removed","in","least","frequent","to","most","frequent","order",".","Note",":","For","vectorizer_strategy","values","other","than","'combined","'",",","the","resulting","ngrams","may","use","more","than","this","value",".","It","is","recommended","to","only","increase","this","value","above","0.15","if","confident","that","higher","values","will","not","result","in","out-of-memory","errors",".","*","*","kwargs",":","Refer","to",":","class",":","`","AbstractFeatureGenerator","`","documentation","for","details","on","valid","key","word","arguments","."]
code: def _fit_transform_ngrams(self, X):
      if not self.features_in:
          return DataFrame(index=X.index)
      features_nlp_to_remove = []
      if self.vectorizer_strategy == "combined":
          self.vectorizer_features = ["__nlp__"]
      elif self.vectorizer_strategy == "separate":
          self.vectorizer_features = copy.deepcopy(self.features_in)
      elif self.vectorizer_strategy == "both":
          self.vectorizer_features = ["__nlp__"] + copy.deepcopy(
              self.features_in
          )
      else:
          raise ValueError(
              f"vectorizer_strategy must be one of {['combined', 'separate', 'both']}, but value is: {self.vectorizer_features}"
          )
      self._log(
          20,
          f"Fitting {self.vectorizer_default_raw.__class__.__name__} for text features: "
          + str(self.features_in),
          self.log_prefix + "\t",
      )
      self._log(
          15, f"{self.vectorizer_default_raw}", self.log_prefix + "\t\t"
      )
      for nlp_feature in self.vectorizer_features:
          # TODO: Preprocess text?
          if nlp_feature == "__nlp__":  # Combine Text Fields
              features_in_str = X[self.features_in].astype(str)
              text_list = list(
                  set(
                      [". ".join(row) for row in features_in_str.values]
                  )
              )
          else:
              text_list = list(
                  X[nlp_feature].astype(str).drop_duplicates().values
              )
          vectorizer_raw = copy.deepcopy(self.vectorizer_default_raw)
          try:
              # Don't use transform_matrix output because it may contain fewer rows due to drop_duplicates call.
              vectorizer_fit, _ = self._train_vectorizer(
                  text_list, vectorizer_raw
              )
              self._log(
                  20,
                  f"{vectorizer_fit.__class__.__name__} fit with vocabulary size = {len(vectorizer_fit.vocabulary_)}",
                  self.log_prefix + "\t",
              )
          except ValueError:
              self._log(
                  30,
                  f"Removing text_ngram feature due to error: '{nlp_feature}'",
                  self.log_prefix + "\t",
              )
              if nlp_feature == "__nlp__":
                  self.vectorizer_features = []
                  features_nlp_to_remove = self.features_in
                  break
              else:
                  features_nlp_to_remove.append(nlp_feature)
          else:
              self.vectorizers.append(vectorizer_fit)
      self._remove_features_in(features_nlp_to_remove)
  
      downsample_ratio = None
      nlp_failure_count = 0
      X_text_ngram = None
      keep_trying_nlp = True
      while keep_trying_nlp:
          try:
              X_text_ngram = self._generate_ngrams(
                  X=X, downsample_ratio=downsample_ratio
              )
              keep_trying_nlp = False
          except Exception as err:
              nlp_failure_count += 1
              traceback.print_tb(err.__traceback__)
  
              X_text_ngram = None
              skip_nlp = False
              for vectorizer in self.vectorizers:
                  vocab_size = len(vectorizer.vocabulary_)
                  if vocab_size <= 50:
                      skip_nlp = True
                      break
              else:
                  if nlp_failure_count >= 3:
                      skip_nlp = True
  
              if skip_nlp:
                  self._log(
                      30,
                      "Warning: ngrams generation resulted in OOM error, removing ngrams features. "
                      "If you want to use ngrams for this problem, increase memory allocation for AutoGluon.",
                      self.log_prefix + "\t",
                  )
                  self._log(10, str(err))
                  self.vectorizers = []
                  self.features_in = []
                  keep_trying_nlp = False
              else:
                  self._log(
                      20,
                      "Warning: ngrams generation resulted in OOM error, attempting to reduce ngram feature count. "
                      "If you want to optimally use ngrams for this problem, increase memory allocation for AutoGluon.",
                      self.log_prefix + "\t",
                  )
                  self._log(10, str(err))
                  downsample_ratio = 0.25
      if X_text_ngram is None:
          X_text_ngram = DataFrame(index=X.index)
      return X_text_ngram
  
imports: import copy
  import logging
  import traceback
  import numpy
  import pandas
  from pandas import DataFrame, Series
  from sklearn.feature_selection import SelectKBest, f_classif, f_regression
  from autogluon.common.features.types import S_IMAGE_BYTEARRAY, S_IMAGE_PATH, S_TEXT, S_TEXT_NGRAM
  from autogluon.common.utils.lite import disable_if_lite_mode
  from ..vectorizers import downscale_vectorizer, get_ngram_freq, vectorizer_auto_ml_default
  from .abstract import AbstractFeatureGenerator
label_desc: Use image node_id 5 for calling the TextNgramFeatureGenerator obj's underlying member method code with example usage: obj._fit_transform_ngrams(X) and returns: X_text_ngram, DataFrame