class_docstr: Ensemble Latent Dirichlet Allocation (eLDA), a method of training a topic model ensemble.
  
  Extracts stable topics that are consistently learned across multiple LDA models. eLDA has the added benefit that
  the user does not need to know the exact number of topics the topic model should extract ahead of time.
class_docstr_tok: ["Ensemble","Latent","Dirichlet","Allocation","(","eLDA",")",",","a","method","of","training","a","topic","model","ensemble",".","Extracts","stable","topics","that","are","consistently","learned","across","multiple","LDA","models",".","eLDA","has","the","added","benefit","that","the","user","does","not","need","to","know","the","exact","number","of","topics","the","topic","model","should","extract","ahead","of","time","."]
docstr: Create and train a new EnsembleLda model.
  
  Will start training immediatelly, except if iterations, passes or num_models is 0 or if the corpus is missing.
  
  Parameters
  ----------
  topic_model_class : str, topic model, optional
      Examples:
          * 'ldamulticore' (default, recommended)
          * 'lda'
          * ldamodel.LdaModel
          * ldamulticore.LdaMulticore
  ensemble_workers : int, optional
      Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
      num_models should be a multiple of ensemble_workers.
  
      Setting it to 0 or 1 will both use the non-multiprocessing version. Default: 1
  num_models : int, optional
      How many LDA models to train in this ensemble.
      Default: 3
  min_cores : int, optional
      Minimum cores a cluster of topics has to contain so that it is recognized as stable topic.
  epsilon : float, optional
      Defaults to 0.1. Epsilon for the CBDBSCAN clustering that generates the stable topics.
  ensemble_workers : int, optional
      Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
      num_models should be a multiple of ensemble_workers.
  
      Setting it to 0 or 1 will both use the nonmultiprocessing version. Default: 1
  memory_friendly_ttda : boolean, optional
      If True, the models in the ensemble are deleted after training and only a concatenation of each model's
      topic term distribution (called ttda) is kept to save memory.
  
      Defaults to True. When False, trained models are stored in a list in self.tms, and no models that are not
      of a gensim model type can be added to this ensemble using the add_model function.
  
      If False, any topic term matrix can be suplied to add_model.
  min_samples : int, optional
      Required int of nearby topics for a topic to be considered as 'core' in the CBDBSCAN clustering.
  masking_method : function, optional
      Choose one of :meth:`~gensim.models.ensemblelda.mass_masking` (default) or
      :meth:`~gensim.models.ensemblelda.rank_masking` (percentile, faster).
  
      For clustering, distances between topic-term distributions are asymmetric.  In particular, the distance
      (technically a divergence) from distribution A to B is more of a measure of if A is contained in B.  At a
      high level, this involves using distribution A to mask distribution B and then calculating the cosine
      distance between the two.  The masking can be done in two ways:
  
      1. mass: forms mask by taking the top ranked terms until their cumulative mass reaches the
      'masking_threshold'
  
      2. rank: forms mask by taking the top ranked terms (by mass) until the 'masking_threshold' is reached.
      For example, a ranking threshold of 0.11 means the top 0.11 terms by weight are used to form a mask.
  masking_threshold : float, optional
      Default: None, which uses ``0.95`` for "mass", and ``0.11`` for masking_method "rank".  In general, too
      small a mask threshold leads to inaccurate calculations (no signal) and too big a mask leads to noisy
      distance calculations.  Defaults are often a good sweet spot for this hyperparameter.
  distance_workers : int, optional
      When ``distance_workers`` is ``None``, it defaults to ``os.cpu_count()`` for maximum performance. Default is
      1, which is not multiprocessed. Set to ``> 1`` to enable multiprocessing.
  **gensim_kw_args
      Parameters for each gensim model (e.g. :py:class:`gensim.models.LdaModel`) in the ensemble.
docstr_tok: ["Create","and","train","a","new","EnsembleLda","model",".","Will","start","training","immediatelly",",","except","if","iterations",",","passes","or","num_models","is","0","or","if","the","corpus","is","missing",".","Parameters","--","--","--","--","--","topic_model_class",":","str",",","topic","model",",","optional","Examples",":","*","'ldamulticore","'","(","default",",","recommended",")","*","'lda'","*","ldamodel.LdaModel","*","ldamulticore.LdaMulticore","ensemble_workers",":","int",",","optional","Spawns","that","many","processes","and","distributes","the","models","from","the","ensemble","to","those","as","evenly","as","possible",".","num_models","should","be","a","multiple","of","ensemble_workers",".","Setting","it","to","0","or","1","will","both","use","the","non-multiprocessing","version",".","Default",":","1","num_models",":","int",",","optional","How","many","LDA","models","to","train","in","this","ensemble",".","Default",":","3","min_cores",":","int",",","optional","Minimum","cores","a","cluster","of","topics","has","to","contain","so","that","it","is","recognized","as","stable","topic",".","epsilon",":","float",",","optional","Defaults","to","0.1",".","Epsilon","for","the","CBDBSCAN","clustering","that","generates","the","stable","topics",".","ensemble_workers",":","int",",","optional","Spawns","that","many","processes","and","distributes","the","models","from","the","ensemble","to","those","as","evenly","as","possible",".","num_models","should","be","a","multiple","of","ensemble_workers",".","Setting","it","to","0","or","1","will","both","use","the","nonmultiprocessing","version",".","Default",":","1","memory_friendly_ttda",":","boolean",",","optional","If","True",",","the","models","in","the","ensemble","are","deleted","after","training","and","only","a","concatenation","of","each","model's","topic","term","distribution","(","called","ttda",")","is","kept","to","save","memory",".","Defaults","to","True",".","When","False",",","trained","models","are","stored","in","a","list","in","self.tms",",","and","no","models","that","are","not","of","a","gensim","model","type","can","be","added","to","this","ensemble","using","the","add_model","function",".","If","False",",","any","topic","term","matrix","can","be","suplied","to","add_model",".","min_samples",":","int",",","optional","Required","int","of","nearby","topics","for","a","topic","to","be","considered","as","'core","'","in","the","CBDBSCAN","clustering",".","masking_method",":","function",",","optional","Choose","one","of",":","meth",":","`","~gensim.models.ensemblelda.mass_masking","`","(","default",")","or",":","meth",":","`","~gensim.models.ensemblelda.rank_masking","`","(","percentile",",","faster",")",".","For","clustering",",","distances","between","topic-term","distributions","are","asymmetric",".","In","particular",",","the","distance","(","technically","a","divergence",")","from","distribution","A","to","B","is","more","of","a","measure","of","if","A","is","contained","in","B",".","At","a","high","level",",","this","involves","using","distribution","A","to","mask","distribution","B","and","then","calculating","the","cosine","distance","between","the","two",".","The","masking","can","be","done","in","two","ways",":","1.","mass",":","forms","mask","by","taking","the","top","ranked","terms","until","their","cumulative","mass","reaches","the","'masking_threshold'","2.","rank",":","forms","mask","by","taking","the","top","ranked","terms","(","by","mass",")","until","the","'masking_threshold","'","is","reached",".","For","example",",","a","ranking","threshold","of","0.11","means","the","top","0.11","terms","by","weight","are","used","to","form","a","mask",".","masking_threshold",":","float",",","optional","Default",":","None",",","which","uses","``","0.95","``","for","``","mass","''",",","and","``","0.11","``","for","masking_method","``","rank","''",".","In","general",",","too","small","a","mask","threshold","leads","to","inaccurate","calculations","(","no","signal",")","and","too","big","a","mask","leads","to","noisy","distance","calculations",".","Defaults","are","often","a","good","sweet","spot","for","this","hyperparameter",".","distance_workers",":","int",",","optional","When","``","distance_workers","``","is","``","None","``",",","it","defaults","to","``","os.cpu_count","(",")","``","for","maximum","performance",".","Default","is","1",",","which","is","not","multiprocessed",".","Set","to","``",">","1","``","to","enable","multiprocessing",".","*","*","gensim_kw_args","Parameters","for","each","gensim","model","(","e.g",".",":","py",":","class",":","`","gensim.models.LdaModel","`",")","in","the","ensemble","."]
code: def __init__(
      self,
      topic_model_class="ldamulticore",
      num_models=3,
      min_cores=None,  # default value from _generate_stable_topics()
      epsilon=0.1,
      ensemble_workers=1,
      memory_friendly_ttda=True,
      min_samples=None,
      masking_method=mass_masking,
      masking_threshold=None,
      distance_workers=1,
      random_state=None,
      **gensim_kw_args,
  ):
      """Create and train a new EnsembleLda model.
  
      Will start training immediatelly, except if iterations, passes or num_models is 0 or if the corpus is missing.
  
      Parameters
      ----------
      topic_model_class : str, topic model, optional
          Examples:
              * 'ldamulticore' (default, recommended)
              * 'lda'
              * ldamodel.LdaModel
              * ldamulticore.LdaMulticore
      ensemble_workers : int, optional
          Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
          num_models should be a multiple of ensemble_workers.
  
          Setting it to 0 or 1 will both use the non-multiprocessing version. Default: 1
      num_models : int, optional
          How many LDA models to train in this ensemble.
          Default: 3
      min_cores : int, optional
          Minimum cores a cluster of topics has to contain so that it is recognized as stable topic.
      epsilon : float, optional
          Defaults to 0.1. Epsilon for the CBDBSCAN clustering that generates the stable topics.
      ensemble_workers : int, optional
          Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
          num_models should be a multiple of ensemble_workers.
  
          Setting it to 0 or 1 will both use the nonmultiprocessing version. Default: 1
      memory_friendly_ttda : boolean, optional
          If True, the models in the ensemble are deleted after training and only a concatenation of each model's
          topic term distribution (called ttda) is kept to save memory.
  
          Defaults to True. When False, trained models are stored in a list in self.tms, and no models that are not
          of a gensim model type can be added to this ensemble using the add_model function.
  
          If False, any topic term matrix can be suplied to add_model.
      min_samples : int, optional
          Required int of nearby topics for a topic to be considered as 'core' in the CBDBSCAN clustering.
      masking_method : function, optional
          Choose one of :meth:`~gensim.models.ensemblelda.mass_masking` (default) or
          :meth:`~gensim.models.ensemblelda.rank_masking` (percentile, faster).
  
          For clustering, distances between topic-term distributions are asymmetric.  In particular, the distance
          (technically a divergence) from distribution A to B is more of a measure of if A is contained in B.  At a
          high level, this involves using distribution A to mask distribution B and then calculating the cosine
          distance between the two.  The masking can be done in two ways:
  
          1. mass: forms mask by taking the top ranked terms until their cumulative mass reaches the
          'masking_threshold'
  
          2. rank: forms mask by taking the top ranked terms (by mass) until the 'masking_threshold' is reached.
          For example, a ranking threshold of 0.11 means the top 0.11 terms by weight are used to form a mask.
      masking_threshold : float, optional
          Default: None, which uses ``0.95`` for "mass", and ``0.11`` for masking_method "rank".  In general, too
          small a mask threshold leads to inaccurate calculations (no signal) and too big a mask leads to noisy
          distance calculations.  Defaults are often a good sweet spot for this hyperparameter.
      distance_workers : int, optional
          When ``distance_workers`` is ``None``, it defaults to ``os.cpu_count()`` for maximum performance. Default is
          1, which is not multiprocessed. Set to ``> 1`` to enable multiprocessing.
      **gensim_kw_args
          Parameters for each gensim model (e.g. :py:class:`gensim.models.LdaModel`) in the ensemble.
  
      """
  
      if "id2word" not in gensim_kw_args:
          gensim_kw_args["id2word"] = None
      if "corpus" not in gensim_kw_args:
          gensim_kw_args["corpus"] = None
  
      if (
          gensim_kw_args["id2word"] is None
          and not gensim_kw_args["corpus"] is None
      ):
          logger.warning(
              "no word id mapping provided; initializing from corpus, assuming identity"
          )
          gensim_kw_args["id2word"] = utils.dict_from_corpus(
              gensim_kw_args["corpus"]
          )
      if (
          gensim_kw_args["id2word"] is None
          and gensim_kw_args["corpus"] is None
      ):
          raise ValueError(
              "at least one of corpus/id2word must be specified, to establish "
              "input space dimensionality. Corpus should be provided using the "
              "`corpus` keyword argument."
          )
  
      #
      # The following conditional makes no sense, but we're in a rush to
      # release and we don't care about this submodule enough to deal with it
      # properly, so we disable flake8 for the following line.
      #
      if type(topic_model_class) == type and issubclass(
          topic_model_class, ldamodel.LdaModel
      ):  # noqa
          self.topic_model_class = topic_model_class
      else:
          kinds = {
              "lda": ldamodel.LdaModel,
              "ldamulticore": ldamulticore.LdaMulticore,
          }
          if topic_model_class not in kinds:
              raise ValueError(
                  "topic_model_class should be one of 'lda', 'ldamulticode' or a model "
                  "inheriting from LdaModel"
              )
          self.topic_model_class = kinds[topic_model_class]
  
      self.num_models = num_models
      self.gensim_kw_args = gensim_kw_args
  
      self.memory_friendly_ttda = memory_friendly_ttda
  
      self.distance_workers = distance_workers
      self.masking_threshold = masking_threshold
      self.masking_method = masking_method
  
      # this will provide the gensim api to the ensemble basically
      self.classic_model_representation = None
  
      # the ensembles state
      self.random_state = utils.get_random_state(random_state)
      self.sstats_sum = 0
      self.eta = None
      self.tms = []
      # initialize empty 2D topic term distribution array (ttda) (number of topics x number of terms)
      self.ttda = np.empty((0, len(gensim_kw_args["id2word"])))
      self.asymmetric_distance_matrix_outdated = True
  
      # in case the model will not train due to some
      # parameters, stop here and don't train.
      if num_models <= 0:
          return
      if gensim_kw_args.get("corpus") is None:
          return
      if (
          "iterations" in gensim_kw_args
          and gensim_kw_args["iterations"] <= 0
      ):
          return
      if "passes" in gensim_kw_args and gensim_kw_args["passes"] <= 0:
          return
  
      logger.info(
          f"generating {num_models} topic models using {ensemble_workers} workers"
      )
  
      if ensemble_workers > 1:
          _generate_topic_models_multiproc(
              self, num_models, ensemble_workers
          )
      else:
          _generate_topic_models(self, num_models)
  
      self._generate_asymmetric_distance_matrix()
      self._generate_topic_clusters(epsilon, min_samples)
      self._generate_stable_topics(min_cores)
  
      # create model that can provide the usual gensim api to the stable topics from the ensemble
      self.generate_gensim_representation()
  
imports: import logging
  import os
  from multiprocessing import Process, Pipe, ProcessError
  import importlib
  from typing import Set, Optional, List
  import numpy
  from scipy.spatial.distance import cosine
  from dataclasses import dataclass
  from gensim import utils
  from gensim.models import ldamodel, ldamulticore, basemodel
  from gensim.utils import SaveLoad
label_desc: Use image node_id 1 to create a new EnsembleLda object from inherited base classes: SaveLoad with example: obj = EnsembleLda(topic_model_class, num_models, min_cores, epsilon, ensemble_workers, memory_friendly_ttda, min_samples, masking_method, masking_threshold, distance_workers, random_state)