class_docstr: Train and use Online Latent Dirichlet Allocation model as presented in
  `'Online Learning for LDA' by Hoffman et al.`_
  
  Examples
  -------
  Initialize a model using a Gensim corpus
  
  .. sourcecode:: pycon
  
      >>> from gensim.test.utils import common_corpus
      >>>
      >>> lda = LdaModel(common_corpus, num_topics=10)
  
  You can then infer topic distributions on new, unseen documents.
  
  .. sourcecode:: pycon
  
      >>> doc_bow = [(1, 0.3), (2, 0.1), (0, 0.09)]
      >>> doc_lda = lda[doc_bow]
  
  The model can be updated (trained) with new documents.
  
  .. sourcecode:: pycon
  
      >>> # In practice (corpus =/= initial training corpus), but we use the same here for simplicity.
      >>> other_corpus = common_corpus
      >>>
      >>> lda.update(other_corpus)
  
  Model persistency is achieved through :meth:`~gensim.models.ldamodel.LdaModel.load` and
  :meth:`~gensim.models.ldamodel.LdaModel.save` methods.
class_docstr_tok: ["Train","and","use","Online","Latent","Dirichlet","Allocation","model","as","presented","in","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","Examples","--","--","--","-","Initialize","a","model","using","a","Gensim","corpus","..","sourcecode",":",":","pycon",">",">",">","from","gensim.test.utils","import","common_corpus",">",">",">",">",">",">","lda","=","LdaModel","(","common_corpus",",","num_topics=10",")","You","can","then","infer","topic","distributions","on","new",",","unseen","documents",".","..","sourcecode",":",":","pycon",">",">",">","doc_bow","=","[","(","1",",","0.3",")",",","(","2",",","0.1",")",",","(","0",",","0.09",")","]",">",">",">","doc_lda","=","lda","[","doc_bow","]","The","model","can","be","updated","(","trained",")","with","new","documents",".","..","sourcecode",":",":","pycon",">",">",">","#","In","practice","(","corpus","=\/=","initial","training","corpus",")",",","but","we","use","the","same","here","for","simplicity",".",">",">",">","other_corpus","=","common_corpus",">",">",">",">",">",">","lda.update","(","other_corpus",")","Model","persistency","is","achieved","through",":","meth",":","`","~gensim.models.ldamodel.LdaModel.load","`","and",":","meth",":","`","~gensim.models.ldamodel.LdaModel.save","`","methods","."]
docstr: Train the model with new documents, by EM-iterating over the corpus until the topics converge, or until
  the maximum number of allowed iterations is reached. `corpus` must be an iterable.
  
  In distributed mode, the E step is distributed over a cluster of machines.
  
  Notes
  -----
  This update also supports updating an already trained model (`self`) with new documents from `corpus`;
  the two models are then merged in proportion to the number of old vs. new documents.
  This feature is still experimental for non-stationary input streams.
  
  For stationary input (no topic drift in new documents), on the other hand,
  this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_
  and is guaranteed to converge for any `decay` in (0.5, 1].
  Additionally, for smaller corpus sizes,
  an increasing `offset` may be beneficial (see Table 1 in the same paper).
  
  Parameters
  ----------
  corpus : iterable of list of (int, float), optional
      Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`) used to update the
      model.
  chunksize :  int, optional
      Number of documents to be used in each training chunk.
  decay : float, optional
      A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
      when each new document is examined. Corresponds to :math:`\kappa` from
      `'Online Learning for LDA' by Hoffman et al.`_
  offset : float, optional
      Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
      Corresponds to :math:`\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
  passes : int, optional
      Number of passes through the corpus during training.
  update_every : int, optional
      Number of documents to be iterated through for each update.
      Set to 0 for batch learning, > 1 for online iterative learning.
  eval_every : int, optional
      Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x.
  iterations : int, optional
      Maximum number of iterations through the corpus when inferring the topic distribution of a corpus.
  gamma_threshold : float, optional
      Minimum change in the value of the gamma parameters to continue iterating.
  chunks_as_numpy : bool, optional
      Whether each chunk passed to the inference step should be a numpy.ndarray or not. Numpy can in some settings
      turn the term IDs into floats, these will be converted back into integers in inference, which incurs a
      performance hit. For distributed computing it may be desirable to keep the chunks as `numpy.ndarray`.
docstr_tok: ["Train","the","model","with","new","documents",",","by","EM-iterating","over","the","corpus","until","the","topics","converge",",","or","until","the","maximum","number","of","allowed","iterations","is","reached",".","`","corpus","`","must","be","an","iterable",".","In","distributed","mode",",","the","E","step","is","distributed","over","a","cluster","of","machines",".","Notes","--","--","-","This","update","also","supports","updating","an","already","trained","model","(","`","self","`",")","with","new","documents","from","`","corpus","`",";","the","two","models","are","then","merged","in","proportion","to","the","number","of","old","vs.","new","documents",".","This","feature","is","still","experimental","for","non-stationary","input","streams",".","For","stationary","input","(","no","topic","drift","in","new","documents",")",",","on","the","other","hand",",","this","equals","the","online","update","of","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","and","is","guaranteed","to","converge","for","any","`","decay","`","in","(","0.5",",","1","]",".","Additionally",",","for","smaller","corpus","sizes",",","an","increasing","`","offset","`","may","be","beneficial","(","see","Table","1","in","the","same","paper",")",".","Parameters","--","--","--","--","--","corpus",":","iterable","of","list","of","(","int",",","float",")",",","optional","Stream","of","document","vectors","or","sparse","matrix","of","shape","(","`","num_documents","`",",","`","num_terms","`",")","used","to","update","the","model",".","chunksize",":","int",",","optional","Number","of","documents","to","be","used","in","each","training","chunk",".","decay",":","float",",","optional","A","number","between","(","0.5",",","1","]","to","weight","what","percentage","of","the","previous","lambda","value","is","forgotten","when","each","new","document","is","examined",".","Corresponds","to",":","math",":","`","\\kappa","`","from","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","offset",":","float",",","optional","Hyper-parameter","that","controls","how","much","we","will","slow","down","the","first","steps","the","first","few","iterations",".","Corresponds","to",":","math",":","`","\\tau_0","`","from","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","passes",":","int",",","optional","Number","of","passes","through","the","corpus","during","training",".","update_every",":","int",",","optional","Number","of","documents","to","be","iterated","through","for","each","update",".","Set","to","0","for","batch","learning",",",">","1","for","online","iterative","learning",".","eval_every",":","int",",","optional","Log","perplexity","is","estimated","every","that","many","updates",".","Setting","this","to","one","slows","down","training","by","~2x",".","iterations",":","int",",","optional","Maximum","number","of","iterations","through","the","corpus","when","inferring","the","topic","distribution","of","a","corpus",".","gamma_threshold",":","float",",","optional","Minimum","change","in","the","value","of","the","gamma","parameters","to","continue","iterating",".","chunks_as_numpy",":","bool",",","optional","Whether","each","chunk","passed","to","the","inference","step","should","be","a","numpy.ndarray","or","not",".","Numpy","can","in","some","settings","turn","the","term","IDs","into","floats",",","these","will","be","converted","back","into","integers","in","inference",",","which","incurs","a","performance","hit",".","For","distributed","computing","it","may","be","desirable","to","keep","the","chunks","as","`","numpy.ndarray","`","."]
code: def update(
      self,
      corpus,
      chunksize=None,
      decay=None,
      offset=None,
      passes=None,
      update_every=None,
      eval_every=None,
      iterations=None,
      gamma_threshold=None,
      chunks_as_numpy=False,
  ):
      """Train the model with new documents, by EM-iterating over the corpus until the topics converge, or until
      the maximum number of allowed iterations is reached. `corpus` must be an iterable.
  
      In distributed mode, the E step is distributed over a cluster of machines.
  
      Notes
      -----
      This update also supports updating an already trained model (`self`) with new documents from `corpus`;
      the two models are then merged in proportion to the number of old vs. new documents.
      This feature is still experimental for non-stationary input streams.
  
      For stationary input (no topic drift in new documents), on the other hand,
      this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_
      and is guaranteed to converge for any `decay` in (0.5, 1].
      Additionally, for smaller corpus sizes,
      an increasing `offset` may be beneficial (see Table 1 in the same paper).
  
      Parameters
      ----------
      corpus : iterable of list of (int, float), optional
          Stream of document vectors or sparse matrix of shape (`num_documents`, `num_terms`) used to update the
          model.
      chunksize :  int, optional
          Number of documents to be used in each training chunk.
      decay : float, optional
          A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
          when each new document is examined. Corresponds to :math:`\\kappa` from
          `'Online Learning for LDA' by Hoffman et al.`_
      offset : float, optional
          Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
          Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
      passes : int, optional
          Number of passes through the corpus during training.
      update_every : int, optional
          Number of documents to be iterated through for each update.
          Set to 0 for batch learning, > 1 for online iterative learning.
      eval_every : int, optional
          Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x.
      iterations : int, optional
          Maximum number of iterations through the corpus when inferring the topic distribution of a corpus.
      gamma_threshold : float, optional
          Minimum change in the value of the gamma parameters to continue iterating.
      chunks_as_numpy : bool, optional
          Whether each chunk passed to the inference step should be a numpy.ndarray or not. Numpy can in some settings
          turn the term IDs into floats, these will be converted back into integers in inference, which incurs a
          performance hit. For distributed computing it may be desirable to keep the chunks as `numpy.ndarray`.
  
      """
      # use parameters given in constructor, unless user explicitly overrode them
      if decay is None:
          decay = self.decay
      if offset is None:
          offset = self.offset
      if passes is None:
          passes = self.passes
      if update_every is None:
          update_every = self.update_every
      if eval_every is None:
          eval_every = self.eval_every
      if iterations is None:
          iterations = self.iterations
      if gamma_threshold is None:
          gamma_threshold = self.gamma_threshold
  
      try:
          lencorpus = len(corpus)
      except Exception:
          logger.warning(
              "input corpus stream has no len(); counting documents"
          )
          lencorpus = sum(1 for _ in corpus)
      if lencorpus == 0:
          logger.warning(
              "LdaModel.update() called with an empty corpus"
          )
          return
  
      if chunksize is None:
          chunksize = min(lencorpus, self.chunksize)
  
      self.state.numdocs += lencorpus
  
      if update_every:
          updatetype = "online"
          if passes == 1:
              updatetype += " (single-pass)"
          else:
              updatetype += " (multi-pass)"
          updateafter = min(
              lencorpus, update_every * self.numworkers * chunksize
          )
      else:
          updatetype = "batch"
          updateafter = lencorpus
      evalafter = min(
          lencorpus, (eval_every or 0) * self.numworkers * chunksize
      )
  
      updates_per_pass = max(1, lencorpus / updateafter)
      logger.info(
          "running %s LDA training, %s topics, %i passes over "
          "the supplied corpus of %i documents, updating model once "
          "every %i documents, evaluating perplexity every %i documents, "
          "iterating %ix with a convergence threshold of %f",
          updatetype,
          self.num_topics,
          passes,
          lencorpus,
          updateafter,
          evalafter,
          iterations,
          gamma_threshold,
      )
  
      if updates_per_pass * passes < 10:
          logger.warning(
              "too few updates, training might not converge; "
              "consider increasing the number of passes or iterations to improve accuracy"
          )
  
      # rho is the "speed" of updating; TODO try other fncs
      # pass_ + num_updates handles increasing the starting t for each pass,
      # while allowing it to "reset" on the first pass of each update
      def rho():
          return pow(
              offset + pass_ + (self.num_updates / chunksize), -decay
          )
  
      if self.callbacks:
          # pass the list of input callbacks to Callback class
          callback = Callback(self.callbacks)
          callback.set_model(self)
          # initialize metrics list to store metric values after every epoch
          self.metrics = defaultdict(list)
  
      for pass_ in range(passes):
          if self.dispatcher:
              logger.info("initializing %s workers", self.numworkers)
              self.dispatcher.reset(self.state)
          else:
              other = LdaState(
                  self.eta, self.state.sstats.shape, self.dtype
              )
          dirty = False
  
          reallen = 0
          chunks = utils.grouper(
              corpus,
              chunksize,
              as_numpy=chunks_as_numpy,
              dtype=self.dtype,
          )
          for chunk_no, chunk in enumerate(chunks):
              reallen += len(
                  chunk
              )  # keep track of how many documents we've processed so far
  
              if eval_every and (
                  (reallen == lencorpus)
                  or (
                      (chunk_no + 1) % (eval_every * self.numworkers)
                      == 0
                  )
              ):
                  self.log_perplexity(chunk, total_docs=lencorpus)
  
              if self.dispatcher:
                  # add the chunk to dispatcher's job queue, so workers can munch on it
                  logger.info(
                      "PROGRESS: pass %i, dispatching documents up to #%i/%i",
                      pass_,
                      chunk_no * chunksize + len(chunk),
                      lencorpus,
                  )
                  # this will eventually block until some jobs finish, because the queue has a small finite length
                  self.dispatcher.putjob(chunk)
              else:
                  logger.info(
                      "PROGRESS: pass %i, at document #%i/%i",
                      pass_,
                      chunk_no * chunksize + len(chunk),
                      lencorpus,
                  )
                  gammat = self.do_estep(chunk, other)
  
                  if self.optimize_alpha:
                      self.update_alpha(gammat, rho())
  
              dirty = True
              del chunk
  
              # perform an M step. determine when based on update_every, don't do this after every chunk
              if (
                  update_every
                  and (chunk_no + 1) % (update_every * self.numworkers)
                  == 0
              ):
                  if self.dispatcher:
                      # distributed mode: wait for all workers to finish
                      logger.info(
                          "reached the end of input; now waiting for all remaining jobs to finish"
                      )
                      other = self.dispatcher.getstate()
                  self.do_mstep(rho(), other, pass_ > 0)
                  del other  # frees up memory
  
                  if self.dispatcher:
                      logger.info("initializing workers")
                      self.dispatcher.reset(self.state)
                  else:
                      other = LdaState(
                          self.eta, self.state.sstats.shape, self.dtype
                      )
                  dirty = False
  
          if reallen != lencorpus:
              raise RuntimeError(
                  "input corpus size changed during training (don't use generators as input)"
              )
  
          # append current epoch's metric values
          if self.callbacks:
              current_metrics = callback.on_epoch_end(pass_)
              for metric, value in current_metrics.items():
                  self.metrics[metric].append(value)
  
          if dirty:
              # finish any remaining updates
              if self.dispatcher:
                  # distributed mode: wait for all workers to finish
                  logger.info(
                      "reached the end of input; now waiting for all remaining jobs to finish"
                  )
                  other = self.dispatcher.getstate()
              self.do_mstep(rho(), other, pass_ > 0)
              del other
              dirty = False
  
imports: import logging
  import numbers
  import os
  import time
  from collections import defaultdict
  import numpy
  from scipy.special import gammaln, psi
  from scipy.special import polygamma
  from gensim import interfaces, utils, matutils
  from gensim.matutils import kullback_leibler, hellinger, jaccard_distance, jensen_shannon, dirichlet_expectation, logsumexp, mean_absolute_difference
  from gensim.models import basemodel, CoherenceModel
  from gensim.models.callbacks import Callback
label_desc: Use image node_id 11 for calling the LdaModel obj's underlying member method code with example usage: obj.update(corpus, chunksize, decay, offset, passes, update_every, eval_every, iterations, gamma_threshold, chunks_as_numpy) and returns: pow