class_docstr: The constructor estimates the author-topic model parameters based on a training corpus.
class_docstr_tok: ["The","constructor","estimates","the","author-topic","model","parameters","based","on","a","training","corpus","."]
docstr: Train the model with new documents, by EM-iterating over `corpus` until the topics converge (or until the
  maximum number of allowed iterations is reached).
  
  Notes
  -----
  This update also supports updating an already trained model (`self`) with new documents from `corpus`;
  the two models are then merged in proportion to the number of old vs. new documents.
  This feature is still experimental for non-stationary input streams.
  
  For stationary input (no topic drift in new documents), on the other hand, this equals the
  online update of `'Online Learning for LDA' by Hoffman et al.`_
  and is guaranteed to converge for any `decay` in (0.5, 1]. Additionally, for smaller corpus sizes, an
  increasing `offset` may be beneficial (see Table 1 in the same paper).
  
  If update is called with authors that already exist in the model, it will resume training on not only new
  documents for that author, but also the previously seen documents. This is necessary for those authors' topic
  distributions to converge.
  
  Every time `update(corpus, author2doc)` is called, the new documents are to appended to all the previously seen
  documents, and author2doc is combined with the previously seen authors.
  
  To resume training on all the data seen by the model, simply call
  :meth:`~gensim.models.atmodel.AuthorTopicModel.update`.
  
  It is not possible to add new authors to existing documents, as all documents in `corpus` are assumed to be
  new documents.
  
  Parameters
  ----------
  corpus : iterable of list of (int, float)
      The corpus in BoW format.
  author2doc : dict of (str, list of int), optional
      A dictionary where keys are the names of authors and values are lists of document IDs that the author
      contributes to.
  doc2author : dict of (int, list of str), optional
      A dictionary where the keys are document IDs and the values are lists of author names.
  chunksize : int, optional
      Controls the size of the mini-batches.
  decay : float, optional
      A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
      when each new document is examined. Corresponds to :math:`\kappa` from
      `'Online Learning for LDA' by Hoffman et al.`_
  offset : float, optional
      Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
      Corresponds to :math:`\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
  passes : int, optional
      Number of times the model makes a pass over the entire training data.
  update_every : int, optional
      Make updates in topic probability for latest mini-batch.
  eval_every : int, optional
      Calculate and estimate log perplexity for latest mini-batch.
  iterations : int, optional
      Maximum number of times the model loops over each document
  gamma_threshold : float, optional
      Threshold value of gamma(topic difference between consecutive two topics)
      until which the iterations continue.
  chunks_as_numpy : bool, optional
      Whether each chunk passed to :meth:`~gensim.models.atmodel.AuthorTopicModel.inference` should be a numpy
      array of not. Numpy can in some settings turn the term IDs into floats, these will be converted back into
      integers in inference, which incurs a performance hit. For distributed computing (not supported now)
      it may be desirable to keep the chunks as numpy arrays.
docstr_tok: ["Train","the","model","with","new","documents",",","by","EM-iterating","over","`","corpus","`","until","the","topics","converge","(","or","until","the","maximum","number","of","allowed","iterations","is","reached",")",".","Notes","--","--","-","This","update","also","supports","updating","an","already","trained","model","(","`","self","`",")","with","new","documents","from","`","corpus","`",";","the","two","models","are","then","merged","in","proportion","to","the","number","of","old","vs.","new","documents",".","This","feature","is","still","experimental","for","non-stationary","input","streams",".","For","stationary","input","(","no","topic","drift","in","new","documents",")",",","on","the","other","hand",",","this","equals","the","online","update","of","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","and","is","guaranteed","to","converge","for","any","`","decay","`","in","(","0.5",",","1","]",".","Additionally",",","for","smaller","corpus","sizes",",","an","increasing","`","offset","`","may","be","beneficial","(","see","Table","1","in","the","same","paper",")",".","If","update","is","called","with","authors","that","already","exist","in","the","model",",","it","will","resume","training","on","not","only","new","documents","for","that","author",",","but","also","the","previously","seen","documents",".","This","is","necessary","for","those","authors","'","topic","distributions","to","converge",".","Every","time","`","update","(","corpus",",","author2doc",")","`","is","called",",","the","new","documents","are","to","appended","to","all","the","previously","seen","documents",",","and","author2doc","is","combined","with","the","previously","seen","authors",".","To","resume","training","on","all","the","data","seen","by","the","model",",","simply","call",":","meth",":","`","~gensim.models.atmodel.AuthorTopicModel.update","`",".","It","is","not","possible","to","add","new","authors","to","existing","documents",",","as","all","documents","in","`","corpus","`","are","assumed","to","be","new","documents",".","Parameters","--","--","--","--","--","corpus",":","iterable","of","list","of","(","int",",","float",")","The","corpus","in","BoW","format",".","author2doc",":","dict","of","(","str",",","list","of","int",")",",","optional","A","dictionary","where","keys","are","the","names","of","authors","and","values","are","lists","of","document","IDs","that","the","author","contributes","to",".","doc2author",":","dict","of","(","int",",","list","of","str",")",",","optional","A","dictionary","where","the","keys","are","document","IDs","and","the","values","are","lists","of","author","names",".","chunksize",":","int",",","optional","Controls","the","size","of","the","mini-batches",".","decay",":","float",",","optional","A","number","between","(","0.5",",","1","]","to","weight","what","percentage","of","the","previous","lambda","value","is","forgotten","when","each","new","document","is","examined",".","Corresponds","to",":","math",":","`","\\kappa","`","from","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","offset",":","float",",","optional","Hyper-parameter","that","controls","how","much","we","will","slow","down","the","first","steps","the","first","few","iterations",".","Corresponds","to",":","math",":","`","\\tau_0","`","from","`","'Online","Learning","for","LDA","'","by","Hoffman","et","al.","`","_","passes",":","int",",","optional","Number","of","times","the","model","makes","a","pass","over","the","entire","training","data",".","update_every",":","int",",","optional","Make","updates","in","topic","probability","for","latest","mini-batch",".","eval_every",":","int",",","optional","Calculate","and","estimate","log","perplexity","for","latest","mini-batch",".","iterations",":","int",",","optional","Maximum","number","of","times","the","model","loops","over","each","document","gamma_threshold",":","float",",","optional","Threshold","value","of","gamma","(","topic","difference","between","consecutive","two","topics",")","until","which","the","iterations","continue",".","chunks_as_numpy",":","bool",",","optional","Whether","each","chunk","passed","to",":","meth",":","`","~gensim.models.atmodel.AuthorTopicModel.inference","`","should","be","a","numpy","array","of","not",".","Numpy","can","in","some","settings","turn","the","term","IDs","into","floats",",","these","will","be","converted","back","into","integers","in","inference",",","which","incurs","a","performance","hit",".","For","distributed","computing","(","not","supported","now",")","it","may","be","desirable","to","keep","the","chunks","as","numpy","arrays","."]
code: def update(
      self,
      corpus=None,
      author2doc=None,
      doc2author=None,
      chunksize=None,
      decay=None,
      offset=None,
      passes=None,
      update_every=None,
      eval_every=None,
      iterations=None,
      gamma_threshold=None,
      chunks_as_numpy=False,
  ):
      """Train the model with new documents, by EM-iterating over `corpus` until the topics converge (or until the
      maximum number of allowed iterations is reached).
  
      Notes
      -----
      This update also supports updating an already trained model (`self`) with new documents from `corpus`;
      the two models are then merged in proportion to the number of old vs. new documents.
      This feature is still experimental for non-stationary input streams.
  
      For stationary input (no topic drift in new documents), on the other hand, this equals the
      online update of `'Online Learning for LDA' by Hoffman et al.`_
      and is guaranteed to converge for any `decay` in (0.5, 1]. Additionally, for smaller corpus sizes, an
      increasing `offset` may be beneficial (see Table 1 in the same paper).
  
      If update is called with authors that already exist in the model, it will resume training on not only new
      documents for that author, but also the previously seen documents. This is necessary for those authors' topic
      distributions to converge.
  
      Every time `update(corpus, author2doc)` is called, the new documents are to appended to all the previously seen
      documents, and author2doc is combined with the previously seen authors.
  
      To resume training on all the data seen by the model, simply call
      :meth:`~gensim.models.atmodel.AuthorTopicModel.update`.
  
      It is not possible to add new authors to existing documents, as all documents in `corpus` are assumed to be
      new documents.
  
      Parameters
      ----------
      corpus : iterable of list of (int, float)
          The corpus in BoW format.
      author2doc : dict of (str, list of int), optional
          A dictionary where keys are the names of authors and values are lists of document IDs that the author
          contributes to.
      doc2author : dict of (int, list of str), optional
          A dictionary where the keys are document IDs and the values are lists of author names.
      chunksize : int, optional
          Controls the size of the mini-batches.
      decay : float, optional
          A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
          when each new document is examined. Corresponds to :math:`\\kappa` from
          `'Online Learning for LDA' by Hoffman et al.`_
      offset : float, optional
          Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
          Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
      passes : int, optional
          Number of times the model makes a pass over the entire training data.
      update_every : int, optional
          Make updates in topic probability for latest mini-batch.
      eval_every : int, optional
          Calculate and estimate log perplexity for latest mini-batch.
      iterations : int, optional
          Maximum number of times the model loops over each document
      gamma_threshold : float, optional
          Threshold value of gamma(topic difference between consecutive two topics)
          until which the iterations continue.
      chunks_as_numpy : bool, optional
          Whether each chunk passed to :meth:`~gensim.models.atmodel.AuthorTopicModel.inference` should be a numpy
          array of not. Numpy can in some settings turn the term IDs into floats, these will be converted back into
          integers in inference, which incurs a performance hit. For distributed computing (not supported now)
          it may be desirable to keep the chunks as numpy arrays.
  
      """
      # use parameters given in constructor, unless user explicitly overrode them
      if decay is None:
          decay = self.decay
      if offset is None:
          offset = self.offset
      if passes is None:
          passes = self.passes
      if update_every is None:
          update_every = self.update_every
      if eval_every is None:
          eval_every = self.eval_every
      if iterations is None:
          iterations = self.iterations
      if gamma_threshold is None:
          gamma_threshold = self.gamma_threshold
  
      # TODO: if deepcopy is not used here, something goes wrong. When unit tests are run (specifically "testPasses"),
      # the process simply gets killed.
      author2doc = deepcopy(author2doc)
      doc2author = deepcopy(doc2author)
  
      # TODO: it is not possible to add new authors to an existing document (all input documents are treated
      # as completely new documents). Perhaps this functionality could be implemented.
      # If it's absolutely necessary, the user can delete the documents that have new authors, and call update
      # on them with the new and old authors.
  
      if corpus is None:
          # Just keep training on the already available data.
          # Assumes self.update() has been called before with input documents and corresponding authors.
          assert (
              self.total_docs > 0
          ), "update() was called with no documents to train on."
          train_corpus_idx = [d for d in range(self.total_docs)]
          num_input_authors = len(self.author2doc)
      else:
          if doc2author is None and author2doc is None:
              raise ValueError(
                  "at least one of author2doc/doc2author must be specified, to establish input space dimensionality"
              )
  
          # If either doc2author or author2doc is missing, construct them from the other.
          if doc2author is None:
              doc2author = construct_doc2author(corpus, author2doc)
          elif author2doc is None:
              author2doc = construct_author2doc(doc2author)
  
          # Number of authors that need to be updated.
          num_input_authors = len(author2doc)
  
          try:
              len_input_corpus = len(corpus)
          except TypeError:
              logger.warning(
                  "input corpus stream has no len(); counting documents"
              )
              len_input_corpus = sum(1 for _ in corpus)
          if len_input_corpus == 0:
              logger.warning(
                  "AuthorTopicModel.update() called with an empty corpus"
              )
              return
  
          self.total_docs += len_input_corpus
  
          # Add new documents in corpus to self.corpus.
          self.extend_corpus(corpus)
  
          # Obtain a list of new authors.
          new_authors = []
          # Sorting the author names makes the model more reproducible.
          for a in sorted(author2doc.keys()):
              if not self.author2doc.get(a):
                  new_authors.append(a)
  
          num_new_authors = len(new_authors)
  
          # Add new authors do author2id/id2author dictionaries.
          for a_id, a_name in enumerate(new_authors):
              self.author2id[a_name] = a_id + self.num_authors
              self.id2author[a_id + self.num_authors] = a_name
  
          # Increment the number of total authors seen.
          self.num_authors += num_new_authors
  
          # Initialize the variational distributions q(theta|gamma)
          gamma_new = self.random_state.gamma(
              100.0, 1.0 / 100.0, (num_new_authors, self.num_topics)
          )
          self.state.gamma = np.vstack([self.state.gamma, gamma_new])
  
          # Combine author2doc with self.author2doc.
          # First, increment the document IDs by the number of previously seen documents.
          for a, doc_ids in author2doc.items():
              doc_ids = [
                  d + self.total_docs - len_input_corpus
                  for d in doc_ids
              ]
  
          # For all authors in the input corpus, add the new documents.
          for a, doc_ids in author2doc.items():
              if self.author2doc.get(a):
                  # This is not a new author, append new documents.
                  self.author2doc[a].extend(doc_ids)
              else:
                  # This is a new author, create index.
                  self.author2doc[a] = doc_ids
  
          # Add all new documents to self.doc2author.
          for d, a_list in doc2author.items():
              self.doc2author[d] = a_list
  
          # Train on all documents of authors in input_corpus.
          train_corpus_idx = set()
          # Collect all documents of authors.
          for doc_ids in self.author2doc.values():
              train_corpus_idx.update(doc_ids)
  
          # Make the list of training documents unique.
          train_corpus_idx = sorted(train_corpus_idx)
  
      # train_corpus_idx is only a list of indexes, so "len" is valid.
      lencorpus = len(train_corpus_idx)
  
      if chunksize is None:
          chunksize = min(lencorpus, self.chunksize)
  
      self.state.numdocs += lencorpus
  
      if update_every:
          updatetype = "online"
          updateafter = min(
              lencorpus, update_every * self.numworkers * chunksize
          )
      else:
          updatetype = "batch"
          updateafter = lencorpus
      evalafter = min(
          lencorpus, (eval_every or 0) * self.numworkers * chunksize
      )
  
      updates_per_pass = max(1, lencorpus / updateafter)
      logger.info(
          "running %s author-topic training, %s topics, %s authors, "
          "%i passes over the supplied corpus of %i documents, updating model once "
          "every %i documents, evaluating perplexity every %i documents, "
          "iterating %ix with a convergence threshold of %f",
          updatetype,
          self.num_topics,
          num_input_authors,
          passes,
          lencorpus,
          updateafter,
          evalafter,
          iterations,
          gamma_threshold,
      )
  
      if updates_per_pass * passes < 10:
          logger.warning(
              "too few updates, training might not converge; "
              "consider increasing the number of passes or iterations to improve accuracy"
          )
  
      # rho is the "speed" of updating; TODO try other fncs
      # pass_ + num_updates handles increasing the starting t for each pass,
      # while allowing it to "reset" on the first pass of each update
      def rho():
          return pow(
              offset + pass_ + (self.num_updates / chunksize), -decay
          )
  
      for pass_ in range(passes):
          if self.dispatcher:
              logger.info("initializing %s workers", self.numworkers)
              self.dispatcher.reset(self.state)
          else:
              # gamma is not needed in "other", thus its shape is (0, 0).
              other = AuthorTopicState(
                  self.eta, self.state.sstats.shape, (0, 0)
              )
          dirty = False
  
          reallen = 0
          for chunk_no, chunk_doc_idx in enumerate(
              utils.grouper(
                  train_corpus_idx, chunksize, as_numpy=chunks_as_numpy
              )
          ):
              chunk = [self.corpus[d] for d in chunk_doc_idx]
              reallen += len(
                  chunk
              )  # keep track of how many documents we've processed so far
  
              if eval_every and (
                  (reallen == lencorpus)
                  or (
                      (chunk_no + 1) % (eval_every * self.numworkers)
                      == 0
                  )
              ):
                  # log_perplexity requires the indexes of the documents being evaluated, to know what authors
                  # correspond to the documents.
                  self.log_perplexity(
                      chunk, chunk_doc_idx, total_docs=lencorpus
                  )
  
              if self.dispatcher:
                  # add the chunk to dispatcher's job queue, so workers can munch on it
                  logger.info(
                      "PROGRESS: pass %i, dispatching documents up to #%i/%i",
                      pass_,
                      chunk_no * chunksize + len(chunk),
                      lencorpus,
                  )
                  # this will eventually block until some jobs finish, because the queue has a small finite length
                  self.dispatcher.putjob(chunk)
              else:
                  logger.info(
                      "PROGRESS: pass %i, at document #%i/%i",
                      pass_,
                      chunk_no * chunksize + len(chunk),
                      lencorpus,
                  )
                  # do_estep requires the indexes of the documents being trained on, to know what authors
                  # correspond to the documents.
                  gammat = self.do_estep(
                      chunk,
                      self.author2doc,
                      self.doc2author,
                      rho(),
                      other,
                      chunk_doc_idx,
                  )
  
                  if self.optimize_alpha:
                      self.update_alpha(gammat, rho())
  
              dirty = True
              del chunk
  
              # perform an M step. determine when based on update_every, don't do this after every chunk
              if (
                  update_every
                  and (chunk_no + 1) % (update_every * self.numworkers)
                  == 0
              ):
                  if self.dispatcher:
                      # distributed mode: wait for all workers to finish
                      logger.info(
                          "reached the end of input; now waiting for all remaining jobs to finish"
                      )
                      other = self.dispatcher.getstate()
                  self.do_mstep(rho(), other, pass_ > 0)
                  del other  # frees up memory
  
                  if self.dispatcher:
                      logger.info("initializing workers")
                      self.dispatcher.reset(self.state)
                  else:
                      other = AuthorTopicState(
                          self.eta, self.state.sstats.shape, (0, 0)
                      )
                  dirty = False
          # endfor single corpus iteration
          if reallen != lencorpus:
              raise RuntimeError(
                  "input corpus size changed during training (don't use generators as input)"
              )
  
          if dirty:
              # finish any remaining updates
              if self.dispatcher:
                  # distributed mode: wait for all workers to finish
                  logger.info(
                      "reached the end of input; now waiting for all remaining jobs to finish"
                  )
                  other = self.dispatcher.getstate()
              self.do_mstep(rho(), other, pass_ > 0)
              del other
  
imports: import logging
  from itertools import chain
  from copy import deepcopy
  from shutil import copyfile
  from os.path import isfile
  from os import remove
  import numpy
  from scipy.special import gammaln
  from gensim import utils
  from gensim.models import LdaModel
  from gensim.models.ldamodel import LdaState
  from gensim.matutils import dirichlet_expectation, mean_absolute_difference
  from gensim.corpora import MmCorpus
label_desc: Use image node_id 9 for calling the AuthorTopicModel obj's underlying member method code with example usage: obj.update(corpus, author2doc, doc2author, chunksize, decay, offset, passes, update_every, eval_every, iterations, gamma_threshold, chunks_as_numpy) and returns: pow