class_docstr: Objects of this class realize the transformation between word-document co-occurrence matrix (int)
  into a locally/globally weighted TF-IDF matrix (positive floats).
  
  Examples
  --------
  .. sourcecode:: pycon
  
      >>> import gensim.downloader as api
      >>> from gensim.models import TfidfModel
      >>> from gensim.corpora import Dictionary
      >>>
      >>> dataset = api.load("text8")
      >>> dct = Dictionary(dataset)  # fit dictionary
      >>> corpus = [dct.doc2bow(line) for line in dataset]  # convert corpus to BoW format
      >>>
      >>> model = TfidfModel(corpus)  # fit model
      >>> vector = model[corpus[0]]  # apply model to the first corpus document
class_docstr_tok: ["Objects","of","this","class","realize","the","transformation","between","word-document","co-occurrence","matrix","(","int",")","into","a","locally\/globally","weighted","TF-IDF","matrix","(","positive","floats",")",".","Examples","--","--","--","--","..","sourcecode",":",":","pycon",">",">",">","import","gensim.downloader","as","api",">",">",">","from","gensim.models","import","TfidfModel",">",">",">","from","gensim.corpora","import","Dictionary",">",">",">",">",">",">","dataset","=","api.load","(","``","text8","''",")",">",">",">","dct","=","Dictionary","(","dataset",")","#","fit","dictionary",">",">",">","corpus","=","[","dct.doc2bow","(","line",")","for","line","in","dataset","]","#","convert","corpus","to","BoW","format",">",">",">",">",">",">","model","=","TfidfModel","(","corpus",")","#","fit","model",">",">",">","vector","=","model","[","corpus","[","0","]","]","#","apply","model","to","the","first","corpus","document"]
docstr: Compute TF-IDF by multiplying a local component (term frequency) with a global component
  (inverse document frequency), and normalizing the resulting documents to unit length.
  Formula for non-normalized weight of term :math:`i` in document :math:`j` in a corpus of :math:`D` documents
  
  .. math:: weight_{i,j} = frequency_{i,j} * log_2 \frac{D}{document\_freq_{i}}
  
  or, more generally
  
  .. math:: weight_{i,j} = wlocal(frequency_{i,j}) * wglobal(document\_freq_{i}, D)
  
  so you can plug in your own custom :math:`wlocal` and :math:`wglobal` functions.
  
  Parameters
  ----------
  corpus : iterable of iterable of (int, int), optional
      Input corpus
  id2word : {dict, :class:`~gensim.corpora.Dictionary`}, optional
      Mapping token - id, that was used for converting input data to bag of words format.
  dictionary : :class:`~gensim.corpora.Dictionary`
      If `dictionary` is specified, it must be a `corpora.Dictionary` object and it will be used.
      to directly construct the inverse document frequency mapping (then `corpus`, if specified, is ignored).
  wlocals : callable, optional
      Function for local weighting, default for `wlocal` is :func:`~gensim.utils.identity`
      (other options: :func:`numpy.sqrt`, `lambda tf: 0.5 + (0.5 * tf / tf.max())`, etc.).
  wglobal : callable, optional
      Function for global weighting, default is :func:`~gensim.models.tfidfmodel.df2idf`.
  normalize : {bool, callable}, optional
      Normalize document vectors to unit euclidean length? You can also inject your own function into `normalize`.
  smartirs : str, optional
      SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System,
      a mnemonic scheme for denoting tf-idf weighting variants in the vector space model.
      The mnemonic for representing a combination of weights takes the form XYZ,
      for example 'ntc', 'bpn' and so on, where the letters represents the term weighting of the document vector.
  
      Term frequency weighing:
          * `b` - binary,
          * `t` or `n` - raw,
          * `a` - augmented,
          * `l` - logarithm,
          * `d` - double logarithm,
          * `L` - log average.
  
      Document frequency weighting:
          * `x` or `n` - none,
          * `f` - idf,
          * `t` - zero-corrected idf,
          * `p` - probabilistic idf.
  
      Document normalization:
          * `x` or `n` - none,
          * `c` - cosine,
          * `u` - pivoted unique,
          * `b` - pivoted character length.
  
      Default is 'nfc'.
      For more information visit `SMART Information Retrieval System
      <https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System>`_.
  pivot : float or None, optional
      In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length
      normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 -
      slope) * pivot`.
  
      You can either set the `pivot` by hand, or you can let Gensim figure it out automatically with the following
      two steps:
  
          * Set either the `u` or `b` document normalization in the `smartirs` parameter.
          * Set either the `corpus` or `dictionary` parameter. The `pivot` will be automatically determined from
            the properties of the `corpus` or `dictionary`.
  
      If `pivot` is None and you don't follow steps 1 and 2, then pivoted document length normalization will be
      disabled. Default is None.
  
      See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/.
  slope : float, optional
      In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length
      normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 -
      slope) * pivot`.
  
      Setting the `slope` to 0.0 uses only the `pivot` as the norm, and setting the `slope` to 1.0 effectively
      disables pivoted document length normalization. Singhal [2]_ suggests setting the `slope` between 0.2 and
      0.3 for best results. Default is 0.25.
  
      See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/.
  
  References
  ----------
  .. [1] Singhal, A., Buckley, C., & Mitra, M. (1996). `Pivoted Document Length
     Normalization <http://singhal.info/pivoted-dln.pdf>`_. *SIGIR Forum*, 51, 176–184.
  .. [2] Singhal, A. (2001). `Modern information retrieval: A brief overview <http://singhal.info/ieee2001.pdf>`_.
     *IEEE Data Eng. Bull.*, 24(4), 35–43.
docstr_tok: ["Compute","TF-IDF","by","multiplying","a","local","component","(","term","frequency",")","with","a","global","component","(","inverse","document","frequency",")",",","and","normalizing","the","resulting","documents","to","unit","length",".","Formula","for","non-normalized","weight","of","term",":","math",":","`","i","`","in","document",":","math",":","`","j","`","in","a","corpus","of",":","math",":","`","D","`","documents","..","math",":",":","weight_","{","i",",","j","}","=","frequency_","{","i",",","j","}","*","log_2","\\frac","{","D","}","{","document\\_freq_","{","i","}","}","or",",","more","generally","..","math",":",":","weight_","{","i",",","j","}","=","wlocal","(","frequency_","{","i",",","j","}",")","*","wglobal","(","document\\_freq_","{","i","}",",","D",")","so","you","can","plug","in","your","own","custom",":","math",":","`","wlocal","`","and",":","math",":","`","wglobal","`","functions",".","Parameters","--","--","--","--","--","corpus",":","iterable","of","iterable","of","(","int",",","int",")",",","optional","Input","corpus","id2word",":","{","dict",",",":","class",":","`","~gensim.corpora.Dictionary","`","}",",","optional","Mapping","token","-","id",",","that","was","used","for","converting","input","data","to","bag","of","words","format",".","dictionary",":",":","class",":","`","~gensim.corpora.Dictionary","`","If","`","dictionary","`","is","specified",",","it","must","be","a","`","corpora.Dictionary","`","object","and","it","will","be","used",".","to","directly","construct","the","inverse","document","frequency","mapping","(","then","`","corpus","`",",","if","specified",",","is","ignored",")",".","wlocals",":","callable",",","optional","Function","for","local","weighting",",","default","for","`","wlocal","`","is",":","func",":","`","~gensim.utils.identity","`","(","other","options",":",":","func",":","`","numpy.sqrt","`",",","`","lambda","tf",":","0.5","+","(","0.5","*","tf","\/","tf.max","(",")",")","`",",","etc.",")",".","wglobal",":","callable",",","optional","Function","for","global","weighting",",","default","is",":","func",":","`","~gensim.models.tfidfmodel.df2idf","`",".","normalize",":","{","bool",",","callable","}",",","optional","Normalize","document","vectors","to","unit","euclidean","length","?","You","can","also","inject","your","own","function","into","`","normalize","`",".","smartirs",":","str",",","optional","SMART","(","System","for","the","Mechanical","Analysis","and","Retrieval","of","Text",")","Information","Retrieval","System",",","a","mnemonic","scheme","for","denoting","tf-idf","weighting","variants","in","the","vector","space","model",".","The","mnemonic","for","representing","a","combination","of","weights","takes","the","form","XYZ",",","for","example","'ntc","'",",","'bpn","'","and","so","on",",","where","the","letters","represents","the","term","weighting","of","the","document","vector",".","Term","frequency","weighing",":","*","`","b","`","-","binary",",","*","`","t","`","or","`","n","`","-","raw",",","*","`","a","`","-","augmented",",","*","`","l","`","-","logarithm",",","*","`","d","`","-","double","logarithm",",","*","`","L","`","-","log","average",".","Document","frequency","weighting",":","*","`","x","`","or","`","n","`","-","none",",","*","`","f","`","-","idf",",","*","`","t","`","-","zero-corrected","idf",",","*","`","p","`","-","probabilistic","idf",".","Document","normalization",":","*","`","x","`","or","`","n","`","-","none",",","*","`","c","`","-","cosine",",","*","`","u","`","-","pivoted","unique",",","*","`","b","`","-","pivoted","character","length",".","Default","is","'nfc","'",".","For","more","information","visit","`","SMART","Information","Retrieval","System","<","https",":","\/\/en.wikipedia.org\/wiki\/SMART_Information_Retrieval_System",">","`","_",".","pivot",":","float","or","None",",","optional","In","information","retrieval",",","TF-IDF","is","biased","against","long","documents","[","1","]","_.","Pivoted","document","length","normalization","solves","this","problem","by","changing","the","norm","of","a","document","to","`","slope","*","old_norm","+","(","1.0","-","slope",")","*","pivot","`",".","You","can","either","set","the","`","pivot","`","by","hand",",","or","you","can","let","Gensim","figure","it","out","automatically","with","the","following","two","steps",":","*","Set","either","the","`","u","`","or","`","b","`","document","normalization","in","the","`","smartirs","`","parameter",".","*","Set","either","the","`","corpus","`","or","`","dictionary","`","parameter",".","The","`","pivot","`","will","be","automatically","determined","from","the","properties","of","the","`","corpus","`","or","`","dictionary","`",".","If","`","pivot","`","is","None","and","you","do","n't","follow","steps","1","and","2",",","then","pivoted","document","length","normalization","will","be","disabled",".","Default","is","None",".","See","also","the","blog","post","at","https",":","\/\/rare-technologies.com\/pivoted-document-length-normalisation\/",".","slope",":","float",",","optional","In","information","retrieval",",","TF-IDF","is","biased","against","long","documents","[","1","]","_.","Pivoted","document","length","normalization","solves","this","problem","by","changing","the","norm","of","a","document","to","`","slope","*","old_norm","+","(","1.0","-","slope",")","*","pivot","`",".","Setting","the","`","slope","`","to","0.0","uses","only","the","`","pivot","`","as","the","norm",",","and","setting","the","`","slope","`","to","1.0","effectively","disables","pivoted","document","length","normalization",".","Singhal","[","2","]","_","suggests","setting","the","`","slope","`","between","0.2","and","0.3","for","best","results",".","Default","is","0.25",".","See","also","the","blog","post","at","https",":","\/\/rare-technologies.com\/pivoted-document-length-normalisation\/",".","References","--","--","--","--","--","..","[","1","]","Singhal",",","A.",",","Buckley",",","C.",",","&","Mitra",",","M.","(","1996",")",".","`","Pivoted","Document","Length","Normalization","<","http",":","\/\/singhal.info\/pivoted-dln.pdf",">","`","_",".","*","SIGIR","Forum","*",",","51",",","176\u2013184",".","..","[","2","]","Singhal",",","A",".","(","2001",")",".","`","Modern","information","retrieval",":","A","brief","overview","<","http",":","\/\/singhal.info\/ieee2001.pdf",">","`","_",".","*","IEEE","Data","Eng",".","Bull",".","*",",","24","(","4",")",",","35\u201343","."]
code: def __init__(
      self,
      corpus=None,
      id2word=None,
      dictionary=None,
      wlocal=utils.identity,
      wglobal=df2idf,
      normalize=True,
      smartirs=None,
      pivot=None,
      slope=0.25,
  ):
      r"""Compute TF-IDF by multiplying a local component (term frequency) with a global component
      (inverse document frequency), and normalizing the resulting documents to unit length.
      Formula for non-normalized weight of term :math:`i` in document :math:`j` in a corpus of :math:`D` documents
  
      .. math:: weight_{i,j} = frequency_{i,j} * log_2 \frac{D}{document\_freq_{i}}
  
      or, more generally
  
      .. math:: weight_{i,j} = wlocal(frequency_{i,j}) * wglobal(document\_freq_{i}, D)
  
      so you can plug in your own custom :math:`wlocal` and :math:`wglobal` functions.
  
      Parameters
      ----------
      corpus : iterable of iterable of (int, int), optional
          Input corpus
      id2word : {dict, :class:`~gensim.corpora.Dictionary`}, optional
          Mapping token - id, that was used for converting input data to bag of words format.
      dictionary : :class:`~gensim.corpora.Dictionary`
          If `dictionary` is specified, it must be a `corpora.Dictionary` object and it will be used.
          to directly construct the inverse document frequency mapping (then `corpus`, if specified, is ignored).
      wlocals : callable, optional
          Function for local weighting, default for `wlocal` is :func:`~gensim.utils.identity`
          (other options: :func:`numpy.sqrt`, `lambda tf: 0.5 + (0.5 * tf / tf.max())`, etc.).
      wglobal : callable, optional
          Function for global weighting, default is :func:`~gensim.models.tfidfmodel.df2idf`.
      normalize : {bool, callable}, optional
          Normalize document vectors to unit euclidean length? You can also inject your own function into `normalize`.
      smartirs : str, optional
          SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System,
          a mnemonic scheme for denoting tf-idf weighting variants in the vector space model.
          The mnemonic for representing a combination of weights takes the form XYZ,
          for example 'ntc', 'bpn' and so on, where the letters represents the term weighting of the document vector.
  
          Term frequency weighing:
              * `b` - binary,
              * `t` or `n` - raw,
              * `a` - augmented,
              * `l` - logarithm,
              * `d` - double logarithm,
              * `L` - log average.
  
          Document frequency weighting:
              * `x` or `n` - none,
              * `f` - idf,
              * `t` - zero-corrected idf,
              * `p` - probabilistic idf.
  
          Document normalization:
              * `x` or `n` - none,
              * `c` - cosine,
              * `u` - pivoted unique,
              * `b` - pivoted character length.
  
          Default is 'nfc'.
          For more information visit `SMART Information Retrieval System
          <https://en.wikipedia.org/wiki/SMART_Information_Retrieval_System>`_.
      pivot : float or None, optional
          In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length
          normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 -
          slope) * pivot`.
  
          You can either set the `pivot` by hand, or you can let Gensim figure it out automatically with the following
          two steps:
  
              * Set either the `u` or `b` document normalization in the `smartirs` parameter.
              * Set either the `corpus` or `dictionary` parameter. The `pivot` will be automatically determined from
                the properties of the `corpus` or `dictionary`.
  
          If `pivot` is None and you don't follow steps 1 and 2, then pivoted document length normalization will be
          disabled. Default is None.
  
          See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/.
      slope : float, optional
          In information retrieval, TF-IDF is biased against long documents [1]_. Pivoted document length
          normalization solves this problem by changing the norm of a document to `slope * old_norm + (1.0 -
          slope) * pivot`.
  
          Setting the `slope` to 0.0 uses only the `pivot` as the norm, and setting the `slope` to 1.0 effectively
          disables pivoted document length normalization. Singhal [2]_ suggests setting the `slope` between 0.2 and
          0.3 for best results. Default is 0.25.
  
          See also the blog post at https://rare-technologies.com/pivoted-document-length-normalisation/.
  
      References
      ----------
      .. [1] Singhal, A., Buckley, C., & Mitra, M. (1996). `Pivoted Document Length
         Normalization <http://singhal.info/pivoted-dln.pdf>`_. *SIGIR Forum*, 51, 176–184.
      .. [2] Singhal, A. (2001). `Modern information retrieval: A brief overview <http://singhal.info/ieee2001.pdf>`_.
         *IEEE Data Eng. Bull.*, 24(4), 35–43.
  
      """
      self.id2word = id2word
      self.wlocal, self.wglobal, self.normalize = (
          wlocal,
          wglobal,
          normalize,
      )
      self.num_docs, self.num_nnz, self.idfs = None, None, None
      self.smartirs = (
          resolve_weights(smartirs) if smartirs is not None else None
      )
      self.slope = slope
      self.pivot = pivot
      self.eps = 1e-12
  
      if smartirs:
          n_tf, n_df, n_n = self.smartirs
          self.wlocal = partial(smartirs_wlocal, local_scheme=n_tf)
          self.wglobal = partial(smartirs_wglobal, global_scheme=n_df)
  
      if dictionary:
          # user supplied a Dictionary object, which already contains all the
          # statistics we need to construct the IDF mapping. we can skip the
          # step that goes through the corpus (= an optimization).
          if corpus:
              logger.warning(
                  "constructor received both corpus and explicit inverse document frequencies; ignoring the corpus"
              )
          self.num_docs, self.num_nnz = (
              dictionary.num_docs,
              dictionary.num_nnz,
          )
          self.cfs = dictionary.cfs.copy()
          self.dfs = dictionary.dfs.copy()
          self.term_lens = {
              termid: len(term) for termid, term in dictionary.items()
          }
          self.idfs = precompute_idfs(
              self.wglobal, self.dfs, self.num_docs
          )
          if not id2word:
              self.id2word = dictionary
      elif corpus:
          self.initialize(corpus)
      else:
          # NOTE: everything is left uninitialized; presumably the model will
          # be initialized in some other way
          pass
  
      # If smartirs is not None, override pivot and normalize
      if not smartirs:
          return
      if self.pivot is not None:
          if n_n in "ub":
              logger.warning(
                  "constructor received pivot; ignoring smartirs[2]"
              )
          return
      if n_n in "ub" and callable(self.normalize):
          logger.warning(
              "constructor received smartirs; ignoring normalize"
          )
      if n_n in "ub" and not dictionary and not corpus:
          logger.warning(
              "constructor received no corpus or dictionary; ignoring smartirs[2]"
          )
      elif n_n == "u":
          self.pivot = 1.0 * self.num_nnz / self.num_docs
      elif n_n == "b":
          self.pivot = (
              1.0
              * sum(
                  self.cfs[termid] * (self.term_lens[termid] + 1.0)
                  for termid in dictionary.keys()
              )
              / self.num_docs
          )
  
file_path: gensim/gensim/models/tfidfmodel.py
imports: import logging
  from functools import partial
  import re
  import numpy
  from gensim import interfaces, matutils, utils
  from gensim.utils import deprecated
label_desc: Use image node_id 1 to create a new TfidfModel object from inherited base classes: interfaces with example: obj = TfidfModel(corpus, id2word, dictionary, wlocal, wglobal, normalize, smartirs, pivot, slope)