class_docstr: Helper class to simplify the pipeline of getting BoW vectors from plain text.
  
  Notes
  -----
  This is an abstract base class: override the :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` and
  :meth:`~gensim.corpora.textcorpus.TextCorpus.__len__` methods to match your particular input.
  
  Given a filename (or a file-like object) in constructor, the corpus object will be automatically initialized
  with a dictionary in `self.dictionary` and will support the :meth:`~gensim.corpora.textcorpus.TextCorpus.__iter__`
  corpus method.  You have a few different ways of utilizing this class via subclassing or by construction with
  different preprocessing arguments.
  
  The :meth:`~gensim.corpora.textcorpus.TextCorpus.__iter__` method converts the lists of tokens produced by
  :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` to BoW format using
  :meth:`gensim.corpora.dictionary.Dictionary.doc2bow`.
  
  :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` does the following:
  
  #. Calls :meth:`~gensim.corpora.textcorpus.TextCorpus.getstream` to get a generator over the texts.
     It yields each document in turn from the underlying text file or files.
  #. For each document from the stream, calls :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` to produce
     a list of tokens. If metadata=True, it yields a 2-`tuple` with the document number as the second element.
  
  Preprocessing consists of 0+ `character_filters`, a `tokenizer`, and 0+ `token_filters`.
  
  The preprocessing consists of calling each filter in `character_filters` with the document text.
  Unicode is not guaranteed, and if desired, the first filter should convert to unicode.
  The output of each character filter should be another string. The output from the final filter is fed
  to the `tokenizer`, which should split the string into a list of tokens (strings).
  Afterwards, the list of tokens is fed through each filter in `token_filters`. The final output returned from
  :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` is the output from the final token filter.
  
  So to use this class, you can either pass in different preprocessing functions using the
  `character_filters`, `tokenizer`, and `token_filters` arguments, or you can subclass it.
  
  If subclassing: override :meth:`~gensim.corpora.textcorpus.TextCorpus.getstream` to take text from different input
  sources in different formats.
  Override :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` if you must provide different initial
  preprocessing, then call the :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` method to apply
  the normal preprocessing.
  You can also override :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` in order to tag the documents
  (token lists) with different metadata.
  
  The default preprocessing consists of:
  
  #. :func:`~gensim.parsing.preprocessing.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
  #. :func:`~gensim.utils.deaccent`- deaccent (asciifolding)
  #. :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces` - collapse multiple whitespaces into one
  #. :func:`~gensim.utils.simple_tokenize` - tokenize by splitting on whitespace
  #. :func:`~gensim.parsing.preprocessing.remove_short_tokens` - remove words less than 3 characters long
  #. :func:`~gensim.parsing.preprocessing.remove_stopword_tokens` - remove stopwords
class_docstr_tok: ["Helper","class","to","simplify","the","pipeline","of","getting","BoW","vectors","from","plain","text",".","Notes","--","--","-","This","is","an","abstract","base","class",":","override","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","and",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__len__","`","methods","to","match","your","particular","input",".","Given","a","filename","(","or","a","file-like","object",")","in","constructor",",","the","corpus","object","will","be","automatically","initialized","with","a","dictionary","in","`","self.dictionary","`","and","will","support","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__iter__","`","corpus","method",".","You","have","a","few","different","ways","of","utilizing","this","class","via","subclassing","or","by","construction","with","different","preprocessing","arguments",".","The",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__iter__","`","method","converts","the","lists","of","tokens","produced","by",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","to","BoW","format","using",":","meth",":","`","gensim.corpora.dictionary.Dictionary.doc2bow","`",".",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","does","the","following",":","#",".","Calls",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.getstream","`","to","get","a","generator","over","the","texts",".","It","yields","each","document","in","turn","from","the","underlying","text","file","or","files",".","#",".","For","each","document","from","the","stream",",","calls",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","to","produce","a","list","of","tokens",".","If","metadata=True",",","it","yields","a","2-","`","tuple","`","with","the","document","number","as","the","second","element",".","Preprocessing","consists","of","0+","`","character_filters","`",",","a","`","tokenizer","`",",","and","0+","`","token_filters","`",".","The","preprocessing","consists","of","calling","each","filter","in","`","character_filters","`","with","the","document","text",".","Unicode","is","not","guaranteed",",","and","if","desired",",","the","first","filter","should","convert","to","unicode",".","The","output","of","each","character","filter","should","be","another","string",".","The","output","from","the","final","filter","is","fed","to","the","`","tokenizer","`",",","which","should","split","the","string","into","a","list","of","tokens","(","strings",")",".","Afterwards",",","the","list","of","tokens","is","fed","through","each","filter","in","`","token_filters","`",".","The","final","output","returned","from",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","is","the","output","from","the","final","token","filter",".","So","to","use","this","class",",","you","can","either","pass","in","different","preprocessing","functions","using","the","`","character_filters","`",",","`","tokenizer","`",",","and","`","token_filters","`","arguments",",","or","you","can","subclass","it",".","If","subclassing",":","override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.getstream","`","to","take","text","from","different","input","sources","in","different","formats",".","Override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","if","you","must","provide","different","initial","preprocessing",",","then","call","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","method","to","apply","the","normal","preprocessing",".","You","can","also","override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","in","order","to","tag","the","documents","(","token","lists",")","with","different","metadata",".","The","default","preprocessing","consists","of",":","#",".",":","func",":","`","~gensim.parsing.preprocessing.lower_to_unicode","`","-","lowercase","and","convert","to","unicode","(","assumes","utf8","encoding",")","#",".",":","func",":","`","~gensim.utils.deaccent","`","-","deaccent","(","asciifolding",")","#",".",":","func",":","`","~gensim.parsing.preprocessing.strip_multiple_whitespaces","`","-","collapse","multiple","whitespaces","into","one","#",".",":","func",":","`","~gensim.utils.simple_tokenize","`","-","tokenize","by","splitting","on","whitespace","#",".",":","func",":","`","~gensim.parsing.preprocessing.remove_short_tokens","`","-","remove","words","less","than","3","characters","long","#",".",":","func",":","`","~gensim.parsing.preprocessing.remove_stopword_tokens","`","-","remove","stopwords"]
docstr: Parameters
  ----------
  input : str, optional
      Path to top-level directory (file) to traverse for corpus documents.
  dictionary : :class:`~gensim.corpora.dictionary.Dictionary`, optional
      If a dictionary is provided, it will not be updated with the given corpus on initialization.
      If None - new dictionary will be built for the given corpus.
      If `input` is None, the dictionary will remain uninitialized.
  metadata : bool, optional
      If True - yield metadata with each document.
  character_filters : iterable of callable, optional
      Each will be applied to the text of each document in order, and should return a single string with
      the modified text. For Python 2, the original text will not be unicode, so it may be useful to
      convert to unicode as the first character filter.
      If None - using :func:`~gensim.parsing.preprocessing.lower_to_unicode`,
      :func:`~gensim.utils.deaccent` and :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces`.
  tokenizer : callable, optional
      Tokenizer for document, if None - using :func:`~gensim.utils.simple_tokenize`.
  token_filters : iterable of callable, optional
      Each will be applied to the iterable of tokens in order, and should return another iterable of tokens.
      These filters can add, remove, or replace tokens, or do nothing at all.
      If None - using :func:`~gensim.parsing.preprocessing.remove_short_tokens` and
      :func:`~gensim.parsing.preprocessing.remove_stopword_tokens`.
  
  Examples
  --------
  .. sourcecode:: pycon
  
      >>> from gensim.corpora.textcorpus import TextCorpus
      >>> from gensim.test.utils import datapath
      >>> from gensim import utils
      >>>
      >>>
      >>> class CorpusMiislita(TextCorpus):
      ...     stopwords = set('for a of the and to in on'.split())
      ...
      ...     def get_texts(self):
      ...         for doc in self.getstream():
      ...             yield [word for word in utils.to_unicode(doc).lower().split() if word not in self.stopwords]
      ...
      ...     def __len__(self):
      ...         self.length = sum(1 for _ in self.get_texts())
      ...         return self.length
      >>>
      >>>
      >>> corpus = CorpusMiislita(datapath('head500.noblanks.cor.bz2'))
      >>> len(corpus)
      250
      >>> document = next(iter(corpus.get_texts()))
docstr_tok: ["Parameters","--","--","--","--","--","input",":","str",",","optional","Path","to","top-level","directory","(","file",")","to","traverse","for","corpus","documents",".","dictionary",":",":","class",":","`","~gensim.corpora.dictionary.Dictionary","`",",","optional","If","a","dictionary","is","provided",",","it","will","not","be","updated","with","the","given","corpus","on","initialization",".","If","None","-","new","dictionary","will","be","built","for","the","given","corpus",".","If","`","input","`","is","None",",","the","dictionary","will","remain","uninitialized",".","metadata",":","bool",",","optional","If","True","-","yield","metadata","with","each","document",".","character_filters",":","iterable","of","callable",",","optional","Each","will","be","applied","to","the","text","of","each","document","in","order",",","and","should","return","a","single","string","with","the","modified","text",".","For","Python","2",",","the","original","text","will","not","be","unicode",",","so","it","may","be","useful","to","convert","to","unicode","as","the","first","character","filter",".","If","None","-","using",":","func",":","`","~gensim.parsing.preprocessing.lower_to_unicode","`",",",":","func",":","`","~gensim.utils.deaccent","`","and",":","func",":","`","~gensim.parsing.preprocessing.strip_multiple_whitespaces","`",".","tokenizer",":","callable",",","optional","Tokenizer","for","document",",","if","None","-","using",":","func",":","`","~gensim.utils.simple_tokenize","`",".","token_filters",":","iterable","of","callable",",","optional","Each","will","be","applied","to","the","iterable","of","tokens","in","order",",","and","should","return","another","iterable","of","tokens",".","These","filters","can","add",",","remove",",","or","replace","tokens",",","or","do","nothing","at","all",".","If","None","-","using",":","func",":","`","~gensim.parsing.preprocessing.remove_short_tokens","`","and",":","func",":","`","~gensim.parsing.preprocessing.remove_stopword_tokens","`",".","Examples","--","--","--","--","..","sourcecode",":",":","pycon",">",">",">","from","gensim.corpora.textcorpus","import","TextCorpus",">",">",">","from","gensim.test.utils","import","datapath",">",">",">","from","gensim","import","utils",">",">",">",">",">",">",">",">",">","class","CorpusMiislita","(","TextCorpus",")",":","...","stopwords","=","set","(","'for","a","of","the","and","to","in","on'.split","(",")",")","...","...","def","get_texts","(","self",")",":","...","for","doc","in","self.getstream","(",")",":","...","yield","[","word","for","word","in","utils.to_unicode","(","doc",")",".lower","(",")",".split","(",")","if","word","not","in","self.stopwords","]","...","...","def","__len__","(","self",")",":","...","self.length","=","sum","(","1","for","_","in","self.get_texts","(",")",")","...","return","self.length",">",">",">",">",">",">",">",">",">","corpus","=","CorpusMiislita","(","datapath","(","'head500.noblanks.cor.bz2","'",")",")",">",">",">","len","(","corpus",")","250",">",">",">","document","=","next","(","iter","(","corpus.get_texts","(",")",")",")"]
code: def __init__(
      self,
      input=None,
      dictionary=None,
      metadata=False,
      character_filters=None,
      tokenizer=None,
      token_filters=None,
  ):
      """
  
      Parameters
      ----------
      input : str, optional
          Path to top-level directory (file) to traverse for corpus documents.
      dictionary : :class:`~gensim.corpora.dictionary.Dictionary`, optional
          If a dictionary is provided, it will not be updated with the given corpus on initialization.
          If None - new dictionary will be built for the given corpus.
          If `input` is None, the dictionary will remain uninitialized.
      metadata : bool, optional
          If True - yield metadata with each document.
      character_filters : iterable of callable, optional
          Each will be applied to the text of each document in order, and should return a single string with
          the modified text. For Python 2, the original text will not be unicode, so it may be useful to
          convert to unicode as the first character filter.
          If None - using :func:`~gensim.parsing.preprocessing.lower_to_unicode`,
          :func:`~gensim.utils.deaccent` and :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces`.
      tokenizer : callable, optional
          Tokenizer for document, if None - using :func:`~gensim.utils.simple_tokenize`.
      token_filters : iterable of callable, optional
          Each will be applied to the iterable of tokens in order, and should return another iterable of tokens.
          These filters can add, remove, or replace tokens, or do nothing at all.
          If None - using :func:`~gensim.parsing.preprocessing.remove_short_tokens` and
          :func:`~gensim.parsing.preprocessing.remove_stopword_tokens`.
  
      Examples
      --------
      .. sourcecode:: pycon
  
          >>> from gensim.corpora.textcorpus import TextCorpus
          >>> from gensim.test.utils import datapath
          >>> from gensim import utils
          >>>
          >>>
          >>> class CorpusMiislita(TextCorpus):
          ...     stopwords = set('for a of the and to in on'.split())
          ...
          ...     def get_texts(self):
          ...         for doc in self.getstream():
          ...             yield [word for word in utils.to_unicode(doc).lower().split() if word not in self.stopwords]
          ...
          ...     def __len__(self):
          ...         self.length = sum(1 for _ in self.get_texts())
          ...         return self.length
          >>>
          >>>
          >>> corpus = CorpusMiislita(datapath('head500.noblanks.cor.bz2'))
          >>> len(corpus)
          250
          >>> document = next(iter(corpus.get_texts()))
  
      """
      self.input = input
      self.metadata = metadata
  
      self.character_filters = character_filters
      if self.character_filters is None:
          self.character_filters = [
              lower_to_unicode,
              deaccent,
              strip_multiple_whitespaces,
          ]
  
      self.tokenizer = tokenizer
      if self.tokenizer is None:
          self.tokenizer = simple_tokenize
  
      self.token_filters = token_filters
      if self.token_filters is None:
          self.token_filters = [
              remove_short_tokens,
              remove_stopword_tokens,
          ]
  
      self.length = None
      self.dictionary = None
      self.init_dictionary(dictionary)
  
imports: from __future__ import with_statement
  import logging
  import os
  import random
  import re
  import sys
  from gensim import interfaces, utils
  from gensim.corpora.dictionary import Dictionary
  from gensim.parsing.preprocessing import remove_stopword_tokens, remove_short_tokens, lower_to_unicode, strip_multiple_whitespaces
  from gensim.utils import deaccent, simple_tokenize
  from smart_open import open
label_desc: Use image node_id 1 to create a new TextCorpus object from inherited base classes: interfaces with example: obj = TextCorpus(input, dictionary, metadata, character_filters, tokenizer, token_filters)