class_docstr: Helper class to simplify the pipeline of getting BoW vectors from plain text.
  
  Notes
  -----
  This is an abstract base class: override the :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` and
  :meth:`~gensim.corpora.textcorpus.TextCorpus.__len__` methods to match your particular input.
  
  Given a filename (or a file-like object) in constructor, the corpus object will be automatically initialized
  with a dictionary in `self.dictionary` and will support the :meth:`~gensim.corpora.textcorpus.TextCorpus.__iter__`
  corpus method.  You have a few different ways of utilizing this class via subclassing or by construction with
  different preprocessing arguments.
  
  The :meth:`~gensim.corpora.textcorpus.TextCorpus.__iter__` method converts the lists of tokens produced by
  :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` to BoW format using
  :meth:`gensim.corpora.dictionary.Dictionary.doc2bow`.
  
  :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` does the following:
  
  #. Calls :meth:`~gensim.corpora.textcorpus.TextCorpus.getstream` to get a generator over the texts.
     It yields each document in turn from the underlying text file or files.
  #. For each document from the stream, calls :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` to produce
     a list of tokens. If metadata=True, it yields a 2-`tuple` with the document number as the second element.
  
  Preprocessing consists of 0+ `character_filters`, a `tokenizer`, and 0+ `token_filters`.
  
  The preprocessing consists of calling each filter in `character_filters` with the document text.
  Unicode is not guaranteed, and if desired, the first filter should convert to unicode.
  The output of each character filter should be another string. The output from the final filter is fed
  to the `tokenizer`, which should split the string into a list of tokens (strings).
  Afterwards, the list of tokens is fed through each filter in `token_filters`. The final output returned from
  :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` is the output from the final token filter.
  
  So to use this class, you can either pass in different preprocessing functions using the
  `character_filters`, `tokenizer`, and `token_filters` arguments, or you can subclass it.
  
  If subclassing: override :meth:`~gensim.corpora.textcorpus.TextCorpus.getstream` to take text from different input
  sources in different formats.
  Override :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` if you must provide different initial
  preprocessing, then call the :meth:`~gensim.corpora.textcorpus.TextCorpus.preprocess_text` method to apply
  the normal preprocessing.
  You can also override :meth:`~gensim.corpora.textcorpus.TextCorpus.get_texts` in order to tag the documents
  (token lists) with different metadata.
  
  The default preprocessing consists of:
  
  #. :func:`~gensim.parsing.preprocessing.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
  #. :func:`~gensim.utils.deaccent`- deaccent (asciifolding)
  #. :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces` - collapse multiple whitespaces into one
  #. :func:`~gensim.utils.simple_tokenize` - tokenize by splitting on whitespace
  #. :func:`~gensim.parsing.preprocessing.remove_short_tokens` - remove words less than 3 characters long
  #. :func:`~gensim.parsing.preprocessing.remove_stopword_tokens` - remove stopwords
class_docstr_tok: ["Helper","class","to","simplify","the","pipeline","of","getting","BoW","vectors","from","plain","text",".","Notes","--","--","-","This","is","an","abstract","base","class",":","override","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","and",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__len__","`","methods","to","match","your","particular","input",".","Given","a","filename","(","or","a","file-like","object",")","in","constructor",",","the","corpus","object","will","be","automatically","initialized","with","a","dictionary","in","`","self.dictionary","`","and","will","support","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__iter__","`","corpus","method",".","You","have","a","few","different","ways","of","utilizing","this","class","via","subclassing","or","by","construction","with","different","preprocessing","arguments",".","The",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.__iter__","`","method","converts","the","lists","of","tokens","produced","by",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","to","BoW","format","using",":","meth",":","`","gensim.corpora.dictionary.Dictionary.doc2bow","`",".",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","does","the","following",":","#",".","Calls",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.getstream","`","to","get","a","generator","over","the","texts",".","It","yields","each","document","in","turn","from","the","underlying","text","file","or","files",".","#",".","For","each","document","from","the","stream",",","calls",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","to","produce","a","list","of","tokens",".","If","metadata=True",",","it","yields","a","2-","`","tuple","`","with","the","document","number","as","the","second","element",".","Preprocessing","consists","of","0+","`","character_filters","`",",","a","`","tokenizer","`",",","and","0+","`","token_filters","`",".","The","preprocessing","consists","of","calling","each","filter","in","`","character_filters","`","with","the","document","text",".","Unicode","is","not","guaranteed",",","and","if","desired",",","the","first","filter","should","convert","to","unicode",".","The","output","of","each","character","filter","should","be","another","string",".","The","output","from","the","final","filter","is","fed","to","the","`","tokenizer","`",",","which","should","split","the","string","into","a","list","of","tokens","(","strings",")",".","Afterwards",",","the","list","of","tokens","is","fed","through","each","filter","in","`","token_filters","`",".","The","final","output","returned","from",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","is","the","output","from","the","final","token","filter",".","So","to","use","this","class",",","you","can","either","pass","in","different","preprocessing","functions","using","the","`","character_filters","`",",","`","tokenizer","`",",","and","`","token_filters","`","arguments",",","or","you","can","subclass","it",".","If","subclassing",":","override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.getstream","`","to","take","text","from","different","input","sources","in","different","formats",".","Override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","if","you","must","provide","different","initial","preprocessing",",","then","call","the",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.preprocess_text","`","method","to","apply","the","normal","preprocessing",".","You","can","also","override",":","meth",":","`","~gensim.corpora.textcorpus.TextCorpus.get_texts","`","in","order","to","tag","the","documents","(","token","lists",")","with","different","metadata",".","The","default","preprocessing","consists","of",":","#",".",":","func",":","`","~gensim.parsing.preprocessing.lower_to_unicode","`","-","lowercase","and","convert","to","unicode","(","assumes","utf8","encoding",")","#",".",":","func",":","`","~gensim.utils.deaccent","`","-","deaccent","(","asciifolding",")","#",".",":","func",":","`","~gensim.parsing.preprocessing.strip_multiple_whitespaces","`","-","collapse","multiple","whitespaces","into","one","#",".",":","func",":","`","~gensim.utils.simple_tokenize","`","-","tokenize","by","splitting","on","whitespace","#",".",":","func",":","`","~gensim.parsing.preprocessing.remove_short_tokens","`","-","remove","words","less","than","3","characters","long","#",".",":","func",":","`","~gensim.parsing.preprocessing.remove_stopword_tokens","`","-","remove","stopwords"]
docstr: Apply preprocessor one by one and generate result.
  
  Warnings
  --------
  This is useful for debugging issues with the corpus preprocessing pipeline.
  
  Parameters
  ----------
  text : str
      Document text read from plain-text file.
  
  Yields
  ------
  (callable, object)
      Pre-processor, output from pre-processor (based on `text`)
docstr_tok: ["Apply","preprocessor","one","by","one","and","generate","result",".","Warnings","--","--","--","--","This","is","useful","for","debugging","issues","with","the","corpus","preprocessing","pipeline",".","Parameters","--","--","--","--","--","text",":","str","Document","text","read","from","plain-text","file",".","Yields","--","--","--","(","callable",",","object",")","Pre-processor",",","output","from","pre-processor","(","based","on","`","text","`",")"]
code: def step_through_preprocess(self, text):
      """Apply preprocessor one by one and generate result.
  
      Warnings
      --------
      This is useful for debugging issues with the corpus preprocessing pipeline.
  
      Parameters
      ----------
      text : str
          Document text read from plain-text file.
  
      Yields
      ------
      (callable, object)
          Pre-processor, output from pre-processor (based on `text`)
  
      """
      for character_filter in self.character_filters:
          text = character_filter(text)
          yield (character_filter, text)
  
      tokens = self.tokenizer(text)
      yield (self.tokenizer, tokens)
  
      for token_filter in self.token_filters:
          yield (token_filter, token_filter(tokens))
  
imports: from __future__ import with_statement
  import logging
  import os
  import random
  import re
  import sys
  from gensim import interfaces, utils
  from gensim.corpora.dictionary import Dictionary
  from gensim.parsing.preprocessing import remove_stopword_tokens, remove_short_tokens, lower_to_unicode, strip_multiple_whitespaces
  from gensim.utils import deaccent, simple_tokenize
  from smart_open import open
label_desc: Use image node_id 6 for calling the TextCorpus obj's underlying member method code with example usage: obj.step_through_preprocess(text) without return types