class_docstr: Represents a "token" (usually a word) extracted from the source text being
  indexed.
  
  See "Advanced analysis" in the user guide for more information.
  
  Because object instantiation in Python is slow, tokenizers should create
  ONE SINGLE Token object and YIELD IT OVER AND OVER, changing the attributes
  each time.
  
  This trick means that consumers of tokens (i.e. filters) must never try to
  hold onto the token object between loop iterations, or convert the token
  generator into a list. Instead, save the attributes between iterations,
  not the object::
  
      def RemoveDuplicatesFilter(self, stream):
          # Removes duplicate words.
          lasttext = None
          for token in stream:
              # Only yield the token if its text doesn't
              # match the previous token.
              if lasttext != token.text:
                  yield token
              lasttext = token.text
  
  ...or, call token.copy() to get a copy of the token object.
class_docstr_tok: ["Represents","a","``","token","''","(","usually","a","word",")","extracted","from","the","source","text","being","indexed",".","See","``","Advanced","analysis","''","in","the","user","guide","for","more","information",".","Because","object","instantiation","in","Python","is","slow",",","tokenizers","should","create","ONE","SINGLE","Token","object","and","YIELD","IT","OVER","AND","OVER",",","changing","the","attributes","each","time",".","This","trick","means","that","consumers","of","tokens","(","i.e",".","filters",")","must","never","try","to","hold","onto","the","token","object","between","loop","iterations",",","or","convert","the","token","generator","into","a","list",".","Instead",",","save","the","attributes","between","iterations",",","not","the","object",":",":","def","RemoveDuplicatesFilter","(","self",",","stream",")",":","#","Removes","duplicate","words",".","lasttext","=","None","for","token","in","stream",":","#","Only","yield","the","token","if","its","text","doesn't","#","match","the","previous","token",".","if","lasttext","!","=","token.text",":","yield","token","lasttext","=","token.text","...","or",",","call","token.copy","(",")","to","get","a","copy","of","the","token","object","."]
docstr: :param positions: Whether tokens should have the token position in the
      'pos' attribute.
  :param chars: Whether tokens should have character offsets in the
      'startchar' and 'endchar' attributes.
  :param removestops: whether to remove stop words from the stream (if
      the tokens pass through a stop filter).
  :param mode: contains a string describing the purpose for which the
      analyzer is being called, i.e. 'index' or 'query'.
docstr_tok: [":","param","positions",":","Whether","tokens","should","have","the","token","position","in","the","'pos","'","attribute",".",":","param","chars",":","Whether","tokens","should","have","character","offsets","in","the","'startchar","'","and","'endchar","'","attributes",".",":","param","removestops",":","whether","to","remove","stop","words","from","the","stream","(","if","the","tokens","pass","through","a","stop","filter",")",".",":","param","mode",":","contains","a","string","describing","the","purpose","for","which","the","analyzer","is","being","called",",","i.e",".","'index","'","or","'query","'","."]
code: def __init__(
      self,
      positions=False,
      chars=False,
      removestops=True,
      mode="",
      **kwargs,
  ):
      """
      :param positions: Whether tokens should have the token position in the
          'pos' attribute.
      :param chars: Whether tokens should have character offsets in the
          'startchar' and 'endchar' attributes.
      :param removestops: whether to remove stop words from the stream (if
          the tokens pass through a stop filter).
      :param mode: contains a string describing the purpose for which the
          analyzer is being called, i.e. 'index' or 'query'.
      """
  
      self.positions = positions
      self.chars = chars
      self.stopped = False
      self.boost = 1.0
      self.removestops = removestops
      self.mode = mode
      self.__dict__.update(kwargs)
  
file_path: whoosh/src/whoosh/analysis/acore.py
imports: from whoosh.compat import iteritems
label_desc: Use image node_id 1 to create a new Token object from inherited base classes: object with example: obj = Token(positions, chars, removestops, mode)