class_docstr: Represents a "token" (usually a word) extracted from the source text being
  indexed.
  
  See "Advanced analysis" in the user guide for more information.
  
  Because object instantiation in Python is slow, tokenizers should create
  ONE SINGLE Token object and YIELD IT OVER AND OVER, changing the attributes
  each time.
  
  This trick means that consumers of tokens (i.e. filters) must never try to
  hold onto the token object between loop iterations, or convert the token
  generator into a list. Instead, save the attributes between iterations,
  not the object::
  
      def RemoveDuplicatesFilter(self, stream):
          # Removes duplicate words.
          lasttext = None
          for token in stream:
              # Only yield the token if its text doesn't
              # match the previous token.
              if lasttext != token.text:
                  yield token
              lasttext = token.text
  
  ...or, call token.copy() to get a copy of the token object.
class_docstr_tok: ["Represents","a","``","token","''","(","usually","a","word",")","extracted","from","the","source","text","being","indexed",".","See","``","Advanced","analysis","''","in","the","user","guide","for","more","information",".","Because","object","instantiation","in","Python","is","slow",",","tokenizers","should","create","ONE","SINGLE","Token","object","and","YIELD","IT","OVER","AND","OVER",",","changing","the","attributes","each","time",".","This","trick","means","that","consumers","of","tokens","(","i.e",".","filters",")","must","never","try","to","hold","onto","the","token","object","between","loop","iterations",",","or","convert","the","token","generator","into","a","list",".","Instead",",","save","the","attributes","between","iterations",",","not","the","object",":",":","def","RemoveDuplicatesFilter","(","self",",","stream",")",":","#","Removes","duplicate","words",".","lasttext","=","None","for","token","in","stream",":","#","Only","yield","the","token","if","its","text","doesn't","#","match","the","previous","token",".","if","lasttext","!","=","token.text",":","yield","token","lasttext","=","token.text","...","or",",","call","token.copy","(",")","to","get","a","copy","of","the","token","object","."]
code: def copy(self):
      # This is faster than using the copy module
      return Token(**self.__dict__)
  
file_path: whoosh/src/whoosh/analysis/acore.py
imports: from whoosh.compat import iteritems
label_desc: Use image node_id 3 for calling the Token obj's underlying member method code with example usage: obj.copy() and returns: Token