class_docstr: A Scanner is used to read tokens from a stream of characters
  using the token set specified by a Plex.Lexicon.
  
  Constructor:
  
    Scanner(lexicon, stream, name = '')
  
      See the docstring of the __init__ method for details.
  
  Methods:
  
    See the docstrings of the individual methods for more
    information.
  
    read() --> (value, text)
      Reads the next lexical token from the stream.
  
    position() --> (name, line, col)
      Returns the position of the last token read using the
      read() method.
  
    begin(state_name)
      Causes scanner to change state.
  
    produce(value [, text])
      Causes return of a token value to the caller of the
      Scanner.
docstr: Read the next lexical token from the stream and return a
  tuple (value, text), where |value| is the value associated with
  the token as specified by the Lexicon, and |text| is the actual
  string read from the stream. Returns (None, '') on end of file.
docstr_tok: ["Read","the","next","lexical","token","from","the","stream","and","return","a","tuple","(","value",",","text",")",",","where","|value|","is","the","value","associated","with","the","token","as","specified","by","the","Lexicon",",","and","|text|","is","the","actual","string","read","from","the","stream",".","Returns","(","None",",","``",")","on","end","of","file","."]
code: def read(self):
      """
      Read the next lexical token from the stream and return a
      tuple (value, text), where |value| is the value associated with
      the token as specified by the Lexicon, and |text| is the actual
      string read from the stream. Returns (None, '') on end of file.
      """
      queue = self.queue
      while not queue:
          self.text, action = self.scan_a_token()
          if action is None:
              self.produce(None)
              self.eof()
          else:
              value = action.perform(self, self.text)
              if value is not None:
                  self.produce(value)
      result = queue[0]
      del queue[0]
      return result
  
file_path: catboost/contrib/tools/cython/Cython/Plex/Scanners.py
imports: from __future__ import absolute_import
  import cython
  from .None import Errors
  from .Regexps import BOL, EOL, EOF
label_desc: Use image node_id 2 for calling the Scanner obj's underlying member method code with example usage: obj.read() and returns: result