def generate_tokens(self):
if self._io_error:<EOL><INDENT>self.report_error(<NUM_LIT:1>, <NUM_LIT:0>, '<STR_LIT>' % self._io_error, readlines)<EOL><DEDENT>tokengen = tokenize.generate_tokens(self.readline)<EOL>try:<EOL><INDENT>for token in tokengen:<EOL><INDENT>if token[<NUM_LIT:2>][<NUM_LIT:0>] > self.total_lines:<EOL><INDENT>return<EOL><DEDENT>self.maybe_check_physical(token)<EOL>yield token<EOL><DEDENT><DEDENT>except (SyntaxError, tokenize.TokenError):<EOL><INDENT>self.report_invalid_syntax()<EOL><DEDENT>
Tokenize the file, run physical line checks and yield tokens.