class_docstr: An incremental decoder that extends Python's built-in UTF-8 decoder.
  
  This encoder needs to take in bytes, possibly arriving in a stream, and
  output the correctly decoded text. The general strategy for doing this
  is to fall back on the real UTF-8 decoder whenever possible, because
  the real UTF-8 decoder is way optimized, but to call specialized methods
  we define here for the cases the real encoder isn't expecting.
class_docstr_tok: ["An","incremental","decoder","that","extends","Python","'s","built-in","UTF-8","decoder",".","This","encoder","needs","to","take","in","bytes",",","possibly","arriving","in","a","stream",",","and","output","the","correctly","decoded","text",".","The","general","strategy","for","doing","this","is","to","fall","back","on","the","real","UTF-8","decoder","whenever","possible",",","because","the","real","UTF-8","decoder","is","way","optimized",",","but","to","call","specialized","methods","we","define","here","for","the","cases","the","real","encoder","is","n't","expecting","."]
docstr: When we have improperly encoded surrogates, we can still see the
  bits that they were meant to represent.
  
  The surrogates were meant to encode a 20-bit number, to which we
  add 0x10000 to get a codepoint. That 20-bit number now appears in
  this form:
  
    11101101 1010abcd 10efghij 11101101 1011klmn 10opqrst
  
  The CESU8_RE above matches byte sequences of this form. Then we need
  to extract the bits and assemble a codepoint number from them.
docstr_tok: ["When","we","have","improperly","encoded","surrogates",",","we","can","still","see","the","bits","that","they","were","meant","to","represent",".","The","surrogates","were","meant","to","encode","a","20-bit","number",",","to","which","we","add","0x10000","to","get","a","codepoint",".","That","20-bit","number","now","appears","in","this","form",":","11101101","1010abcd","10efghij","11101101","1011klmn","10opqrst","The","CESU8_RE","above","matches","byte","sequences","of","this","form",".","Then","we","need","to","extract","the","bits","and","assemble","a","codepoint","number","from","them","."]
code: def _buffer_decode_surrogates(sup, input, errors, final):
      """
      When we have improperly encoded surrogates, we can still see the
      bits that they were meant to represent.
  
      The surrogates were meant to encode a 20-bit number, to which we
      add 0x10000 to get a codepoint. That 20-bit number now appears in
      this form:
  
        11101101 1010abcd 10efghij 11101101 1011klmn 10opqrst
  
      The CESU8_RE above matches byte sequences of this form. Then we need
      to extract the bits and assemble a codepoint number from them.
      """
      if len(input) < 6:
          if final:
              # We found 0xed near the end of the stream, and there aren't
              # six bytes to decode. Delegate to the superclass method to
              # handle it as normal UTF-8. It might be a Hangul character
              # or an error.
              return sup(input, errors, final)
          else:
              # We found a surrogate, the stream isn't over yet, and we don't
              # know enough of the following bytes to decode anything, so
              # consume zero bytes and wait.
              return "", 0
      else:
          if CESU8_RE.match(input):
              # Given this is a CESU-8 sequence, do some math to pull out
              # the intended 20-bit value, and consume six bytes.
              codepoint = (
                  ((input[1] & 0x0F) << 16)
                  + ((input[2] & 0x3F) << 10)
                  + ((input[4] & 0x0F) << 6)
                  + (input[5] & 0x3F)
                  + 0x10000
              )
              return chr(codepoint), 6
          else:
              # This looked like a CESU-8 sequence, but it wasn't one.
              # 0xed indicates the start of a three-byte sequence, so give
              # three bytes to the superclass to decode as usual.
              return sup(input[:3], errors, False)
  
file_path: python-ftfy/ftfy/bad_codecs/utf8_variants.py
imports: import re
  import codecs
  from typing import Tuple
  from encodings.utf_8 import IncrementalDecoder, IncrementalEncoder
label_desc: Use image node_id 3 for calling the IncrementalDecoder obj's underlying member method code with example usage: obj._buffer_decode_surrogates(sup, input, errors, final) and returns: sup, str,  int, chr,  int, sup