class_docstr: Version of parlai.utils.bpe.Gpt2BpeHelper that can be TorchScripted.
class_docstr_tok: ["Version","of","parlai.utils.bpe.Gpt2BpeHelper","that","can","be","TorchScripted","."]
docstr: An implementation of UTF8 character iteration in TorchScript. There are no
  bitwise operations in torchscript, so we compare directly to integer values.
  There isn't a lot of validation, for instance if you pass in an improperly
  encoded string with an out-of-place continuation byte, or with a non-left-to-
  right byte order, you'll get unexpected results and likely throw. Torch itself
  takes in unicode strings and encodes them as UTF8, so that should be actively
  hard to do.
  
  The logic is simple: looking at the current start-of-character byte.
  If its high bit is 0, it's a 1-byte character. Otherwise, the number of
  bytes is the number of leading 1s in its binary representation, so
  find that number by comparing it directly to ints with the appropriate
  representation, then append that many bytes as a character and move past
  them to the next start byte.
  
  From pytext.torchscript.utils.
docstr_tok: ["An","implementation","of","UTF8","character","iteration","in","TorchScript",".","There","are","no","bitwise","operations","in","torchscript",",","so","we","compare","directly","to","integer","values",".","There","is","n't","a","lot","of","validation",",","for","instance","if","you","pass","in","an","improperly","encoded","string","with","an","out-of-place","continuation","byte",",","or","with","a","non-left-to-","right","byte","order",",","you","'ll","get","unexpected","results","and","likely","throw",".","Torch","itself","takes","in","unicode","strings","and","encodes","them","as","UTF8",",","so","that","should","be","actively","hard","to","do",".","The","logic","is","simple",":","looking","at","the","current","start-of-character","byte",".","If","its","high","bit","is","0",",","it","'s","a","1-byte","character",".","Otherwise",",","the","number","of","bytes","is","the","number","of","leading","1s","in","its","binary","representation",",","so","find","that","number","by","comparing","it","directly","to","ints","with","the","appropriate","representation",",","then","append","that","many","bytes","as","a","character","and","move","past","them","to","the","next","start","byte",".","From","pytext.torchscript.utils","."]
code: def utf8_chars(self, s: str) -> List[str]:
      """
      An implementation of UTF8 character iteration in TorchScript. There are no
      bitwise operations in torchscript, so we compare directly to integer values.
      There isn't a lot of validation, for instance if you pass in an improperly
      encoded string with an out-of-place continuation byte, or with a non-left-to-
      right byte order, you'll get unexpected results and likely throw. Torch itself
      takes in unicode strings and encodes them as UTF8, so that should be actively
      hard to do.
  
      The logic is simple: looking at the current start-of-character byte.
      If its high bit is 0, it's a 1-byte character. Otherwise, the number of
      bytes is the number of leading 1s in its binary representation, so
      find that number by comparing it directly to ints with the appropriate
      representation, then append that many bytes as a character and move past
      them to the next start byte.
  
      From pytext.torchscript.utils.
      """
      chars: List[str] = []
      i = 0
      while i < len(s):
          byte = ord(s[i])
          if byte < 0b10000000:
              chars.append(s[i])
              i += 1
          else:
              if byte < 0b11100000:
                  num_bytes = 2
              elif byte < 0b11110000:
                  num_bytes = 3
              elif byte < 0b11111000:
                  num_bytes = 4
              elif byte < 0b11111100:
                  num_bytes = 5
              elif byte < 0b11111110:
                  num_bytes = 6
              elif byte < 0b11111111:
                  num_bytes = 7
              else:
                  num_bytes = 8
              chars.append(s[i : i + num_bytes])
              i += num_bytes
      return chars
  
imports: from typing import Dict, List, Tuple
  import torch.jit
label_desc: Use image node_id 9 for calling the ScriptableGpt2BpeHelper obj's underlying member method code with example usage: obj.utf8_chars(s) and returns: chars