class_docstr: Byte-pair encoding implementation in TorchScript.
  
  vocab_file should be a file-like object separated by newlines, where each line
  consists of a word and a count separated by whitespace. Words in the vocab
  therefore can't contain space (according to python regex \s). The vocab file
  should be sorted according to the importance of each token, and they will be
  merged in this priority; the actual score values are irrelevant.
  
  eow_token should be a string that is appended to the last character and token,
  and that token is used at each step in the process and returned at the end.
  You should set this to be consistent with the EOW signature used however you
  generated your ScriptBPE vocab file.
  
  >>> import io
  >>> vocab_file = io.StringIO('''
  hello_EOW 20
  world_EOW 18
  th  17
  is_EOW 16
  bpe_EOW 15
  ! 14
  h 13
  t 6
  s_EOW 2
  i -1
  ii -2
  ''')
  >>> bpe = ScriptBPE.from_vocab_file(vocab_file)
  >>> bpe.tokenize(["hello", "world", "this", "is", "bpe"])
  ["hello_EOW", "world_EOW", "th", "is_EOW", "is_EOW", "bpe_EOW"]
  >>> bpe.tokenize(["iiiis"])
  ["ii", "i", "is_EOW"]
class_docstr_tok: ["Byte-pair","encoding","implementation","in","TorchScript",".","vocab_file","should","be","a","file-like","object","separated","by","newlines",",","where","each","line","consists","of","a","word","and","a","count","separated","by","whitespace",".","Words","in","the","vocab","therefore","ca","n't","contain","space","(","according","to","python","regex","\\s",")",".","The","vocab","file","should","be","sorted","according","to","the","importance","of","each","token",",","and","they","will","be","merged","in","this","priority",";","the","actual","score","values","are","irrelevant",".","eow_token","should","be","a","string","that","is","appended","to","the","last","character","and","token",",","and","that","token","is","used","at","each","step","in","the","process","and","returned","at","the","end",".","You","should","set","this","to","be","consistent","with","the","EOW","signature","used","however","you","generated","your","ScriptBPE","vocab","file",".",">",">",">","import","io",">",">",">","vocab_file","=","io.StringIO","(","``","'","hello_EOW","20","world_EOW","18","th","17","is_EOW","16","bpe_EOW","15","!","14","h","13","t","6","s_EOW","2","i","-1","ii","-2","''","'",")",">",">",">","bpe","=","ScriptBPE.from_vocab_file","(","vocab_file",")",">",">",">","bpe.tokenize","(","[","``","hello","''",",","``","world","''",",","``","this","''",",","``","is","''",",","``","bpe","''","]",")","[","``","hello_EOW","''",",","``","world_EOW","''",",","``","th","''",",","``","is_EOW","''",",","``","is_EOW","''",",","``","bpe_EOW","''","]",">",">",">","bpe.tokenize","(","[","``","iiiis","''","]",")","[","``","ii","''",",","``","i","''",",","``","is_EOW","''","]"]
docstr: vocab is a dictionary from BPE segments, including any EOW elements,
  to their priority in joining. Priority must be an integer, should not be
  negative, and should not contain ties. In the case of negative priorities,
  segments with negative priorities will be ignored. In the case of ties,
  ties will be broken according to left-to-right byte order precedence, but
  this behavior isn't guaranteed and may change in the future.
  
  eow should be a string which corresponds to the EOW used in the vocab
  dictionary.
docstr_tok: ["vocab","is","a","dictionary","from","BPE","segments",",","including","any","EOW","elements",",","to","their","priority","in","joining",".","Priority","must","be","an","integer",",","should","not","be","negative",",","and","should","not","contain","ties",".","In","the","case","of","negative","priorities",",","segments","with","negative","priorities","will","be","ignored",".","In","the","case","of","ties",",","ties","will","be","broken","according","to","left-to-right","byte","order","precedence",",","but","this","behavior","is","n't","guaranteed","and","may","change","in","the","future",".","eow","should","be","a","string","which","corresponds","to","the","EOW","used","in","the","vocab","dictionary","."]
code: def __init__(self, vocab: Dict[str, int], eow: str = "_EOW"):
      """vocab is a dictionary from BPE segments, including any EOW elements,
      to their priority in joining. Priority must be an integer, should not be
      negative, and should not contain ties. In the case of negative priorities,
      segments with negative priorities will be ignored. In the case of ties,
      ties will be broken according to left-to-right byte order precedence, but
      this behavior isn't guaranteed and may change in the future.
  
      eow should be a string which corresponds to the EOW used in the vocab
      dictionary."""
      super().__init__()
      self.vocab = torch.jit.Attribute(vocab, Dict[str, int])
      self.eow = torch.jit.Attribute(eow, str)
  
file_path: pytext/pytext/torchscript/tokenizer/bpe.py
imports: import io
  from typing import Dict, List
  import torch
  from pytext.torchscript.utils import utf8_chars
  from pytext.utils.file_io import PathManager
label_desc: Use image node_id 1 to create a new ScriptBPE object from inherited base classes: torch.jit with example: obj = ScriptBPE(vocab, eow)