class_docstr: Construct an FAIRSEQ Transformer tokenizer. Based on Byte-Pair Encoding. The tokenization process is the following:
  
  - Moses preprocessing and tokenization.
  - Normalizing all inputs text.
  - The arguments `special_tokens` and the function `set_special_tokens`, can be used to add additional symbols (like
    "__classify__") to a vocabulary.
  - The argument `langs` defines a pair of languages.
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      langs (`List[str]`, *optional*):
          A list of two languages to translate from and to, for instance `["en", "ru"]`.
      src_vocab_file (`str`, *optional*):
          File containing the vocabulary for the source language.
      tgt_vocab_file (`st`, *optional*):
          File containing the vocabulary for the target language.
      merges_file (`str`, *optional*):
          File containing the merges.
      do_lower_case (`bool`, *optional*, defaults to `False`):
          Whether or not to lowercase the input when tokenizing.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      bos_token (`str`, *optional*, defaults to `"<s>"`):
          The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
  
          <Tip>
  
          When building a sequence using special tokens, this is not the token that is used for the beginning of
          sequence. The token used is the `cls_token`.
  
          </Tip>
  
      sep_token (`str`, *optional*, defaults to `"</s>"`):
          The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
          sequence classification or for a text and a question for question answering. It is also used as the last
          token of a sequence built with special tokens.
      pad_token (`str`, *optional*, defaults to `"<pad>"`):
          The token used for padding, for example when batching sequences of different lengths.
class_docstr_tok: ["Construct","an","FAIRSEQ","Transformer","tokenizer",".","Based","on","Byte-Pair","Encoding",".","The","tokenization","process","is","the","following",":","-","Moses","preprocessing","and","tokenization",".","-","Normalizing","all","inputs","text",".","-","The","arguments","`","special_tokens","`","and","the","function","`","set_special_tokens","`",",","can","be","used","to","add","additional","symbols","(","like","``","__classify__","''",")","to","a","vocabulary",".","-","The","argument","`","langs","`","defines","a","pair","of","languages",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","langs","(","`","List","[","str","]","`",",","*","optional","*",")",":","A","list","of","two","languages","to","translate","from","and","to",",","for","instance","`","[","``","en","''",",","``","ru","''","]","`",".","src_vocab_file","(","`","str","`",",","*","optional","*",")",":","File","containing","the","vocabulary","for","the","source","language",".","tgt_vocab_file","(","`","st","`",",","*","optional","*",")",":","File","containing","the","vocabulary","for","the","target","language",".","merges_file","(","`","str","`",",","*","optional","*",")",":","File","containing","the","merges",".","do_lower_case","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","lowercase","the","input","when","tokenizing",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","bos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","s",">","''","`",")",":","The","beginning","of","sequence","token","that","was","used","during","pretraining",".","Can","be","used","a","sequence","classifier","token",".","<","Tip",">","When","building","a","sequence","using","special","tokens",",","this","is","not","the","token","that","is","used","for","the","beginning","of","sequence",".","The","token","used","is","the","`","cls_token","`",".","<","\/Tip",">","sep_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","\/s",">","''","`",")",":","The","separator","token",",","which","is","used","when","building","a","sequence","from","multiple","sequences",",","e.g",".","two","sequences","for","sequence","classification","or","for","a","text","and","a","question","for","question","answering",".","It","is","also","used","as","the","last","token","of","a","sequence","built","with","special","tokens",".","pad_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","pad",">","''","`",")",":","The","token","used","for","padding",",","for","example","when","batching","sequences","of","different","lengths","."]
code: def moses_tokenize(self, text, lang):
      if lang not in self.cache_moses_tokenizer:
          moses_tokenizer = self.sm.MosesTokenizer(lang=lang)
          self.cache_moses_tokenizer[lang] = moses_tokenizer
      return self.cache_moses_tokenizer[lang].tokenize(
          text,
          aggressive_dash_splits=True,
          return_str=False,
          escape=True,
      )
  
imports: import json
  import os
  import re
  import unicodedata
  from typing import Dict, List, Optional, Tuple
  from ...tokenization_utils import PreTrainedTokenizer
  from ...utils import logging
label_desc: Use image node_id 5 for calling the FSMTTokenizer obj's underlying member method code with example usage: obj.moses_tokenize(text, lang) and returns: self