class_docstr: Construct a Transformer-XL tokenizer adapted from Vocab class in [the original
  code](https://github.com/kimiyoung/transformer-xl). The Transformer-XL tokenizer is a word-level tokenizer (no
  sub-word tokenization).
  
  This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to
  this superclass for more information regarding those methods.
  
  Args:
      special (`List[str]`, *optional*):
          A list of special tokens (to be treated by the original implementation of this tokenizer).
      min_freq (`int`, *optional*, defaults to 0):
          The minimum number of times a token has to be present in order to be kept in the vocabulary (otherwise it
          will be mapped to `unk_token`).
      max_size (`int`, *optional*):
          The maximum size of the vocabulary. If left unset, it will default to the size of the vocabulary found
          after excluding the tokens according to the `min_freq` rule.
      lower_case (`bool`, *optional*, defaults to `False`):
          Whether or not to lowercase the input when tokenizing.
      delimiter (`str`, *optional*):
          The delimiter used between tokens.
      vocab_file (`str`, *optional*):
          File containing the vocabulary (from the original implementation).
      pretrained_vocab_file (`str`, *optional*):
          File containing the vocabulary as saved with the `save_pretrained()` method.
      never_split (`List[str]`, *optional*):
          List of tokens that should never be split. If no list is specified, will simply use the existing special
          tokens.
      unk_token (`str`, *optional*, defaults to `"<unk>"`):
          The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
          token instead.
      eos_token (`str`, *optional*, defaults to `"<eos>"`):
          The end of sequence token.
      additional_special_tokens (`List[str]`, *optional*, defaults to `['<formula>']`):
          A list of additional special tokens (for the HuggingFace functionality).
      language (`str`, *optional*, defaults to `"en"`):
          The language of this tokenizer (used for mose preprocessing).
class_docstr_tok: ["Construct","a","Transformer-XL","tokenizer","adapted","from","Vocab","class","in","[","the","original","code","]","(","https",":","\/\/github.com\/kimiyoung\/transformer-xl",")",".","The","Transformer-XL","tokenizer","is","a","word-level","tokenizer","(","no","sub-word","tokenization",")",".","This","tokenizer","inherits","from","[","`","PreTrainedTokenizer","`","]","which","contains","most","of","the","main","methods",".","Users","should","refer","to","this","superclass","for","more","information","regarding","those","methods",".","Args",":","special","(","`","List","[","str","]","`",",","*","optional","*",")",":","A","list","of","special","tokens","(","to","be","treated","by","the","original","implementation","of","this","tokenizer",")",".","min_freq","(","`","int","`",",","*","optional","*",",","defaults","to","0",")",":","The","minimum","number","of","times","a","token","has","to","be","present","in","order","to","be","kept","in","the","vocabulary","(","otherwise","it","will","be","mapped","to","`","unk_token","`",")",".","max_size","(","`","int","`",",","*","optional","*",")",":","The","maximum","size","of","the","vocabulary",".","If","left","unset",",","it","will","default","to","the","size","of","the","vocabulary","found","after","excluding","the","tokens","according","to","the","`","min_freq","`","rule",".","lower_case","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","or","not","to","lowercase","the","input","when","tokenizing",".","delimiter","(","`","str","`",",","*","optional","*",")",":","The","delimiter","used","between","tokens",".","vocab_file","(","`","str","`",",","*","optional","*",")",":","File","containing","the","vocabulary","(","from","the","original","implementation",")",".","pretrained_vocab_file","(","`","str","`",",","*","optional","*",")",":","File","containing","the","vocabulary","as","saved","with","the","`","save_pretrained","(",")","`","method",".","never_split","(","`","List","[","str","]","`",",","*","optional","*",")",":","List","of","tokens","that","should","never","be","split",".","If","no","list","is","specified",",","will","simply","use","the","existing","special","tokens",".","unk_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","unk",">","''","`",")",":","The","unknown","token",".","A","token","that","is","not","in","the","vocabulary","can","not","be","converted","to","an","ID","and","is","set","to","be","this","token","instead",".","eos_token","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","<","eos",">","''","`",")",":","The","end","of","sequence","token",".","additional_special_tokens","(","`","List","[","str","]","`",",","*","optional","*",",","defaults","to","`","[","'","<","formula",">","'","]","`",")",":","A","list","of","additional","special","tokens","(","for","the","HuggingFace","functionality",")",".","language","(","`","str","`",",","*","optional","*",",","defaults","to","`","``","en","''","`",")",":","The","language","of","this","tokenizer","(","used","for","mose","preprocessing",")","."]
docstr: sents : a list of sentences, each a list of tokenized symbols
code: def count_sents(self, sents, verbose=False):
      """
      sents : a list of sentences, each a list of tokenized symbols
      """
      if verbose:
          logger.info(f"counting {len(sents)} sents ...")
      for idx, symbols in enumerate(sents):
          if verbose and idx > 0 and idx % 500000 == 0:
              logger.info(f"    line {idx}")
          self.counter.update(symbols)
  
imports: import glob
  import os
  import pickle
  import re
  from collections import Counter, OrderedDict
  from typing import List, Optional, Tuple
  import numpy
  from ....tokenization_utils import PreTrainedTokenizer
  from ....utils import cached_file, is_sacremoses_available, is_torch_available, logging, requires_backends, strtobool, torch_only_method
label_desc: Use image node_id 5 for calling the TransfoXLTokenizer obj's underlying member method code with example usage: obj.count_sents(sents, verbose) without return types