class_docstr: Use Hugging Face tokenizers.
class_docstr_tok: ["Use","Hugging","Face","tokenizers","."]
code: def __init__(self, opt: Opt, shared=None):
      if not shared:
          self.hf_tokenizer = self.get_tokenizer(opt)
          self.tok2ind = self.hf_tokenizer.get_vocab()
          self.ind2tok = {v: k for k, v in self.tok2ind.items()}
      else:
          self.hf_tokenizer = shared["hf_tokenizer"]
          self.tok2ind = shared["tok2ind"]
          self.ind2tok = shared["ind2tok"]
  
      self.freq = defaultdict(int)
      for tok in self.tok2ind:
          self.freq[tok] = 1
      self.minfreq = opt.get(
          "dict_minfreq", DictionaryAgent.default_minfreq
      )
  
      self._unk_token_idx = self.hf_tokenizer.unk_token_id
      self.override_special_tokens(opt)
  
      self.lower = opt.get("dict_lower", DictionaryAgent.default_lower)
      self.tokenizer = "hf"
      self.opt = opt
      self.max_length = (
          self.opt.get("text_truncate")
          or self.hf_tokenizer.model_max_length
      )
  
imports: import os
  import re
  from abc import ABC, abstractmethod, abstractproperty
  from collections import defaultdict
  from typing import List
  from parlai.core.dict import DictionaryAgent
  from parlai.core.opt import Opt
  from parlai.utils.io import PathManager
label_desc: Use image node_id 1 to create a new HuggingFaceDictionaryAgent object from inherited base classes: DictionaryAgent, ABC with example: obj = HuggingFaceDictionaryAgent(opt, shared)