class_docstr: Builds and/or loads a dictionary.
  
  The dictionary provides access to the frequency of each token, functions to
  translate sentences from tokens to their vectors (list of ints, each int is the
  index of a token in the dictionary) and back from vectors to tokenized text.
class_docstr_tok: ["Builds","and\/or","loads","a","dictionary",".","The","dictionary","provides","access","to","the","frequency","of","each","token",",","functions","to","translate","sentences","from","tokens","to","their","vectors","(","list","of","ints",",","each","int","is","the","index","of","a","token","in","the","dictionary",")","and","back","from","vectors","to","tokenized","text","."]
docstr: Return a sequence of tokens from the iterable.
  
  Also handles special tokens for some tokenizers
code: def tokenize(self, text, building=False):
      """
      Return a sequence of tokens from the iterable.
  
      Also handles special tokens for some tokenizers
      """
      if self.tokenizer in ("re", "split", "space"):
          for special_token in self.additional_special_tokens:
              index = text.find(special_token)
              if index == -1:
                  continue
              left = text[:index]
              right = text[index + len(special_token) :]
              tokens_left = (
                  self.tokenize(left, building) if left else []
              )
              tokens_right = (
                  self.tokenize(right, building) if right else []
              )
              return tokens_left + [special_token] + tokens_right
  
      if self.lower:
          text = text.lower()
  
      # calls the selected tokenizer function e.g. 're' => re_tokenize(text)
      word_tokens = self.tokenizer_fun(text)
  
      if not building and self.max_ngram_size > 1:
          # search for ngrams during parse-time
          # TODO(ahm): support build-time ngrams using word2vec heuristic?
          word_tokens = find_ngrams(
              self.tok2ind, word_tokens, self.max_ngram_size
          )
      return word_tokens
  
imports: from typing import Optional
  from parlai.core.params import ParlaiParser
  from parlai.core.opt import Opt
  from parlai.core.build_data import modelzoo_path
  from parlai.utils.bpe import bpe_factory, BPEHelper
  from .agents import Agent
  from .build_data import make_dir
  from collections import defaultdict
  import copy
  import numpy
  import os
  import json
  import re
  import parlai.utils.logging
  from parlai.utils.io import PathManager
  from typing import List
  import enum
label_desc: Use image node_id 21 for calling the DictionaryAgent obj's underlying member method code with example usage: obj.tokenize(text, building) and returns: word_tokens, unknown