class_docstr: A Vocabulary maps strings to integers, allowing for strings to be mapped to an
  out-of-vocabulary token.
  
  Vocabularies are fit to a particular dataset, which we use to decide which tokens are
  in-vocabulary.
  
  Vocabularies also allow for several different namespaces, so you can have separate indices for
  'a' as a word, and 'a' as a character, for instance, and so we can use this object to also map
  tag and label strings to indices, for a unified :class:`~.fields.field.Field` API.  Most of the
  methods on this class allow you to pass in a namespace; by default we use the 'tokens'
  namespace, and you can omit the namespace argument everywhere and just use the default.
  
  This class is registered as a `Vocabulary` with four different names, which all point to
  different `@classmethod` constructors found in this class.  `from_instances` is registered as
  "from_instances", `from_files` is registered as "from_files", `from_files_and_instances` is
  registered as "extend", and `empty` is registered as "empty".  If you are using a configuration
  file to construct a vocabulary, you can use any of those strings as the "type" key in the
  configuration file to use the corresponding `@classmethod` to construct the object.
  "from_instances" is the default.  Look at the docstring for the `@classmethod` to see what keys
  are allowed in the configuration file (when there is an `instances` argument to the
  `@classmethod`, it will be passed in separately and does not need a corresponding key in the
  configuration file).
  
  # Parameters
  
  counter : `Dict[str, Dict[str, int]]`, optional (default=`None`)
      A collection of counts from which to initialize this vocabulary.  We will examine the
      counts and, together with the other parameters to this class, use them to decide which
      words are in-vocabulary.  If this is `None`, we just won't initialize the vocabulary with
      anything.
  
  min_count : `Dict[str, int]`, optional (default=`None`)
      When initializing the vocab from a counter, you can specify a minimum count, and every
      token with a count less than this will not be added to the dictionary.  These minimum
      counts are `namespace-specific`, so you can specify different minimums for labels versus
      words tokens, for example.  If a namespace does not have a key in the given dictionary, we
      will add all seen tokens to that namespace.
  
  max_vocab_size : `Union[int, Dict[str, int]]`, optional (default=`None`)
      If you want to cap the number of tokens in your vocabulary, you can do so with this
      parameter.  If you specify a single integer, every namespace will have its vocabulary fixed
      to be no larger than this.  If you specify a dictionary, then each namespace in the
      `counter` can have a separate maximum vocabulary size.  Any missing key will have a value
      of `None`, which means no cap on the vocabulary size.
  
  non_padded_namespaces : `Iterable[str]`, optional
      By default, we assume you are mapping word / character tokens to integers, and so you want
      to reserve word indices for padding and out-of-vocabulary tokens.  However, if you are
      mapping NER or SRL tags, or class labels, to integers, you probably do not want to reserve
      indices for padding and out-of-vocabulary tokens.  Use this field to specify which
      namespaces should `not` have padding and OOV tokens added.
  
      The format of each element of this is either a string, which must match field names
      exactly,  or `*` followed by a string, which we match as a suffix against field names.
  
      We try to make the default here reasonable, so that you don't have to think about this.
      The default is `("*tags", "*labels")`, so as long as your namespace ends in "tags" or
      "labels" (which is true by default for all tag and label fields in this code), you don't
      have to specify anything here.
  
  pretrained_files : `Dict[str, str]`, optional
      If provided, this map specifies the path to optional pretrained embedding files for each
      namespace. This can be used to either restrict the vocabulary to only words which appear
      in this file, or to ensure that any words in this file are included in the vocabulary
      regardless of their count, depending on the value of `only_include_pretrained_words`.
      Words which appear in the pretrained embedding file but not in the data are NOT included
      in the Vocabulary.
  
  min_pretrained_embeddings : `Dict[str, int]`, optional
      Specifies for each namespace a minimum number of lines (typically the
      most common words) to keep from pretrained embedding files, even for words not
      appearing in the data. By default the minimum number of lines to keep is 0.
      You can automatically include all lines for a namespace by setting the minimum number of lines
      to `-1`.
  
  only_include_pretrained_words : `bool`, optional (default=`False`)
      This defines the strategy for using any pretrained embedding files which may have been
      specified in `pretrained_files`.
  
      If `False`, we use an inclusive strategy and include both words in the `counter`
      that have a count of at least `min_count` and words from the pretrained file
      that are within the first `N` lines defined by `min_pretrained_embeddings`.
  
      If `True`, we use an exclusive strategy where words are only included in the `Vocabulary`
      if they are in the pretrained embedding file. Their count must also be at least `min_count`
      or they must be listed in the embedding file within the first `N` lines defined
      by `min_pretrained_embeddings`.
  
  tokens_to_add : `Dict[str, List[str]]`, optional (default=`None`)
      If given, this is a list of tokens to add to the vocabulary, keyed by the namespace to add
      the tokens to.  This is a way to be sure that certain items appear in your vocabulary,
      regardless of any other vocabulary computation.
  
  padding_token : `str`,  optional (default=`DEFAULT_PADDING_TOKEN`)
      If given, this the string used for padding.
  
  oov_token : `str`,  optional (default=`DEFAULT_OOV_TOKEN`)
      If given, this the string used for the out of vocabulary (OOVs) tokens.
class_docstr_tok: ["A","Vocabulary","maps","strings","to","integers",",","allowing","for","strings","to","be","mapped","to","an","out-of-vocabulary","token",".","Vocabularies","are","fit","to","a","particular","dataset",",","which","we","use","to","decide","which","tokens","are","in-vocabulary",".","Vocabularies","also","allow","for","several","different","namespaces",",","so","you","can","have","separate","indices","for","'","a","'","as","a","word",",","and","'","a","'","as","a","character",",","for","instance",",","and","so","we","can","use","this","object","to","also","map","tag","and","label","strings","to","indices",",","for","a","unified",":","class",":","`","~.fields.field.Field","`","API",".","Most","of","the","methods","on","this","class","allow","you","to","pass","in","a","namespace",";","by","default","we","use","the","'tokens'","namespace",",","and","you","can","omit","the","namespace","argument","everywhere","and","just","use","the","default",".","This","class","is","registered","as","a","`","Vocabulary","`","with","four","different","names",",","which","all","point","to","different","`","@","classmethod","`","constructors","found","in","this","class",".","`","from_instances","`","is","registered","as","''","from_instances","''",",","`","from_files","`","is","registered","as","``","from_files","''",",","`","from_files_and_instances","`","is","registered","as","``","extend","''",",","and","`","empty","`","is","registered","as","``","empty","''",".","If","you","are","using","a","configuration","file","to","construct","a","vocabulary",",","you","can","use","any","of","those","strings","as","the","``","type","''","key","in","the","configuration","file","to","use","the","corresponding","`","@","classmethod","`","to","construct","the","object",".","``","from_instances","''","is","the","default",".","Look","at","the","docstring","for","the","`","@","classmethod","`","to","see","what","keys","are","allowed","in","the","configuration","file","(","when","there","is","an","`","instances","`","argument","to","the","`","@","classmethod","`",",","it","will","be","passed","in","separately","and","does","not","need","a","corresponding","key","in","the","configuration","file",")",".","#","Parameters","counter",":","`","Dict","[","str",",","Dict","[","str",",","int","]","]","`",",","optional","(","default=","`","None","`",")","A","collection","of","counts","from","which","to","initialize","this","vocabulary",".","We","will","examine","the","counts","and",",","together","with","the","other","parameters","to","this","class",",","use","them","to","decide","which","words","are","in-vocabulary",".","If","this","is","`","None","`",",","we","just","wo","n't","initialize","the","vocabulary","with","anything",".","min_count",":","`","Dict","[","str",",","int","]","`",",","optional","(","default=","`","None","`",")","When","initializing","the","vocab","from","a","counter",",","you","can","specify","a","minimum","count",",","and","every","token","with","a","count","less","than","this","will","not","be","added","to","the","dictionary",".","These","minimum","counts","are","`","namespace-specific","`",",","so","you","can","specify","different","minimums","for","labels","versus","words","tokens",",","for","example",".","If","a","namespace","does","not","have","a","key","in","the","given","dictionary",",","we","will","add","all","seen","tokens","to","that","namespace",".","max_vocab_size",":","`","Union","[","int",",","Dict","[","str",",","int","]","]","`",",","optional","(","default=","`","None","`",")","If","you","want","to","cap","the","number","of","tokens","in","your","vocabulary",",","you","can","do","so","with","this","parameter",".","If","you","specify","a","single","integer",",","every","namespace","will","have","its","vocabulary","fixed","to","be","no","larger","than","this",".","If","you","specify","a","dictionary",",","then","each","namespace","in","the","`","counter","`","can","have","a","separate","maximum","vocabulary","size",".","Any","missing","key","will","have","a","value","of","`","None","`",",","which","means","no","cap","on","the","vocabulary","size",".","non_padded_namespaces",":","`","Iterable","[","str","]","`",",","optional","By","default",",","we","assume","you","are","mapping","word","\/","character","tokens","to","integers",",","and","so","you","want","to","reserve","word","indices","for","padding","and","out-of-vocabulary","tokens",".","However",",","if","you","are","mapping","NER","or","SRL","tags",",","or","class","labels",",","to","integers",",","you","probably","do","not","want","to","reserve","indices","for","padding","and","out-of-vocabulary","tokens",".","Use","this","field","to","specify","which","namespaces","should","`","not","`","have","padding","and","OOV","tokens","added",".","The","format","of","each","element","of","this","is","either","a","string",",","which","must","match","field","names","exactly",",","or","`","*","`","followed","by","a","string",",","which","we","match","as","a","suffix","against","field","names",".","We","try","to","make","the","default","here","reasonable",",","so","that","you","do","n't","have","to","think","about","this",".","The","default","is","`","(","``","*","tags","''",",","``","*","labels","''",")","`",",","so","as","long","as","your","namespace","ends","in","``","tags","''","or","``","labels","''","(","which","is","true","by","default","for","all","tag","and","label","fields","in","this","code",")",",","you","don't","have","to","specify","anything","here",".","pretrained_files",":","`","Dict","[","str",",","str","]","`",",","optional","If","provided",",","this","map","specifies","the","path","to","optional","pretrained","embedding","files","for","each","namespace",".","This","can","be","used","to","either","restrict","the","vocabulary","to","only","words","which","appear","in","this","file",",","or","to","ensure","that","any","words","in","this","file","are","included","in","the","vocabulary","regardless","of","their","count",",","depending","on","the","value","of","`","only_include_pretrained_words","`",".","Words","which","appear","in","the","pretrained","embedding","file","but","not","in","the","data","are","NOT","included","in","the","Vocabulary",".","min_pretrained_embeddings",":","`","Dict","[","str",",","int","]","`",",","optional","Specifies","for","each","namespace","a","minimum","number","of","lines","(","typically","the","most","common","words",")","to","keep","from","pretrained","embedding","files",",","even","for","words","not","appearing","in","the","data",".","By","default","the","minimum","number","of","lines","to","keep","is","0",".","You","can","automatically","include","all","lines","for","a","namespace","by","setting","the","minimum","number","of","lines","to","`","-1","`",".","only_include_pretrained_words",":","`","bool","`",",","optional","(","default=","`","False","`",")","This","defines","the","strategy","for","using","any","pretrained","embedding","files","which","may","have","been","specified","in","`","pretrained_files","`",".","If","`","False","`",",","we","use","an","inclusive","strategy","and","include","both","words","in","the","`","counter","`","that","have","a","count","of","at","least","`","min_count","`","and","words","from","the","pretrained","file","that","are","within","the","first","`","N","`","lines","defined","by","`","min_pretrained_embeddings","`",".","If","`","True","`",",","we","use","an","exclusive","strategy","where","words","are","only","included","in","the","`","Vocabulary","`","if","they","are","in","the","pretrained","embedding","file",".","Their","count","must","also","be","at","least","`","min_count","`","or","they","must","be","listed","in","the","embedding","file","within","the","first","`","N","`","lines","defined","by","`","min_pretrained_embeddings","`",".","tokens_to_add",":","`","Dict","[","str",",","List","[","str","]","]","`",",","optional","(","default=","`","None","`",")","If","given",",","this","is","a","list","of","tokens","to","add","to","the","vocabulary",",","keyed","by","the","namespace","to","add","the","tokens","to",".","This","is","a","way","to","be","sure","that","certain","items","appear","in","your","vocabulary",",","regardless","of","any","other","vocabulary","computation",".","padding_token",":","`","str","`",",","optional","(","default=","`","DEFAULT_PADDING_TOKEN","`",")","If","given",",","this","the","string","used","for","padding",".","oov_token",":","`","str","`",",","optional","(","default=","`","DEFAULT_OOV_TOKEN","`",")","If","given",",","this","the","string","used","for","the","out","of","vocabulary","(","OOVs",")","tokens","."]
docstr: This method returns a bare vocabulary instantiated with `cls()` (so, `Vocabulary()` if you
  haven't made a subclass of this object).  The only reason to call `Vocabulary.empty()`
  instead of `Vocabulary()` is if you are instantiating this object from a config file.  We
  register this constructor with the key "empty", so if you know that you don't need to
  compute a vocabulary (either because you're loading a pre-trained model from an archive
  file, you're using a pre-trained transformer that has its own vocabulary, or something
  else), you can use this to avoid having the default vocabulary construction code iterate
  through the data.
code: def empty(cls) -> "Vocabulary":
      """
      This method returns a bare vocabulary instantiated with `cls()` (so, `Vocabulary()` if you
      haven't made a subclass of this object).  The only reason to call `Vocabulary.empty()`
      instead of `Vocabulary()` is if you are instantiating this object from a config file.  We
      register this constructor with the key "empty", so if you know that you don't need to
      compute a vocabulary (either because you're loading a pre-trained model from an archive
      file, you're using a pre-trained transformer that has its own vocabulary, or something
      else), you can use this to avoid having the default vocabulary construction code iterate
      through the data.
      """
      return cls()
  
imports: import codecs
  import copy
  import logging
  import os
  import re
  from collections import defaultdict
  from typing import Any, Callable, Dict, Iterable, List, Optional, Set, Union, TYPE_CHECKING
  from transformers import PreTrainedTokenizer
  from allennlp.common import Registrable
  from allennlp.common.file_utils import cached_path, FileLock
  from allennlp.common.checks import ConfigurationError
  from allennlp.common.tqdm import Tqdm
  from allennlp.common.util import namespace_match
label_desc: Use image node_id 7 for calling the Vocabulary obj's underlying member method code with example usage: obj.empty(cls) and returns: cls