class_docstr: Punctuation capitalization dataset which allows not to load all data in memory simultaneously. A tarred dataset
  is created from text and label files using script
  `examples/nlp/token_classification/data/create_punctuation_capitalization_tarred_dataset.py
  <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/create_punctuation_capitalization_tarred_dataset.py>`_
  or function
  :func:`~nemo.collections.nlp.data.token_classification.punctuation_capitalization_tarred_dataset.create_tarred_dataset`.
  
  Args:
      metadata_file (:obj:`Union[os.PathLike, str]`): a path to tarred dataset metadata file. Metadata file and files
          referenced in metadata file are created by
          `examples/nlp/token_classification/data/create_punctuation_capitalization_tarred_dataset.py
          <https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/token_classification/data/create_punctuation_capitalization_tarred_dataset.py>`_.
          Metadata file is a JSON file which contains ``'num_batches'``, ``'tar_files'``,
          ``'punct_label_vocab_file'``, ``'capit_label_vocab_file'`` items. The first item is total number of batches
          in a dataset, the second is a list of paths to tar files relative to directory containing
          ``metadata_file``. Items ``'punct_label_vocab_file'`` and ``'capit_label_vocab_file'`` are paths to
          ``.csv`` files which contain unique punctuation a capitalization label vocabularies. Vocabulary file paths
          are relative to directory containing the ``metadata_file``. Each line in ``'punct_label_vocab_file'`` and
          ``'capit_label_vocab_file'`` contains 1 label. The first lines in ``'punct_label_vocab_file'`` and
          ``'capit_label_vocab_file'`` files are neutral labels which also serve as pad labels. Neutral labels for
          punctuation and capitalization must be equal to the ``pad_label`` parameter.
      tokenizer (:obj:`TokenizerSpec`): a tokenizer instance used for tokenization of dataset source. A tokenizer
          instance is used for getting ids of [CLS], [PAD], and [SEP] tokens which are used for masks creation.
      pad_label (:obj:`str`): a label that is used for padding and for absence of punctuation or
          capitalization. Used for checking items ``'punct_label_vocab'`` and ``'capit_label_vocab'`` of dictionary
          in ``metadata_file``.
      label_info_save_dir (:obj:`Union[os.PathLike, str]`, `optional`): a path to a directory where label
          vocabularies are copied when method :meth:`save_labels_and_get_file_paths` is called. This parameter is
          useful if tarred dataset directory is read-only.
      ignore_extra_tokens (:obj:`bool`, `optional`, defaults to :obj:`False`): whether to use only first token in a
          word for loss computation and training. If set to ``True``, then loss will be computed only for the first
          tokens of words.
      ignore_start_end (:obj:`bool`, `optional`, defaults to :obj:`True`): whether to compute loss for [CLS] and
          [SEP] tokens. If set to ``True``, then loss will not be computed for [CLS] and [SEP] tokens.
      world_size (:obj:`int`, `optional`, defaults to :obj:`1`): a number of processes used for model training. It is
          used together with a ``global_rank`` parameter to decide which tar files will be used in the current
          process.
      global_rank (:obj:`int`, `optional`, defaults to :obj:`0`): a number of current process in the pool of workers
          used for model training. It is used together with ``world_size`` parameter to decide which tar files will
          be used in the current process.
      shuffle_n (:obj:`int`, `optional`, defaults to :obj:`1`): a number of shuffled batches in a buffer.
          ``shuffle_n`` batches are loaded into memory, shuffled, and then yielded by a dataset instance.
      shard_strategy (:obj:`str`, defaults to :obj:``'scatter'``): Tarred dataset shard distribution strategy chosen as
          a str value during ddp.
          -   ``'scatter'``: The default shard strategy applied by WebDataset, where each node gets
              a unique set of shards, which are permanently pre-allocated and never changed at runtime.
          -   ``'replicate'``: Optional shard strategy, where each node gets all the set of shards
              available in the tarred dataset, which are permanently pre-allocated and never changed at runtime.
              The benefit of replication is that it allows each node to sample data points from the entire
              dataset independently of other nodes, and reduces dependence on value of :param:`shuffle_n`.
  
              .. warning::
                  Replicated strategy allows every node to sample the entire set of available tar files,
                  and therefore more than one node may sample the same tarfile, and even sample the same
                  data points! As such, there is no assured guarantee that all samples in the dataset will be
                  sampled at least once during 1 epoch. Scattered strategy, on the other hand, on specific
                  occasions (when the number of shards is not divisible with ``world_size``), will not sample
                  the entire dataset. For these reasons it is not advisable to use tarred datasets as validation
                  or test datasets.
class_docstr_tok: ["Punctuation","capitalization","dataset","which","allows","not","to","load","all","data","in","memory","simultaneously",".","A","tarred","dataset","is","created","from","text","and","label","files","using","script","`","examples\/nlp\/token_classification\/data\/create_punctuation_capitalization_tarred_dataset.py","<","https",":","\/\/github.com\/NVIDIA\/NeMo\/blob\/main\/examples\/nlp\/token_classification\/data\/create_punctuation_capitalization_tarred_dataset.py",">","`","_","or","function",":","func",":","`","~nemo.collections.nlp.data.token_classification.punctuation_capitalization_tarred_dataset.create_tarred_dataset","`",".","Args",":","metadata_file","(",":","obj",":","`","Union","[","os.PathLike",",","str","]","`",")",":","a","path","to","tarred","dataset","metadata","file",".","Metadata","file","and","files","referenced","in","metadata","file","are","created","by","`","examples\/nlp\/token_classification\/data\/create_punctuation_capitalization_tarred_dataset.py","<","https",":","\/\/github.com\/NVIDIA\/NeMo\/blob\/main\/examples\/nlp\/token_classification\/data\/create_punctuation_capitalization_tarred_dataset.py",">","`","_",".","Metadata","file","is","a","JSON","file","which","contains","``","'num_batches","'","``",",","``","'tar_files","'","``",",","``","'punct_label_vocab_file","'","``",",","``","'capit_label_vocab_file","'","``","items",".","The","first","item","is","total","number","of","batches","in","a","dataset",",","the","second","is","a","list","of","paths","to","tar","files","relative","to","directory","containing","``","metadata_file","``",".","Items","``","'punct_label_vocab_file","'","``","and","``","'capit_label_vocab_file","'","``","are","paths","to","``",".csv","``","files","which","contain","unique","punctuation","a","capitalization","label","vocabularies",".","Vocabulary","file","paths","are","relative","to","directory","containing","the","``","metadata_file","``",".","Each","line","in","``","'punct_label_vocab_file","'","``","and","``","'capit_label_vocab_file","'","``","contains","1","label",".","The","first","lines","in","``","'punct_label_vocab_file","'","``","and","``","'capit_label_vocab_file","'","``","files","are","neutral","labels","which","also","serve","as","pad","labels",".","Neutral","labels","for","punctuation","and","capitalization","must","be","equal","to","the","``","pad_label","``","parameter",".","tokenizer","(",":","obj",":","`","TokenizerSpec","`",")",":","a","tokenizer","instance","used","for","tokenization","of","dataset","source",".","A","tokenizer","instance","is","used","for","getting","ids","of","[","CLS","]",",","[","PAD","]",",","and","[","SEP","]","tokens","which","are","used","for","masks","creation",".","pad_label","(",":","obj",":","`","str","`",")",":","a","label","that","is","used","for","padding","and","for","absence","of","punctuation","or","capitalization",".","Used","for","checking","items","``","'punct_label_vocab","'","``","and","``","'capit_label_vocab","'","``","of","dictionary","in","``","metadata_file","``",".","label_info_save_dir","(",":","obj",":","`","Union","[","os.PathLike",",","str","]","`",",","`","optional","`",")",":","a","path","to","a","directory","where","label","vocabularies","are","copied","when","method",":","meth",":","`","save_labels_and_get_file_paths","`","is","called",".","This","parameter","is","useful","if","tarred","dataset","directory","is","read-only",".","ignore_extra_tokens","(",":","obj",":","`","bool","`",",","`","optional","`",",","defaults","to",":","obj",":","`","False","`",")",":","whether","to","use","only","first","token","in","a","word","for","loss","computation","and","training",".","If","set","to","``","True","``",",","then","loss","will","be","computed","only","for","the","first","tokens","of","words",".","ignore_start_end","(",":","obj",":","`","bool","`",",","`","optional","`",",","defaults","to",":","obj",":","`","True","`",")",":","whether","to","compute","loss","for","[","CLS","]","and","[","SEP","]","tokens",".","If","set","to","``","True","``",",","then","loss","will","not","be","computed","for","[","CLS","]","and","[","SEP","]","tokens",".","world_size","(",":","obj",":","`","int","`",",","`","optional","`",",","defaults","to",":","obj",":","`","1","`",")",":","a","number","of","processes","used","for","model","training",".","It","is","used","together","with","a","``","global_rank","``","parameter","to","decide","which","tar","files","will","be","used","in","the","current","process",".","global_rank","(",":","obj",":","`","int","`",",","`","optional","`",",","defaults","to",":","obj",":","`","0","`",")",":","a","number","of","current","process","in","the","pool","of","workers","used","for","model","training",".","It","is","used","together","with","``","world_size","``","parameter","to","decide","which","tar","files","will","be","used","in","the","current","process",".","shuffle_n","(",":","obj",":","`","int","`",",","`","optional","`",",","defaults","to",":","obj",":","`","1","`",")",":","a","number","of","shuffled","batches","in","a","buffer",".","``","shuffle_n","``","batches","are","loaded","into","memory",",","shuffled",",","and","then","yielded","by","a","dataset","instance",".","shard_strategy","(",":","obj",":","`","str","`",",","defaults","to",":","obj",":","``","'scatter","'","``",")",":","Tarred","dataset","shard","distribution","strategy","chosen","as","a","str","value","during","ddp",".","-","``","'scatter","'","``",":","The","default","shard","strategy","applied","by","WebDataset",",","where","each","node","gets","a","unique","set","of","shards",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","-","``","'replicate","'","``",":","Optional","shard","strategy",",","where","each","node","gets","all","the","set","of","shards","available","in","the","tarred","dataset",",","which","are","permanently","pre-allocated","and","never","changed","at","runtime",".","The","benefit","of","replication","is","that","it","allows","each","node","to","sample","data","points","from","the","entire","dataset","independently","of","other","nodes",",","and","reduces","dependence","on","value","of",":","param",":","`","shuffle_n","`",".","..","warning",":",":","Replicated","strategy","allows","every","node","to","sample","the","entire","set","of","available","tar","files",",","and","therefore","more","than","one","node","may","sample","the","same","tarfile",",","and","even","sample","the","same","data","points","!","As","such",",","there","is","no","assured","guarantee","that","all","samples","in","the","dataset","will","be","sampled","at","least","once","during","1","epoch",".","Scattered","strategy",",","on","the","other","hand",",","on","specific","occasions","(","when","the","number","of","shards","is","not","divisible","with","``","world_size","``",")",",","will","not","sample","the","entire","dataset",".","For","these","reasons","it","is","not","advisable","to","use","tarred","datasets","as","validation","or","test","datasets","."]
code: def __len__(self) -> int:
      return self.length
  
imports: import inspect
  import json
  import multiprocessing
  import os
  import pickle
  import re
  import shutil
  import tempfile
  from collections import deque
  from pathlib import Path
  from typing import Any, Callable, Dict, Iterator, List, Optional, Set, Tuple, Type, Union
  import numpy
  import torch
  import webdataset
  from joblib import Parallel, delayed
  from omegaconf import DictConfig
  from torch.utils.data import IterableDataset
  from nemo.collections.common.tokenizers import TokenizerSpec
  from nemo.collections.nlp.data.token_classification.punctuation_capitalization_dataset import LABEL_ID_DIR_FOR_NEMO_CHECKPOINT, BertPunctuationCapitalizationDataset, Progress, create_label_ids, create_masks_and_segment_ids, load_label_ids, raise_not_equal_labels_error
  from nemo.collections.nlp.modules.common.tokenizer_utils import get_tokenizer
  from nemo.core.neural_types import AudioSignal, ChannelType, LabelsType, LengthsType, MaskType, NeuralType
  from nemo.utils import logging
label_desc: Use image node_id 8 for calling the BertPunctuationCapitalizationTarredDataset obj's underlying member method code with example usage: obj.__len__() and returns: self