class_docstr: DataHandler is the central place to prepare data for model training/testing.
  The class is responsible of:
  
  * Define pipeline to process data and generate batch of tensors to be
    consumed by model. Each batch is a (input, target, extra_data) tuple, in
    which input can be feed directly into model.
  * Initialize global context, such as build vocab, load pretrained embeddings.
    Store the context as metadata, and provide function to serialize/deserialize
    the metadata
  
  The data processing pipeline contains the following steps:
  
  * Read data from file into a list of raw data examples
  * Convert each row of row data to a TorchText Example. This logic happens
    in process_row function and will:
  
    * Invoke featurizer, which contains data processing steps to apply
      for both training and inference time, e.g: tokenization
    * Use the raw data and results from featurizer to do any preprocessing
  
  * Generate a TorchText.Dataset that contains the list of Example, the Dataset
    also has a list of TorchText.Field, which defines how to do padding and
    numericalization while batching data.
  * Return a BatchIterator which will give a tuple of (input, target, context)
    tensors for each iteration. By default the tensors have a 1:1 mapping to
    the TorchText.Field fields, but this behavior can be overwritten by
    _input_from_batch, _target_from_batch, _context_from_batch functions.
  
  Attributes:
      raw_columns (List[str]): columns to read from data source. The order should
          match the data stored in that file.
      featurizer (Featurizer): perform data preprocessing that should be shared
          between training and inference
      features (Dict[str, Field]): a dict of name -> field that used to process data
          as model input
      labels (Dict[str, Field]): a dict of name -> field that used to process data
          as training target
      extra_fields (Dict[str, Field]): fields that process any extra data used
          neither as model input nor target. This is None by default
      text_feature_name (str): name of the text field, used to define the default
          sort key of data
      shuffle (bool): if the dataset should be shuffled, true by default
      sort_within_batch (bool): if data within same batch should be sorted, true
          by default
      train_path (str): path of training data file
      eval_path (str): path of evaluation data file
      test_path (str): path of test data file
      train_batch_size (int): training batch size, 128 by default
      eval_batch_size (int): evaluation batch size, 128 by default
      test_batch_size (int): test batch size, 128 by default
      max_seq_len (int): maximum length of tokens to keep in sequence
      pass_index (bool): if the original index of data in the batch should be
          passed along to downstream steps, default is true
class_docstr_tok: ["DataHandler","is","the","central","place","to","prepare","data","for","model","training\/testing",".","The","class","is","responsible","of",":","*","Define","pipeline","to","process","data","and","generate","batch","of","tensors","to","be","consumed","by","model",".","Each","batch","is","a","(","input",",","target",",","extra_data",")","tuple",",","in","which","input","can","be","feed","directly","into","model",".","*","Initialize","global","context",",","such","as","build","vocab",",","load","pretrained","embeddings",".","Store","the","context","as","metadata",",","and","provide","function","to","serialize\/deserialize","the","metadata","The","data","processing","pipeline","contains","the","following","steps",":","*","Read","data","from","file","into","a","list","of","raw","data","examples","*","Convert","each","row","of","row","data","to","a","TorchText","Example",".","This","logic","happens","in","process_row","function","and","will",":","*","Invoke","featurizer",",","which","contains","data","processing","steps","to","apply","for","both","training","and","inference","time",",","e.g",":","tokenization","*","Use","the","raw","data","and","results","from","featurizer","to","do","any","preprocessing","*","Generate","a","TorchText.Dataset","that","contains","the","list","of","Example",",","the","Dataset","also","has","a","list","of","TorchText.Field",",","which","defines","how","to","do","padding","and","numericalization","while","batching","data",".","*","Return","a","BatchIterator","which","will","give","a","tuple","of","(","input",",","target",",","context",")","tensors","for","each","iteration",".","By","default","the","tensors","have","a","1:1","mapping","to","the","TorchText.Field","fields",",","but","this","behavior","can","be","overwritten","by","_input_from_batch",",","_target_from_batch",",","_context_from_batch","functions",".","Attributes",":","raw_columns","(","List","[","str","]",")",":","columns","to","read","from","data","source",".","The","order","should","match","the","data","stored","in","that","file",".","featurizer","(","Featurizer",")",":","perform","data","preprocessing","that","should","be","shared","between","training","and","inference","features","(","Dict","[","str",",","Field","]",")",":","a","dict","of","name","-",">","field","that","used","to","process","data","as","model","input","labels","(","Dict","[","str",",","Field","]",")",":","a","dict","of","name","-",">","field","that","used","to","process","data","as","training","target","extra_fields","(","Dict","[","str",",","Field","]",")",":","fields","that","process","any","extra","data","used","neither","as","model","input","nor","target",".","This","is","None","by","default","text_feature_name","(","str",")",":","name","of","the","text","field",",","used","to","define","the","default","sort","key","of","data","shuffle","(","bool",")",":","if","the","dataset","should","be","shuffled",",","true","by","default","sort_within_batch","(","bool",")",":","if","data","within","same","batch","should","be","sorted",",","true","by","default","train_path","(","str",")",":","path","of","training","data","file","eval_path","(","str",")",":","path","of","evaluation","data","file","test_path","(","str",")",":","path","of","test","data","file","train_batch_size","(","int",")",":","training","batch","size",",","128","by","default","eval_batch_size","(","int",")",":","evaluation","batch","size",",","128","by","default","test_batch_size","(","int",")",":","test","batch","size",",","128","by","default","max_seq_len","(","int",")",":","maximum","length","of","tokens","to","keep","in","sequence","pass_index","(","bool",")",":","if","the","original","index","of","data","in","the","batch","should","be","passed","along","to","downstream","steps",",","default","is","true"]
code: def init_feature_metadata(
      self,
      train_data: textdata.Dataset,
      eval_data: textdata.Dataset,
      test_data: textdata.Dataset,
  ):
      # field metadata
      self.metadata.features = {}
      # build vocabs for features
      for name, feat in self.features.items():
          weights = None
          if feat.use_vocab:
              pretrained_embeddings = None
              pretrained_embeddings_path = getattr(
                  feat, "pretrained_embeddings_path", None
              )
              if pretrained_embeddings_path:
                  print(
                      "load pretrained embeddings from {}".format(
                          pretrained_embeddings_path
                      )
                  )
                  pretrained_embeddings = (
                      embeddings_utils.PretrainedEmbedding(
                          pretrained_embeddings_path, feat.lower
                      )
                  )
  
              if hasattr(feat, "vocab"):  # Don't rebuild vocab
                  print(
                      f"Vocab for feature {name} has been built. Not rebuilding."
                  )
              else:
                  print(f"Building vocab for feature {name}.")
                  vocab_data = self._get_data_to_build_vocab(
                      feat,
                      train_data,
                      eval_data,
                      test_data,
                      pretrained_embeddings,
                  )
                  feat.build_vocab(*vocab_data, min_freq=feat.min_freq)
              print(
                  "{} field's vocabulary size is {}".format(
                      name, len(feat.vocab)
                  )
              )
  
              # Initialize pretrained embedding weights.
              if pretrained_embeddings:
                  weights = pretrained_embeddings.initialize_embeddings_weights(
                      feat.vocab.stoi,
                      VocabMeta.UNK_TOKEN,
                      feat.embed_dim,
                      feat.embedding_init_strategy,
                  )  # this is of type torch.Tensor
  
          meta = feat.get_meta()
          meta.pretrained_embeds_weight = weights
          self.metadata.features[name] = meta
  
imports: import csv
  import math
  import multiprocessing
  from copy import deepcopy
  from typing import Any, Dict, Generator, Iterable, List, MutableMapping, Optional, Set, Tuple, Type, Union
  import torch
  from pytext.common.constants import BatchContext, DatasetFieldName, DFColumn, VocabMeta
  from pytext.config.component import Component, ComponentType
  from pytext.config.field_config import Target
  from pytext.config.pytext_config import ConfigBase
  from pytext.data.featurizer import Featurizer
  from pytext.fields import Field, FieldMeta, RawField, VocabUsingField
  from pytext.legacy import data
  from pytext.utils import cuda, distributed
  from pytext.utils.data import parse_json_array
  from pytext.utils.file_io import PathManager
  from pytext.utils.path import get_absolute_path
  from pytext.utils.usage import log_class_usage
  from pytorch.text.fb.utils import embeddings
  from .utils import align_target_labels
label_desc: Use image node_id 14 for calling the DataHandler obj's underlying member method code with example usage: obj.init_feature_metadata(train_data, eval_data, test_data) without return types