class_docstr: Data is an abstraction that handles all of the following:
  
  - Initialize model metadata parameters
  - Create batches of tensors for model training or prediction
  
  It can accomplish these in any way it needs to. The base implementation
  utilizes `pytext.data.sources.DataSource`, and sends batches to
  `pytext.data.tensorizers.Tensorizer` to create tensors.
  
  The `tensorizers` dict passed to the initializer should be considered something like
  a signature for the model. Each batch should be a dictionary with the same keys
  as the `tensorizers` dict, and values should be tensors arranged in the way
  specified by that tensorizer. The tensorizers dict doubles as a simple baseline
  implementation of that same signature, but subclasses of Data can override the
  implementation using other methods. This value is how the model specifies what
  inputs it's looking for.
docstr: Create batches of tensors to pass to model train_batch.
  This function yields dictionaries that mirror the `tensorizers` dict passed to
  `__init__`, ie. the keys will be the same, and the tensors will be the shape
  expected from the respective tensorizers.
  
  `stage` is used to determine which data source is used to create batches.
  if data_source is provided, it is used instead of the configured data_sorce
  this is to allow setting a different data_source for testing a model.
  
  Passing in `load_early` = True disables loading all data in memory and using
  PoolingBatcher, so that we get the first batch as quickly as possible.
docstr_tok: ["Create","batches","of","tensors","to","pass","to","model","train_batch",".","This","function","yields","dictionaries","that","mirror","the","`","tensorizers","`","dict","passed","to","`","__init__","`",",","ie",".","the","keys","will","be","the","same",",","and","the","tensors","will","be","the","shape","expected","from","the","respective","tensorizers",".","`","stage","`","is","used","to","determine","which","data","source","is","used","to","create","batches",".","if","data_source","is","provided",",","it","is","used","instead","of","the","configured","data_sorce","this","is","to","allow","setting","a","different","data_source","for","testing","a","model",".","Passing","in","`","load_early","`","=","True","disables","loading","all","data","in","memory","and","using","PoolingBatcher",",","so","that","we","get","the","first","batch","as","quickly","as","possible","."]
code: def batches(self, stage: Stage, data_source=None, load_early=False):
      """Create batches of tensors to pass to model train_batch.
      This function yields dictionaries that mirror the `tensorizers` dict passed to
      `__init__`, ie. the keys will be the same, and the tensors will be the shape
      expected from the respective tensorizers.
  
      `stage` is used to determine which data source is used to create batches.
      if data_source is provided, it is used instead of the configured data_sorce
      this is to allow setting a different data_source for testing a model.
  
      Passing in `load_early` = True disables loading all data in memory and using
      PoolingBatcher, so that we get the first batch as quickly as possible.
      """
      data_source = (
          data_source if data_source is not None else self.data_source
      )
      rows = {
          Stage.TRAIN: data_source.train,
          Stage.TEST: data_source.test,
          Stage.EVAL: data_source.eval,
      }[stage]
  
      # We add row indices here so that the original order can be reproduced
      # after shuffling the data if necessary.
      indexed_rows = self.add_row_indices(rows)
  
      # rows and numberized_rows are generators which can iterate over large
      # datasets; be careful not to do any operations which will expend them.
      if self.in_memory and not load_early:
          numberized_rows = self.numberized_cache.get(stage, None)
          if numberized_rows is None:
              numberized_rows = self.cache(
                  self.numberize_rows(indexed_rows), stage
              )
          else:
              print(f"Get numberized rows from cache in stage: {stage}")
      else:
          numberized_rows = self.numberize_rows(indexed_rows)
      sort_key = self.sort_key
  
      def key(row):
          return self.tensorizers[sort_key].sort_key(
              row.numberized[sort_key]
          )
  
      if load_early:
          batcher = Batcher(
              self.batcher.train_batch_size,
              self.batcher.eval_batch_size,
              self.batcher.test_batch_size,
          )
      else:
          batcher = self.batcher
      batches = batcher.batchify(
          numberized_rows,
          sort_key=(key if sort_key else None),
          stage=stage,
      )
      return pad_and_tensorize_batches(self.tensorizers, batches)
  
imports: import functools
  import itertools
  import math
  import random
  from typing import Any, Dict, Iterable, List, MutableMapping, NamedTuple, Optional, Type
  from pytext.common.constants import RawExampleFieldName, Stage
  from pytext.config.component import Component, ComponentType, create_component, Registry
  from pytext.utils.usage import log_class_usage
  from .sources import DataSource, RawExample, TSVDataSource
  from .sources.data_source import GeneratorIterator, RowShardedDataSource, ShardedDataSource
  from .tensorizers import initialize_tensorizers, MetricTensorizer, Tensorizer
label_desc: Use image node_id 6 for calling the Data obj's underlying member method code with example usage: obj.batches(stage, data_source, load_early) and returns: pad_and_tensorize_batches, self