class_docstr: This class provides the structure for defining neural networks in a compositional way.
  An `AbstractNeuralModel` accepts as input `TRawDatapoint`, a single example. Each model can have some
  metadata. For example, this may be the vocabulary of words to be represented and their mappings to integer ids.
  The computed metadata is then used to build a `TNeuralModule` (a `nn.Module`) which contains
  the definition of the neural network architecture and its parameters. During training and testing, each `TRawDatapoint`
  is converted to a tensorized format `TTensorizedDatapoint`. By batching multiple `TTensorizedDatapoint` a minibatch
  is created which is the input of the `TNeuralModule`.
  
  To create an `AbstractNeuralModel`:
  
  * Compute the metadata.
      * Initialize metadata in `initialize_metadata`. Store any metadata structures within the model.
      * Update this structure for each example in `update_metadata_from`. Explicitly unpack/transform any parts of the
          `TRawDatapoint` and invoke the `update_metadata_from` for all child models.
      * Finally, compute the model metadata in `finalize_metadata` and store them as fields in your implementation
          of the `AbstractNeuralModel`. Optionally, discard any temporary structures within the model.
  
  * Implement `build_neural_module()` that uses the metadata previously computed to create a `TNeuralModule`, ie.
      a concrete `nn.Module` or `ModuleWithMetrics`. Note that no elements of the neural network should be stored
      in this object but they should be returned.
  * Define `tensorize()` accepting a single `TRawDatapoint` and yielding a `TTensorizedDatapoint`.
      Each `TTensorizedDatapoint` should be considered a single sample for your neural network. Each model should
      call its child models (if any) to delegate tensorization of the relevant parts of the `TRawDatapoint`
  
  * Finally, to create minibatches:
      * Implement `initialize_minibatch` which creates a dictionary that will be populated with the information from
          multiple `TTensorizedDatapoint`s. Explicitly invoke child models to initialize aspect of the minibatch
          that are their responsibility.
      * Implement `extend_minibatch_with` to extend the minibatch with an additional sample. Return `True`
          if the minibatch can be further extended. Explicitly deconstruct the `TTensorizedDatapoint` and pass its
          parts to the responsible child models.
      * Implement `finalize_minibatch` to finalize the minibatch (invoking this for any child models). This
          should return a dictionary of PyTorch tensors ready to be consumed by the `forward()` of the `TNeuralModule`.
docstr: Given a dataset, return an iterator of the tensorized data. If `return_input_data` is True
  then the original data is also returned.
  
  :param dataset_iterator: An iterator of raw data.
  :param parallelize: If True, the tensorization will be parallelized. This speeds up tensorization but
      makes debugging harder.
  :param use_multiprocessing: If parallelize==True, then use multiprocessing instead of multithreading.
      If parallelize==False this option is ignored.
  :param return_input_data: Should the raw input data also be returned?
docstr_tok: ["Given","a","dataset",",","return","an","iterator","of","the","tensorized","data",".","If","`","return_input_data","`","is","True","then","the","original","data","is","also","returned",".",":","param","dataset_iterator",":","An","iterator","of","raw","data",".",":","param","parallelize",":","If","True",",","the","tensorization","will","be","parallelized",".","This","speeds","up","tensorization","but","makes","debugging","harder",".",":","param","use_multiprocessing",":","If","parallelize==True",",","then","use","multiprocessing","instead","of","multithreading",".","If","parallelize==False","this","option","is","ignored",".",":","param","return_input_data",":","Should","the","raw","input","data","also","be","returned","?"]
code: def tensorize_dataset(
      self,
      dataset_iterator: Iterator[TRawDatapoint],
      *,
      parallelize: bool = True,
      use_multiprocessing: bool = True,
      return_input_data: bool = False,
  ) -> Iterator[Tuple[TTensorizedDatapoint, Optional[TRawDatapoint]]]:
      """
      Given a dataset, return an iterator of the tensorized data. If `return_input_data` is True
      then the original data is also returned.
  
      :param dataset_iterator: An iterator of raw data.
      :param parallelize: If True, the tensorization will be parallelized. This speeds up tensorization but
          makes debugging harder.
      :param use_multiprocessing: If parallelize==True, then use multiprocessing instead of multithreading.
          If parallelize==False this option is ignored.
      :param return_input_data: Should the raw input data also be returned?
      """
      assert (
          self.__metadata_initialized
      ), "Metadata has not been initialized."
      if parallelize:
          base_iterator = self._TensorizedDataIter(
              self, dataset_iterator, return_input_data
          )
  
          if use_multiprocessing:
              for tensorized_sample in BufferedIterator(
                  base_iterator, max_queue_size=500
              ):
                  if tensorized_sample[0] is not None:
                      yield tensorized_sample
          else:
              with futures.ThreadPoolExecutor() as pool:
                  for tensorized_sample in pool.map(
                      lambda d: (
                          self.tensorize(d),
                          d if return_input_data else None,
                      ),
                      dataset_iterator,
                      chunksize=20,
                  ):
                      if tensorized_sample[0] is not None:
                          yield tensorized_sample
      else:
          for datapoint in dataset_iterator:
              tensorized_data_sample = self.tensorize(datapoint)
              if tensorized_data_sample is not None:
                  yield (
                      tensorized_data_sample,
                      datapoint if return_input_data else None,
                  )
  
imports: from typing_extensions import final
  import gzip
  import os
  import torch
  from abc import ABC, abstractmethod
  from concurrent import futures
  from dpu_utils.utils.iterators import BufferedIterator, ThreadedIterator, shuffled_iterator
  from itertools import islice
  from pathlib import Path
  from torch import nn
  from typing import Any, Dict, Generic, Iterable, Iterator, List, Mapping, NamedTuple, Optional, Tuple, Type, TypeVar, Union
label_desc: Use image node_id 13 for calling the AbstractNeuralModel obj's underlying member method code with example usage: obj.tensorize_dataset(dataset_iterator) without return types