class_docstr: An iterator for reading records from a :class:`~Dataset`.
  
  For Datasets, each iteration call represents a complete read of all items in the
  Dataset.
  
  If using Ray Train, each trainer actor should get its own iterator by calling
  :meth:`ray.train.get_dataset_shard("train")
  <ray.train.get_dataset_shard>`.
  
  Examples:
      >>> import ray
      >>> ds = ray.data.range(5)
      >>> ds
      Dataset(num_blocks=..., num_rows=5, schema={id: int64})
      >>> ds.iterator()
      DataIterator(Dataset(num_blocks=..., num_rows=5, schema={id: int64}))
class_docstr_tok: ["An","iterator","for","reading","records","from","a",":","class",":","`","~Dataset","`",".","For","Datasets",",","each","iteration","call","represents","a","complete","read","of","all","items","in","the","Dataset",".","If","using","Ray","Train",",","each","trainer","actor","should","get","its","own","iterator","by","calling",":","meth",":","`","ray.train.get_dataset_shard","(","``","train","''",")","<","ray.train.get_dataset_shard",">","`",".","Examples",":",">",">",">","import","ray",">",">",">","ds","=","ray.data.range","(","5",")",">",">",">","ds","Dataset","(","num_blocks=","...",",","num_rows=5",",","schema=","{","id",":","int64","}",")",">",">",">","ds.iterator","(",")","DataIterator","(","Dataset","(","num_blocks=","...",",","num_rows=5",",","schema=","{","id",":","int64","}",")",")"]
docstr: Return a Torch IterableDataset over this dataset.
  
  This is only supported for datasets convertible to Arrow records.
  
  It is recommended to use the returned ``IterableDataset`` directly
  instead of passing it into a torch ``DataLoader``.
  
  Each element in IterableDataset will be a tuple consisting of 2
  elements. The first item contains the feature tensor(s), and the
  second item is the label tensor. Those can take on different
  forms, depending on the specified arguments.
  
  For the features tensor (N is the ``batch_size`` and n, m, k
  are the number of features per tensor):
  
  * If ``feature_columns`` is a ``List[str]``, the features will be
    a tensor of shape (N, n), with columns corresponding to
    ``feature_columns``
  
  * If ``feature_columns`` is a ``List[List[str]]``, the features will be
    a list of tensors of shape [(N, m),...,(N, k)], with columns of each
    tensor corresponding to the elements of ``feature_columns``
  
  * If ``feature_columns`` is a ``Dict[str, List[str]]``, the features
    will be a dict of key-tensor pairs of shape
    {key1: (N, m),..., keyN: (N, k)}, with columns of each
    tensor corresponding to the value of ``feature_columns`` under the
    key.
  
  If ``unsqueeze_label_tensor=True`` (default), the label tensor will be
  of shape (N, 1). Otherwise, it will be of shape (N,).
  If ``label_column`` is specified as ``None``, then no column from the
  ``Dataset`` will be treated as the label, and the output label tensor
  will be ``None``.
  
  Note that you probably want to call ``.split()`` on this dataset if
  there are to be multiple Torch workers consuming the data.
  
  Time complexity: O(1)
  
  Args:
      label_column: The name of the column used as the
          label (second element of the output list). Can be None for
          prediction, in which case the second element of returned
          tuple will also be None.
      feature_columns: The names of the columns
          to use as the features. Can be a list of lists or
          a dict of string-list pairs for multi-tensor output.
          If None, then use all columns except the label column as
          the features.
      label_column_dtype: The torch dtype to
          use for the label column. If None, then automatically infer
          the dtype.
      feature_column_dtypes: The dtypes to use for the feature
          tensors. This should match the format of ``feature_columns``,
          or be a single dtype, in which case it will be applied to
          all tensors. If None, then automatically infer the dtype.
      batch_size: How many samples per batch to yield at a time.
          Defaults to 1.
      prefetch_batches: The number of batches to fetch ahead of the current batch
          to fetch. If set to greater than 0, a separate threadpool will be used
          to fetch the objects to the local node, format the batches, and apply
          the collate_fn. Defaults to 1.
      drop_last: Set to True to drop the last incomplete batch,
          if the dataset size is not divisible by the batch size. If
          False and the size of dataset is not divisible by the batch
          size, then the last batch will be smaller. Defaults to False.
      local_shuffle_buffer_size: If non-None, the data will be randomly shuffled
          using a local in-memory shuffle buffer, and this value will serve as the
          minimum number of rows that must be in the local in-memory shuffle
          buffer in order to yield a batch. When there are no more rows to add to
          the buffer, the remaining rows in the buffer will be drained. This
          buffer size must be greater than or equal to ``batch_size``, and
          therefore ``batch_size`` must also be specified when using local
          shuffling.
      local_shuffle_seed: The seed to use for the local random shuffle.
      unsqueeze_label_tensor: If set to True, the label tensor
          will be unsqueezed (reshaped to (N, 1)). Otherwise, it will
          be left as is, that is (N, ). In general, regression loss
          functions expect an unsqueezed tensor, while classification
          loss functions expect a squeezed one. Defaults to True.
      unsqueeze_feature_tensors: If set to True, the features tensors
          will be unsqueezed (reshaped to (N, 1)) before being concatenated into
          the final features tensor. Otherwise, they will be left as is, that is
          (N, ). Defaults to True.
  
  Returns:
      A torch IterableDataset.
docstr_tok: ["Return","a","Torch","IterableDataset","over","this","dataset",".","This","is","only","supported","for","datasets","convertible","to","Arrow","records",".","It","is","recommended","to","use","the","returned","``","IterableDataset","``","directly","instead","of","passing","it","into","a","torch","``","DataLoader","``",".","Each","element","in","IterableDataset","will","be","a","tuple","consisting","of","2","elements",".","The","first","item","contains","the","feature","tensor","(","s",")",",","and","the","second","item","is","the","label","tensor",".","Those","can","take","on","different","forms",",","depending","on","the","specified","arguments",".","For","the","features","tensor","(","N","is","the","``","batch_size","``","and","n",",","m",",","k","are","the","number","of","features","per","tensor",")",":","*","If","``","feature_columns","``","is","a","``","List","[","str","]","``",",","the","features","will","be","a","tensor","of","shape","(","N",",","n",")",",","with","columns","corresponding","to","``","feature_columns","``","*","If","``","feature_columns","``","is","a","``","List","[","List","[","str","]","]","``",",","the","features","will","be","a","list","of","tensors","of","shape","[","(","N",",","m",")",",","...",",","(","N",",","k",")","]",",","with","columns","of","each","tensor","corresponding","to","the","elements","of","``","feature_columns","``","*","If","``","feature_columns","``","is","a","``","Dict","[","str",",","List","[","str","]","]","``",",","the","features","will","be","a","dict","of","key-tensor","pairs","of","shape","{","key1",":","(","N",",","m",")",",","...",",","keyN",":","(","N",",","k",")","}",",","with","columns","of","each","tensor","corresponding","to","the","value","of","``","feature_columns","``","under","the","key",".","If","``","unsqueeze_label_tensor=True","``","(","default",")",",","the","label","tensor","will","be","of","shape","(","N",",","1",")",".","Otherwise",",","it","will","be","of","shape","(","N",",",")",".","If","``","label_column","``","is","specified","as","``","None","``",",","then","no","column","from","the","``","Dataset","``","will","be","treated","as","the","label",",","and","the","output","label","tensor","will","be","``","None","``",".","Note","that","you","probably","want","to","call","``",".split","(",")","``","on","this","dataset","if","there","are","to","be","multiple","Torch","workers","consuming","the","data",".","Time","complexity",":","O","(","1",")","Args",":","label_column",":","The","name","of","the","column","used","as","the","label","(","second","element","of","the","output","list",")",".","Can","be","None","for","prediction",",","in","which","case","the","second","element","of","returned","tuple","will","also","be","None",".","feature_columns",":","The","names","of","the","columns","to","use","as","the","features",".","Can","be","a","list","of","lists","or","a","dict","of","string-list","pairs","for","multi-tensor","output",".","If","None",",","then","use","all","columns","except","the","label","column","as","the","features",".","label_column_dtype",":","The","torch","dtype","to","use","for","the","label","column",".","If","None",",","then","automatically","infer","the","dtype",".","feature_column_dtypes",":","The","dtypes","to","use","for","the","feature","tensors",".","This","should","match","the","format","of","``","feature_columns","``",",","or","be","a","single","dtype",",","in","which","case","it","will","be","applied","to","all","tensors",".","If","None",",","then","automatically","infer","the","dtype",".","batch_size",":","How","many","samples","per","batch","to","yield","at","a","time",".","Defaults","to","1.","prefetch_batches",":","The","number","of","batches","to","fetch","ahead","of","the","current","batch","to","fetch",".","If","set","to","greater","than","0",",","a","separate","threadpool","will","be","used","to","fetch","the","objects","to","the","local","node",",","format","the","batches",",","and","apply","the","collate_fn",".","Defaults","to","1.","drop_last",":","Set","to","True","to","drop","the","last","incomplete","batch",",","if","the","dataset","size","is","not","divisible","by","the","batch","size",".","If","False","and","the","size","of","dataset","is","not","divisible","by","the","batch","size",",","then","the","last","batch","will","be","smaller",".","Defaults","to","False",".","local_shuffle_buffer_size",":","If","non-None",",","the","data","will","be","randomly","shuffled","using","a","local","in-memory","shuffle","buffer",",","and","this","value","will","serve","as","the","minimum","number","of","rows","that","must","be","in","the","local","in-memory","shuffle","buffer","in","order","to","yield","a","batch",".","When","there","are","no","more","rows","to","add","to","the","buffer",",","the","remaining","rows","in","the","buffer","will","be","drained",".","This","buffer","size","must","be","greater","than","or","equal","to","``","batch_size","``",",","and","therefore","``","batch_size","``","must","also","be","specified","when","using","local","shuffling",".","local_shuffle_seed",":","The","seed","to","use","for","the","local","random","shuffle",".","unsqueeze_label_tensor",":","If","set","to","True",",","the","label","tensor","will","be","unsqueezed","(","reshaped","to","(","N",",","1",")",")",".","Otherwise",",","it","will","be","left","as","is",",","that","is","(","N",",",")",".","In","general",",","regression","loss","functions","expect","an","unsqueezed","tensor",",","while","classification","loss","functions","expect","a","squeezed","one",".","Defaults","to","True",".","unsqueeze_feature_tensors",":","If","set","to","True",",","the","features","tensors","will","be","unsqueezed","(","reshaped","to","(","N",",","1",")",")","before","being","concatenated","into","the","final","features","tensor",".","Otherwise",",","they","will","be","left","as","is",",","that","is","(","N",",",")",".","Defaults","to","True",".","Returns",":","A","torch","IterableDataset","."]
code: def to_torch(
      self,
      *,
      label_column: Optional[str] = None,
      feature_columns: Optional[
          Union[List[str], List[List[str]], Dict[str, List[str]]]
      ] = None,
      label_column_dtype: Optional["torch.dtype"] = None,
      feature_column_dtypes: Optional[
          Union[
              "torch.dtype",
              List["torch.dtype"],
              Dict[str, "torch.dtype"],
          ]
      ] = None,
      batch_size: int = 1,
      prefetch_batches: int = 1,
      drop_last: bool = False,
      local_shuffle_buffer_size: Optional[int] = None,
      local_shuffle_seed: Optional[int] = None,
      unsqueeze_label_tensor: bool = True,
      unsqueeze_feature_tensors: bool = True,
      # Deprecated.
      prefetch_blocks: int = 0,
  ) -> "torch.utils.data.IterableDataset":
      """Return a Torch IterableDataset over this dataset.
  
      This is only supported for datasets convertible to Arrow records.
  
      It is recommended to use the returned ``IterableDataset`` directly
      instead of passing it into a torch ``DataLoader``.
  
      Each element in IterableDataset will be a tuple consisting of 2
      elements. The first item contains the feature tensor(s), and the
      second item is the label tensor. Those can take on different
      forms, depending on the specified arguments.
  
      For the features tensor (N is the ``batch_size`` and n, m, k
      are the number of features per tensor):
  
      * If ``feature_columns`` is a ``List[str]``, the features will be
        a tensor of shape (N, n), with columns corresponding to
        ``feature_columns``
  
      * If ``feature_columns`` is a ``List[List[str]]``, the features will be
        a list of tensors of shape [(N, m),...,(N, k)], with columns of each
        tensor corresponding to the elements of ``feature_columns``
  
      * If ``feature_columns`` is a ``Dict[str, List[str]]``, the features
        will be a dict of key-tensor pairs of shape
        {key1: (N, m),..., keyN: (N, k)}, with columns of each
        tensor corresponding to the value of ``feature_columns`` under the
        key.
  
      If ``unsqueeze_label_tensor=True`` (default), the label tensor will be
      of shape (N, 1). Otherwise, it will be of shape (N,).
      If ``label_column`` is specified as ``None``, then no column from the
      ``Dataset`` will be treated as the label, and the output label tensor
      will be ``None``.
  
      Note that you probably want to call ``.split()`` on this dataset if
      there are to be multiple Torch workers consuming the data.
  
      Time complexity: O(1)
  
      Args:
          label_column: The name of the column used as the
              label (second element of the output list). Can be None for
              prediction, in which case the second element of returned
              tuple will also be None.
          feature_columns: The names of the columns
              to use as the features. Can be a list of lists or
              a dict of string-list pairs for multi-tensor output.
              If None, then use all columns except the label column as
              the features.
          label_column_dtype: The torch dtype to
              use for the label column. If None, then automatically infer
              the dtype.
          feature_column_dtypes: The dtypes to use for the feature
              tensors. This should match the format of ``feature_columns``,
              or be a single dtype, in which case it will be applied to
              all tensors. If None, then automatically infer the dtype.
          batch_size: How many samples per batch to yield at a time.
              Defaults to 1.
          prefetch_batches: The number of batches to fetch ahead of the current batch
              to fetch. If set to greater than 0, a separate threadpool will be used
              to fetch the objects to the local node, format the batches, and apply
              the collate_fn. Defaults to 1.
          drop_last: Set to True to drop the last incomplete batch,
              if the dataset size is not divisible by the batch size. If
              False and the size of dataset is not divisible by the batch
              size, then the last batch will be smaller. Defaults to False.
          local_shuffle_buffer_size: If non-None, the data will be randomly shuffled
              using a local in-memory shuffle buffer, and this value will serve as the
              minimum number of rows that must be in the local in-memory shuffle
              buffer in order to yield a batch. When there are no more rows to add to
              the buffer, the remaining rows in the buffer will be drained. This
              buffer size must be greater than or equal to ``batch_size``, and
              therefore ``batch_size`` must also be specified when using local
              shuffling.
          local_shuffle_seed: The seed to use for the local random shuffle.
          unsqueeze_label_tensor: If set to True, the label tensor
              will be unsqueezed (reshaped to (N, 1)). Otherwise, it will
              be left as is, that is (N, ). In general, regression loss
              functions expect an unsqueezed tensor, while classification
              loss functions expect a squeezed one. Defaults to True.
          unsqueeze_feature_tensors: If set to True, the features tensors
              will be unsqueezed (reshaped to (N, 1)) before being concatenated into
              the final features tensor. Otherwise, they will be left as is, that is
              (N, ). Defaults to True.
  
      Returns:
          A torch IterableDataset.
      """
      import torch
  
      from ray.air._internal.torch_utils import (
          convert_pandas_to_torch_tensor,
      )
      from ray.data._internal.torch_iterable_dataset import (
          TorchIterableDataset,
      )
  
      # If an empty collection is passed in, treat it the same as None
      if not feature_columns:
          feature_columns = None
  
      if feature_column_dtypes and not isinstance(
          feature_column_dtypes, torch.dtype
      ):
          if isinstance(feature_columns, dict):
              if not isinstance(feature_column_dtypes, dict):
                  raise TypeError(
                      "If `feature_columns` is a dict, "
                      "`feature_column_dtypes` must be None, `torch.dtype`,"
                      f" or dict, got {type(feature_column_dtypes)}."
                  )
              if set(feature_columns) != set(feature_column_dtypes):
                  raise ValueError(
                      "`feature_columns` and `feature_column_dtypes` "
                      "must have the same keys."
                  )
              if any(
                  not subcolumns
                  for subcolumns in feature_columns.values()
              ):
                  raise ValueError("column list may not be empty")
          elif isinstance(feature_columns[0], (list, tuple)):
              if not isinstance(feature_column_dtypes, (list, tuple)):
                  raise TypeError(
                      "If `feature_columns` is a list of lists, "
                      "`feature_column_dtypes` must be None, `torch.dtype`,"
                      f" or a sequence, got {type(feature_column_dtypes)}."
                  )
              if len(feature_columns) != len(feature_column_dtypes):
                  raise ValueError(
                      "`feature_columns` and `feature_column_dtypes` "
                      "must have the same length."
                  )
              if any(not subcolumns for subcolumns in feature_columns):
                  raise ValueError("column list may not be empty")
  
      def make_generator():
          for batch in self.iter_batches(
              batch_size=batch_size,
              batch_format="pandas",
              prefetch_blocks=prefetch_blocks,
              prefetch_batches=prefetch_batches,
              drop_last=drop_last,
              local_shuffle_buffer_size=local_shuffle_buffer_size,
              local_shuffle_seed=local_shuffle_seed,
          ):
              if label_column:
                  label_tensor = convert_pandas_to_torch_tensor(
                      batch,
                      [label_column],
                      label_column_dtype,
                      unsqueeze=unsqueeze_label_tensor,
                  )
                  batch.pop(label_column)
              else:
                  label_tensor = None
  
              if isinstance(feature_columns, dict):
                  features_tensor = {
                      key: convert_pandas_to_torch_tensor(
                          batch,
                          feature_columns[key],
                          feature_column_dtypes[key]
                          if isinstance(feature_column_dtypes, dict)
                          else feature_column_dtypes,
                          unsqueeze=unsqueeze_feature_tensors,
                      )
                      for key in feature_columns
                  }
              else:
                  features_tensor = convert_pandas_to_torch_tensor(
                      batch,
                      columns=feature_columns,
                      column_dtypes=feature_column_dtypes,
                      unsqueeze=unsqueeze_feature_tensors,
                  )
  
              yield (features_tensor, label_tensor)
  
      return TorchIterableDataset(make_generator)
  
imports: import abc
  import time
  from typing import TYPE_CHECKING, Any, Callable, Dict, Iterable, Iterator, List, Optional, Tuple, TypeVar, Union
  import numpy
  from ray.data._internal.block_batching.iter_batches import iter_batches
  from ray.data._internal.stats import DatasetStats, StatsManager
  from ray.data.block import Block, BlockAccessor, BlockMetadata, DataBatch, _apply_strict_mode_batch_format
  from ray.types import ObjectRef
  from ray.util.annotations import PublicAPI
label_desc: Use image node_id 9 for calling the DataIterator obj's underlying member method code with example usage: obj.to_torch() and returns: TorchIterableDataset