class_docstr: A `SparkDatasetConverter` object holds one materialized spark dataframe and
  can be used to make one or more tensorflow datasets or torch dataloaders.
  The `SparkDatasetConverter` object is picklable and can be used in remote
  processes.
  See `make_spark_converter`
class_docstr_tok: ["A","`","SparkDatasetConverter","`","object","holds","one","materialized","spark","dataframe","and","can","be","used","to","make","one","or","more","tensorflow","datasets","or","torch","dataloaders",".","The","`","SparkDatasetConverter","`","object","is","picklable","and","can","be","used","in","remote","processes",".","See","`","make_spark_converter","`"]
docstr: Make a tensorflow dataset.
  
  This method will do the following two steps:
    1) Open a petastorm reader on the materialized dataset dir.
    2) Create a tensorflow dataset based on the reader created in (1)
  
  :param batch_size: The number of items to return per batch. Default ``None``.
      If ``None``, current implementation will set batch size to be 32, in future,
      ``None`` value will denotes auto tuned best value for batch size.
  :param prefetch: Prefetch size for tensorflow dataset. If ``None`` will use
      tensorflow autotune size. Note only available on tensorflow>=1.14
  :param num_epochs: An epoch is a single pass over all rows in the dataset.
      Setting ``num_epochs`` to ``None`` will result in an infinite number
      of epochs.
  :param workers_count: An int for the number of workers to use in the
      reader pool. This only is used for the thread or process pool.
      ``None`` denotes auto tune best value (current implementation when auto tune,
      it will always use 4 workers, but it may be improved in future)
      Default value ``None``.
  :param shuffling_queue_capacity: An int specifying the number of items to fill into a queue
      from which items are sampled each step to form batches. The larger the capacity, the
      better shuffling of the elements within the dataset. The default value of ``None``
      results in no shuffling.
  :param petastorm_reader_kwargs: arguments for `petastorm.make_batch_reader()`,
      exclude these arguments: ``dataset_url``, ``num_epochs``, ``workers_count``.
  
  :return: a context manager for a `tf.data.Dataset` object.
           when exit the returned context manager, the reader
           will be closed.
code: def make_tf_dataset(
      self,
      batch_size=None,
      prefetch=None,
      num_epochs=None,
      workers_count=None,
      shuffling_queue_capacity=None,
      **petastorm_reader_kwargs,
  ):
      """Make a tensorflow dataset.
  
      This method will do the following two steps:
        1) Open a petastorm reader on the materialized dataset dir.
        2) Create a tensorflow dataset based on the reader created in (1)
  
      :param batch_size: The number of items to return per batch. Default ``None``.
          If ``None``, current implementation will set batch size to be 32, in future,
          ``None`` value will denotes auto tuned best value for batch size.
      :param prefetch: Prefetch size for tensorflow dataset. If ``None`` will use
          tensorflow autotune size. Note only available on tensorflow>=1.14
      :param num_epochs: An epoch is a single pass over all rows in the dataset.
          Setting ``num_epochs`` to ``None`` will result in an infinite number
          of epochs.
      :param workers_count: An int for the number of workers to use in the
          reader pool. This only is used for the thread or process pool.
          ``None`` denotes auto tune best value (current implementation when auto tune,
          it will always use 4 workers, but it may be improved in future)
          Default value ``None``.
      :param shuffling_queue_capacity: An int specifying the number of items to fill into a queue
          from which items are sampled each step to form batches. The larger the capacity, the
          better shuffling of the elements within the dataset. The default value of ``None``
          results in no shuffling.
      :param petastorm_reader_kwargs: arguments for `petastorm.make_batch_reader()`,
          exclude these arguments: ``dataset_url``, ``num_epochs``, ``workers_count``.
  
      :return: a context manager for a `tf.data.Dataset` object.
               when exit the returned context manager, the reader
               will be closed.
      """
      self._check_and_set_overriden_petastorm_args(
          petastorm_reader_kwargs,
          num_epochs=num_epochs,
          workers_count=workers_count,
      )
      return TFDatasetContextManager(
          self.file_urls,
          batch_size=batch_size,
          prefetch=prefetch,
          petastorm_reader_kwargs=petastorm_reader_kwargs,
          shuffling_queue_capacity=shuffling_queue_capacity,
      )
  
imports: import atexit
  import datetime
  import logging
  import os
  import shutil
  import threading
  import time
  import uuid
  from distutils.version import LooseVersion
  from multiprocessing.pool import ThreadPool
  from typing import List, Any
  import pyspark
  from pyarrow import LocalFileSystem
  from pyspark.sql.session import SparkSession
  from pyspark.sql.types import ArrayType, DoubleType, FloatType
  from six.moves.urllib.parse import urlparse
  from petastorm import make_batch_reader
  from petastorm.fs_utils import FilesystemResolver, get_filesystem_and_path_or_paths, normalize_dir_url
  from fsspec.core import strip_protocol
label_desc: Use image node_id 4 for calling the SparkDatasetConverter obj's underlying member method code with example usage: obj.make_tf_dataset(batch_size, prefetch, num_epochs, workers_count, shuffling_queue_capacity) and returns: TFDatasetContextManager