python/ray/data/dataset_pipeline.py
def split(
        self, n: int, *, equal: bool = False, locality_hints: List[Any] = None
    ) -> List["DatasetPipeline[T]"]:
        """Split the pipeline into ``n`` disjoint pipeline shards.

        This returns a list of sub-pipelines that can be passed to Ray tasks
        and actors and used to read the pipeline records in parallel.

        Examples:
            >>> import ray
            >>> pipe = ray.data.range(10).repeat(50) # doctest: +SKIP
            >>> workers = ... # doctest: +SKIP
            >>> # Split up a pipeline to process over `n` worker actors.
            >>> shards = pipe.split( # doctest: +SKIP
            ...     len(workers), locality_hints=workers)
            >>> for shard, worker in zip(shards, workers): # doctest: +SKIP
            ...     worker.consume.remote(shard) # doctest: +SKIP

        Time complexity: O(1)

        Implementation detail: this launches a coordinator actor that is used
        to execute the pipeline and push data blocks to each pipeline shard.
        Reading from an individual shard will be blocked if other shards are
        falling behind. A warning will be printed if a shard has been blocked
        on read for more than 10 seconds.

        Args:
            n: Number of child pipelines to return.
            equal: Whether to guarantee each split has an equal
                number of records. This may drop records if they cannot be
                divided equally among the splits.
            locality_hints: A list of Ray actor handles of size ``n``. The
                system will try to co-locate the blocks of the ith pipeline
                shard with the ith actor to maximize data locality.

        Returns:
            A list of ``n`` disjoint pipeline splits.
        """
        return self._split(
            n,
            lambda ds, equal=equal: ds.split(
                n, equal=equal, locality_hints=locality_hints
            ),
        )
Split the pipeline into ``n`` disjoint pipeline shards.

        This returns a list of sub-pipelines that can be passed to Ray tasks
        and actors and used to read the pipeline records in parallel.

        Examples:
            >>> import ray
            >>> pipe = ray.data.range(10).repeat(50) # doctest: +SKIP
            >>> workers = ... # doctest: +SKIP
            >>> # Split up a pipeline to process over `n` worker actors.
            >>> shards = pipe.split( # doctest: +SKIP
            ...     len(workers), locality_hints=workers)
            >>> for shard, worker in zip(shards, workers): # doctest: +SKIP
            ...     worker.consume.remote(shard) # doctest: +SKIP

        Time complexity: O(1)

        Implementation detail: this launches a coordinator actor that is used
        to execute the pipeline and push data blocks to each pipeline shard.
        Reading from an individual shard will be blocked if other shards are
        falling behind. A warning will be printed if a shard has been blocked
        on read for more than 10 seconds.

        Args:
            n: Number of child pipelines to return.
            equal: Whether to guarantee each split has an equal
                number of records. This may drop records if they cannot be
                divided equally among the splits.
            locality_hints: A list of Ray actor handles of size ``n``. The
                system will try to co-locate the blocks of the ith pipeline
                shard with the ith actor to maximize data locality.

        Returns:
            A list of ``n`` disjoint pipeline splits.
        
Split the pipeline into ``n`` disjoint pipeline shards.
This returns a list of sub-pipelines that can be passed to Ray tasks
and actors and used to read the pipeline records in parallel.
Split the pipeline into ` ` n ` ` disjoint pipeline shards . This returns a list of sub - pipelines that can be passed to Ray tasks and actors and used to read the pipeline records in parallel .
def split(
        self, n: int, *, equal: bool = False, locality_hints: List[Any] = None
    ) -> List["DatasetPipeline[T]"]:
        return self._split(
            n,
            lambda ds, equal=equal: ds.split(
                n, equal=equal, locality_hints=locality_hints
            ),
        )
def split ( self , n : int , * , equal : bool = False , locality_hints : List [ Any ] = None ) -> List [ "DatasetPipeline[T]" ] : return self . _split ( n , lambda ds , equal = equal : ds . split ( n , equal = equal , locality_hints = locality_hints ) , )
Split the pipeline into ``n`` disjoint pipeline shards.
Split the pipeline into ` ` n ` ` disjoint pipeline shards .
"""Split the pipeline into ``n`` disjoint pipeline shards.

        This returns a list of sub-pipelines that can be passed to Ray tasks
        and actors and used to read the pipeline records in parallel.

        Examples:
            >>> import ray
            >>> pipe = ray.data.range(10).repeat(50) # doctest: +SKIP
            >>> workers = ... # doctest: +SKIP
            >>> # Split up a pipeline to process over `n` worker actors.
            >>> shards = pipe.split( # doctest: +SKIP
            ...     len(workers), locality_hints=workers)
            >>> for shard, worker in zip(shards, workers): # doctest: +SKIP
            ...     worker.consume.remote(shard) # doctest: +SKIP

        Time complexity: O(1)

        Implementation detail: this launches a coordinator actor that is used
        to execute the pipeline and push data blocks to each pipeline shard.
        Reading from an individual shard will be blocked if other shards are
        falling behind. A warning will be printed if a shard has been blocked
        on read for more than 10 seconds.

        Args:
            n: Number of child pipelines to return.
            equal: Whether to guarantee each split has an equal
                number of records. This may drop records if they cannot be
                divided equally among the splits.
            locality_hints: A list of Ray actor handles of size ``n``. The
                system will try to co-locate the blocks of the ith pipeline
                shard with the ith actor to maximize data locality.

        Returns:
            A list of ``n`` disjoint pipeline splits.
        """
A list of ``n`` disjoint pipeline splits.
A list of ` ` n ` ` disjoint pipeline splits .
self
n
int
Number of child pipelines to return.
Number of child pipelines to return .
equal
bool
Whether to guarantee each split has an equal
number of records. This may drop records if they cannot be
divided equally among the splits.
Whether to guarantee each split has an equal number of records . This may drop records if they cannot be divided equally among the splits .
locality_hints
List[Any]
A list of Ray actor handles of size ``n``. The
system will try to co-locate the blocks of the ith pipeline
shard with the ith actor to maximize data locality.
A list of Ray actor handles of size ` ` n ` ` . The system will try to co - locate the blocks of the ith pipeline shard with the ith actor to maximize data locality .
examples
None