

Retrieve list of urls from each S3 bucket based on prefix

Support timeout to prevent hanging indefinitely
Support to specify S3 bucket region



Load data from S3 urls

Support buffered and multi-part download
Support to specify S3 bucket region



AWS native DataPipes are still in the beta phase. And, we will keep tuning them to improve their performance.
(Prototype) DataLoader2
DataLoader2 became available in prototype mode. We are introducing new ways to interact between DataPipes, DataLoading API, and backends (aka ReadingServices). Feature is stable in terms of API, but functionally not complete yet. We welcome early adopters and feedback, as well as potential contributors.
For more details, please checkout the link.
functorch