Size of WebDataset in Pytorch
When it comes to the Pytorch Dataloader which takes a default dataset (e.g. datasets.ImageFolder), we can find the size of a dataset that is used by the dataloader with len(dataloader). However, what about WebDataset?
As WebDataset is a PyTorch Dataset, is it possible to get the size of a loader which takes a WebDataset?
https://webdataset.github.io/webdataset/

WebDataset doesn't provide a __len__ method, as it conforms to the PyTorch IterableDataset interface. IterableDataset is designed for stream-like data, and considers it wrong to have a len().
If you have code that depends on len() to be available, you can set the length to some value using with_length():
&gt;&gt;&gt; dataset = wds.WebDataset(url)
&gt;&gt;&gt; len(dataset)
Traceback (most recent call last):
  File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
TypeError: object of type 'WebDataset' has no len()
&gt;&gt;&gt; dataset = dataset.with_length(10)
&gt;&gt;&gt; len(dataset)
10

