ValueError: no gopen handler defined
I am new to using webdataset library from pytorch. I have created .tar files of a sample dataset present locally in my system using webdataset.TarWriter(). The .tar files creation seems to be successful as I could extract them separately on windows platform and verify the same dataset files.
Now, I create train_dataset = wds.Dataset(url) where url is the local file path of the .tar files. after this, I perform the following operations:
train_loader = torch.utils.data.DataLoader(train_dataset, num_workers=0, batch_size=10)
sample = next(iter(train_loader))
print(sample)

It is resulting me in error like this
The same code works fine if I use a web url example: &quot;http://storage.googleapis.com/nvdata-openimages/openimages-train-000000.tar&quot; mentioned in webdataset documentation: https://reposhub.com/python/deep-learning/tmbdev-webdataset.html
I couldn't understand the error so far. Any idea on how to solve this problem?

I have had the same error since yesterday, I finally found the culprit. WebDataset/tarIterators.py makes use of WebDataset/gopen.py. In gopen.py urllib.parse.urlparse is called to parse the url to be opened, in your case the url is D:/PhD/....
gopen_schemes = dict(
    __default__=gopen_error,
    pipe=gopen_pipe,
    http=gopen_curl,
    https=gopen_curl,
    sftp=gopen_curl,
    ftps=gopen_curl,
    scp=gopen_curl)

def gopen(url, mode=&quot;rb&quot;, bufsize=8192, **kw):
    &quot;&quot;&quot;Open the URL. This uses the `gopen_schemes` dispatch table to dispatch based on scheme.
    Support for the following schemes is built-in: pipe, file, http, https, sftp, ftps, scp.
    When no scheme is given the url is treated as a file. You can use the OPEN_VERBOSE argument to get info about files being opened.
    :param url: the source URL
    :param mode: the mode (&quot;rb&quot;, &quot;r&quot;)
    :param bufsize: the buffer size
    &quot;&quot;&quot;
    global fallback_gopen
    verbose = int(os.environ.get(&quot;GOPEN_VERBOSE&quot;, 0))
    if verbose:
        print(&quot;GOPEN&quot;, url, info, file=sys.stderr)
    assert mode in [&quot;rb&quot;, &quot;wb&quot;], mode
    if url == &quot;-&quot;:
        if mode == &quot;rb&quot;:
            return sys.stdin.buffer
        elif mode == &quot;wb&quot;:
            return sys.stdout.buffer
        else:
            raise ValueError(f&quot;unknown mode {mode}&quot;)
    pr = urlparse(url)
    if pr.scheme == &quot;&quot;:
        bufsize = int(os.environ.get(&quot;GOPEN_BUFFER&quot;, -1))
        return open(url, mode, buffering=bufsize)
    if pr.scheme == &quot;file&quot;:
        bufsize = int(os.environ.get(&quot;GOPEN_BUFFER&quot;, -1))
        return open(pr.path, mode, buffering=bufsize)
    handler = gopen_schemes[&quot;__default__&quot;]
    handler = gopen_schemes.get(pr.scheme, handler)
    return handler(url, mode, bufsize, **kw)

As you can see in the dictionary the __default__ function is gopen_error. This is the function returning the error you are seeing. pr = urlparse(url) on your url will generate an urlparse where the scheme (pr.scheme) is 'd' because your disk is named D. However, it should be 'file' for the function to work as intended. Since it is not equal to 'file' or any of the other schemes in the dictionary (http, https, sftp, etc), the default function will be used, which returns the error. I circumvented this issue by adding d=gopen_file to the gopen_schemes dictionary. I hope this helps you further temporarily. I will address this issue on the WebDataset GitHub page as well and keep this page updated if I get a more practical update.
Good luck!
