RuntimeError: PytorchStreamReader failed locating file data.pkl: file not found
I have been trying to train some data using a model that utilizes src+img. When running the training script, I'm running into an error namely: RuntimeError: PytorchStreamReader failed locating file data.pkl: file not found
the .pkl file in this should have been the pickled .pt file.
First I figured that maybe I did not specify the preprocessed training.pt file correctly, but it is actually correctly specified.
data.pkl hasn't been dumped anywhere (or I failed to find it). I am guessing it has something do with pickle accordimg to docs:
&quot;This save/load process uses the most intuitive syntax and involves the least amount of code. Saving a model in this way will save the entire module using Python’s pickle module. The disadvantage of this approach is that the serialized data is bound to the specific classes and the exact directory structure used when the model is saved. The reason for this is because pickle does not save the model class itself. Rather, it saves a path to the file containing the class, which is used during load time. Because of this, your code can break in various ways when used in other projects or after refactors.&quot;
I've tried a multitude of things, like changing data.pkl in the script in order to see if there was a generated corrupted file, but this seems to not be the case.
I would appreciate it if anyone is willing to help:
full error:
  File &quot;train_mm.py&quot;, line 448, in &lt;module&gt;
    main()
  File &quot;train_mm.py&quot;, line 423, in main
    first_dataset = next(lazily_load_dataset(&quot;train&quot;))
  File &quot;train_mm.py&quot;, line 314, in lazily_load_dataset
    yield lazy_dataset_loader(pt, corpus_type)
  File &quot;train_mm.py&quot;, line 305, in lazy_dataset_loader
    dataset = torch.load(pt_file)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/serialization.py&quot;, line 607, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File &quot;/usr/local/lib/python3.7/dist-packages/torch/serialization.py&quot;, line 878, in _load
    data_file = io.BytesIO(zip_file.get_record(pickle_file))
RuntimeError: PytorchStreamReader failed locating file data.pkl: file not found

solved: I re-ran the preprocessing script and it generated new .pt files and the error was resolved

This issue has been resolved. the .pt file was heavily corrupted. After deleting the corrupt .pt file and re-running the preprocess script and consequently training script, I did not get the error anymore.
