Why is PyTorch's DataLoader not deterministic?
If you want to shuffle the data in a deterministic way, how about shuffling the dataset beforehand e.g. in a simple list of filenames, then simply reading that list deterministically in a single-processed loop, with shuffle = False in the DataLoader??

Another things that may cause non-deterministic behaviour is using multiple processes - then there are operations that are passed out and managed by the operating system, which doesn't pay attention to any of the random seeds you set. Performance is dependent on available resources i.e. affected by other activities running on your host machine.

In addition to that, any interaction between CPU and GPU could be causing non-deterministic behaviour, as data transfer is non-deterministic (related Nvidia thread). Data packets can be split differently every time, but there are apparent CUDA-level solutions in the pipeline.