Why would Pytorch be non-deterministic for batch only? I am training a LSTM network on CPU and can achieve deterministic results when not using a dataloader. But when I used Pytorchs dataloader I achieve non-deterministic training error results, despite the actual batches being loaded from the dataloader being deterministic.

I have set pretty much every seed I could think of here

random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)

torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False  
torch.backends.cudnn.enabled   = False


The two pieces of code are:

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    for batch_idx, (X_train, y_train) in enumerate(train_loader):

        # Zero out gradients
        optimiser.zero_grad()

        # Forward pass
        y_pred = model_test(X_train)

        # Loss Function
        loss = loss_fn(y_pred, y_train)

        # Backward pass
        loss.backward()

        # Update parameters
        optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())


and

optimiser = torch.optim.Adam(model_test.parameters(), learning_rate)
set_seed(42)

for t in range(num_epochs):

    # Zero out gradients
    optimiser.zero_grad()

    # Forward pass
    y_pred = model_test(X_train)

    # Loss Function
    loss = loss_fn(y_pred, y_train)

    # Backward pass
    loss.backward()

    # Update parameters
    optimiser.step()

    if t % 100 == 0: print("Epoch ", t, "MSE: ", loss.item())


I have seen some posts on Github talking about there being issues with determinism on GPU, but this is just on CPU.
The cudnn implementaton of the LSTM has determinism issues that appear to be fixed in the 7.6.1 release. Check your cudnn version.

https://github.com/pytorch/pytorch/issues/18110