Reproducibility in PyTorch with K-Fold Cross Validation
I have recently started a new project using PyTorch and I am still new in AI. In order to perform  better on my dataset during training process I used cross-validation technique. Everyone seems to work fine but I am struggling with reproducibility. I even tried to set SEED number for each k-fold iteration but it does not seem to work at all. Changes in loss and accuracy are insignificant but they are. Before using cross-validation everything worked perfect. Thank you in advance.
Here is a for loop for my k-fold. I used a solution from:
k-fold cross validation using DataLoaders in PyTorch
K_FOLD = 5
fraction = 1 / K_FOLD
unit = int(dataset_length * fraction)

for i in range(K_FOLD):


    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    torch.cuda.manual_seed_all(SEED)            # if you are using multi-GPU.
    np.random.seed(SEED)                        # Numpy module.
    random.seed(SEED)                           # Python random module.
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True


    print(&quot;-----------K-FOLD {}------------&quot;.format(i+1))
    tr_ll = 0
    print(&quot;Train left begin:&quot;, tr_ll)
    tr_lr = i * unit
    print(&quot;Train left end:&quot;, tr_lr)
    val_l = tr_lr
    print(&quot;Validation begin:&quot;, val_l)
    val_r = i * unit + unit
    print(&quot;Validation end:&quot;, val_r)
    tr_rl = val_r
    print(&quot;Train right begin:&quot;, tr_rl)
    tr_rr = dataset_length
    print(&quot;Train right end:&quot;, tr_rr)
    # msg
    #         print(&quot;train indices: [%d,%d),[%d,%d), test indices: [%d,%d)&quot;
    #               % (tr_ll,tr_lr,tr_rl,tr_rr,val_l,val_r))

    train_left_indices = list(range(tr_ll, tr_lr))
    train_right_indices = list(range(tr_rl, tr_rr))

    train_indices = train_left_indices + train_right_indices
    val_indices = list(range(val_l, val_r))
    # print(&quot;TRAIN Indices:&quot;, train_indices, &quot;VAL Indices:&quot;, val_indices)

    train_set = torch.utils.data.dataset.Subset(DATASET, train_indices)
    val_set = torch.utils.data.dataset.Subset(DATASET, val_indices)
    # print(&quot;Length of train set:&quot;, len(train_set), &quot;Length of val set:&quot;, len(val_set))
    
    image_datasets = {&quot;train&quot;: train_set, &quot;val&quot;: val_set}
    loader = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=10, shuffle=True)
                     for x in sets}
    dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}


    # training
    trained_model = train_model(AlexNet, CRITERION, OPTIMIZER,
                            dataloader=loader, dataset_sizes=dataset_sizes, num_epochs=EPOCHS, k_fold=i)


According to the latest docs, looks like you'll also need:
torch.use_deterministic_algorithms(True)

