PyTorch: Dropout (?) causes different model convergence for training+validation V. training-only
We are facing a very strange issue. We tested the exact same model into two different “execution” settings. In the first case, given a certain amount of epochs, we train using mini-batches for one epoch, and thereafter we test on the validation set following the same criteria. Then, we go for the next epoch. Clearly, before each training epoch, we use model.train(), and before validation we turn on model.eval().
Then we take the exact same model (same init, same dataset, same epochs, etc.) and we just train it without validation after each epoch.
Just looking at performance on training set, we observed that, even if we fixed all seeds, the two training procedures evolve differently and produce quite different metrics results (losses, accuracy, and so on). Specifically, the training-only procedure is less performing.
We also observe the following things:

It is not a reproducibility issue, because multiple executions of the
same procedure produce exactly the same results (and this is
intended);
Removing the dropout, it appears that the problem vanishes;
Batchnorm1d layer, that still has different behaviours between
training and evaluation, seems to work properly;
The issue still happens if we move from training onto TPUs to CPUs.
We are working and tried Pythorch 1.6, Pythorch nightly, XLA 1.6.

We quite lost one full day in trying to tackle this issue (and no, we cannot avoid using dropout). Does anyone have any idea about how to solve this fact?
Thank you very much!
p.s. Here the code employed for the training (on CPU).
def sigmoid(x):
    return 1 / (1 + torch.exp(-x))


def _run(model, EPOCHS, training_data_in, validation_data_in=None):
    
    def train_fn(train_dataloader, model, optimizer, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.
        
        model.train()

        for batch_idx, (ecg, spo2, labels) in enumerate(train_dataloader, 1):

            optimizer.zero_grad() 
                
            outputs = model(ecg)

            loss = criterion(outputs, labels)
                        
            loss.backward() # calculate the gradients
            torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
            optimizer.step() # update the network weights
                                                
            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model
            
            running_accuracy += (predicted == labels).sum().item() / labels.size(0)   
            
            fp = ((predicted - labels) == 1.).sum().item() 
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn
            
        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }
            
        return retval
            

        
    def valid_fn(valid_dataloader, model, criterion):

        running_loss = 0.
        running_accuracy = 0.
        running_tp = 0.
        running_tn = 0.
        running_fp = 0.
        running_fn = 0.

        model.eval()
        
        for batch_idx, (ecg, spo2, labels) in enumerate(valid_dataloader, 1):

            outputs = model(ecg)

            loss = criterion(outputs, labels)
            
            running_loss += loss.item()
            predicted = torch.round(sigmoid(outputs.data)) # here determining the sigmoid, not included in the model

            running_accuracy += (predicted == labels).sum().item() / labels.size(0)  
            
            fp = ((predicted - labels) == 1.).sum().item()
            fn = ((predicted - labels) == -1.).sum().item()
            tp = ((predicted + labels) == 2.).sum().item()
            tn = ((predicted + labels) == 0.).sum().item()
            running_tp += tp
            running_fp += fp
            running_tn += tn
            running_fn += fn
            
        retval = {'loss':running_loss / batch_idx,
                'accuracy':running_accuracy / batch_idx,
                'tp':running_tp,
                'tn':running_tn,
                'fp':running_fp,
                'fn':running_fn
                }
            
        return retval
    
    
    
    # Defining data loaders

    train_dataloader = torch.utils.data.DataLoader(training_data_in, batch_size=BATCH_SIZE, shuffle=True, num_workers=1)
    
    if validation_data_in != None:
        validation_dataloader = torch.utils.data.DataLoader(validation_data_in, batch_size=BATCH_SIZE, shuffle=False, num_workers=1)


    # Defining the loss function
    criterion = nn.BCEWithLogitsLoss()
    
    
    # Defining the optimizer
    import torch.optim as optim
    optimizer = optim.AdamW(model.parameters(), lr=3e-4, amsgrad=False, eps=1e-07) 


    # Training code
    
    metrics_history = {&quot;loss&quot;:[], &quot;accuracy&quot;:[], &quot;precision&quot;:[], &quot;recall&quot;:[], &quot;f1&quot;:[], &quot;specificity&quot;:[], &quot;accuracy_bis&quot;:[], &quot;tp&quot;:[], &quot;tn&quot;:[], &quot;fp&quot;:[], &quot;fn&quot;:[],
                    &quot;val_loss&quot;:[], &quot;val_accuracy&quot;:[], &quot;val_precision&quot;:[], &quot;val_recall&quot;:[], &quot;val_f1&quot;:[], &quot;val_specificity&quot;:[], &quot;val_accuracy_bis&quot;:[], &quot;val_tp&quot;:[], &quot;val_tn&quot;:[], &quot;val_fp&quot;:[], &quot;val_fn&quot;:[],}
    
    train_begin = time.time()
    for epoch in range(EPOCHS):
        start = time.time()

        print(&quot;EPOCH:&quot;, epoch+1)

        train_metrics = train_fn(train_dataloader=train_dataloader, 
                                model=model,
                                optimizer=optimizer, 
                                criterion=criterion)
        
        metrics_history[&quot;loss&quot;].append(train_metrics[&quot;loss&quot;])
        metrics_history[&quot;accuracy&quot;].append(train_metrics[&quot;accuracy&quot;])
        metrics_history[&quot;tp&quot;].append(train_metrics[&quot;tp&quot;])
        metrics_history[&quot;tn&quot;].append(train_metrics[&quot;tn&quot;])
        metrics_history[&quot;fp&quot;].append(train_metrics[&quot;fp&quot;])
        metrics_history[&quot;fn&quot;].append(train_metrics[&quot;fn&quot;])
        
        precision = train_metrics[&quot;tp&quot;] / (train_metrics[&quot;tp&quot;] + train_metrics[&quot;fp&quot;]) if train_metrics[&quot;tp&quot;] &gt; 0 else 0
        recall = train_metrics[&quot;tp&quot;] / (train_metrics[&quot;tp&quot;] + train_metrics[&quot;fn&quot;]) if train_metrics[&quot;tp&quot;] &gt; 0 else 0
        specificity = train_metrics[&quot;tn&quot;] / (train_metrics[&quot;tn&quot;] + train_metrics[&quot;fp&quot;]) if train_metrics[&quot;tn&quot;] &gt; 0 else 0
        f1 = 2*precision*recall / (precision + recall) if precision*recall &gt; 0 else 0
        metrics_history[&quot;precision&quot;].append(precision)
        metrics_history[&quot;recall&quot;].append(recall)
        metrics_history[&quot;f1&quot;].append(f1)
        metrics_history[&quot;specificity&quot;].append(specificity)
        
        
        
        if validation_data_in != None:    
            # Calculate the metrics on the validation data, in the same way as done for training
            with torch.no_grad(): # don't keep track of the info necessary to calculate the gradients

                val_metrics = valid_fn(valid_dataloader=validation_dataloader, 
                                    model=model,
                                    criterion=criterion)

                metrics_history[&quot;val_loss&quot;].append(val_metrics[&quot;loss&quot;])
                metrics_history[&quot;val_accuracy&quot;].append(val_metrics[&quot;accuracy&quot;])
                metrics_history[&quot;val_tp&quot;].append(val_metrics[&quot;tp&quot;])
                metrics_history[&quot;val_tn&quot;].append(val_metrics[&quot;tn&quot;])
                metrics_history[&quot;val_fp&quot;].append(val_metrics[&quot;fp&quot;])
                metrics_history[&quot;val_fn&quot;].append(val_metrics[&quot;fn&quot;])

                val_precision = val_metrics[&quot;tp&quot;] / (val_metrics[&quot;tp&quot;] + val_metrics[&quot;fp&quot;]) if val_metrics[&quot;tp&quot;] &gt; 0 else 0
                val_recall = val_metrics[&quot;tp&quot;] / (val_metrics[&quot;tp&quot;] + val_metrics[&quot;fn&quot;]) if val_metrics[&quot;tp&quot;] &gt; 0 else 0
                val_specificity = val_metrics[&quot;tn&quot;] / (val_metrics[&quot;tn&quot;] + val_metrics[&quot;fp&quot;]) if val_metrics[&quot;tn&quot;] &gt; 0 else 0
                val_f1 = 2*val_precision*val_recall / (val_precision + val_recall) if val_precision*val_recall &gt; 0 else 0
                metrics_history[&quot;val_precision&quot;].append(val_precision)
                metrics_history[&quot;val_recall&quot;].append(val_recall)
                metrics_history[&quot;val_f1&quot;].append(val_f1)
                metrics_history[&quot;val_specificity&quot;].append(val_specificity)


            print(&quot;  &gt; Training/validation loss:&quot;, round(train_metrics['loss'], 4), round(val_metrics['loss'], 4))
            print(&quot;  &gt; Training/validation accuracy:&quot;, round(train_metrics['accuracy'], 4), round(val_metrics['accuracy'], 4))
            print(&quot;  &gt; Training/validation precision:&quot;, round(precision, 4), round(val_precision, 4))
            print(&quot;  &gt; Training/validation recall:&quot;, round(recall, 4), round(val_recall, 4))
            print(&quot;  &gt; Training/validation f1:&quot;, round(f1, 4), round(val_f1, 4))
            print(&quot;  &gt; Training/validation specificity:&quot;, round(specificity, 4), round(val_specificity, 4))
        else:
            print(&quot;  &gt; Training loss:&quot;, round(train_metrics['loss'], 4))
            print(&quot;  &gt; Training accuracy:&quot;, round(train_metrics['accuracy'], 4))
            print(&quot;  &gt; Training precision:&quot;, round(precision, 4))
            print(&quot;  &gt; Training recall:&quot;, round(recall, 4))
            print(&quot;  &gt; Training f1:&quot;, round(f1, 4))
            print(&quot;  &gt; Training specificity:&quot;, round(specificity, 4))


        print(&quot;Completed in:&quot;, round(time.time() - start, 1), &quot;seconds \n&quot;)

    print(&quot;Training completed in:&quot;, round((time.time()- train_begin)/60, 1), &quot;minutes&quot;)    

    
    
    # Save the model weights
    torch.save(model.state_dict(), './nnet_model.pt')
    
    
    # Save the metrics history
    torch.save(metrics_history, 'training_history')

And here is the function that initializes the model and set the seeds, called before each execution of the code of &quot;_run&quot;:
def reinit_model():
    torch.manual_seed(42)
    np.random.seed(42)
    random.seed(42)
    net = Net() # the model
    return net


Ok, I found the issue.
The problem is determined by the fact that, apparently, running the evaluation some random seeds are changed, and this affects the training phase.
The solution is thus as follows:

at the beginning of function &quot;_run()&quot;, set all seeds states to the desired value, e.g., 42. Then, save those seeds to disk.
at the beginning of function &quot;train_fn()&quot;, read the seeds states from disk, and set them
at the end of function &quot;train_fn()&quot;, save the seeds states to disk

For instance, running on TPU with XLA, the following instructions have to be used:

at the beginning of function &quot;_run()&quot;: xm.set_rng_state(42), xm.save(xm.get_rng_state(), 'xm_seed')
at the beginning of function &quot;train_fn()&quot;: xm.set_rng_state(torch.load('xm_seed'), device=device)  (you can also print here the seed for verification purposes with xm.master_print(xm.get_rng_state())
at the end of function &quot;train_fn_()&quot;: xm.save(xm.get_rng_state(), 'xm_seed')

