Evaluate the model during training affects its performance PyTorch
In PyTorch, I want to evaluate my model on the validation set every eval_step during training, and I wrote code like this:
def tune(model, loader_train, loader_dev, optimizer, epochs, eval_step):
    for epoch in range(epochs):
        for step,x in enumerate(loader_train):
            optimizer.zero_grad()
            loss = model(x)
            loss.backward()
            optimizer.step()
            
            if step % eval_step == 0:
                model.eval()
                test(model, loader_dev)
                model.train()

When eval_step = int(len(loader_train)/2) and eval_step = int(len(loader_train)/8), they lead to quite different metric result after training through one whole epoch (which means the second output for the former differs the eighth output for the latter).
Could anyone explain why?

The length of loader_train is 20000 (it depends on batch size), and here is my testing script:
def test(model, loader_dev):
    preds = []
    labels = []
    for step,x in enumerate(loader_dev):
        preds.append(model(x).view(-1))
        labels.apend(x['label'].view(-1))

    metric = cal_metric(preds, labels)
    logger.info(metric)


I think you probably set 'shffule=True' in your dataloader. Even though you fix 'random seed', dataloader in torch will generate different results if you use another dataloader while using current dataloader. In the scenario you describe, it may cause your model get data input in different order and then result in different metric result.
