pytorch: model.forward() effects the training process even if the result is not part of the loss function
I have the following code to train a simple model:

train_loader, another_loader = get_loaders()
model = torch_utils.get_model()

ce = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
another_loader_iter = iter(another_loader)

for epoch in range(10):
    for i, (X, y) in enumerate(train_loader):
        model.train()
        X, y = X.to(device), y.to(device)
        pred = model.forward(X)
        loss1 = ce(pred, y)
        for j in range(2):
            X2, y2 = next(another_loader)
            pred2 = model.forward(X2) #**This line effects the training process - why???**

        new_loss = loss1
        optimizer.zero_grad()
        new_loss.backward()
        optimizer.step()


When I comment out pred2 = model.forward(X2) the model trains as it should but I want to use pred2 to add a new expression to the loss function.

I'd suggest the following changes:

for epoch in range(10):
    for i, (X, y) in enumerate(train_loader):
        model.train()
        X, y = X.to(device), y.to(device)
        pred = model(X)
        loss1 = ce(pred, y)
        for j in range(2):
            model.eval()
            X2, y2 = next(another_loader)
            pred2 = model(X2)

        new_loss = loss1
        optimizer.zero_grad()
        new_loss.backward()
        optimizer.step()


I'd also say you don't need new_loss = loss1, but think you're including more stuff here in your original code, right?

In any case, check if these changes help you.
