PyTorch: defined layer was not involved in the forward propagation but influenced the loss value
Recently, I met a confusing phenomenon when I used Pytorch to do simple experiments on logistic regression.
The question is when I fixed the random seed like this:
def set_seed(seed, cuda=True):
    np.random.seed(seed)
    torch.manual_seed(seed)
    if cuda:
        torch.cuda.manual_seed(seed)

and defined the following model with 2 layers:
class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)

    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x

trained the network with:
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

The orignal loss value was 0.6422, which is reproducible.
However, when I added an additional layer that was not involoved in the forward process like this:
class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)
        self.add = nn.Linear(10,10)

    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x


The original loss value changed to 0.7431, which was not equal to the previous one and the model performance dropped simultaneously.
I really wonder the reason to this. Thank you!

This is completely expected if there are other sources of randomness (something that consumes the RNG) before computing the loss. As you didn't provide a Minimal, Reproducible Example, I'd guess that you're using a DataLoader with shuffle=True. In this case, even though you do not use the self.add layer, when you initialize it, it consumes the RNG; therefore leading to a different order to the samples. If the randomness is coming from a DataLoader with shuffle=True, you can control that by providing a different RNG to the DataLoader. Something like this:
import numpy as np

import torch
from torch import nn

import torchvision
from torchvision.transforms import ToTensor

def set_seed(seed, cuda=True):
    np.random.seed(seed)
    torch.manual_seed(seed)
    if cuda:
        torch.cuda.manual_seed(seed)

class net(nn.Module):
    def __init__(self):
        super(net, self).__init__()
        self.hidden = nn.Linear(784, 100)
        self.output = nn.Linear(100, 10)
        # self.add = nn.Linear(10, 10)  # try with and without
    def forward(self, x):
        x = self.hidden(x)
        x = self.output(x)
        return x

set_seed(0)

m = net()

bs = 4
ds = torchvision.datasets.MNIST(root=&quot;.&quot;, train=True, transform=ToTensor(), download=True)
rng_dl = torch.Generator()
dl = torch.utils.data.DataLoader(ds, batch_size=bs, shuffle=True, num_workers=0, generator=rng_dl)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(m.parameters(), lr=0.1)

for x, y in dl:
    y_hat = m(x.view(bs, -1))
    l = criterion(y_hat, y)
    print(l)
    exit()

Keep in mind that it could be several other things, such as data augmentation and other calls to functions that rely on random ops. If you can provide an MRE, I could try and give a more specific answer.
