Why did I get 2 different results from two models with same parameters and inputs?
I loaded resnet18 into my two models (model1 and model2), with pretrained weights.

I want to use them as feature extractors
For model1: I freezed the parameters except the last linear layer model1.fc, then train it. After training, I set model1.fc into torch.nn.Identity()
For model2: I directly set model2.fc into torch.nn.Identity()

Then these 2 models should be the same, but I get different forward result from the same inputs.

If the training of model1 is not done, they will have the same result, maybe something wrong with the freezing of  parameter.
However, I checked their parameters after the training of model1 and setting the last layer of both models to identity layer, and they seems to be the same.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms, models

# Load weights pretrained on ImageNet
def load_weights(model):
    model_dir = "....."
    model.load_state_dict(tor`enter code here`ch.utils.model_zoo.load_url("https://download.pytorch.org/models/resnet18-5c106cde.pth", model_dir=model_dir))
    return model

model1=models.resnet18()
model1=load_weights(model1)
for param in model1.parameters():
    param.requires_grad = False
model1.fc=nn.Linear(512, 2)
model1.cuda()
optimizer = optim.SGD(model1.fc.parameters(), lr=1e-2, momentum=0.9)
result_freeze = \
    run_training(model1, optimizer, device, train_loader, val_loader,num_epochs=10)

model2=models.resnet18()
model2=load_weights(model2)
model2.fc=nn.Identity()
model2.cuda()

model1.fc=nn.Identity()
model1.cuda()



# checking forward results(extracting features) 

# The batch size is one here
for batch_idx, (data, target) in enumerate(X_train): 
        data, target = data.to(device), target.to(device)
        d=data
        X_train_feature[batch_idx]=model1(data).cpu().detach().numpy()
        y_train[batch_idx]=target.cpu().detach().numpy()
        X_train2_feature[batch_idx]=model2(d).cpu().detach().numpy()
        y_train2[batch_idx]=target.cpu().detach().numpy()
        print(sum(X_train_feature[batch_idx]==X_train2_feature[batch_idx]))
        print(sum(y_train[batch_idx]==y_train2[batch_idx]))
        print(torch.sum(d==data))
for batch_idx, (data, target) in enumerate(X_test):
        data, target = data.to(device), target.to(device)
        d=data
        X_test_feature[batch_idx]=model1(data).cpu().detach().numpy()
        y_test[batch_idx]=target.cpu().detach().numpy()
        X_test2_feature[batch_idx]=model2(d).cpu().detach().numpy()
        y_test2[batch_idx]=target.cpu().detach().numpy()
        print(sum(X_test_feature[batch_idx]==X_test2_feature[batch_idx]))
        print(sum(y_test[batch_idx]==y_test2[batch_idx]))
        print(torch.sum(d==data))

# checking parameters
for a,b in zip(model1.parameters(),model2.parameters()):
    print(torch.sum(a!=b)) 


Expect to get the same forward results from model1 and model2, but they are different. And If they produce different forward results, why do they have exactly the same parameters?

Have you taken into account changes that might occur to BatchNorm layers?
Batch norm layers do not behave like normal layers - their internal parameters are modified by computing running mean and std of the data, and not by gradient descent.
Try setting model1.eval() before the finetune and then check.
