PyTorch model not converging
I'm training a binary classification model on a series of images.
The model was derived from resnet18 in torchvision and I made the last FC as nn.Linear(512, 1)
The loss function is BCELoss
However, the model doesn't show any sign of converging even after 5000 iterations.
I'm suspecting I might do something wrong in the training stage? But I can't find where's the bug.
Here's my code:
Model:
## Model
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models

resnet18 = models.resnet18(pretrained= True)
resnet18.fc = nn.Linear(512, 1)

Parameters, loss, optimizers:
## parameter
epochs = 200
learning_rate = 0.1
momen = 0.9
batch = 8
criterion = nn.BCELoss()
resnet18.to(device)
opt = optim.SGD(resnet18.parameters(), lr = learning_rate, momentum = momen)

Dataloaders:
# Generators
training_set = Dataset(X_train)
training_generator = torch.utils.data.DataLoader(training_set, batch_size= batch, shuffle=True)

validation_set = Dataset(X_test)
validation_generator = torch.utils.data.DataLoader(validation_set, batch_size=1, shuffle=False)

Training:
# training
history = []
for t in range(epochs):  
  for i, data in enumerate(training_generator, 0):

    inputs, labels = data
    # check if input size == batch size #
    if inputs.shape[0] &lt; batch:
      break

#    print(&quot;labels&quot;, labels, labels.dtype)

    # move data to GPU #
    inputs, labels = inputs.to(device), labels.to(device)

    opt.zero_grad()
    # Prediction #
    y_pred = resnet18(inputs).view(batch,)
    y_pred = (y_pred &gt; 0).float().requires_grad_()
#    print(&quot;y_pred&quot;, y_pred, y_pred.dtype)

    # Calculating loss #
    loss = criterion(y_pred, labels.view(batch,))
    loss.backward()
    opt.step()
    
    if i % 10 == 0:
      history.append(loss.item())
      print(&quot;Epoch: {}, iter: {}, loss: {}&quot;.format(t, i, loss.item())


torch.save(resnet18, 'trained_resnet18.pt')

Edit:
The loss values are like this:
Epoch: 3, iter: 310, loss: 0.0
Epoch: 3, iter: 320, loss: 37.5
Epoch: 3, iter: 330, loss: 37.5
Epoch: 3, iter: 340, loss: 0.0
Epoch: 3, iter: 350, loss: 37.5
Epoch: 3, iter: 360, loss: 50.0
Epoch: 3, iter: 370, loss: 37.5
Epoch: 3, iter: 380, loss: 25.0
Epoch: 3, iter: 390, loss: 12.5



I belive the error lies in the following line:
y_pred = (y_pred &gt; 0).float().requires_grad_()

You try to binarize the model prediction in a weird way, I suggest do the following instead:
y_pred = torch.sigmoid(y_pred)

And pass this to the loss function.
Explanation
The output of the model can be any value, but we want to normalize that values to reside in the [0,1] range. This is exactly what the sigmoid function does. Once we have the values in the range of [0,1] the comparison with the binary labels will make sense, closer to 1 will be &quot;1&quot; and the opposite.
You can refer to the following link: https://www.youtube.com/watch?v=WsFasV46KgQ
