CNN Classifier only guesses one thing - PyTorch
I'm trying to make a model predict the race of a 75x75 image's ethnicity, but when ever I train the model, the accuracy always stays completely still at 53.2%. I didn't realize why until I actually ran it on some of photos. It turned out, that no matter what the photo was, it would always predict 'other'. I'm not entirely sure why though.
I copied the code over from the official PyTorch Quickstart tutorial, and in that dataset or the standard MNIST data, it worked fine. I changed the dataset to the UTKFace, and then it started only predicting one label, all the time.
Here's my code:
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import ToTensor
import torch.nn.functional as F

training_data = ImageFolder(
    root = &quot;data_training/&quot;,
    transform = ToTensor(),
)

testing_data = ImageFolder(
    root = &quot;data_testing/&quot;,
    transform = ToTensor()
)

training_dataloader = DataLoader(training_data, batch_size=64, shuffle=True)
test_dataloader = DataLoader(testing_data, batch_size=64, shuffle=True)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        
        self.fc1 = nn.Linear(1296, 1024)
        self.fc2 = nn.Linear(1024, 1024)
        self.fc3 = nn.Linear(1024, 512)
        self.fc4 = nn.Linear(512, 84)
        self.fc5 = nn.Linear(84, 5)
        
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        
        
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.relu(self.fc4(x))
        
        x = self.fc5(x)
        return x
    
model = NeuralNetwork().to(&quot;cpu&quot;)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        X, y = X.to(&quot;cpu&quot;), y.to(&quot;cpu&quot;)
        # Compute prediction error
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f&quot;loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]&quot;)

def tests(dataloader, model):
    size = len(dataloader.dataset)
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(&quot;cpu&quot;), y.to(&quot;cpu&quot;)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
    test_loss /= size
    correct /= size
    print(f&quot;Test Error: \n Accuracy: {(100*correct):&gt;0.1f}%, Avg loss: {test_loss:&gt;8f} \n&quot;)

epochs = 10
for t in range(epochs):
    print(f&quot;Epoch {t+1}\n-------------------------------&quot;)
    train(training_dataloader, model, loss_fn, optimizer)
    tests(test_dataloader, model)

torch.save(model.state_dict(), &quot;model.pth&quot;)

The training logs:
Epoch 1
-------------------------------
loss: 1.628994  [    0/23705]
loss: 1.620698  [ 6400/23705]
loss: 1.615423  [12800/23705]
loss: 1.596390  [19200/23705]
Test Error: 
 Accuracy: 53.2%, Avg loss: 0.024725 

Epoch 2
-------------------------------
loss: 1.593613  [    0/23705]
loss: 1.581375  [ 6400/23705]
loss: 1.583656  [12800/23705]
loss: 1.591942  [19200/23705]
Test Error: 
 Accuracy: 53.2%, Avg loss: 0.024165 

Epoch 3
-------------------------------
loss: 1.541260  [    0/23705]
loss: 1.592345  [ 6400/23705]
loss: 1.540908  [12800/23705]
loss: 1.540741  [19200/23705]
Test Error: 
 Accuracy: 53.2%, Avg loss: 0.023705 

Epoch 4
-------------------------------
loss: 1.566888  [    0/23705]
loss: 1.524875  [ 6400/23705]
loss: 1.540764  [12800/23705]
loss: 1.510044  [19200/23705]
Test Error: 
 Accuracy: 53.2%, Avg loss: 0.023323 

Epoch 5
-------------------------------
loss: 1.530084  [    0/23705]
loss: 1.498773  [ 6400/23705]
loss: 1.537755  [12800/23705]
loss: 1.508989  [19200/23705]
Test Error: 
 Accuracy: 53.2%, Avg loss: 0.022993 
....

No matter how many epochs I set it to, or how many layers I add in to try to get it to overfit, it always just seems to guess the same thing over and over again, with no signs of improvement.
I separated the UTKFace dataset into folders based on the ethnicity category of the name. There are 23705 images in the training data and 10134 in the testing.
I'm not sure why this is happening. Is my dataset not large enough? Are there not enough layers?

The number of layers and the dataset size don't explain this behavior for this example. Your CNN is behaving as a constant function, so far I don't know why, but these might be some clues:

Since you have separated your data by label into folders, if you are training your model using only one of those folders you will obtain a constant function.
The last layer of your neural network has no activation function! This is, in method forward you are doing x = self.fc5(x) instead of x = F.&lt;function&gt;(self.fc5(x)).
Where do you indicate, when loading the training data, which is the label for each image? Are you sure that training_dataloader is loading the images with their correct label?

