Why network with linear layers can't learn anything
I am trying to build a neural network for binary classification, unfortunately it always predicts the value 0, even though one fifth of the training set data is 1. I have no idea why it is so. My dataset looks as this, so there are a couple of categorical variables and a couple of continuous, (target is the one we predict):

Here one can download the data:
https://drive.google.com/drive/folders/1PsG2rRdbxyocyqvLSa7zSy_aVDMRJ2Ug?usp=sharing
You can read it with
df = pd.read_csv(&quot;train.csv&quot;, index_col=0)

Now I am preparing the data for neural network:
x_train=df.drop(labels=['target'], axis=1).values
y_train=df['target'].values
X_train, X_val, y_train, y_val = train_test_split(
x_train,
y_train,
test_size=0.2)

LR = 0.001 
EPOCH = 50
BATCH_SIZE = 64

torch_X_train = torch.tensor(X_train)
torch_y_train = torch.tensor(y_train)

torch_X_val = torch.tensor(X_val)
torch_y_val = torch.tensor(y_val)

train = torch.utils.data.TensorDataset(torch_X_train,torch_y_train)
validate = torch.utils.data.TensorDataset(torch_X_val,torch_y_val)

train_loader = torch.utils.data.DataLoader(train, batch_size = 
BATCH_SIZE, shuffle = True)
val_loader = torch.utils.data.DataLoader(validate, batch_size = 
BATCH_SIZE, shuffle = False)

And define a simple network with just linear layers:
class NN(nn.Module):
def __init__(self, input_size):
    super().__init__()
    self.to_class=nn.Sequential(
    nn.Linear(input_size,512),
    nn.ReLU(),
    nn.Linear(512,256),
    nn.ReLU(),
    nn.Linear(256,32),
    nn.ReLU(),
    nn.Linear(32,2)
    )
def forward(self,inputs):
    pred= self.to_class(inputs)
    return F.softmax(pred, dim=1)

And lastly I train it
net=NN(7)


loss_func = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=LR)

train_loss = np.zeros(EPOCH)
val_loss = np.zeros(EPOCH)
acc_train = []
acc_val=[]

for epoch in range(EPOCH):
    correct = 0
    total = 0
    for data in train_loader:
        X, y = data
        optimizer.zero_grad()
        output=net(X.float())
        total += output.size(0)
        pred = output.argmax(dim=1, keepdim=True)
        correct += pred.eq(y.view_as(pred)).sum().item()
        loss = loss_func(output.squeeze(), y) 
        loss.backward()                     
        optimizer.step() 
        train_loss[epoch] += loss
    train_loss[epoch] /= len(train_loader)
    acc_train.append(correct/total*100)
print('epoch %d:\t train_accuracy %.5f\ttrain loss: %.5f'%(epoch,acc_train[epoch], train_loss[epoch]))

But the train loss is not going anywhere and the predictions are always one class! Can someone explain this phenomenon and hint how I can improve it?


Please see the doc
This is the standard training loop
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

It seems you are missing optimizer.zero_grad().
Please try to add it and see how it does.

If this doesn't work, please use the same net on a different dataset, for example MNIST, directly downloaded  from pytorch. This net should be able to solve MNIST, so this way we can debug if the problem is the net or something related to the data or the labels.

Another thing is balancing the labels: if 80% are 0 and 20% are 1 (or vice versa) you may want to use weighted cross entropy

Another thing you may want is leaky relu instead of relu. This prevents vanishing gradients for some problems.
