title: PyTorch: model accuracy stays the same after first iteration even if I add another layer
body: <p>The dataset I'm trying to perform binary classification on is the <a href="https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data" rel="nofollow noreferrer">Adult dataset</a>.</p>
  <p>The only preprocessing I've done was to remove rows with missing values and replace the two values of the target with an integer instead of a string.</p>
  <p>Done that, I encoded it using the <a href="https://contrib.scikit-learn.org/category_encoders/targetencoder.html" rel="nofollow noreferrer">TargetEncoder</a> and split it into train and test set.</p>
  <p>This is the model I'm using:</p>
  <pre class="lang-py prettyprint-override"><code>class SimpleLinear(Module):
      &quot;&quot;&quot;
      Simple linearmodel of two linear layers, the first with tanh and the last with softmax
  
      The weights of the Linear layers are initialized using the xavier initialization.
      The final layer uses the softmax function.
      &quot;&quot;&quot;
  
      def __init__(self, input_len: int, output_len: int, middle_dim=64):
          super().__init__()
  
          self.input_len = input_len
          self.output_len = output_len
          self.middle_dim = middle_dim
  
          self.fc1 = Linear(input_len, middle_dim)
          xavier_normal_(self.fc1.weight)
          self.drop = Dropout(0.3)
          self.fc4 = Linear(middle_dim, output_len)
          xavier_normal_(self.fc4.weight)
  
      def forward(self, x):
          # Pass the input tensor through each of our operations
          x = tanh(self.fc1(x))
          x = self.fc4(self.drop(x))
          return softmax(x, dim=1)
  </code></pre>
  <p>And this is the training loop:</p>
  <pre class="lang-py prettyprint-override"><code>def train(model: Module, optimizer: Optimizer, loss_fn: Callable, train_loader: DataLoader, test_loader: DataLoader,
            epochs: int = 20, device: str = &quot;cpu&quot;) -&gt; None:
      &quot;&quot;&quot;
      Training loop for training a model.
  
      Args:
          model: model to train.
          optimizer: optimizer to use.
          loss_fn: loss function.
          train_loader: DataLoader for loading the training data.
          test_loader: Dataloader for loading the test data
          epochs: Training epochs. (Default  = 20)
          device: Where to train the model, &quot;cpu&quot; or &quot;cuda&quot;. (Default = &quot;cpu&quot;)
  
      Returns:
  
      &quot;&quot;&quot;
      for epoch in range(epochs):
          training_loss = 0.0
          test_loss = 0.0
          model.train()
  
          for batch in train_loader:
              optimizer.zero_grad()
              inputs, targets = batch
              inputs = inputs.to(device)
              targets = targets.to(device)
              output = model(inputs)
              loss = loss_fn(output, targets)
              loss.backward()
              optimizer.step()
              training_loss += loss.data.item()
          training_loss /= len(train_loader)
  
          model.eval()
  
          num_correct = 0
          num_examples = 0
  
          for batch in test_loader:
              inputs, targets = batch
              inputs = inputs.to(device)
              output = model(inputs)
              targets = targets.to(device)
              loss = loss_fn(output, targets)
              test_loss += loss.data.item()
              correct = torch.eq(torch.max(F.softmax(output, dim=1), dim=1)[1], targets).view(-1)
              num_correct += torch.sum(correct).item()
              num_examples += correct.shape[0]
          test_loss /= len(test_loader)
  
          print('Epoch: {}, Training Loss: {:.2f}, Test Loss: {:2f}, accuracy = {:.2f}'.format(epoch, training_loss,
                                                                                               test_loss,
                                                                                               num_correct / num_examples))
  </code></pre>
  <p>I create and train the  model in this way, getting the shown outputs:</p>
  <pre><code>model = SimpleLinear(input_len=14, output_len=2, middle_dim=64)
  model.to(device)
  optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
  loss_fn   = nn.CrossEntropyLoss()
  
  train_dataloader = DataLoader(train_dataset, batch_size=16)
  test_dataloader = DataLoader(test_dataset, batch_size=16)
  train(model, optimizer, loss_fn,train_dataloader, test_dataloader, epochs=20, device=device)
  </code></pre>
  <pre><code>Epoch: 0, Training Loss: 0.56, Test Loss: 0.562031, accuracy = 0.75
  Epoch: 1, Training Loss: 0.56, Test Loss: 0.561948, accuracy = 0.75
  Epoch: 2, Training Loss: 0.56, Test Loss: 0.561480, accuracy = 0.75
  Epoch: 3, Training Loss: 0.56, Test Loss: 0.551922, accuracy = 0.75
  Epoch: 4, Training Loss: 0.55, Test Loss: 0.543033, accuracy = 0.75
  Epoch: 5, Training Loss: 0.56, Test Loss: 0.553800, accuracy = 0.75
  </code></pre>
  <p>As it can be seen the accuracy never increases, the weird thing is that if I change the <code>middle_dim</code> parameter to increase the size of the hidden layer, or I change the learning rate / optimizer (I tried SGD) nothing changes.</p>
  <p>I also tried adding another hidden layer to see if the model was underfitting:</p>
  <pre class="lang-py prettyprint-override"><code>class SimpleLinear(Module):
  
      def __init__(self, input_len: int, output_len: int, middle_dim=128):
          super().__init__()
  
          self.input_len = input_len
          self.output_len = output_len
          self.middle_dim = middle_dim
  
          self.fc1 = Linear(input_len, middle_dim)
          xavier_normal_(self.fc1.weight)
          self.drop = Dropout(0.3)
  
          self.fc2 = Linear(middle_dim, int(middle_dim / 2))
          xavier_normal_(self.fc2.weight)
  
          self.drop2 = Dropout(0.3)
  
          self.fc4 = Linear(int(middle_dim / 2), output_len)
          xavier_normal_(self.fc4.weight)
  
      def forward(self, x):
          # Pass the input tensor through each of our operations
          x = tanh(self.fc1(x))
          x = self.drop(x)
          x = self.fc2(x)
          x = tanh(x)
          x = self.fc4(self.drop2(x))
          return softmax(x, dim=1)
  </code></pre>
  <p>but I still get the same accuracy, right from the first iteration. What can the causes of this be?</p>
  <p>I checked the distribution of the target variables in the train and test datasets, and in both 2/3 of the values belong to the class 0 and 1/3 to the class 1. Can it be this the cause of this results? But why would it be and why the accuracy would be the same right from the start?</p>
  <p>EDIT:
  After @jhso's comment, I made another model using a sigmoid:</p>
  <pre class="lang-py prettyprint-override"><code>class SimpleBinaryClassifier(nn.Module):
      def __init__(self,input_shape):
          super().__init__()
          self.fc1 = nn.Linear(input_shape,64)
          self.fc2 = nn.Linear(64,32)
          self.fc3 = nn.Linear(32,1)
      def forward(self,x):
          x = torch.relu(self.fc1(x))
          x = torch.relu(self.fc2(x))
          x = torch.sigmoid(self.fc3(x))
          return x
  </code></pre>
  <p>I used the <code>BCELoss</code> and fixed the training script by removing the softmax and unsqueezing the targets (here I paste just the second half containing the testing code, the first half is the same as before but with targets.unsqueeze)</p>
  <pre class="lang-py prettyprint-override"><code>for batch in test_loader:
              inputs, targets = batch
              inputs = inputs.to(device)
              output = model(inputs)
              targets = targets.to(device)
              loss = loss_fn(output, targets.unsqueeze(1).float())
              test_loss += loss.data.item()
              correct = torch.eq(output, targets).view(-1)
              num_correct += torch.sum(correct).item()
              num_examples += correct.shape[0]
          test_loss /= len(test_loader)
  </code></pre>
  <p>but now the outputs after each iteration are straight up identical:</p>
  <pre><code>Epoch: 0, Training Loss: 75.09, Test Loss: 75.127100, accuracy = 0.25
  ...
  Epoch: 19, Training Loss: 75.09, Test Loss: 75.127100, accuracy = 0.25
  </code></pre>
  <p>Here is a sample output of the model:</p>
  <pre><code>a, b = test_dataset[1]
  print(f&quot;Features: {a}\nTarget: {b}&quot;)
  model(torch.from_numpy(a).to(device))
  </code></pre>
  <pre><code>Features: [6.0000000e+01 2.9462990e-01 2.0247300e+05 2.6319817e-01 1.1000000e+01
   4.5501101e-01 4.1134309e-02 4.5572770e-01 2.6385459e-01 3.1395462e-01
   0.0000000e+00 0.0000000e+00 4.0000000e+01 2.5441119e-01]
  Target: 0
  tensor([1.], device='cuda:0', grad_fn=&lt;SigmoidBackward0&gt;)
  </code></pre>