title: Why does multi-class classification fails with sigmoid?
body: <p>MNIST trained with Sigmoid fails while Softmax works fine</p>
  
  <p>I am trying to investigate how different activation affects the final results, so I implemented a simple net for MNIST with PyTorch.</p>
  
  <p>I am using NLLLoss (Negative log likelihood) as it implements Cross Entropy Loss when used with softmax.</p>
  
  <p>When I have softmax as activation of the last layer, it works great.
  But when I used sigmoid instead, I noticed that things fall apart</p>
  
  <p>Here is my network code</p>
  
  <pre><code>def forward(self, x):
          x = F.relu(F.max_pool2d(self.conv1(x), 2))
          x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
          x = x.view(-1, 80)
          x = F.relu(self.fc1(x))
          x = F.dropout(x, training=self.training)
          x = self.fc2(x)
          return F.XXXX(x)
  </code></pre>
  
  <p>where XXXX is the activation function</p>
  
  <p>both Sigmoid and Softmax output values between (0, 1).
  Yes Softmax guarantees the sum of 1 but I am not sure if this answers why the training fails with Sigmoid.
  Is there any detail I am not catching here?</p>