why does the Accuracy decrease when using a ReLu activation after Linear layers
so I started using Pytorch and I'm building a very basic CNN on the FashionMNIST Dataset. I noticed some weird Behaviour while using the NN and I don't know why this is happening, in the Forward Function the Accuracy of the NN decrease when I use a Relu function after every Linear Layer. 

here is the Code for my custom NN:

# custom class neural network 
class FashionMnistClassifier(nn.Module):
  def __init__(self, n_inputs, n_out):
    super().__init__()
    self.cnn1 = nn.Conv2d(n_inputs, out_channels=32, kernel_size=5).cuda(device)
    self.cnn2 = nn.Conv2d(32, out_channels=64, kernel_size=5).cuda(device)
    #self.cnn3 = nn.Conv2d(n_inputs, out_channels=32, kernel_size=5)
    self.fc1 = nn.Linear(64*4*4, out_features=100).cuda(device)
    self.fc2 = nn.Linear(100, out_features=n_out).cuda(device)
    self.relu = nn.ReLU().cuda(device)
    self.pool = nn.MaxPool2d(kernel_size=2).cuda(device)
    self.soft_max = nn.Softmax().cuda(device)

  def forward(self, x):
    x.cuda(device)
    out = self.relu(self.cnn1(x))
    out = self.pool(out)
    out = self.relu(self.cnn2(out))
    out = self.pool(out)
    #print("out shape in classifier forward func: ", out.shape)
    out = self.fc1(out.view(out.size(0), -1))
    #out = self.relu(out) # if I uncomment these then the Accuracy decrease from 90 to 50!!!
    out = self.fc2(out)
    #out = self.relu(out) # this too
    return out

n_batch = 100
n_outputs = 10
LR = 0.001

model = FashionMnistClassifier(1, 10).cuda(device)
optimizer = optim.Adam(model.parameters(), lr=LR)
criterion = nn.CrossEntropyLoss()


so if I use the ReLu only after the CNN layers I get an Accuracy of 90% but when I uncomment that section and use the Relu activation after the Linear Layers the Accuracy decrease to 50% I have no idea why is this happening since I thought it is always better to use activation after every linear layer to get a better accuracy for Classification. I thought always that we should always use the activation function if we have a Classification Problem and for Linear Regression we don't have to do this, but here in my case although it is a Classification Problem, I get a better performance if I don't use the Activation Function after the Linear Layers. Can someone maybe clarify this to me?

CrossEntropyLoss requires you to pass in unnormalized logits (output from last Linear layer).

If you use ReLU as the output from last layer you are only outputting values in the range [0, inf), while neural network tends to go with small values for incorrect labels and high for the correct ones (we may say it's overconfident in it's predictions). Oh, and the one with highest logit value is chosen by argmax as correct label.

So it won't definitely work with this line:

# out = self.relu(out) # this too


Though it should with the ReLU before it. Just remember, more nonlinearity isn't always good for the network.
