Pytorch ConvNet loss remains unchanged and only one class is predicted
My ConvNet is only predicting a single class and the loss remains unchanged.
I have tried the following:

added class weights to be proportional to the data sizes (1-(class occurrences/total data))
adjusted learning rate attempting to find a sweet spot
adjusted gamma (Multiplicative factor of learning rate decay)
used gamma in the loss function, and not used it in loss function (Mainly have been experimenting with Adam)
Tried a much more complex ConvNet than the 'Simple' one currently in use

I am unsure of where to go from here. It seems no matter what I try the Neural Net always predicts the same class (I have been able to get it to predict the other class by throwing the weights very out of proportion)
Below is the output from running my program. It should have all the relevant information in it to come up with some ideas as to how to fix it. If you need to see some of the source code or are curious about what the dataset looks like, please ask.
Any help is greatly appreciated. I have been stumped on this issue for quite a while now. Thank you!
Train dataset length: 27569
Test dataset length: 4866

Image preprocessing: 
None
Input dimensions: 28 X 28
Output dimension: 2

Model: Simple
NeuralNetwork(
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=784, out_features=512, bias=True)
    (1): ReLU()
    (2): Linear(in_features=512, out_features=512, bias=True)
    (3): ReLU()
    (4): Linear(in_features=512, out_features=2, bias=True)
    (5): ReLU()
  )
)

Optimizer: Adam
Learning rate: 0.0001

Loss function: CEL
class weights: tensor([0.3481, 0.6519], device='cuda:0')
Multiplicative factor of learning rate decay: 0.0005

Train Epoch: 1 [0/27569 (0%)]   Loss: 3785.907959
Train Epoch: 1 [6400/27569 (23%)]   Loss: 0.693147
Train Epoch: 1 [12800/27569 (46%)]  Loss: 0.693147
Train Epoch: 1 [19200/27569 (70%)]  Loss: 0.693147
Train Epoch: 1 [25600/27569 (93%)]  Loss: 0.693147

Test set: Average loss: 0.0110, Accuracy: 3172/4866 (65%)

actual count: [3172, 1694]
predicted count: [4866, 0]


It is very uncommon to have a ReLU after the last Linear layer (where the logits come from). Consider removing it.
In addition, maybe your learning rate is too high. You could try tweaking it a little bit. Check if the loss decreases smoothly between the iterations (which is ideal in most cases), otherwise decrease it
