Get the Cross Entropy Loss in pytorch as in Keras
I am struggling to port a classification model form keras to pytorch. Especially the cross entropy loss seems to return totally different numbers.
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print(&quot;Keras&quot;, K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred)))
print(&quot;PyTorch&quot;, nn.CrossEntropyLoss()(t.tensor(y_pred).argsort(dim=-1).float(), t.tensor(y_true).argmax(dim=-1)))```

prints:

Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32)
PyTorch tensor(1.4587)

Since I have a custom loss function where cross entropy is a part of it, I would need to get similar if not the same numbers.

The problem is that they have different implementations. 

As pytorch docs says, nn.CrossEntropyLoss combines nn.LogSoftmax() and nn.NLLLoss() in one single class. However, tensorflow docs specifies that keras.backend.categorical_crossentropy do not apply Softmax by default unless you set from_logits is True. For this reason, you should not use keras.backend.categorical_crossentropy without having previously apply softmax unless you use from_logits=True.

If you don't want to apply softmax beforehand you should use:

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])


print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred), from_logits=True))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)


Otherwise, you can apply Softmax manually before computing categorical_crossentropy

import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])


print("Keras", K.categorical_crossentropy(K.constant(y_true), K.softmax(K.constant(y_pred))))
# output: Keras tf.Tensor([2.408051], shape=(1,), dtype=float32)
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).float(), t.tensor(y_true).argmax(dim=-1)))
# output: PyTorch tensor(2.4081)


So you should not use keras.backend.categorical_crossentropy with from_logits=False as you were doing in your example.


  tf.keras.backend.categorical_crossentropy
  
  target:   A tensor of the same shape as output. 
  
  output:   A tensor resulting from a softmax (unless from_logits is True, in which case output is expected to be the logits). 
  
  from_logits:  Boolean, whether output is the result of a softmax, or is a tensor of logits.

