How to interpret the value of categorical cross entropy? Categorical cross-entropy loss is usually used in settings where the target in one-hot encoded. Suppose I have a problem where there are 300 possible outcomes, and thus my final fully connected layer will have 300 neurons and the output (after softmax) is expected to be probabilities for each class such that their sum is equal to one.
My question is: how to interpret the value of the categorical cross-entropy loss? What does it mean to have a categorical cross-entropy loss of 5 or 0.9 or 0.1?
Does the interpretation of the loss get affected with the number of possible classes?
There is usually no straightforward interpretation of what cross-entropy means in the context of the given task. In practice, it is more important to follow the trend of how the cross-entropy develops during the training.
The measure comes from the information theory. It says how surprised the model is when it has some belief about the output (a distribution) and you provide it with the ground truth output. A nice (but not very useful) interpretation is that the entropy of $x$ means the same surprisal as observing an output of rolling and $2^x$-sided dice.
The value of cross-entropy indeed depends on the number of classes, just because with more classes you have the output distribution must cover more options and it is more difficult to assign more probability to the ground truth class.