Why does TensorFlow docs discourage using softmax as activation for the last layer? The beginner colab example for tensorflow states:


  Note: It is possible to bake this tf.nn.softmax in as the activation function for the last layer of the network. While this can make the model output more directly interpretable, this approach is discouraged as it's impossible to
  provide an exact and numerically stable loss calculation for all models when using a softmax output. 


My question is, then, why? What do they mean by impossible to provide an exact and numerically stable loss calculation?
It's because of gradient computations: automatic differentiation will compute the gradient for each module and if you have a standalone crossentropy module the over all loss will be unstable (~1/x so it will diverge for small input values) whereas if you use a softmax + crossentropy module all-in-one, then it becomes numerically stable (y-p)

Slides from DeepMind's Simon Osindero lecture at UCL in 2016: