Why is the implementation of cross entropy different in Pytorch and Tensorflow?
I am going through the documentation of Cross Entropy in Pytorch and Tensorflow. I understand that they are modifying the naive implementation of Cross Entropy to solve for the potential numeric over/underflows. However, I am unable to understand as to how these modifications are helping at all.
The implementation of Cross Entropy in Pytorch follows the following logic -

where  is the softmax score and  is the raw score.
This doesn't seem to solve the problem because  also leads to numeric overflow.
Now, we contrast it with Tensorflow's implementation (I got it from a discussion in Github. This might be completely wrong) -
Let  is the vector of all k raw logit scores.
 
While this solves the problem of overflow but it runs into problems of underflow because it is possible that  which would lead to an even smaller 
Can someone please help me in understanding what's going on here?

Answering here by combining answers from the comment section for the benefit of the community.
Since you have addressed the issue of numeric overflow in PyTorch, that is handled by subtracting the max value like below(from here).
scalar_t z = std::exp(input_data[d * dim_stride] - max_input);

Coming to TensorFlow's implementation of Cross entropy the issue of underflow is not that major since it is numerically ignored with the dominating large value.
