Dealing with extreme values in softmax cross entropy?
Where exactly in the computations are these underflows manifesting? See here for a brief explanation around the extremes of the softmax.

Quick fixes could be to either increase the precision of your model (using 64-bit floats instead of, presumably, 32 bit floats), or just introduce a function that caps your values, so anything below zero or exactly zero is just made to be close enough to zero that the computer doesn't freak out. For example, use X = np.log(np.max(x, 1e-9)) before going into the softmax.

In any case, the softmax shouldn't have problems with your input, as the final activations are exponentiated:

$$
\sigma (\mathbf {z} )_{j}={\frac {e^{z_{j}}}{\sum _{k=1}^{K}e^{z_{k}}}}  
$$

This means all values will now be in the range [0, 1].

The cross-entropy equation should also be able to deal with the output of this.

If none of this helps your specific issue, could you share a specific example of your problem?