In softmax classifier, why use exp function to do normalization?
It is more than just numerical. A quick reminder of the softmax:
$$
P(y=j | x) = \frac{e^{x_j}}{\sum_{k=1}^K e^{x_k}}
$$

Where $x$ is an input vector with length equal to the number of classes $K$. The softmax function has 3 very nice properties: 1. it normalizes your data (outputs a proper probability distribution), 2. is differentiable, and 3. it uses the exp you mentioned. A few important points:


The loss function is not directly related to softmax. You can use standard normalization and still use cross-entropy.
A "hardmax" function (i.e. argmax) is not differentiable. The softmax gives at least a minimal amount of probability to all elements in the output vector, and so is nicely differentiable, hence the term "soft" in softmax.
Now I get to your question. The $e$ in softmax is the natural exponential function. Before we normalize, we transform $x$ as in the graph of $e^x$:




If $x$ is 0 then $y=1$, if $x$ is 1, then $y=2.7$, and if $x$ is 2, now $y=7$! A huge step! This is what's called a non-linear transformation of our unnormalized log scores. The interesting property of the exponential function combined with the normalization in the softmax is that high scores in $x$ become much more probable than low scores. 

An example. Say $K=4$, and your log score $x$ is vector $[2, 4, 2, 1]$. The simple argmax function outputs:

$$
[0, 1, 0, 0]
$$

The argmax is the goal, but it's not differentiable and we can't train our model with it :( A simple normalization, which is differentiable, outputs  the following probabilities:

$$
[0.2222, 0.4444, 0.2222, 0.1111]
$$

That's really far from the argmax! :( Whereas the softmax outputs:
$$
[0.1025, 0.7573, 0.1025, 0.0377]
$$

That's much closer to the argmax! Because we use the natural exponential, we hugely increase the probability of the biggest score and decrease the probability of the lower scores when compared with standard normalization. Hence the "max" in softmax.