Why do people use CrossEntropyLoss and not just a softmax probability as the loss? I don't understand why one would add additional complexity to log, probabilities for the loss function of a classification Neural Network. What benefit does that have, as opposed to just using the 0-1.0 values(probabilities of a class) you get from the softmax function at the final layer?

Does this add extra non-linearity that we don't understand why it does good, but just happens to do good a lot of times since we give the Neural Net some more complexity?
Let me explain it with logistic regression first: 


The output of logistic regression is $\hat{y}= \dfrac{1}{1 + e^{-\theta x}}$
Consider the minimum square loss function $\mathcal{L} = (1 -\hat{y})^2$
substitute $\hat{y}$ in the second equation $\mathcal{L} = (\dfrac{e^\theta x}{1 + e^{-\theta x}})^2$


Tha above formula is not convex, so optimization is much more difficult since the gradient may point to local optima.

Whereas cross entropy looks like.$$\mathcal{L} = -y\log \hat{y} - (1-y)\log (1-\hat{y}) $$ which has two advantages. 


It is convex when subtitute $\hat{y}$ for its value (considering that $y$'s value is either 1 or 0)
For each row in your dataset you are taking on count both labels at the same time


When we generalize from binari logistic regression to multiclass classification, we use the same ideas of using convex loss functions with respect to the output and also loss functions which minimize at the same time for all labels.