Metrics for evaluating models that output probabilities
For a binary classifier, the cross-entropy loss is a natural measure of probability accuracy, if you care about relative probabilities. By that I mean if you care that the estimate $\hat{p}$ is within some ratio of the true value. So an estimate of $\hat{p} = 0.1$ is a better estimate if the true value is $p = 0.2$ than if the true value is $p = 0.01$ (even though the latter value is closer and would score better under MSE). It also applies that you care that 0.9 is by the same logic "closer" to 0.8 than it is to 0.97. With cross-entropy loss, extreme confidence (predicting close to $0$ or close to $1$) is penalised more heavily when it is wrong.

For completeness, the loss function (per data point) is:

$$\mathcal{L}(\hat{y},y) = -(y\text{log}(\hat{y})+ (1-y)\text{log}(1-\hat{y}))$$

This is likely to be the same loss function as you are using for your objective (or at least it should be), so for tests, simply also use it as your metric*.

$\hat{y}$ is your predicted probability of being in class A, and can be in range $[0,1]$. Ideally you have ground truth probabilities and $y$ is also in that range. In which case the only problem is that the "perfect" score is no longer $0$ but some positive number. If that bothers you, then you could offset by the perfect score, pre-calculating it on each data set (just set $\hat{y} = y$ for each item and you will find the minimum possible score).

If you don't have ground truth probabilities, but you do have classes, then $y$ will either be 0 or 1, and the metric still works. To get an accurate metric, you will need enough samples that the relative frequencies of each class depending on input has a significant effect. That is, you need more data, both training and test, in order to train for accurate probabilities instead of targeting simpler classification accuracy metrics.

Similar logic also works for multi-class probabilities. However, many off-the-shelf libraries use an optimisation in the loss function - assuming only one true class - which makes using probabilities as ground truth impossible. You might therefore need to write your own loss function and gradient functions based on multi-class cross-entropy loss in that case.



* I am making the assumption here that you use standard conventions for noting loss functions (typically per item), cost functions (typically aggregated across a data set and possibly multiple loss functions) and metric functions which don't have to be differentiable or usable as either of the former. The cost function is usually also fed into optimisers as the objective function - i.e. it has the important job of driving parameters to reach a maximum or minimum value. For gradient based solvers, that means it must be differentiable.