How to understand marginal loglikelihood objective function as loss function (explanation of an article)?
I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.
The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.
You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: https://www.deeplearningbook.org/
or here : https://en.wikipedia.org/wiki/Loss_functions_for_classification
I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in A Practical Bayesian Framework for Backpropagation networks by McKay 1992.
It is explained very well in a video series by Hinton on youtube here:
https://www.youtube.com/watch?v=YcwZFNd3UvI
Hopefully these references help you I wish I could explain in greater detail.