Calibrate the predicted class probability to make it represent a true probability?
"Not all classification models are naturally probabilistic" is the first line of this chapter [1] .
This means that some models do output probabilities, and some do not. This is highly dependant on the model structure as well as of the objective function that it uses.

In particular, for neural network, a commonly used objective function is Cross Entropy. And to my understanding, Cross Entropy does try to approach the true distribution of the target variable, conditionned on the input variable. This would mean that for a hundred similar cases $X_{similar}$, that are similar in terms of the information that the model is capable to understand, if 80% of those cases are dogs, the predicted score will be a descent approach of the probability you want to estimate: $ \hat{y} \approx p(dog|X_{similar}) = 80/100$.

For other models with less "statistical" behaviour, such as Decision Tree, SVM, of models taken into a Boosting or Bagging process, the output distribution ($\hat{y}$) is skewed. In this binary classification setting, these models would try to give a score as close as possible of the target value. In the case at hand, they would consider the $X_{similar}$ as a group, and the best decision for that group in terms of accuracy beeing to classify them all as dogs, $\hat{y}$ would be skewed toward 1 value.

So, my take on this : 


If your goal is to have accurate probabilities, use a model that directly outputs probabilities. If the probabilities are "perfect", a cutoff value of 0.5 would provide the best accuracy (i.e. $\hat{y}\geq{0.5}$ as a decision rule)
If your goal is to have a good classifier in terms of accuracy, use the model that... has the best accuracy. Note that in this case, the best cutoff value may not be 0.5 (it can even be far from it).
If you have both those goals, well, I do not have the answer as this is data and model dependant. If your scores are not probabilities, calibration should make them look more like it indeed. See Calibration in classification for a list of calibration methods.


[1] Wikipedia - Probabilistic Classification  - Probability calibration

[2] Wikipedia - Calibration (statistics) - In Classification