The exact form of the 'Ockham factor' in model selection
Jonathan Rougier Carey Priebe











The exact form of the `Ockham factor'
in model selection
    Jonathan Rougier
Jonathan Rougier (corresponding author) is Honorary Professor, School
of Mathematics, University of Bristol (email:
); Carey Priebe is
Professor, Department of Applied Mathematics and Statistics, Johns
Hopkins University (email: ).

       School of Mathematics 

       University of Bristol, UK
       Carey E. Priebe 

       Department of Applied Mathematics and Statistics 

       Johns Hopkins University, USA
     
=====================================================================================================================================================================================================================================================================================================================================================================================================================================================








We explore the arguments for maximizing the `evidence' as an
algorithm for model selection.  We show, using a new definition of
model complexity which we term `flexibility', that maximizing the
evidence should appeal to both Bayesian and Frequentist statisticians.
This is due to flexibility's unique position in the exact decomposition
of log-evidence into log-fit minus flexibility.  In the Gaussian linear
model, flexibility is asymptotically equal to the Bayesian Information
Criterion (BIC) penalty, but we caution against using BIC in place
of flexibility for model selection.


Keywords:
  Complexity, evidence, Ockham factor, flexibility, BIC penalty






§ INTRODUCTION



As Albert Einstein almost said, “Scientific models should be as
simple as possible, but no simpler”. Ockham's razor is a
common term for this principle of simplicity. <cit.> contains
a wide-ranging review, including what Einstein actually said, while
<cit.> give an engaging and non-technical account of how
Ockham's razor emerges from a Bayesian treatment of model selection.

In philosophy, statistics, and machine learning, Ockham's razor
recommends that we favour less complex models where we can.  But why
this should be, and what we mean by `complex', are subtle issues.
In this paper, we describe one approach to implementing Ockham's
razor for model selection, based on a decomposition of the `evidence'
into a `fit' term and a complexity term which we call `flexibility'.
Our decomposition is exact for all models and all regularizers.
We do not advocate one particular approach to model selection,
but clarify the implications of and relations between various approaches,
describing how one particular approach—maximizing the
evidence—might be justified by both Bayesian and Frequentist
statisticians.

Section <ref> provides background, and
section <ref> provides and discusses various arguments for the
use of the evidence in model selection.  Section <ref>
presents our exact decomposition of evidence into `fit plus
flexibility', and we justify flexibility as a measure of model
complexity.  Section <ref> illustrates using the Gaussian
linear model, for which simple closed-form expressions are available.
The Bayesian Information Criterion (BIC) penalty is shown to be an
approximation to flexibility in this case, but we caution against
its use in model selection in general, and even specifically in those
cases where the BIC penalty and flexibility are asymptotically
equivalent.
Section <ref> concludes with a summary.




§ BACKGROUND



Our starting-point is a set of observations .  A statistical model is proposed,

    f(y; θ) ,     
        y ∈⊂^n 
    θ∈Θ⊂^d .

The defining features of such a model are

    ∀θ∈Θ:    f(·; θ) ≥ 0,  ∫_^n f(y; θ)   y = 1 ,

where the integral may be replaced by a sum if  is countable.
In addition, this model is augmented with a regularizer R : Θ→.
The fitted value of the parameter is

    _θ∈Θ {log f(; θ) - R(θ) } .

Thus the regularizer is functionally equivalent to a prior distribution

    π(θ) ∝ e^-R(θ) ,

which we assume is proper, and in this case the fitted value 
is identical to the Maximum A Posteriori (MAP) estimate of θ.

As this outline makes clear, the estimated value  is a function
of both the statistical model and the regularizer.  To be precise, we would write `model and regularizer' everywhere below, but this would
be tedious; therefore we will write `model', but in every case where
we write `model' we mean `model and regularizer'.

Define the `evidence' of this model as

    E ∫_Θ f(; θ)  π(θ) θ .

This value is also referred to as the `integrated likelihood', or the
`marginal likelihood' <cit.>.
In a Bayesian approach it has the additional interpretation of
the `predictive density' of the observations <cit.>.  
The normalizing constant in the prior distribution
(<ref>) is not required to compute , but it is
required to compute the evidence directly.  <cit.>
provide a review of methods for estimating the evidence, covering a
literature which stretches back thirty years, while <cit.>
present a recent and promising approach using logistic regression
<cit.>.

Suppose that we would like to
select a single model from a set of models under consideration,
indexed by i ∈.  For example,  might represent a
set of Gaussian linear models with different model matrices (see
Section <ref>).  The claim we investigate in this paper is that maximizing the
evidence is a good way to select such a model.  That is, if E_i
is the evidence of model i, then 

    i^* _i ∈ E_i

is the best single model in —or one of the best.

The notion that there is a best single model in  is a bold
claim, because it does not distinguish between the possibly different
requirements of model-selection for inference, and model-selection
for prediction.  In inference, the intention would be to inspect the
fitted components of the model in order to make statements about
the underlying system; for example, to see whether a regression
coefficient is negative, roughly zero, or positive.  In this case,
it is helpful for interpretability to focus on a single model.
In prediction, the structure of the model is less important than
its predictive performance out-of-sample.  In this case, the use of
a single model would be a pragmatic simplification, especially for
operational systems.  So in both cases there is an apparent need for
a single model, but there is no a priori reason to think
that one method will produce a best or nearly-best choice of model
in both cases.




§ ARGUMENTS FOR THE EVIDENCE



For statisticians, the natural approach to choosing a single model
from a set 
is to use statistical decision theory, in which a loss function L(i,
j) quantifies the negative consequences of choosing model i were
model j to be the true model.  While there will be 
applications where a particular loss function suggests itself, in
many cases the zero-one loss function is a pragmatic choice:

    L(i, j) =  0     i = j 
     1    otherwise.

This loss function treats all wrong choices as equally-wrong.  There is
a useful generalization of this loss function, which includes
`undecided' among the actions, but we will not consider it further
here <cit.>.

In a Bayesian approach,
the models in  are given prior probabilities, w_i > 0.
The Bayes Rule for zero-one loss is to select the model with the
largest posterior probability, where the posterior probabilities are
proportional to w_i   E_i.  Thus if  is finite and
the prior
probabilities are uniform, then choosing i^*, the model with
the largest
evidence, is the Bayes Rule.  More generally, if the
evidence is concentrated relative to the prior probabilities,
then choosing i^* is `hopefully' the Bayes Rule.  A Bayesian
statistician might reason: “I don't want to spend too much effort
thinking about my prior probabilities, but I believe that they are
sufficiently
uniform over  that that the model with the
highest evidence will be the Bayes Rule, or a good approximation.”
This reasoning goes back to the `Principle of Stable Estimation' of
L.J. Savage <cit.>.

Clearly, this Bayesian argument is contingent on a willingness
to contemplate and possibly provide prior probabilities for the
models in .  This may be anathema to some Frequentist
statisticians, who might otherwise accept the need for a loss function,
and tentatively accept the choice of the zero-one loss function.
But these statisticians will recognize the validity of the Complete
Class theorems, which in this context state that choosing the model
with the largest evidence cannot be inadmissible under zero-one loss,
because it is the Bayes Rule for a prior which assigns positive
probability to all models; see, e.g., <cit.>,
<cit.>, or <cit.>.  Avoiding
inadmissible decision rules is a low bar but an important one.
So statisticians of all persuasions should at least contemplate
choosing the model with the highest evidence, on the basis of zero-one
loss, but this argument is suggestive, rather than compelling; and more
suggestive in the Bayesian approach than in the Frequentist approach.

The second argument is more Frequentist,
because it claims that i^* is a model-selection algorithm with
attractive properties.  Making this case for
inference tends to be model-dependent and often requires asymptotic
arguments; we concur with <cit.>, whose Principle 7
states
“If you need to use asymptotic arguments, do not forget to let your
number of observations tend to infinity”; we touch on this principle
at the end of section <ref>.  Instead, we
focus on the argument for choosing i^* for prediction, using
a canonical example which also has much practical relevance:
polynomial regression.

In the regression

    y_i = β_0 + β_1   x_i + β_2   x_i^2  + ⋯ + β_p   x_i^p + error_i

there is a simple
index of models in terms of the degree p.  Under
standard fitting procedures such as Ordinary Least Squares, and its
Bayesian treatment with a vague prior on β, models with
higher degree will never fit worse, in terms of root mean squared
error, and will usually fit better.
Nevertheless, the wiggliness of high-degree polynomials
compromises their predictive accuracy, particularly
outside the convex hull of the observations.  This property is captured
by the following schematic:

    Low degree
    poor fit
    poor prediction[rrrr]             High degree
    excellent fit
    poor prediction
       Medium degree
    good fit
    good prediction[u]

in which there is a `sweet spot' between low and high degree which
is optimal for prediction.  Most textbooks contain a detailed
discussion of this property; see, e.g., <cit.>
or <cit.>.  <cit.> provides a wide-ranging
and non-technical discussion with historical examples.

The logic of polynomial regression would seem to apply much more
generally: models with the equivalent of high degree will fit better
but predict worse than models of medium degree.  This can be represented
as `overfitting': models of high degree fit into the 
noise, and thereby carry the noise into the prediction, which 
decreases accuracy.  Or, to put it another way, models need some
`stiffness' in order to resist noise.  The `degree' of polynomial
regression is replaced by the more ambiguous term `complexity',
and the schematic becomes 

    Simple model
    poor fit
    poor prediction[rrrr]             Complex model
    excellent fit
    poor prediction
       Medium complexity
    good fit
    good prediction[u]

see, e.g., <cit.>.
It is only a short step from this schematic to an outline model-selection
criterion for prediction: to reward fit but to
penalize complexity, hoping to settle at or near the sweet-spot
between low and high complexity.

While there
are many forms that such a criterion might take, one form has emerged
as nearly ubiquitous, which is to maximize the scalar quantity

    log f(; ) -

where is a measure of complexity that can depend on the model
and on the observations.  The first author to derive a criterion of
this form was <cit.>, whose penalty did not depend on the
data, and there have been many proposals since:
this is still an active field of research in statistics <cit.>.

Now to return to our main topic, evidence.  David MacKay
<cit.> argued that the log-evidence itself
has the form of (<ref>), at least approximately.  
MacKay's argument had two strands.
First, there was `proof by picture', a compelling illustration that
the evidence will sometimes select less complex models over more
complex models, shown here as Figure <ref>.  This picture
embodies <cit.>'s informal definition of complexity:
“the more complex hypothesis tends to spread the probability more
evenly among all the outcomes” (p. 68).





But the second strand is less compelling.  MacKay applied a
first-order Laplace approximation to compute the evidence, and
showed that, under this approximation,

    log E ≈log f(; ) -

where has an explicit form in terms of the model and the
observations, but its form is not important to our argument.  We know,
as a matter of logic, that in (<ref>) must behave something like a
model complexity penalty.  This follows because the first term on the
righthand side will tend to be larger for more complex models, and
hence if the evidence can sometimes be smaller for more complex
models, as Figure <ref> demonstrates, then must sometimes be larger for more complex models.

However, Laplace approximations, of first or higher order <cit.> can perform poorly in practice, which is
why there has been such sustained interest in developing MCMC methods
for computing posterior expectations over the last three decades <cit.>.
Their performance today is likely to be worse than ever, in our
current era of massively over-parameterized models and exotic
regularizers (e.g. to promote sparsity).  Therefore MacKay's
decomposition is suggestive, but does not provide a strong reason to
believe that the evidence on its own represents a fit term minus a
complexity term suitable for all applications.

This long discussion of the Frequentist argument for using
evidence to choose a single predictive model indicates once again that the
argument is suggestive but not compelling.  There have been a number
of conjectural steps: the generalization from polynomial degree to
model complexity, the idea that the `sweet spot' of optimal complexity
can be found by penalizing the maximized log-likelihood, and then
MacKay's construction which suggests that log-evidence has the
approximate form of penalized log-likelihood.  Therefore it is
gratifying that in the next section we can show that there is an exact
equality between log-evidence, fit, and a complexity penalty, which
holds in complete generality, and where the penalty has a simple and
intuitive form.




§ `FLEXIBILITY'



Our approach uses a simple result that is a
reformulation of Bayes's Theorem:

    E =  f(; θ_0)  π(θ_0) /π^*(θ_0)

which holds for all θ_0 ∈Θ
for which π^*(θ_0) > 0, where π^* is the posterior
distribution.  <cit.> refers to this equality as 
the basic marginal likelihood identity (BMI); it also goes by
the name Candidate's formula, after <cit.>.  

If we set θ_0 = from (<ref>), then we immediately deduce

    log E = log f(; ) - logπ^*() /π()

from (<ref>).
By our argument at the end of the previous section, the second term
on the righthand side must behave something like a model complexity
penalty.
To identify it explicitly, we give it the name


    logπ^*() /π()  ,

which will be positive, except possibly in pathological situations where
there is a conflict between the prior distribution and the likelihood.
We have, under this definition,

    log E = log f(, ) -


an exact decomposition of the evidence, which holds for all models.
This is the unique decomposition for which the `fit' term is log
f(; ).  A different estimate for θ, such as the
Generalized Method of Moments (GMM) estimate, would give a different
penalty term in the decomposition of the evidence.  However, given that
the regularizer is operationally equivalent to a prior distribution,
the penalized likelihood (or MAP) estimate  seems the most
natural value to use for θ_0.

We contend that is a reasonable way to measure model
complexity. A model will be `flexible' if it contains a large number
of degrees of freedom, in the engineering sense (e.g. represented
by Θ, which might be the number of basis functions),
and if its parameters are unconstrained in the prior distribution
(or regularizer).  A model will be `inflexible' (or `stiff')
either if it contains few degrees of freedom, or if its parameters
are constrained in the prior distribution, or both.  A flexible
model will often be able to
concentrate its posterior probability into a small data-determined
region of its parameter space, relative to its prior probability,
and hence its will be high.  An inflexible model in the same
situation will be prevented from concentrating, lacking either the
capacity to shape itself to the data, or else having the capacity but
being prevented from doing so by the prior probability.  Thus its
will be low.  Complex models will typically be flexible, in
this sense, and simple models will be inflexible.

Our exact decomposition of the log-evidence has changed neither
the Bayesian nor the Frequentist arguments for using the evidence for
model selection, although it has sharpened MacKay's
construction.  So does it have any practical value?  We think it does.  eq:decomp shows that

    log f(; ) - 
      = log E - ( - )_  .

Therefore any method which penalizes the maximum of the log-likelihood
with is equivalent to a method which penalizes the log-evidence
with .  A decision not to set = is
equivalent to a decision to penalize the
log-evidence, with a penalty of an explicit but perhaps unanticipated
form.

Our decomposition presents this type of decision in a new light.
Suppose you choose to penalize log-likelihood with something other than
.  Your client, or your client's auditor, might well ask
“What deficiency of the evidence as a criterion for model selection 
are you addressing with your choice of ?”  You might answer,
“The evidence has no epistemic value to me: I regard (<ref>)
as a purely mathematical result, and can provide a rationale for
my in terms of log-likelihood.”  The difficulty with this
position is that log-evidence clearly does have epistemic value,
as demonstrated by the Bayesian and admissibility arguments.  Thus any
rationale for penalizing log-likelihood is more compelling if it can
be cross-referenced to log-evidence.

This viewpoint also operates in the other direction: a decision to
penalize log-evidence is equivalent to a decision to penalize
log-likelihood with something other than .

There will always be applications in model selection where a strong case
can be made for a penalty which is not
.  But we propose that is
a reasonable default choice, which has some cross-party appeal.
Thus a Bayesian might say,

We selected the model which maximized the
evidence, which is similar and sometimes the same as selecting the
model which has the largest posterior probability, which itself is the Bayes Rule for zero-one loss.  But we note that
the same model would have been selected using a penalized log-likelihood
approach in which the model complexity penalty is `flexibility'.

And a Frequentist might say

We selected the model which maximized
the log-likelihood penalized by the `flexibility' penalty for model
complexity; in other words, we maximized the evidence.  But we note
that the same model would have been selected 
under a Bayesian approach which maximized posterior probability with a uniform prior distribution, which is an admissible decision rule for zero-one loss.


In situations where the choice of complexity penalty is not clear-cut,
choosing seems to be a simple and defensible way forward.
The next section considers the case where the has a
closed-form expression, and a simple asymptotic approximation.




§ ILLUSTRATION: GAUSSIAN LINEAR MODEL



Consider the Gaussian linear model
with model-matrix G ∈^n × d and observation error
variance σ^2.  We use a quadratic regularizer parameterized by λ:


    R(θ)     =  λ^2 ‖θ‖^2 
    or, equivalently in terms of the prior distribution,π(θ)     = ( 0_d, P^-1)  
    where P λ^2 I_d is the prior precision.  This implies that the posterior distribution has the formπ^*(θ)     = ( , (P^*)^-1) 
    
    
    where  is the posterior expectation, as well as the MAP estimate, and P^*
    is the posterior precision,
      P^*     = 1/σ^2 G G + P 
        = 1/σ^2 (P^*)^-1 G ;


these are both functions of (σ, λ), which we treat as known.  Hence

    = 1/2log(  P^* / P ) + λ^2/2‖‖^2 .

This is the exact result.  In the limit as λ→∞, P^*
P^-1 = I and = (1/λ^2); thus = (1 / λ^2), confirming that, asymptotically, decreases to zero as the penalty on the quadratic regularizer
increases, although λ might have to be huge to overwhelm the
G G term in P^*.


Now consider the effect of n →∞ when d is fixed.
Under IID sampling,


    n^-1 G G     H 
        m


where H and m are both non-random limits.  Substituting into (<ref>) and rearranging,

    -  d log n / 2 1/2{-d (logσ^2 + logλ^2) + log H + λ^2  ‖ m ‖^2 } .

The second term on the lefthand side is the Bayesian Information Criterion
(BIC) penalty.  Thus, for the Gaussian linear model and IID sampling,

    = BIC penalty + _p(1) ,     d fixed, n →∞.

This large-n calculation
is also applicable for non-Gaussian models which satisfy the conditions 
for asymptotic posterior Normality <cit.>.

However, we advise caution.  First, as already noted,  we cannot
presume an approximately Gaussian posterior distribution in modern
practice, and therefore

    

is a poor generic approximation.  Second, it is not safe to drop
_p(1) terms in model comparison <cit.>.  In the
Gaussian linear model, (<ref>) shows that is
approximately the sum of a term in log n and a term tending to
a constant.  But log n grows very slowly: as memorably expressed
by <cit.>, log n is about 2.3 times the number
of digits in the decimal expression of n.  It could take a truely
massive n for the log n term to dominate the in every
one of the models in , as required to be reasonably sure that
and the BIC penalty would give the same outcome.  In practice, choosing to
approximate with the BIC penalty is effectively a choice to use a
different penalty.  In the terms of section <ref>, this
is equivalent to penalizing the log-evidence.  Unless the application
provides a specific justification for penalizing the log-evidence,
it is more defensible to estimate the evidence directly.




§ SUMMARY



There are many issues to consider in model selection, starting
with whether it is even sensible to select a single model from a
set of candidates.  Pragmatically, though, operating with a single
data-selected model is a powerful simplification in both inference
and prediction.  Accepting this, the next issue is what loss
function is appropriate for selecting a single model.  While it is
not obligatory to formulate model selection as a decision problem, it is
hardly defensible to make a choice without contemplating the consequences
of an error.  The zero-one loss function used in this paper is a
reasonable default choice, but if the candidate models live in a
metric space there would be alternatives which could use the distances
between models more effectively.

After these two steps, selecting the single model which maximizes the
evidence is `hopefully' the Bayes Rule and definitely admissible,
under zero-one loss.  This is the basis of the Bayesian argument for
maximizing the evidence, and a reassuring feature for all statisticians.

But we have also described maximizing the evidence from a Frequentist
perspective, that of penalizing fit using a measure of model
complexity.  The general justification of this penalization approach
is heuristic—avoiding the dangers of overfitting—and therefore
there cannot be a `right' complexity penalty.
Our contribution in this paper is to clarify that there is a unique
measure of complexity, which we term `flexibility', under which the
log-evidence decomposes exactly into `fit minus flexibility'.
Thus there is a second argument for maximizing the evidence, if
`flexibility' is accepted as a reasonable way to quantify and
penalize model complexity.

These are two arguments to the conclusion that maximizing
the evidence is a sensible way to allow the data to select
a single model, one Bayesian and one Frequentist.  Neither argument
on its own is compelling, but there must be some appeal in a criterion
which has
`cross-party' support.  As we describe in Section <ref>,
the decision not to use flexibility as the complexity penalty
should raise questions in the mind of an auditor which might be better
unasked—unless the statistician has a good answer.  This is a weak
justification, to be sure, but model selection and model complexity are
subtle and ambiguous topics, and in many applications we might welcome
even a weak justification.


Finally, if flexibility is accepted as a reasonable way to quantify
and penalize model complexity, then we strongly recommend
estimating the evidence directly, rather than using  and then replacing flexibility with a
simpler term such as the BIC penalty.  




§ ACKNOWLEDGMENTS


We began this paper while CEP was Heilbronn Distinguished Visitor in Data
Science at the University of Bristol, UK, in Spring 2019.  We would
like to thank the two reviewers for their detailed comments on previous
versions of this paper, which improved it considerably.








[AkaikeAkaike1973]akaike73
Akaike, H. (1973).
Information theory and an extension of the maximum likelihood
  principle.
In B. N. Petrovand and F. Csaki (Eds.), Proceedings of the
  Second International Symposium on Information Theory, pp.  267–281.
  Budapest: Akademiai Kiado.

[Andrieu, Freitas, Doucet, and JordanAndrieu
  et al.2003]andrieu03
Andrieu, C., N. D. Freitas, A. Doucet, and M. I. Jordan (2003).
An introduction to MCMC for Machine Learning.
Machine Learning 50, 5–43.

[BergerBerger1985]berger85
Berger, J. (1985).
Statistical Decision Theory and Bayesian Analysis (second
  ed.).
New York: Springer-Verlag New York, Inc.

[BesagBesag1989]besag89
Besag, J. (1989).
A candidate's formula; a curious result in Bayesian prediction.
Biometrika 76(1), 183.

[BishopBishop2006]bishop06
Bishop, C. M. (2006).
Pattern Recognition and Machine Learning.
New York: Springer.

[BoxBox1980]box80
Box, G. (1980).
Sampling and Bayes' inference in scientific modelling and
  robustness.
Journal of the Royal Statistical Society, Series A 143(4), 383–430.
With discussion.

[ChibChib1995]chib95
Chib, S. (1995).
Marginal likelihood from the Gibbs output.
Journal of the American Statistical Association 90,
  1313–1321.

[DeGrootDeGroot1970]degroot70
DeGroot, M. H. (1970).
Optimal Statistical Decisions.
New York: McGraw-Hill, Inc.

[Edwards, Lindman, and SavageEdwards
  et al.1963]edwards63
Edwards, W., H. Lindman, and L. Savage (1963, May).
Bayesian statistical inference for psychological research.
Psychological Review 70(3), 193–242.

[Friel and WyseFriel and
  Wyse2012]friel12
Friel, N. and J. Wyse (2012).
Estimating the evidence – a review.
Statistica Neerlandica 66(3), 288–308.

[Gelfand and DeyGelfand and
  Dey1994]gelfand94
Gelfand, A. and D. Dey (1994).
Bayesian model choice: Asymptotic and exact calculations.
Journal Royal Statistical Society B 56, 501–514.

[Gelman, Carlin, Stern, Dunson, Vehtari, and
  RubinGelman et al.2014]gelman14
Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B.
  Rubin (2014).
Bayesian Data Analysis (3rd ed.).
Boca Raton, Florida: Chapman and Hall/CRC.
Online resources at <http://www.stat.columbia.edu/ gelman/book/>.

[GowersGowers2002]gowers02
Gowers, T. (2002).
Mathematics: A Very Short Introduction.
Oxford: Oxford University Press.

[Gutmann and HyvärinenGutmann and
  Hyvärinen2012]gutmann12
Gutmann, M. U. and A. Hyvärinen (2012).
Noise-contrastive estimation of unnormalized statistical models, with
  applications to natural image statistics.
Journal of Machine Learning Research 13, 307–361.

[Hastie, Tibshirani, and FriedmanHastie
  et al.2009]hastie09
Hastie, T., R. Tibshirani, and J. Friedman (2009).
The Elements of Statistical Learning: Data Mining, Inference,
  and Prediction (second ed.).
New York: Springer.
Available online at
  <http://statweb.stanford.edu/ tibs/ElemStatLearn/>.

[Jefferys and BergerJefferys and
  Berger1992]jefferys92
Jefferys, W. H. and J. O. Berger (1992).
Ockham's razor and Bayesian analysis.
American Scientist 80(1), 64–72.

[Le CamLe Cam1990]lecam90
Le Cam, L. (1990).
Maximum likelihood: An introduction.
International Statistical Review 58(2), 153–171.

[MacKayMacKay1992]mackay92
MacKay, D. J. C. (1992).
Bayesian interpolation.
Neural Computation 4, 415–447.

[MacKayMacKay2003]mackay03
MacKay, D. J. C. (2003).
Information Theory, Inference, and Learning Algorithms.
Cambridge: Cambridge University Press.
Available online, <http://www.inference.org.uk/mackay/itila/>.

[MurphyMurphy2012]murphy12
Murphy, K. (2012).
Machine Learning: A Probabilistic Perspective.
Cambridge MA, USA: MIT Press.

[Robert and CasellaRobert and
  Casella1999]robert99
Robert, C. and G. Casella (1999).
Monte Carlo Statistical Methods.
New York: Springer.

[SchervishSchervish1995]schervish95
Schervish, M. J. (1995).
Theory of Statistics.
New York: Springer.
Corrected 2nd printing, 1997.

[SoberSober2015]sober15
Sober, E. (2015).
Ockham's Razors: A User's Manual.
Cambridge: Cambridge University Press.



