What is a “saturated” model?

What is meant when we say we have a saturated model?

A saturated model is one in which there are as many estimated parameters as data points. By definition, this will lead to a perfect fit, but will be of little use statistically, as you have no data left to estimate variance.

For example, if you have 6 data points and fit a 5th-order polynomial to the data, you would have a saturated model (one parameter for each of the 5 powers of your independant variable plus one for the constant term).

• I seen examples where a model has ten data points and nine parameters. On pointing out that the model has too many parameters, I was told that the R^2 was 0.999 so the model must be correct! – csgillespie Jul 20 '10 at 16:03
• As can be read in my and dave's post, saturated models do not per definition lead to perfect fit. but if you use the n-1 polynominal as the model they will. see Sue Doe Nihm's seminal paper on this topic psych.fullerton.edu/mbirnbaum/papers/Nihm_18_1976.pdf – Henrik Jul 27 '10 at 16:52

A saturated model is a model that is overparameterized to the point that it is basically just interpolating the data. In some settings, such as image compression and reconstruction, this isn't necessarily a bad thing, but if you're trying to build a predictive model it's very problematic.

In short, saturated models lead to extremely high-variance predictors that are being pushed around by the noise more than the actual data.

As a thought experiment, imagine you've got a saturated model, and there is noise in the data, then imagine fitting the model a few hundred times, each time with a different realization of the noise, and then predicting a new point. You're likely to get radically different results each time, both for your fit and your prediction (and polynomial models are especially egregious in this regard); in other words the variance of the fit and the predictor are extremely high.

By contrast a model that is not saturated will (if constructed reasonably) give fits that are more consistent with each other even under different noise realization, and the variance of the predictor will also be reduced.

As everybody else said before, it means that you have as much parameters have you have data points. So, no goodness of fit testing. But this does not mean that "by definition", the model can perfectly fit any data point. I can tell you by personal experience of working with some saturated models that could not predict specific data points. It is quite rare, but possible.

Another important issue is that saturated does not mean useless. For instance, in mathematical models of human cognition, model parameters are associated with specific cognitive processes that have a theoretical background. If a model is saturated, you can test its adequacy by doing focused experiments with manipulations that should affect only specific parameters. If the theoretical predictions match the observed differences (or lack of) in parameter estimates, then one can say that the model is valid.

An example: Imagine for instance a model that has two sets of parameters, one for cognitive processing, and another for motor responses. Imagine now that you have an experiment with two conditions, one in which the participants ability to respond is impaired (they can only use one hand instead of two), and in the other condition there is no impairment. If the model is valid, differences in parameter estimates for both conditions should only occur for the motor response parameters.

Also, be aware that even if one model is non-saturated, it might still be non-identifiable, which means that different combinations of parameter values produce the same result, which compromises any model fit.

If you wanna find more information on these issues in general, you might wanna take look at these papers:

Bamber, D., & van Santen, J. P. H. (1985). How many parameters can a model have and still be testable? Journal of Mathematical Psychology, 29, 443-473.

Bamber, D., & van Santen, J. P. H. (2000). How to Assess a Model's Testability and Identifiability. Journal of Mathematical Psychology, 44, 20-40.

cheers

A model is saturated if and only if it has as many parameters as it has data points (observations). Or put otherwise, in non-saturated models the degrees of freedom are bigger than zero.

This basically means that this model is useless, because it does not describe the data more parsimoniously than the raw data does (and describing data parsimoniously is generally the idea behind using a model). Furthermore, saturated models can (but don't necessarily) provide a (useless) perfect fit because they just interpolate or iterate the data.

Take for example the mean as a model for some data. If you have only one data point (e.g., 5) using the mean (i.e., 5; note that the mean is a saturated model for only one data point) does not help at all. However if you already have two data points (e.g., 5 and 7) using the mean (i.e., 6) as a model provides you with a more parsimonious description than the original data.

• This point about saturated not implying a perfect fit is the most interesting part of this thread. A natural example of such a situation would be monotonic regression. Suppose, e.g., you know your values must increase over time and you do polynomial regression, constraining the polynomials to be increasing. Consider data that have some error, so at some times they decrease a little. Then no matter how many parameters you use (even when it's more than the number of data values), you will never fit these data perfectly. – whuber Sep 10 '11 at 16:11

It is also useful if you need to calculate AIC for a quasi-likelihood model. The estimate of dispersion should come from the saturated model. You would divide the LL you are fitting by the estimated dispersion from the saturated model in the AIC calculation.

In the context of SEM (or path analysis), a saturated model or a just-identified model is a model in which the number of free parameters exactly equals the number of variances and unique covariances. For example the following model is a saturated model because there are 3*4/2 data points (variances and unique covariances) and also 6 free parameters to be estimated:
{"ft_lang_label":"__label__en","ft_lang_prob":0.92484444,"math_prob":0.96047455,"size":6306,"snap":"2019-43-2019-47","text_gpt3_token_len":1373,"char_repetition_ratio":0.15090448,"word_repetition_ratio":0.007561437,"special_character_ratio":0.21693625,"punctuation_ratio":0.112837285,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9852809,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-10-22T16:23:33Z\",\"WARC-Record-ID\":\"<urn:uuid:8aae5f84-2e07-4257-bf18-75e9ba4a6244>\",\"Content-Length\":\"173412\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:a46d5619-e83a-4382-b312-d275a45f97d0>\",\"WARC-Concurrent-To\":\"<urn:uuid:26ed70fd-7f6b-4060-b449-3ee9087d87a1>\",\"WARC-IP-Address\":\"151.101.129.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/283/what-is-a-saturated-model/307\",\"WARC-Payload-Digest\":\"sha1:7TONHLM4R24TMGOJ7DL2FYCV3FTZYIZP\",\"WARC-Block-Digest\":\"sha1:5VT2MNEDBVNPYIWPKDLX3VWS45J6XS4I\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-43/CC-MAIN-2019-43_segments_1570987822458.91_warc_CC-MAIN-20191022155241-20191022182741-00194.warc.gz\"}"}