How does one define the possibility space of valid priors (models)? When one trains a model on data of any complexity one inevitably ends up with a one particular model among a vast many possible models that would make similar (or even, the same) predictions.

For example, we can both train a neutral net on mnist and the model I produce will have different weights and biases from yours, even if they make the same predictions.

But are there any algorithms that don't produce one model, but instead produce a description of the space of all possible models that could describe the data?

Does that even make sense? In some way, as near as I can tell, a model represents a set of priors (biases and weights). Is this sort of thing not done because it's too expensive?

Take the following riddle as a conceptual example, '''A man and his son are in a terrible accident and are rushed to the hospital in critical care. The doctor looks at the boy and exclaims "I can't operate on this boy, he's my son!" How could this be?'''

You might have a prior that says, "most surgeons are men" and another prior that says, "most parents are heterosexual." You have a space of two models that would explain the data, and of course they are mutually exclusive: either the boy has two fathers, or the surgeon is his mother.

The key is, both of these models would work to explain the data and you don't know which model is true until you get more data. Also, just as importantly, you can map each model's relationship to your prior beliefs so when you learn which one is valid you can update you're priors accordingly.

A technique or algorithm that produces a probability space of all possible models seems like a potentially powerful tool. Yet I can't find this kind of thing anywhere.

Are there any algorithms or machine learning techniques that do this? And furthermore how do you even go about mathematically describing a possibility space of all possible valid models according to their likelihood, according to your priors?

Thanks for your help!
From a theoretical perspective, I think this question is a variant of "given a problem, is there a way to determine the absolute best learning algorithm for it?". Why? Because as OP correctly suggests, if there was a way to represent all the possible models/priors, then assuming that an infinite stream of data is provided we would be able to eventually determine the absolute best learning algorithm. Unfortunately the No Free Lunch theorem is a theoretical result which states that there can be no such "absolute best" learning method. (to be honest personally I'm out of my depth with this kind of thing, but Wikipedia gives a nice short explanation of the implications of the theorem for machine learning.)

From a practical perspective, this is a matter of design, i.e. how one chooses to represent the problem. It's easy to overlook how much simplification is made when one represents a problem. In the example proposed by OP, one could imagine an infinity of "alternative worlds": maybe the word "father" doesn't have its usual meaning; maybe it happens in a future where people can have any number of parents; etc. My point is: whenever a problem is formally stated, a vast number of assumptions are made anyway, most of them unconsciously. It's not only a matter of cost (it's important, of course), it's a matter of "targeting" the exact problem: if we tried to leave the space of possibilities completely open, then we cannot even start to solve the problem, in the same way that people have to agree on a common language in order to understand each other. Thus choosing a specific space of priors/models for a particular problem is akin to fixing the language of the problem, and when doing it it's important to take into account the limitations/assumptions associated with it.