Solomonoff's theory of induction, Kolmogorov complexity and Bayesian Inference
Mathematics and computer science doesn't have anything to say about whether simpler hypotheses are more likely.  That's a question about reality, not about math / computer science.

What computer science can provide is the notion of Kolmogorov complexity.  Kolmogorov complexity is one reasonable notion of simplicity: bit-strings with lower Kolmogorov complexity are "simpler" in some sense.

Now we could start with a model of reality, which says that nature randomly picks a process for generating data; the process is obtained by picking a Turing machine at random, with smaller (simpler) Turing machines more likely to be chosen than complex ones; and then nature runs the Turing machine and the data you observe is the output of that Turing machine.  That's a sketch of a possible model of reality; basically, it's an assumption about how nature works.

That assumption then implies (if we fill in some details) a particular prior on hypotheses.  A hypothesis amounts to a Turing machine.  There will be multiple Turing machines that are all consistent with the observations (that could have produced those observations), and Bayes theorem + the prior will let you infer the posterior distribution on which of those hypotheses are most likely.

This is a possible way to obtain a prior, and it seems reasonable to me.  Computer science can't tell you whether that model (that assumption) is a good model of reality. But it can help you work out the consequences of making that assumption.



Finally, you seem to be asking why simpler explanations are more often correct.  Basically, why is Occam's razor useful?  I don't know if there is any completely convincing answer to that.

One possible answer is the empirical answer: it seems to work well in practice.  In other words, we often find in nature that very simple processes/models suffice to explain a wide range of natural observations.  See, e.g., the unreasonable effectiveness of mathematics. 

There's lots more that one can say about Occam's razor.  See, e.g., https://en.wikipedia.org/wiki/Occam%27s_razor#Justifications and https://philosophy.stackexchange.com/questions/tagged/occams-razor?sort=votes.  I think that gets beyond the scope of this site.