Mathematical model for a webpage layout?
At the time of writing, I am not aware of any formal (or at least formalized enough) models for layouts in HTML. All the examples (ALM, etc) indicate, that usually the model is created to guide layout manager's creation. In the case of HTML and web-browsers the development is evolutional and while each web engine contains some sort of algorithmic models to be able to render HTML, nobody (I know of) published underlying math, and nobody "reverse engineered" program code to extract the model.

It should also be noted, that the model could have grown very complex, so any effort to build one retrospectively may be quite costly.

Even though it is not done yet, it does not mean it is impossible. As more rigorous approaches are being applied to web programming (e.g., recent release of Ur/Web - typesafe approach spanning from database to client-side interactions), modelling HTML5 layout management may be on the horizon, especially, as we witness more formal approach in the recommendations for new HTML5/CSS features (for example, http://www.w3.org/TR/css3-flexbox/), because having formal model may help proving consistency of the additions (analogously to formal grammar, which greatly facilitates making additions to the programming language syntax).