Textbook on the mathematics of RBC/DSGE models?
The mathematical theory behind DSGE models can be found in any textbook on stochastic dynamic optimisation.

One common reference that economists use for this is Stokey, Lucas and Prescott. Of course, they focus exclusively on recursive methods, but (perhaps) the lion’s share of dynamic problems in economics are solved in this way.

There is also a treatment of dynamic optimisation (including non-recursive and/or continuous-time methods) in Acemoglu.