Simulating the earths crust
I am afraid that you aren't being specific enough to really answer your question: What about mountain chians valleys and hills are you interested in? There are simple and complex models dating back from the 1960s to present. Are you interested in learning about the material properties of the crust, or just topography in general? What about weathering effects? What about the flow laws that govern the lower and middle crust (depending on who you talk to). 

I would look into the field of dynamic topography and how that is calculated. 

Two books that I think might be useful to read are Fowler and Tourcotte and Schubert, they will help give you an overview of geodynamics as a field, and what engineering solutions we apply to Earth. In the 1-D and 2-D flow sections there is much about the crust. 

A good starting paper might be Bird 1991 (sorry, its probably behind a pay wall, but I couldn't find an open pdf on it). 

As far as codes (and consquently mathematics) The two sections of CIG I would look at are S-T Dynamics and Mantle Convection.

I am sorry I couldn't provide a comprehensive answer, but I think the sources I've linked may give you a good start and thinking about this problem.