How does simulation software work at a fundamental level?
Well, that's a rather diffuse question. From what I have seen, the software packages you mention do Molecular Mechanics (MM), which is a treatment of molecules with classical physics. For each atom in a given molecule, the forces acting upon it due to chemical bonding with other atoms is calculated using a set of empirically determined parameters (a force field), and then an estimate of the potential energy is provided. But given that this question is tagged with "Computational Chemistry", I also want to mention quantum chemistry, i.e. the application of quantum mechanics to atoms and molecules. This field is both far more interesting and far more difficult than the classical MM treatment.

A first shot would be the Wikipedia article about Computational Chemistry which already points you to other relevant topics, i.e. the article about Molecular Mechanics, which is quite concise about the topic (although I don't know how exhaustive it is in this regard, since I don't deal with MM in my research). For quantum chemistry, the necessary knowledge is more spread-out; relevant Wikipedia articles would be Eigenproblems, the Schrödinger equation and Ab initio methods. I don't know any literature about MM, but if you want to look deeper into quantum chemistry, I would recommend the following two books (and lots and lots of time and patience for reading them):


Attila Szabo, Neil Ostlund: "Modern Quantum Chemistry". Provides a good mathematical introduction and thorough treatments of the Hartree-Fock and Moller-Plesset methods. It's cheap, also.
Ira Levine: "Quantum Chemistry". Covers a wider range of methods, but is in my opinion a bit more difficult to consume than Szabo/Ostlund.