Where can I begin to study the math behind modern cryptography? Why do I start the word buffers in MD5 with those values specifically? What in the world motivated the decision to use the numbers 1, 1, 2, 3 for the mix columns stagein AES? And what kind of witchcraft is going on in SHA when I call the sigma_x functions?

What kind of math should I be starting with, assuming I only know how to add, subtract, multiply, divide, and mod?
One of the best avenues for study is to practice breaking existing cyphers.  Start small, with classic cyphers such as monoalphabetic substitution, polyalphabetic cyphers, transpositions, etc.  Do the daily Cryptoquip in the newspaper!  Learn what makes them weak, what makes them exploitable, and what makes them strong.  Then move on to stream cyphers, to mechanized cyphers like the Hebern and Enigma machines, and then to modern electronic block cyphers.  

With each cypher you attack, you first learn how to attack a simplified or reduced round variant, then apply it to the more complex and complete implementations.  Along the way, learn how some of the various protocols and implementations have made cyphers weak, or provided defenses against certain forms of attack.  Finally, study some of the advanced attacks, and replicate their results.

To implement these attacks, you'll discover new topics you need to understand in order to pull them off:  statistics, modulo arithmetic, algebra, recursion, protocols, zero knowledge proofs, bignum math, game theory, Feistel networks, whatever.  These are the same tools you will later use to understand and construct cyphers and protocols of your own.

Once you study attacks on cyphers, protocols, and implementations, you'll better understand why some of the more esoteric design decisions were made, or why certain mechanisms were added.  A fascinating case study for the value of this approach can be a found by reading the history of the Lucifer cypher.

Good luck!