Keyboard layout for math
I believe this is all possible. In my answer on TeX.SE, I give the MathUnicode.keylayout file that I made. Essentially, my advice is: go ahead and play around with this file. 

A couple of thoughts:


The XML specification for .keylayout files is available from Apple.
On line 22 you can see that I refer to the anyOption modifier. This refers to the left or right option key. But you can also write option or rightOption if you wish to distinguish between the two.
I'm not sure whether caps can be treated as a "hold down" key, as you want it, rather than a "press on / press off" key.