How difficult is it to work on mathematics problems as a computer science graduate student?
It's very common to move from maths to CS: for example, in my office of four people in CS, one is doing a PhD in CS after doing an undergraduate maths degree and another is a postdoc who just finished a PhD in maths. Moving from CS to maths seems much less common.

When you say "graduate student" it's unclear whether you mean master's or PhD. It sounds like you're interested in lots of areas of mathematics (you've mentioned geometry, topology, quantum mechanics and string theory in this and your other question), which doesn't seem focused enough for a PhD: for that, you need to pick one, fairly narrow area.  From what little I know about you, it seems that a master's in maths would be more appropriate, perhaps followed by a PhD in either maths or computer science if you find a particular subject whose computational aspects interest you.