Hi, If I really want to focus on the mathematics of signal procesing, Should I do a Masters in EE or Math?
As stated in the comments, this is more about the content than the label. The best is to play with both fields if you want to invest in the future: signal processing is borrowing  increasing from mathematical tools, for instance convex and non-convex optimization, learning, topological and geometrical tools (for dimension reduction), graphs, etc., that might  not be taught in traditional EE. Such techniques can help with the growing quantity of data with little models and knowledge. 

As a personal anecdote, I attended a lecture on "math for EE engineers" for the beginning of a signal processing PhD program in a US university. It was about Euclidean vector spaces (in finite dimension of course). A young PhD student asked whether the basis vectors did belong to the space. I was quite surprised by the level of the question. But not as much as when I hear the teacher say, after a long pause: "mmmm, I'd say... yes, let me think about it" (such a question is indeed interesting in infinite dimension spaces). There are similar  stories from colleagues attending other MsC programs.  

So after picking the right topics, I strongly suggest  that you either complement an EE program with maths, to get better insight and intuition about the theoretical    tools, or dive in maths, while practicing a lot on "real DSP problems" aside. The gap is large between mastering  Fourier theory and being able to properly estimate and interpret  of a power spectrum of a noisy signal.
There are plenty of online resources to assist you in that task.

I really appreciate the concept  behind the law of excessive learning in mathematics, stated by Alexander Borovik: 


  To be able to use maths at certain level it is necessary to learn it
  at the next level