Career in Mathematics as an Electrical Engineer
In the US it is possible to switch after undergraduate, but I'll note that the focus of an engineering degree is very different from mathematics. I'd suggest that you switch as early as possible. 

One reason that first year math isn't as rigorous as it might be, and not as rigorous as it will be later, is that those courses are intended for a general audience, including engineers, who have less need for a theoretical basis in math. The same will be true of early physics courses. 

Outside the US it may be harder to change after an bachelors degree. 

I'd suggest staying where you are only if you want a career in EE or at the junction of EE and CS. But your interests seem to be elsewhere.