Mathematical rigour in Signal and Systems
This xkcd comic comes to mind.



EE is some kind of hybrid of applied physics and mathematics. It's easy for a mathematician to pick up an electronics book and move to the left. It is not as easy for a EE to pick up an advanced mathematics book and move to the right. 

For most practical things, the the math is well understood. For example, filters are commonly developed from canned topologies, and most practicing EEs let MATLAB or other software do the number crunching. It's just not worth the time it would take to work through all of the transforms manually to get the transfer function, especially for filters higher than second order. If the filter is going to be implemented digitally, all it takes is a z transform and some algebra to get to a difference equation.

The majority of the guys that care about the math at the level you're interested in are researchers. You might look at the Proakis signal processing and communications books. They have quite a bit of math in them, but your expectations aren't clear enough to me to say whether you'll be satisfied with them. You would probably be better served to look for straight math books that cover the topics of interest. Unfortunately, I cannot provide any recommendations there. That question may be better suited for Math.SE