Mathematically Inclined Signal and Systems / Signal Processing Book Recommendations
Oppenheim and Willsky's Signals and Systems or Lathi's Linear Systems and Signals are intended for Sophomores who have only a single semester of differential equations under their belts, so it is a bit unfair to criticize them for leaving out the functional analysis and the conformal mapping.  At the sophomore level my favorite book is Siebert's Circuits, Signals, and Systems.  It won't give you the mathematical rigor you desire either, but you can see his great love for the mathematics and he has these wonderful, witty, footnotes that provide a great historical perspective. 

There is a great book (which I love, but do not recommend to you) by the (applied) mathematician Richard Hamming (of the "Hamming window", "Hamming code", "Hamming distance", "Hamming bound" and "Hamming problem") called Digital Filters.  In it he makes a number of snarky comments like:


  Since we are interested mainly in using mathematics, we are obliged in our turn to be ambiguous with respect to mathematical rigor.  Those who believe that mathematical rigor justifies the use of mathematics in applications are referred to Lighthill and Papoulis for rigor; those who believe that it is the usefulness in practice that justifies the mathematics are referred to the rest of this book.  (1998 Dover edition, page 72.)


So in addition to the book by Papoulis that @Matt_L recommended I will add Hamming's (and Siebert's!) recommendation of M.J. Lighthill, Fourier Analysis and Generalized Functions, Cambridge University Press, 1958.