Fourier series representation
However what is conceptually wrong with using say an infinite sum of time shifted rect functions to represent it? 


There is nothing conceptually wrong with it.  Fourier transforms decompose a signal into a sum of complex sinusoids, but you can also decompose a signal into many other things, which may be more useful in certain applications.  The Haar wavelet transform, for instance, breaks up a signal a sum of rectangular pulses:

 source

We use sinusoids in lots of applications because it makes the most sense in those applications.  For instance, why do we almost always decompose audio signals into sinusoids?  Because our cochleas do the same thing: