Phase error correction for Fourier transform basis vectors
This can be expressed with vectors and matrices as $f'(x) = M\cdot f(\nu)$.  Here, the rows of $M$ correspond to an orthogonal set of sinusoidally varying basis vectors.


Woah, there, that's a big step you're doing. So, you have your band-limited signal; it has to be band-limited, otherwise the $x_\text{max}$ limit in $(1)$ would be illegal.

So, since it's limited in Fourier ($\nu$) domain, it has to have infinite support in time domain ($x$). (Imagine you take an infinitely long signal, and window it with a rectangular function: You need to convolve the Fourier transform of that spectrum with a scaled $\text{sinc}$ function, and that has infinite support)

Now, you're right to consider the Fourier transform as a linear operator $M$. What is very hard to do is justify that $M$ is actually a matrix with discrete rows of discrete frequencies. That integral there doesn't give you discrete, separate sinusoids, it gives you a continuum of frequencies!

Basically, the whole of the sampling theory is to prove that you can do that. Short idea: Thanks to Nyquist we know that we can represent any band-limited signal by sampling that signal at a sufficiently high rate, giving us, indeed, a discrete vector. The Time-Discrete Fourier transform of a signal collapses to a sampling rate-periodic spectrum, and we can show that this is also representable by a discrete vector, leading us to an understanding of the Discrete Fourier Transform (DFT), which actually is just a Matrix with complex numbers in it, of size $N\times N$, $N$ being the length of the time observation vector.

But: that discretization into an actual matrix over $\mathbb C$ requires you to first convert to discrete signals. Then, $M$ must be square, and is indeed (easily!) invertible. So, your problem lies in that somehow, you ended up with a non-square $M$, and that can't happen if you adhere to the sampling theorem.