Phase error correction for Fourier transform basis vectors I was wondering what is the best way to account for phase noise in a set of basis vectors?

Example:
I have a measured signal, say $f'(x)$ that should be related to a desired spectrum $f(\nu)$ through the following relation (which is a Fourier transform for x-symmetric, band-limited signals, as pointed at in comments below):

\begin{equation}
f(\nu) = \int_0^{x_\text{max}} f'(x) \cos(2\pi x \nu)\,dx \tag1
\end{equation}

This can be expressed with vectors and matrices as $f'(x) = M\cdot f(\nu)$.  Here, the rows of $M$ correspond to an orthogonal set of sinusoidally varying basis vectors.

In my particular problem, the system has constant phase error that results in an actual relation of:
\begin{equation}
f(\nu) = \int_0^{x_\text{max}} f'(x) \cos\left(2\pi x \nu + \phi(x)\right)\,dx \tag2
\end{equation}
When I calculate this particular calibration matrix $M'$ (s.t. $f'(x) = M'\cdot f(\nu)$) for this system and perform the Moore-Penrose pseudo-inverse ($M'$ is non-square), I find very significant errors resulting in my spectra calculated via $f(v) = M'^{-1}f'(x)$.  I have also confirmed that spectral reconstruction via the pseudo-inverse without the phase noise works well (as I would expect). 

Assuming I know apriori what my phase errors $\phi(x)$ are, is there a better way for me to reconstruct my spectrum $f(v)$ than through just the pseudo-inverse of $M'$?  Any additional insight or ways of looking at this are appreciated.

Edit:
Assume the matrix $M'$ is measured with a known, single frequency input so that the rows of $M'$ are indeed sinusoidally varying basis vectors, and the spectrum is known for $k$ points, and the signal is measured for $m$ $x$-points, so $M'$ is a $k\times m$ matrix ($k&gt;m$). I could simply measure fewer spectral points, so $M'$ is square, though I had thought oversampling might provide some additional information.
This can be expressed with vectors and matrices as $f'(x) = M\cdot f(\nu)$.  Here, the rows of $M$ correspond to an orthogonal set of sinusoidally varying basis vectors.


Woah, there, that's a big step you're doing. So, you have your band-limited signal; it has to be band-limited, otherwise the $x_\text{max}$ limit in $(1)$ would be illegal.

So, since it's limited in Fourier ($\nu$) domain, it has to have infinite support in time domain ($x$). (Imagine you take an infinitely long signal, and window it with a rectangular function: You need to convolve the Fourier transform of that spectrum with a scaled $\text{sinc}$ function, and that has infinite support)

Now, you're right to consider the Fourier transform as a linear operator $M$. What is very hard to do is justify that $M$ is actually a matrix with discrete rows of discrete frequencies. That integral there doesn't give you discrete, separate sinusoids, it gives you a continuum of frequencies!

Basically, the whole of the sampling theory is to prove that you can do that. Short idea: Thanks to Nyquist we know that we can represent any band-limited signal by sampling that signal at a sufficiently high rate, giving us, indeed, a discrete vector. The Time-Discrete Fourier transform of a signal collapses to a sampling rate-periodic spectrum, and we can show that this is also representable by a discrete vector, leading us to an understanding of the Discrete Fourier Transform (DFT), which actually is just a Matrix with complex numbers in it, of size $N\times N$, $N$ being the length of the time observation vector.

But: that discretization into an actual matrix over $\mathbb C$ requires you to first convert to discrete signals. Then, $M$ must be square, and is indeed (easily!) invertible. So, your problem lies in that somehow, you ended up with a non-square $M$, and that can't happen if you adhere to the sampling theorem.