What's the ideal FFT window for measuring a group of signals of differing amplitudes but close in frequency?
As for ideal windows for FFT frequency resolution and dynamic range, many assume that the Gaussian window would be best since it has the minimum time-bandwidth product, and due to this it is the filter most often used in spectrum analyzers. However, this is only true when the time domain extends to infinity. For finite time durations the window that actually has the minimum time-bandwidth product is the Discrete Prolate Spheroidal Sequence (DPSS, or Slepian) window which maximizes the energy concentration of the main lobe. The Kaiser window very closely approaches this optimum and both have an controlling parameter to trade frequency resolution (ability to discern two closely spaced frequencies) and dynamic range (ability to discern two frequency tones of different amplitudes). For further details on the comparison of the two, see this link by Julius Smith: https://ccrma.stanford.edu/~jos/sasp/Kaiser_DPSS_Windows_Compared.html
That said, the signal after IQ demodulation could be windowed and then FFT'd to provide the phase and amplitude (relative to the demodulation signal used) of closely spaced modulation frequencies. For this application I would use a Kaiser window with Beta = 8 which provides 80 dB dynamic range of signals captured over a 2 second duration while being able to resolve two signals that are 4 Hz apart.