Instantaneous Frequency Estimation by Hilbert Transform - Theoretical Justification and Proof I would like to better understand why the instantaneous frequency estimation by Hilbert transformation works (and especially why it doesn't work / lead to precise results in many cases).

The motivation is to estimate signal $x(t)$ by decomposing it into an amplitude envelope $m(t)$ and phase of cosine $\omega_c (t)$ (or carrier waveform):

$$x(t) = m(t)  \cos\left(\omega_c (t)\right)$$

Now, assume that $x(t)$ indeed is a result from such a process.

Questions:

1) There are two "parameters" to be estimated for any $t$, as such some constraints are needed. What are the constraints regarding $m(t)$ and $\omega_c (t)$ that are selected when applying the Hilbert transform decomposition?

2) Is there a proof available somewhere that given the constraints, the estimation indeed finds the correct amplitude envelope and carrier (for continuous and also discrete case)?
See my earlier comments here: Meaning of Hilbert transform  

Common fractal noise isn't an analytic signal (infinitely differentiable).  And a Hilbert transform re-creates the imaginary component of an analytic signal if you have the real component of the complex analytic signal (which one rarely has from real-world data).  

However, sufficiently band-pass filtered data might be similar to a finite length segment of something that looks like an infinite length analytic signal (e.g. is from a source whose behavior can be modeled or estimated by a 2nd order linear differential equation).