Maximum Magnitude Deviation between DFT and DTFT
okay, i'm gonna steal a diagram from this publication:



now this is an approximation that is based on the assumption that the discrete spectral points $|X[k]|$ are close enough to each other that only the discrete peak:

$$m_\textrm{d} = \max_k |X[k]| = |X[m]|$$

and the two adjacent neighbors $|X[m-1]$ and $|X[m+1]|$ need be considered. (these are shown as $k$ in the diagram above, not $m$, but i don't know if i should change the convention of discrete variables we're using and i don't wanna draw out another diagram.)  

the assumption is, within this very tight range of $\omega$, that the peak is adequately approximated by a quadratic:

$$ |X(\omega)| = m_\textrm{c} - a(\omega - \omega_\textrm{m})^2 $$

where $a &gt; 0$, and the "spectral peak":

$$\star \quad |X(\omega_\textrm{m})| = m_\textrm{c} = \max_\omega |X(\omega)|$$

so we have three unknowns: $|X(\omega_\textrm{m})| = m_\textrm{c}$, $a$, and $\omega_\textrm{m}$ and three equations by evaluating this quadratic at the discrete points $|X[m-1]|$, $|X[m]|$, $|X[m+1]|$:

$$\begin{align}
|X[m-1]| &amp;= |X\left(2\pi \tfrac{m-1}{N}\right)| &amp;= m_\textrm{c} - a (2\pi \tfrac{m-1}{N} - \omega_\textrm{m})^2 \\
|X[m]| &amp;= |X\left(2\pi \tfrac{m}{N}\right)| &amp;= m_\textrm{c} - a (2\pi \tfrac{m}{N} - \omega_\textrm{m})^2 \\
|X[m+1]| &amp;= |X\left(2\pi \tfrac{m+1}{N}\right)| &amp;= m_\textrm{c} - a (2\pi \tfrac{m+1}{N} - \omega_\textrm{m})^2 \\
\end{align}$$

so you can solve those three equations for the three unknowns.  we're not so interested in $a$, but we are interested in the frequency location of the peak $\omega_\textrm{m}$ and the height of the peak, $m_\textrm{c}$.  if you slug out solving this set of three equations and three unknowns, you get for those two parameters:

$$\begin{align}
\omega_\textrm{m} &amp;= \frac{2 \pi}{N}\left( m + \tfrac12 \frac{|X[m+1]| - |X[m-1]|}{2|X[m]| - |X[m+1]| - |X[m-1]|} \right) \\
\\
|X(\omega_\textrm{m})|= m_\textrm{c} &amp;= |X[m]| + \frac{\tfrac18 \big( |X[m+1]| - |X[m-1]| \big)^2}{2|X[m]| - |X[m+1]| - |X[m-1]|}   \\
\end{align}$$

Now this makes no assumptions about the nature of the spectrum except that the discrete spectral peak is at $|X[m]|$ and the neighboring spectral samples, $|X[m-1]|$ and $|X[m+1]|$ are no larger.  With no other information, this is about the best or only guess you can make.  It's not always perfect, but it should get you to within one-half FFT bin of the true DTFT peak.

It is perfect if you're looking at the peak of an isolated sinusoid, that was windowed with a gaussian window before going to the FFT, and you're looking at the dB or log magnitude.  Then this quadratic behavior in the log magnitude scale is mathematically the correct model and this quadratic peak interpolation is exactly correct.  So then you replace $|X[m]|$ and $|X[m-1]|$ and $|X[m+1]|$ with the logarithms (or dB values) of the same before using the above formulae to get the continuous-frequency peak.