Optimally approximating the sign function by functions with compactly supported Fourier transform I'm re-posting a question of mine from math.stackexchange in hopes that folks here might have the right kind of expertise.
I'm looking for a systematic way to approximate the sign function
$$\operatorname{sgn}(x) \triangleq \begin{cases}1&amp;\text{ if }x &gt; 0\\
0&amp;\text{ if }x=0\\
-1&amp;\text{ if }x&lt;0\\\end{cases}$$
by functions $f$ whose (distributional) Fourier transforms $\hat{f}$ are compactly supported.
More precisely, let us fix some $\epsilon$ small. Then I am interested in odd, continuous $f$ such that:

$\text{supp}{\hat{f}}\subseteq[-\pi,\pi];$
there exists some $a_f&gt;0$ such that $|f^2(x)-1|&lt;\epsilon$ whenever $|x|\geq a_f$.

Goal: I am looking for a &quot;best&quot; $f$ in the sense $a_f$ is as small as possible.
I figured that one way to do this would to be to look at convolutions $f=a*sgn$ where $a$ is a function that satisfies $\int_\mathbb{R} a(s)\,ds=1$ and $\text{supp }{\hat{a}}\subseteq[-\pi,\pi]$. For example, we could take

$a_1(s)=\dfrac{\sin(\pi s)}{\pi s}$; or
$a_2(s)=\frac{1}{2}\left(\dfrac{\sin(\frac{\pi}{2}s)}{\frac{\pi}{2}s}\right)^2,$

where the Fourier transforms of $a_1$ and $a_2$ are a rectangular and a triangle pulse respectively.
Doing some numerical experiments, I find that for the choice $\epsilon=0.05$, $a_2$ is &quot;better&quot; than $a_1$ by the above criterion.
Question: Is there a more systematic way to find better (or a best) $f$? References that deal with this type of problem would also be appreciated.
This problem is in a way the dual of the problem of approximating a Hilbert transformer by a filter with finite memory. The frequency response of an ideal Hilbert transformer is
$$H(\omega)=-j\,\textrm{sgn}(\omega)\tag{1}$$
and the corresponding ideal impulse response is
$$h(t)=\frac{1}{\pi t}\tag{2}$$
We can approximate $(1)$ and $(2)$ by multiplying the ideal impulse response by a window of finite length:
$$\tilde{h}(t)=h(t)w(t)\tag{3}$$
where the window function $w(t)$ is a real-valued even and smooth function with finite support.
In your case you could use a window function $W(\omega)$ in the frequency domain to approximate a sign function:
$$f(t)=\textrm{p.v.}\frac{1}{j\pi}\int_{-\pi}^{\pi}\frac{W(\omega)}{\omega}e^{j\omega t}d\omega=\textrm{p.v.}\frac{1}{\pi}\int_{-\pi}^{\pi}\frac{W(\omega)}{\omega}\sin(\omega t)d\omega\tag{4}$$
Note that this is just a generalization of what you already did. You used a rectangle and a triangle for $W(\omega)$. However, there are many more &quot;nicer&quot; window functions that are smooth, and that may result in better approximations of the sign function.