Optimally approximating the sign function by functions with compactly supported Fourier transform
This problem is in a way the dual of the problem of approximating a Hilbert transformer by a filter with finite memory. The frequency response of an ideal Hilbert transformer is
$$H(\omega)=-j\,\textrm{sgn}(\omega)\tag{1}$$
and the corresponding ideal impulse response is
$$h(t)=\frac{1}{\pi t}\tag{2}$$
We can approximate $(1)$ and $(2)$ by multiplying the ideal impulse response by a window of finite length:
$$\tilde{h}(t)=h(t)w(t)\tag{3}$$
where the window function $w(t)$ is a real-valued even and smooth function with finite support.
In your case you could use a window function $W(\omega)$ in the frequency domain to approximate a sign function:
$$f(t)=\textrm{p.v.}\frac{1}{j\pi}\int_{-\pi}^{\pi}\frac{W(\omega)}{\omega}e^{j\omega t}d\omega=\textrm{p.v.}\frac{1}{\pi}\int_{-\pi}^{\pi}\frac{W(\omega)}{\omega}\sin(\omega t)d\omega\tag{4}$$
Note that this is just a generalization of what you already did. You used a rectangle and a triangle for $W(\omega)$. However, there are many more &quot;nicer&quot; window functions that are smooth, and that may result in better approximations of the sign function.