FIR filter digital differentiator with low cutoff I'm trying to design a digital differentiator FIR Filter. It features a lowpass, such that above the cutoff frequency the amplification is very low. I get the coefficients by a linear program minimizing the chebychef error of desired and actual frequency response.

It works really well, but I cannot place the cutoff frequency below some 0.1*pi rad/sample. Small cutoff frequencies still have very steep rising amplitude responses in low frequencies and thus need a broad transition band.

The picture shows such a design and the very broad transition band. The red is the desired and blue obtained frequency response. I've weighted the bands accordingly. Also I'm not talking about bandpass, nor lowpass, I design a differentiator - thus the linear slew rate in low frequencies.

There are limits to the possible lowpass frequency, correct? How can I make the cutoff even smaller, or even better: why is this degradation happening?

I know, that the frequency response in my formulation has the form
$$
H(e^{j\omega}) = 2\sum_{k=0}^M j \,h(k)\, \sin(k \omega)
$$
where $M$ is $(N-1)/2$ with order $N$ filter. And thus the shape can be better traced by having longer filters. But the gain actually is very small.

Also I read, that with a derived then sampled Blackman Window (without control over cutoff frequency) one obtains a cutoff of around $\omega_C \approx 0.005$, while I struggle with $0.1$... I want to know why exactly.

This document suggests a method for a first order differentiator, where "only" the derivatives at $\omega = 0$ are matched to the ideal one. It results in an earlier drop off. However, as I understand it, this cannot be achieved with higher order differentiators, since second order is a quadratic function and I am not sure if a (basically taylor) approx in derivatives is sufficient for that. Let alone even higher orders.
What you want is what I call a Jekyl and Hyde design. The low pass is parsimoniously represented in terms of cosines and the derivative in sines. So let Jekyl be Jekyl and Hyde be Hyde.

The way to go very low would be a multirate approach.  Low pass and decimate for a few cascades until the high pass part of your differentiator is a reasonable fraction of the band.  If you need the original sample rate, upsample appropriately.  Your filter plot implies that you really aren't interested in most of your original band, so why retain it?