Making a light detector using a phototransistor and an open drain comparator
So my problem is that comparator doesn't work like a step function,
but more like gradually increasing function

This sounds like the op-amp gives constantly on/off signal to the led, as the voltage of the phototransistor and the Voltage of the potentiometer are getting really close + due to the parasitic/stray capacitances the comparison of the two is fluctuating.