Higher order filters
Main purpose of my filtering is to smoothen the signal, remove those steps. I want to know how much benifit will i get by increasing the order of the filter, mathematically and practically


There are basically two parameters for filters like a Butterworth lowpass: Its order and the cutoff frequency. Both have influence on the response.

Setting the cutoff frequency lower will make the response more smooth. Setting it higher will make the response follow more closely the steps because more of the frequencies are retained and less are attenuated.

The filter order controls how selective the filter is / how "sharply" it separates the passpand (the frequencies that are preserved) from the stop band (the frequencies that are rejected. Inbetween the passband and the stopband you have the transition band which will get narrower the higher order you use. In the time domain higher order filters tend to make the response more wavy. The filter's impulse response oscillates more and its envelope decays more slowly. Choosing the filter order is kind of a trade-off between frequency selectivity and time selectivity. By time selectivity I mean that a tiny pop in your source signal will affect a larger window of time in the filtered signal.

Without knowing exactly what you are shooting for, it's hard to suggest anything. I hope this explanation of how the two tuning knobs affect the response will enable you to move into the right direction.

If you want to do noise reduction it's generally good to know what the PSD of your signal and the noise part looks like. In your case, the noise is highly correlated and depends too much on the signal. You basically have truncation without any prior dithering which would otherwise decouple the noise error from the signal. So, in your case it's hard to predict the PSD of the error. But given both spectra, Wiener filtering could get you closer to the solution. Then, a good filter's cutoff frequency would depend on the point where the signal PSD crosses the noise's PSD. The order would depend on the difference of the PSD slopes between signal and noise in a loglog domain. So, for example, given a brownish signal mixed with white noise, the difference of the PSD slopes in the loglog domain would be 2. Divide this by two (--> 1) and use that as a filter order. The most appropriate cutoff frequency setting would be to make the -6 dB point the frequency where the signal PSD curve intersects the noise PSD curve.

If you care about the phase response, you could apply the IIR filter bidirectionally. Once in the forward direction and then in the backward direction. This cancels the phase response and squares the magnitude response. To compensate for this you'd reduce the filter order by about a factor of two. This might be a good approach if you do this kind of filtering offline. The phase and group delay of higher order IIR filters is higher like you observed.