How to improve accuracy while converting floating point coefficients to fixed point in the case of an all pole IIR filter
Where are the poles of your filter? If they are close to the unit circle, then quantizing them can cause some of them to cross beyond the unit circle, causing the quantized filter to be unstable. You should check the poles of your filter before quantizing and after quantizing. See how close they match.
Secondly, it's risky, albeit not impossible, to implement an order-5 IIR filter directly in fixed-point arithmetic. Usually, it's a good idead to split it in cascaded order-2 filters, each of these order-2 IIR filters is usually called a &quot;biquad&quot;. In your case you would have 2 cascaded biquads with one order-1 IIR filter.
There are many references that you can check out such as this one.
https://www.dsprelated.com/showarticle/1137.php