Why do butterworth filter sometimes make signals diverge?
Basically, this is bound to happen with higher order filters and finite precision. The filters' poles are getting closer and closer to the unit circle for sharper transition bands. But poles very close to the unit circle lead to unstable filters for some level of precision.

Things you could do:


Increase the precision of your numeric type (e.g. double instead of single precision)
Factor the filter into a series of 2nd order sections


Matlab and Octave allow you to directly compute the poles (p), zeros (z) and the overall gain (g) instead of the filter coefficients b and a:

[z,p,g] = butter(...);


and the result can be converted to a SOS matrix (second order sections):

[sos,g] = zp2sos(z,p,g);


Each row in the sos matrix represents a biquad, a 2nd order IIR filter. Using this second order section factorization -- meaning: a couple of 2nd order IIR filter in series -- is less subspectible to rounding errors as far as I know. Actual filtering can be done with sosfilt instead of filter:

y = sosfilt(sos,x)*g;


And if you still have trouble getting it stable, you should probably replace butter with ellip and allow a bit of passband ripple and/or do multiple passes of the same lower order filter.