Undoing a filter operation in MATLAB
In general it's only possible to implement causal and stable filters. There are exceptions where marginally stable filters are used, but this doesn't apply here. So if you want to invert a given filter, this is only possible if its zeros are inside the unit circle of the complex plane (such filters are called minimum-phase filters). The zeros of the given filter become the poles of the inverse filter, so if the zeros are inside the unit circle, the poles of the inverse filter will also be inside the unit circle, and, consequently, the resulting filter is causal and stable.

The problem with the filter in your example is that it has a sixth order zero at $z=-1$, i.e. on the unit circle. Consequently, this filter is not strictly minimum-phase and it can't be inverted by a causal and stable filter. However, it can be inverted approximately by a causal and stable filter. One way to achieve such an approximate inversion is by minimizing the mean squared error between the ideal and the actual filter response. In practice, it is often acceptable to add some delay to the output, which will make it easier to approximate the desired response by a causal and stable filter.

Note that transients don't play a role. It's the inverse filter's instability that is the problem here.

As an example, take a filter with coefficients

b = [1,0,-0.5]; a = [1,0.3,0.1];

The zeros of the numerator polynomial (with coefficients b) are all inside the unit circle, so the inverse filter is causal and stable.


x = rand(100,1);
y = filter(b,a,x);
x2 = filter(a,b,y);
max(abs(x2-x))    % in the order of 1e-16, i.e., machine precision