Designing butterworth filter with pole placement
I think I found the problem. Everything was fine except for the denormalization step for arbitrary cut-off frequencies wc. I changed the following lines to obtain the correct state-space matrices (and to handle the case n=1):


...
if (n == 1)
    a = [1,1];
else
    p = ...
    ...
end

a = real(a);
a = a./(wc.^(n:-1:0))   % new!
a = fliplr(a);
...

% last_row = (wc.^(N:-1:1)).*last_row;    % removed!

if( n == 1)
    this.x = last_row;
else
    k = ones(1,N-1);
    this.x = diag(k,1) ;
    this.x(end,:) = last_row;
end

...

this.y(N,:) = 1/a(end);    % changed!

...



I used the function ss2tf to compute the polynomial coefficients from the state-space matrices computed above, and the coefficients were correct for different values of wc. This means that the matrices are correct. If the output signal is still different from what you expect, the problem must be in the filtering routine.