Incorrect Butterworth filter outcome using Rational Transfer Funcion I've been playing with Second-Order Butterworth high-pass filter for a while.
I've calculated the coefficients using Matlab function and I wanted to use them to filter simple sine wave generated by Matlab. 
Unfortunately the results are not correct/consistent, verified by Matlab filter(b,a, signal) function.
My Matlab code is as following (ommiting plotting part):

clear all;
fp = 10000; %sampling frequency
ts = 0:1/fp:0.05; %number of samples, length 
squareF=100; %frequency of signal
squareWave = sin(2*pi*squareF*ts); %generate the sine
fc = 500;
[b,a] = butter(2 ,fc/(fp/2), 'high'); %numenator, denumenator
for n = 3:501
    filteredWave(n) = (b(1)+b(2)*squareWave(n-1)+b(3)*squareWave(n-2))/(a(1)+a(2)*squareWave(n-1)+a(3)*squareWave(n-2))*squareWave(n);
    %Rational Transfer Funcion
end
filteredCorrectly = filter(b, a, squareWave);


Filter plots are below.

Where is the error?
The output can be calculated using the coefficients as follows:

$$a_0y(n)+a_1y(n-1)+a_2y(n-2)=b_0x(n)+b_1x(n-1)+b_2x(n-2)$$

In your formula you are not doing this. Instead, you are doing a division with terms that only involve the input and not the output, which doesn't make sense as the filter is IIR.

I suspect that you got confused with the transfer function definition of an IIR filter, which is given by

$$H(z)=\frac{Y(z)}{X(z)}=\frac{\sum\limits_{k=0}^2 b_kz^{-k}}{\sum\limits_{k=0}^2 a_kz^{-k}}$$

Note that you are using an equation in the $z$-domain applied to time-domain signals (to put it simply). That's why the loop you wrote is not working.