what is matlab function is taking as sign of second order IIR filter?
The filter function assumes this form (for your case):

$$
\tilde{H}(z) = \frac{b_0+b_1z^{-1}+b_2z^{-2}}{1+a_1z^{-1}+a_2z^{-2}}
$$

where the $1$ on the bottom is from the assumption that $a_0=1$, and call it $\tilde{H}(z)$ so not to be confused with your $H(z)$. Since you have written the denominator with the signs of $a_1$ and $a_2$ inverted, then you just have to fix that before using filter. 

Suppose someone wrote the transfer function from your question down $\big(H(z)\big)$, handed you the coefficients, and said use MATLAB to find the output. The only thing you need to do before is to invert $a_1$ and $a_2$: 
filter(..., [a0, -a1, -a2], ...).

When using besself, it returns the coefficients in the form according to the transfer function $\tilde{H}(z)$, and you can then directly use those coefficients.