Determine filter type (HPF or LPF) by its coefficients I want to determine optimal initial states for a FIR/IIR filter to get rid of oscillations and shifts at the initial interval. 

I've found that these initial states depend on filter type in the following way (up to a constant):

for k=1:order % (Direct Form II Transposed)
   z_hp(k) = sum(1*b(1+k:end)- 0*a(1+k:end)); % for HPF:  x=const, y=0
   z_lp(k) = sum(1*b(1+k:end)- 1*a(1+k:end)); % for LPF:  x=const, y=const
end


Then I should choose z_hp or z_lp - how can I do this given only filter coefficients?

By now the solution is empirical - passing 10 points of constant value to the filter and compare the shift in output:

x = ones(1,10);
y_hp = filter(b,a,x,z_hp);
y_lp = filter(b,a,x,z_lp); 

if (1-y_lp) &gt; y_hp
    z = z_hp;
else
    z = z_lp;
end


I wonder if I there is a simple formula for this.
I am not sure if my answer will correctly suits your question as it seems a little confusing. Anyway, I understand that you want to eliminate the initial transition of your filter. What I propose is to initialize your filter state so that the output equals the input at $t=0$

I had to solve that problem in the past. What I did was to assume that all delays were initialized to the same value. Doing that, you simplify your problem a lot. I don't have a lot of theory to back this technique, but that seems to work quite good on my side.

Let's start with Direct Form II



If we consider that all delays are set to the same value $K$, we find this output equation.

$$ y = (-a_1k-a_2k+x)b_0+b_1k+b_2k $$
A little algebra : 
$$ y = b_0k(-a_1-a_2-...)+b_0x+k(b_1+b_2+...)$$
$$ y = -kb_0\sum_{n=1}^{N_a}{a_n} + b_0x+k\sum_{n=1}^{N_b}{b_n}$$

Now, since we know we want the output to be equal to the input, we can say $y=x$

$$ x = -kb_0\sum_{n=1}^{N_a}{a_n} + b_0x+k\sum_{n=1}^{N_b}{b_n}$$

Then

$$ x-b_0x = -kb_0\sum_{n=1}^{N_a}{a_n}+k\sum_{n=1}^{N_b}{b_n}$$
$$ x(1-b_0) = -kb_0\sum_{n=1}^{N_a}{a_n}+k\sum_{n=1}^{N_b}{b_n}$$
$$ \frac{x(1-b_0)}{-b_0\sum_{n=1}^{N_a}{a_n}+\sum_{n=1}^{N_b}{b_n}} = k $$

So there you go. Initialize all your delays with $K$ and your filter will start with an output equal to the input. 

You can do this for other forms as well.

Hope that helps