Determining imaginary axis crossing of a root locus
I'm not sure about your calculation but Matlab yields the correct result. In this problem, the common approach is to use  Routh-Hurwitz criterion and search for  a row of zeros that yields the possibility for imaginary axis roots. For convert the system to the closed-loop transfer function, hence 
$$
\frac{K}{s^4 + 10s^3 + 88s^2 + 256s + K}
$$

The Routh table is 

$$
\begin{matrix}
s^4 &amp;&amp;&amp;&amp; 1    &amp;&amp;&amp;&amp; 88  &amp;&amp;&amp;&amp; K \\
s^3 &amp;&amp;&amp;&amp; 10   &amp;&amp;&amp;&amp; 256  \\
s^2 &amp;&amp;&amp;&amp; 62.4 &amp;&amp;&amp;&amp; K    \\
s^1 &amp;&amp;&amp;&amp; \frac{15974.4-10K}{62.4}  \\
s^0 &amp;&amp;&amp;&amp; K 
\end{matrix}
$$

The \$ s^1 \$ row is the only row that can yield a row of zeros. From the preceding row, we obtain 

$$
\begin{align}
&amp;15974.4 - 10K = 0 \\
K &amp;= \frac{15974.4}{10} = 1597.44
\end{align}
$$

Now we take a look at the row above \$s^1\$ and construct the following polynomial, hence 

$$
\begin{align}
62.4 s^2 + K &amp;= 0 \\
62.4 s^2 + 1597.44 &amp;= 0 \\ 
s^2 &amp;= \frac{-1597.44}{62.4} \\
s_{1,2} &amp;= \pm j \sqrt{25.6} \\
s_{1,2} &amp;= \pm j 5.0596 \\
\end{align}
$$

The root locus crosses the imaginary axis at \$\pm j5.0596\$ at the gain \$K=1597.44\$. Consequently, the gain \$K\$ must be less than 1597.44 for the system to be stable.