Find the largest value of \$K_1\$ and \$K_2\$ for instability of the system
You seem to be doing things right. I verified them using Mathematica. So either you have \$G_1\$ and \$G_2\$ wrong, or the question is wrong.



There are two possibilities:


The max value of \$K1\$ is \$\frac{9}{2}\$, and the max value of \$K2\$ is \$\infty\$.
\$K1\$ is some finite value greater than \$\frac{9}{2}\$, and \$0&lt;\text{K2}&lt;\frac{20}{2 \ \text{K1}-9}\$.