Why doesn't KVL work for the bottom loop?
My equation was -5*ia - 6V + 18*-6R = 0


The current through the unknown resistor is not \$6R\$, it's \$6/R\$. So you should have

$$-5i_a - 6\ {\rm V} -\left(18\ \Omega\right)\left(\frac{6\ {\rm V}}{R}\right)=0$$

(I've assumed the blue probe of the voltmeter is the positive probe, because otherwise you're going to end up with a negative resistor value)

(Also, notice that if you had included the units in your equation, you would have seen right away that you were adding a voltage to a term with units of \$[\Omega^2V]\$, so the equation can't be correct)

You will get the same result by using the resistor divider formula

$$ 6\ {\rm V} = -5i_a\frac{R}{R+18\ \Omega}$$

because the resistor divider formula can be derived from KVL and Ohm's Law.