Coupling cofficient of coil in Resonant Series RLC Circuit
Hint - if the two inductors were perfectly coupled (k = 1) then you would get a total reactance of j27 ohms. Are you able to see that? If you are then clearly, to obtain a total reactance of j18 ohms k must be less than 1.

How did I get j27 ohms - it's a sanity check - reactance or inductance is proportional to turns squared and if coupling is 100% then the turns for L1 and L2 are perfectly coupled and can be regarded as being on the same former so, new impedance is \$j(\sqrt{12} + \sqrt{3})^2\$ = j27 ohms.

I think the mistake I see is that you equate the reactance to -j18 when in fact M should equal j3.