How can infer instability from the frequency response of control system?
Some comments to your question:

(1) A frequency response analysis can describe the systems behaviour only if the system is LINEAR (which is not the case for the thermostat system).  

(2) For linear systems, it is best to open the feedback loop (however, for finding the conditions and the correct place for the opening requires some deeper knowledge). Then, you can analyze the loop gain response and apply one of the stability crireria (Nyquist, Bode). This method allows to determine the degree (quality) of stability (phase and/or gain margin)

(3) There is a rough method to evaluate the closed-system response (stable yes/no): The magnitude will exhibit a peaking in the "critical" frequency region where the system could oscillate. In this case, the SLOPE of the phase function at the peaking frequency must be NEGATIVE for a stable system (and will be positive in case of instability).   

(4) A circuit involving thermostat devices should be analyzed in the time domain only.