Is this circuit linear or not?
\$R_{\theta JA}\$ is one small part of the story. The main part of the story is \$R_{\theta JC}\$ (at 1.5 °C/W), because this is the junction-to-case thermal resistance and it adds to the heatsink thermal resistance to give the lowest (normally) path for heat.

So, if the thermal resistance of the heatsink is (say) 6 °C/W then the total thermal resistance that matters is 7.5 °C/W (plus the typical figure of 0.5 °C/W for the interface between transistor case and heatsink). A total of about 8 °C/W.

Therefore 17.5 watts would heat the junction by an extra 140 °C for a 6 °C/W heatsink. In an ambient of 25 °C this means a junction temperature of 165 °C and pretty close to the 175 °C limit for the device.

You also need to ensure that local ambient doesn't rise due to this heatflow. It could easily rise by another 20 °C if the heat is not effectively taken away. So now you might be into the wrong side of the specification and you should expect trouble if the heatsink is only 6 °C/W.

Given that you only saw a temperature increase of 50 °C I suspect you are using a heatsink with thermal resistance much lower than what I guessed at.


  Also I scoped the gate voltage using an oscilloscope and there is a
  switching.


You have a 2 µF capacitor on the op-amp output, and this massively alters the open loop gain of that part. Basically it takes it several steps forward to becoming an oscillator due to the extra (almost) 90 degrees phase shift you have added due to the 2 µF capacitor and the op-amp's output resistance of several ohms. Don't do this.

You are also using an LM211 and that part is a comparator - these are not guaranteed to be stable with negative feedback, so this is another problem. You have a few problems here.