HV9961 led driver: trying 170V led output
I said:


What is the diode package?
How is it heatsunk?
How hot does it get?
What is its FORWARD voltage .   

Reverse voltage should be only about 330 VDC.
Dissipation steady state should be maybe slightly over 0.5 Watt. (Fig1 page 3, 0.4A, ~~~= 50% on.
I'd not expect dI/dT to be an issue.
Maybe recovery losses but ...?
Try a say 10 Ohm resistor n series with D1 - it drops about 3V in operation (so minimal loss %) but may soften the Q1 turnoff. Maybe a small cap across D1. Small. 

If you have used a high thermal resistance pkg device, heatinking is minimal and it is filing due to overtemperature. 


  You said: I used SMB package for the diode STTH2R06U.
  The diode temperature increases very quickly when the design is powered and it becomes very hot before crashing. 


SMB Rth-jc is 25C/Watt so you should have no problems IF the diode has adequate heat sinking. BUT fig 11 in datasheet shows Rth-ja of 90 C/W - without any heatsinking you are liable to "have problems" if air access is other than superb.

The datasheet says that 3cm^2 of FR4 PCB copper gives about 60 C/W.


  I cannot measure temperature and there is no heatsink for my SMB package. But here I need a quick course about "Thermal resistance junction to ambient versus copper surface under each lead". 


If you burn yourself on it you probably need more heat sink.
It COULD run far hotter than that and be OK but ... .
If you are JUST able to keep touching the case it is at about 55 C.
If a drop of water on it boils or if when you lick your finger and dab at it with the wetted finger it hisses then it's over 100 degrees C.
Even that's perhaps OK but also maybe not.
If it DOESN'T ever get hiss test hot it's probably OK. 

Below: j=junction, c=case, a=air.
Rth = thermal resistance.
Rth_jc is Rth from junction to case.
Rth_ca = case to air.
With no heatsing Rthja = 90 C/W. ie junction is 90C > air temp with no heatsink.
Rthjc is 25 C/W.
So Rth_ca = Rth_ja - Rth_jc = 90-25 = 65 C/W.
Si if air is 20C and case just sizzles = 100C then
Tj = 100 + 25/65 x (Tc-Ta) = 100 + 31 ~= 130 C.  

Tj_allowed is &lt; 175C so IF you can keep the leads below 100C (sizzling hot) the temperature SHOULD be OK. Should.