Buck converter problem
A buck implementing a synchronous rectification scheme is a bi-directional converter: it can either reduce the source voltage like a classical buck does or, if fed from the output, it can boost the output and increase the original input. If the output is lightly loaded and you open the input line, it is very likely that the converter operates in boost mode for a few cycles. If the delivered voltage exceeds the internal switch maximum rating, you fuse it.
Below is a simple SIMPLIS cycle-by-cycle circuitry of a 5-V buck converter operated in voltage-mode control. The upper-switch S3 is closed in the beginning of the run and I will open it after some time while monitoring the input and power switch rails:

After the simulation is done, you see a peak in the input rail when the S3 switch opens:

You can see how the input voltage increases and approaches 50 V. Of course, it depends on the IC internal architecture but also how the input line is decoupled. However, as these spikes can be extremely energetic, a few might be enough to destroy the switcher. I am not saying this is necessarily the explanation in your case but it is a phenomenon at work in these synchronous buck switchers that designers must be aware of.
Additional edit
The below diagrams show how the configuration changes in buck or boost mode. In buck mode, the regular source voltage is \$V_{in}\$ while in reversed operation, the output capacitor \$C_{out}\$ supplies a boost converter.