Grounding and unisolated power supply
If it's a desktop PC then the metal case, all the shields of all the cables, and the internal GND, are all tied to protection earth.

So USB GND is tied to Earth.

The other side of your USB-Serial converter has GND tied to mains.

Thus you shorted Live to Protection Earth and exploded whatever bits of electronics were in the way... I hope the lights were out in the lab after this. If you still had power, this means you have no RCD so you are at risk.

Now, if you had used a laptop running on batteries, then nothing would have blown, since the laptop is floating. However... every bit of metal in and on that laptop would have been at live mains potential.

Also everything that is connected to it via USB.

This includes lots of stuff you're supposed to put your fingers on: mouse, keyboard, that shiny aluminium case... so many ways to kill yourself!...

The insulation transformer turns your uninsulated SMPS into an isolated one, so it could work. I have no idea why you chose to make an uninsulated supply, maybe to save a few cents?...

Note: put the isolation transformer on the DUT, not on the PC! If you supply the PC from an insulation transformer, then you'll be in the "laptop" case, and you're gonna fry.

If you insist on the uninsulated SMPS, a much better solution to talk to your micro is to simply isolate the USB-serial interface with optocouplers. Then, there s no need for an isolation transformer. Serial is easy to isolate. Also this will save your PC USB port from unhealthy voltages.

But of course, no fingers shall go near the live device...