Capacitive AC to DC power supply noise
Please don't wire up that circuit as shown. It's illustrative of principles but is quite dangerous without additional components. We don't want to lose any of our fellow enthusiasts!
To be more direct about it - don't use that type of circuit to power your electronics. It's incredibly unsafe unless it and everything connected to it are totally physically isolated. That includes ground nodes, microcontroller GPIOs, everything. It's also very dangerous to work on unless you're completely competent working with high voltages. Please, use a transformer.
To make it safe, things like the following are needed: A fuse, an inrush current limiting resistor or thermistor, C1 must be an X-cap, R1 must be rated for hundreds of volts. We all have to learn somewhere; it's hard to know all of this before we ever hook anything up. So find someone with experience with offline power supplies to look over your shoulder as you go. This isn't the kind of circuit to be cutting one's teeth on without great care.
Why are things blowing up?
If you plug into mains at the instant that mains voltage is near its peak then that surge can shoot through to the output and destroy low voltage components that are connected to it and then propagate down ground nodes, supply nodes, all over the place. There could be some hundreds of voltage anywhere. An inrush limiting resistor will help this a lot. Also, keep Z1 and C2 in parallel but add a current limiting resistor between them and the diode bridge. This will also help suppress voltage surges and noise spikes.
On the other hand, if you plug in as the mains voltage crosses zero then it will be a much smoother turn-on.
There are also all kinds of high voltage spikes and noise that can be seen on AC mains that won't be adequately filtered by the capacitor and zener diode.
So, find a transformer to step down to 9V or 12V or so and then regulate down from there. And, still use a fuse and be careful. Good luck!