Short while trying to measure line voltage on a 3 way switch
Well it's possible you're unrealizingly touching something else, like the shell of the metal junction box, but that would cause arcing damage to your test lead.

Yes.  The switch you are probing is defective.  

Cracked switches happen all the time.  When you lean on it a little bit with the probe, you are causing something internally to make contact and POOF.  That's why it's not causing arc damage to your test lead. 

Needless to say, stop doing that. 

I would first replace that 3-way with a plain 3-way.  Since you seem to be replacing several 3-way sets with smart switches, I would do this one second, so you can use the 3-way liberated from your first repair to fix this one.  

Do not try to short-circuit (heh) the process and just upgrade the broken switch.  It's possible something else is broken and you want to remove that possibility before you proceed, otherwise you'll have a miserable time troubleshooting.