Smooth brush destroys my mesh
It turns out, my pen tablet has the option to switch from Add to Substract, to do the opposite of what the brush was doing.
Whenever this happened was because I had it in substract mode, I didn't realize this because, in previous versions of Blender, Shift to smooth always smoothed stuff, but not anymore it seems.