Tyre popped when going from 26 x 1.95 to 26 x 2.125
I would say, you just didn't put the tire on well. When you inflate the tire to half its recommended pressure you should check that the tire seats well for the whole diameter around the rim on both sides. If it came out even a little, push it back in (may require some deflating). Otherwise the pressure may push the tire out like you had, and then the tube can explode.  

For your questions: both the tire and the tube can be used again. But check accurately that the tire's bead is not damaged.