How do I test a 5V doorbell transformer?
The easiest way to test the output of a transformer is with a multimeter which can measure both volts and ohms.

If the transformer is wired into the 120VAC, and the power is on, remove any wires on the 5V side and place the multimeter probes/clips on the two connection screws on that side. You need to read the transformer output rating to see if this is AC or DC output. Set the multimeter to the correct volt range for AC or DC (5V should fall within the range) and see what the reading is.

If the reading is in the general area of 5V (say between 4 and 6), you are probably OK. If not, the transformer may be bad. 

Just knowing the ohms on the 5V side won't help because it is the ratio of turns between both sides of the transformer that counts and the wire size determines both the resistance and current output and is irrelevant in determining the output voltage.