How do I set my multimeter to test doorbell transformer
Yes your selecting an AC voltage setting is correct for a typical doorbell transformer. 



If you can find the transformer itself the first test would be to remove one or both low voltage wires from the transformer output and then measure the output with the meter. This will tell you if the transformer is OK or if there is some problem with the AC line voltage feed to the transformer or the transformer itself has failed. This isolation could save a good amount of time trouble shooting the low voltage wiring at the doorbell and switch.