Can I measure the relative output gain of a power amplifier with a multimeter?
Most amplifiers will have a gain that varies very little with output load so this is optional but probably a good idea because some amplifiers may go unstable on no-load and you might get rail-to-rail oscillations no matter what input signal you apply.

If your meter is good for the whole audio range then the method you propose is fine but, make sure that when on AC, the meter rejects dc levels because these could produce a big error in your estimations.

Also, make sure that the output isn't clipping. If it clips, the output level is restricted so, take two measurements at 50% levels to see that as a ratio, the two numbers are the same.

For measuring power, just use the equivalent resistance of the speakers impedance i.e. if the speaker is 8 ohms then use an 8 ohm resistor. Using the DC resistance value of the speaker coil is incorrect because that isn't the full picture of the impedance presented by an 8 ohm speaker at 400 Hz (a typical standard): -



Picture stolen from here.