Is recalibrating the battery after flashing a new ROM really neccessary, and if yes, why? The description of the app, Battery Calibration, claims that after changing ROMs you should recalibrate your battery gauge. It has good reviews so it seems like the general concept is sound and trying the app at least won't cause harm.

What I'm wondering is what the battery statistics have to do with the ROM. My understanding is that the batterystats file contains a mapping of charge values in mV to %, so why does it need to be regenerated if a new ROM is flashed?
No.

This App just removes the file "batterystats.bin" in the "data/system" directory. It has nothing to do with your battery life and also not the status indicator of your system.

It only keeps the Infos in the "Battery Usage" screen. It will also be resetted every time you connect to a power source.

Further Information and Sources:
XDA-Developers and Dianne Hackborn (Google employee)