How do I determine the proper FM transmitter strength?
It is both complex, and easy in the same time. Easy, because there is a single formula involved, complex because you will not be able to determine some of the variables.

If you had omnidirectional antennas (which you have since there is no predetermined direction of transmission) they are calculated in with 0 dB gain, so it is all about receiver sensitivity, and the transmission loss which you can determine only for the free air transmission. So the electrical power you refer to does not mean to much here, like with cars, you care about power at the wheels, not only the raw engine power at the flywheel :-)

What you ask is called a power budget calculation, and you can learn a bit about it here: http://en.wikipedia.org/wiki/Link_budget

Then of course, you should know how much of your transmitted power was actually handed over to the antenna (it is a matter of matching the transmitter to the antenna resulting in something called SWR - for Standing Wave Ratio). All in all, you should take the maximum allowed by regulations, and hope it would be enough. And yes, make sure you are within some allowed frequency band (some ISM band is safe to use, I assume)