Can modems be used to communicate with RC Submarines?
It could work, but there would be some major disadvantages to the approach:


Achieving a sufficient signal-to-noise ratio (SNR) will be very difficult, as you likely cannot control sources of noise. You'll probably need high-power directional transducers to achieve a strong enough signal to drown out the noise. Sorry for the pun.
A commercial off-the-shelf modem has been designed to work with a copper transmission line and electrical signals. Although the speed of sound in water is higher than in air, it still comes nowhere close to the propagation rate of electrical charges in copper. The modem will definitely have trouble dealing with the added latency.


Using pressure waves to communicate across water can be done, but you'll be much better off if you go ultrasonic. Here's why:


For starters, high power directional ultrasonic underwater transducers are cheap and very easy to find. Just look for the business end of a discarded echo sounder.
You'll get a better SNR with less effort, since there are simply fewer ultrasonic noise sources around. Unless you'll be operating your submarine in dolphin territory, of course ;-).
As a rule of thumb, you'll require less power to transmit a higher frequency signal, which is probably nice unless you're building a nuclear sub.