Modulation scheme for a noisy channel with a complicated transfer function
Far field, so there is potentially significant reverb


That's basically a multipath channel with a large delay spread.

In communications engineering, the measures taken to revert that effect would be called equalization.

So, that's less of a modulation problem, but a channel estimation problem. Estimating channels with large delay spread / high frequency-selectivity is relatively hard, and so is equalizing them once you know the channel. You can deal with that by sending a known sequence ("preamble") against which the receiver correlates.

Since the result of that correlation is the autocorrelation function of the preamble, convolved with the channel impulse response, you can get pretty far with that. Especially considering you're free to pick a preamble that is spectrally white, i.e. has a sharp (dirac) impulse as ACF.

Based on that, you could build a zero-forcing equalizer, which is essentially simply taking that impulse response, inverting its Fourier transform, and transforming back to time domain.

Problem with zero-forcing equalizers is that you get strong noise amplification at the frequencies the channel is essentially "deaf". In audio engineering, that's what happens when the guy at the FOH doesn't pay attention, and suddenly you get a really unpleasant tone at some frequency when someone starts using their microphone.

Other options include MMSE equalizers, LMS adaptive filters, etc.; there's a large corpus of literature on both equalization for communications, and equalization for making audio work.

I'd actually, for simplicity reasons, recommend a different approach:

Instead of trying to equalize this very long, and hence very frequency-selective channel, you'd simply take it, and divide it into many narrower channels.

If you choose to pick these equidistantly and use a synchronous boxcar-shaped pulse shaping on each of these, you end up with OFDM (Orthogonal Frequency Division Multiplexing).

The idea is that instead of this complex equalizer that you need to use the full channel with a high bandwidth data signal, you get a lot of narrower channels that you divide your data to, and because they are so narrow, an equalizer is just a phase and amplitude correction – much, much easier!

And to boost the benefits, OFDM is mathematically just an IDFT - DFT pair, and can hence be implemented really efficiently using FFTs.


  Typical residential background noise


yeah, so noise. Some frequencies might be especially disturbed, but most of this noise will be relatively broadband. 

So, channel coding with a relatively long code might be a good idea. Probably with an outer and an inner code, to convert burst noise to something benign and then throw a high-rate code on the rest.


  phone microphone quality, phone case, and room transfer function can be all over the place


Yeah, so that's covered by equalization.


  microphone handling noise and hand partially blocking the mic


burst Noise and fading.


  There may be mild time-variance, but probably not a lot


well, that's why you'll need to re-estimate your channel regularly.



If you want something to play with: GNU Radio is a software framework for live signal processing and can be directly attached to microphone sources or speaker sinks.

It comes with OFDM blocks, but there's also projects like gr-adapt that offer adaptive filters to compensate channels.

I'd start with the OFDM examples, and just replace the radio hardware block with a block that interpolates the complex baseband to match your audio sampling rate, and shift it up to a carrier frequency before discarding the imaginary part.

Do the reverse (add zero imaginary part, shift down to baseband, decimate to baseband bandwidth) at the receiver  – Tadah, OFDM audio testbed.

Pick an OFDM FFT length that is large enough to divide the channel into subchannels of  a couple dozen Hz bandwidth, at first (that system will then have very long symbols, and thus be sensitive to channel changes, but it should be robust against complex channel impulse responses).

Start with a really "easy" modulation – BPSK. If that works, and a constellation diagram says your SNR permits it, you can go for higher modulations like QPSK.

My guess is that staying within the confines of PSK is probably a good idea. OFDM isn't really easy on the analog chain – it requires a large dynamic range. In fact, I think I remember that if you want to avoid more than one in thousand signals clipping, you need a 10 dB headroom over average power.