Which digital modulation scheme would be suitable for transmitting digital data through the output of a PC soundcard
You've got a pretty good set of circumstances here; you should be able to meet your goal without too much trouble. I don't see anything in your description that would eliminate a whole class of modulation (e.g. phase-shift keying, frequency-shift keying, etc.). Some of the factors that would go into the choice of a suitable format would include:

The required spectral efficiency (i.e. how much data throughput you need relative to the available bandwidth)
The complexity requirements for your receiver (which is usually the most complicated part of the system)
How much effort you're willing to put into developing the implementation.
Other circumstances specific to your application (e.g. if you have poor timing accuracy at one or both ends, known interference, or a poor channel response)

So, ticking these off one by one for your system, we can come up with a few guidelines:

It seems that your biggest constraint is your channel response (which is limited by your sound card's DAC). If you have 40 kHz of one-sided bandwidth available, then you're going to be limited to a symbol rate that is somewhat below that. For a target data rate of at least 40 kilobits per second, you'll want some scheme that transmits multiple bits per symbol.

Provided that your embedded platform isn't loaded down with too many other functions, a modern 120-MHz ARM processor should easily be able to handle demodulation of most any format in the tens-of-kilobits-per-second range.
I'm not sure specifically which model you're running with, but many recent processors provide very tight integration of onboard ADCs with the memory and interrupt subsystem, perhaps allowing you to (without manual CPU intervention) automatically sample the input signal at a specified rate, store the samples in onboard memory, and trigger a processor interrupt only when a certain-sized block of samples is available to be processed. I know that some Atmel devices at a minimum provide this sort of functionality; I've had good success with them in the past.

I assume that this is probably one component of some larger system, so you don't want to have to build something really complex in order to meet your requirements. Luckily, based upon what you've said, I think you should be able to implement something that meets your requirements pretty simply. I assume that since the two devices are cabled together, you're likely to have a very high signal-to-noise ratio (40 dB or greater), which for digital modulations is typically measured as $\frac{E_b}{N_0}$. That makes life much easier, as you likely could skip additions like error-correction encoding altogether (or at least implement a very simple scheme that would catch the very infrequent errors that you would expect at such an SNR).

As far as special circumstances go, for this system, I wouldn't expect much of any. I would expect the oscillator accuracy on the PC side to be pretty good (at a minimum crystal-controlled, so you're in the range of &lt;50 ppm or so; possibly much better if the oscillator is calibrated using some other more precise source). The embedded side is likely to be the same; I assume you're using a crystal oscillator as the clock source. Since the two ends are cabled together, I'll assume that you don't have interference of note.


So, smashing all of these together into a single recommendation, I would probably start down the path of a quadrature phase-shift keying (QPSK) approach at 24 kilosymbols per second. At 2 bits per symbol, this yields a data rate of 48 kilobits per second, which is in excess of your requirement. This particular rate makes your implementation a bit easier; since the output DAC is running at 96 kHz, this results in 4 samples per symbol (it's always easier to run at an integer number of samples per symbol time). I would probably try to design the embedded side so that it would sample at the same 96 kHz rate if possible; this avoids the need to do any resampling on the more resource-starved end.
To avoid any problems with the DC notch that your sound card DAC employs, you could modulate the QPSK signal onto a carrier at 24 kHz. Then, the modulated signal's spectrum would have a null at DC, which would line up with your notch. It's possible that the notch could end up not being a problem at all (especially if it is truly just a few Hz wide as your suggested). In that case, you could potentially get by with an even simpler scheme that just works at baseband, bypassing carrier modulation altogether.
QPSK is a good choice because of its simplicity, both at the transmitter and receiver. At your SNR, you could accomplish greater spectral efficiency using a more complicated scheme like quadrature amplitude modulation (QAM), but the constant-envelope property of PSK signals is attractive from a receiver complexity standpoint. As a note, if you truly needed more bits per symbol in the future, you could move to a higher-order PSK constellation like 8- or 16-PSK. However, these are suboptimal from a bit-error-rate performance standpoint when compared to QAM constellations.
As far as a library implementation goes, I'm not aware of anything that you would just be able to drop in and go, especially for an embedded platform. Your receiver implementation is likely to be tied in with the hardware interface to some extent. You may be able to find some existing implementations for the various steps required for the demodulator, but you'll need to at least tweak what you might find to work well on your platform. The GNU Radio project is a good place to look if you just want to see C++ implementations of many different communications signal processing operations, and it might even yield a useful framework for implementing the transmitter onboard your PC. As a summary, the high-level steps that your receiver would need to do would include:

If a nonzero carrier frequency was used:

Frequency synchronization: locate and track the carrier frequency offset due to oscillator mismatches between the transmitter and receiver (in many cases, that frequency offset will be approximately constant over time)
Carrier demodulation: translate the signal to baseband, yielding two in-phase and quadrature (I/Q) signals (often expressed as a complex baseband signal).


Matched filtering: pass the baseband signal through a filter matched to the pulse shape used at the transmitter (you can likely get away with a rectangular pulse)

Timing synchronization: locate and track the times that correspond to symbol transitions; this can be done by tracking the peak locations in the matched filter output modulo the symbol time

Bit slicing: convert the matched filter outputs at symbol sample times to hard bit decisions

Serialization: make sure you write the multiple bits per symbol out in the right order!


This might sound like a complicated process, but building a practical receiver for even a simple situation like this can be very illuminating. Just comment if there's anything else I left out.