How do you program a completely transparent crossover? Or, if that's too broad, what is/are the most popular algorithms?

Background: I have no formal DSP training but much informal tinkering. I am trying to program a crossover for an audio effect. Another audio effect I've used (ReaXComp) has completely transparent crossovers, or so it sounds to my ears. I assumed that crossovers were just two matching filters, a lowpass and highpass filter of some kind, at the same frequency with a Q value of $\sqrt{\frac{1}{2}}$ (that seems to be the "default" Q value). However, when I run an audio signal through a lot of such crossovers it sounds... weird.

Later, I discovered through experimentation that you can put a two-pole low pass filter on a sound, and then subtract the original sound from the output of that and it sounds like a high pass filter, which is exatly what I want: a low and high pass filter that when added together create mathematically the exact same sound. Unfortunately, if you want a steeper filter and apply an identical filter again to get a 4-pole filter (again at a q value of $\sqrt{\frac{1}{2}}$) the resulting high band just starts to sound weird, not like a high pass filter at all. It basically sounds like the original sound with some sort of weird comb filtering. So I finally decided to come here and ask how to do it properly.

Edit:

robert bristow-johnson asks in the comments:

okay, RealXComp is a multiband compressor. so you want to split the audio into subbands for multiband compression and, if none of the individual-band compressors hit their knee, you want it all to add up to the original signal (perhaps with a little delay)? is that it? is that what you're trying to do?

Yes, Robert, that's exactly what I'm trying to do, sorry if that wasn't clear, I'm a bit of a DSP noob
I'm no expert but I'll explain how I would approach this. In general, crossover design involves selecting your respective cutoff frequency (or frequencies, in your case) and desired attenuation or filter order, and deciding on a filter type. Trivially, a Butterworth filter is a popular choice as they are easy to design and implement. According to Wikipedia, a better choice may be the Linkwitz-Riley filter:


  A Linkwitz–Riley (L-R) filter is an infinite impulse response filter used in Linkwitz–Riley audio crossovers...It is also known as a Butterworth squared filter. A Linkwitz-Riley "L-R" crossover consists of a parallel combination of a low-pass and a high-pass L-R filter. The filters are usually designed by cascading two Butterworth filters, each of which has −3 dB gain at the cut-off frequency. The resulting Linkwitz–Riley filter has a −6 dB gain at the cutoff frequency. This means that summing the low-pass and high-pass outputs, the gain at the crossover frequency will be 0 dB, so the crossover behaves like an all-pass filter, having a flat amplitude response with a smoothly changing phase response. This is the biggest advantage of L-R crossovers compared to Butterworth crossovers, whose summed output has a +3 dB peak around the crossover frequency.


Wikipedia on L-R Filters

I would design these such that the bands of interest are all separated out nicely, either through some sort of series configuration or by explicitly designing bandpass filters instead of lowpass/highpass filters (except for your lowest and highest bands of interest, of course).

This could either be done by designing the filters as analog and then using a bilinear transform to bring them into the Z-domain, resulting in a form that could be implemented in your language of choice, OR could be done digitally from the get-go in Matlab or Python (using SciPy's iirdesign() function).

SciPy's IIR Design Function

Then, you would need to implement them in whatever language/platform you're using (although I would hazard a guess that its C++ on Windows/Macfor a VST?).

Additionally, the concept of a "transparent" filter is something of a oxymoron. Either you are going to affect your signal or you aren't, and everything will involve some compromise. For instance, better filtering in the frequency domain may result in poorer performance in the time domain. Choosing what you can "ignore" is key to a good design.