Analog crosspoint switches
It's difficult to figure out what you actually want, when all your question says is what you don't want.

Non-blocking. Let's say you have N inputs and M outputs. If you have a matrix of NM switches, then you have a totally non-blocking connection capability, and any number of inputs and outputs can be connected simultaneously. Is this what you want? This is not how the international telephone system works. It obviously cannot work like this, as you'd need a (few billion)^2 switches, and a lot of wires.

Blocking. Let's say you have an N:1 input multiplexer, and a 1:M output demultiplexer. This is the fewest number of switches to connect any input to any output, but it can support only one connection. Any one connection blocks the others. This is not how the international telephone system works either.

Channelised. Let's say you have a number P of N:1 input multiplexers, and P 1:M output demultiplexers. Now you can support up to P simultaneous connections between input and output. This economises on the number of switches, while allowing more than one input/output connection, is scalable by adding more channels, and permits the use of 1:N multiplexers. Apart from the hierarchy, that there are several tiers of multiplexers, this is how the international telephone system works.

Would considering a channelised connection ease your apparently impossible switch requirements?