Digital potentiometer mixing matrix
I'm not saying this is a perfect solution for the OP's partially defined problem but it gives food for thought.

Analogue cross-point switches (as developed for the first digital telephone exchanges) give you the ability to do the signal routing and then what you are left with is controlling amplitude.

First the analogue cross-point switch (this one is 16 x 16): -



The one above is the ADV3205 from analogue devices but they also supply 8x8 types. So the problem is somewhat reduced to controlling the amplitudes of the inputs by 4 digital pots for 4 inputs.

If you want all four inputs to be weighted individually to each of the four outputs then you might need to use 16 digital pots and a 16x16 cross-point switch.

However, there is an alternative to digipots and that is the use of PWM to modulate the amplitude of an audio signal like this: -



So if you can generate several independent PWM outputs you can use simple analogue switches to chop up an analogue input to reduce its amplitude. You need to have PWM frequencies that are significantly higher than the analogue highest frequency of course.