Assign one audio input to multiple outputs
You're thinking about this backwards. What you're describing is a full-matrix mixer, in which each of the outputs can have a mix of any of the inputs. In other words, treat each output independently as an N-channel mixer, where N is the number of inputs. Then build as many mixers as you have outputs.

You might need buffer amplifiers (unity gain, low output impedance) on the input channels in order to have adequate drive for all of those mixer inputs.