ANNs with multiple activation outputs
I think this is an interesting approach, but it's hard to tell if it can make better or faster learning networks.

If I understand correctly, this could be looked at as using a function returning a tuple as activation function. The number of elements in the tuple must be equal to the number of neurons in the next layer.

For example you could use an activation function like f(x) = (ReLU(x), sigmoid(x)) if the next layer contains two neurons.

I think the biggest problem is to determine why one value should be passed to exactly THAT neuron and another value to another neuron - why not opposite? This must either be determined randomly or by doing some advanced analysis in beforehand.

Anyway, I think it's an interesting idea which should be tested out, but it's hard to say whether it will give better or worse results before trying it out.