How do I build a permutation invariance neural network in keras?
Concatenate was fine.



By concatenating, you give to your network access to all information it needs to predict the output. The order does not matter.

By averaging, you crunch your data into something smaller : something that represent your data in a less informative way. That's why your loose accuracy : you are giving to your network less information to learn from.



What you need to do is get more data. Your initial approach is fine, you just need enough data for your network to learn that when the data represent a 90 degree-rotation, the output should be the same.