How to learn 3D orientations reliably?
I found an intermediate solution between minimal representation and full rotation matrix, which is using two unit vectors indicating two orthogonal directions (e.g. forward and right) with respect to the orientation. These can be learned simply through squared differences and are simple to use (in my case the animation runtime already had functions to compute an orientation given two directions, taking into account that they may not be perfectly orthogonal). This representation takes 6 values per orientation which is not as good as 3 or 4 but not too terrible either, and in my experiments it seems to work just as fine, only without the unstable cases.