How to learn 3D orientations reliably? I am working on neural network models for 3D skeletal character animation, where I learn joint positions and orientations. The problem comes with the orientations. There are several ways I can choose to represent a 3D rotation, but all of them have some form of discontinuity that is making my model unable to produce stable correct rotations in all cases. The alternatives I have considered are:


Quaternions. Quaternions (or unit quaternions) are very commonly used for 3D orientation, and are generally fine except a unit quaternion and its opposite represent the same orientation. It is not hard to come up with a loss formula the accounts for this (1 - abs(sum(componentwise_product(q1, q2))) works fine), but the problem is that in some cases the network somehow learns to produce the same quaternions with both signs, and kind of randomly "flips" from one sign to the other at times, with some frames in between, resulting in a sort of "flickering". One possibility would be to take only "half" of the quaternions, e.g. by fixing one component to be positive (then learn by squared differences for example). It doesn't always work, though. For example, suppose I fix the first component to be positive and I have a quaternion like [0.018, 0.743, 0.557, 0.371], and another one like [-0.017,  0.870,  0.348,  0.348]; these are rather similar quaternions, but if I make the first component positive then the second one will be [0.017,  -0.870,  -0.348,  -0.348], which is numerically very different, and the result is that sometimes the network cannot learn the value properly and comes up with some numbers in between (which are incorrect).
Rotation vectors. A rotation vector has the direction of the axis of rotation and the size of the angle of the rotation. They are used for example in the original paper of phase-functioned neural networks, but they still have bad cases. Suppose vector sizes go from 0 to π (greater angles would be taken along the opposite axis). If you have a rotation that is just around the value of π, the rotation vector will be "swinging" from one direction to the opposite one, again making it hard for the network to learn it.
Euler angles. These are just three rotation angles that are applied along specific axes (X, Y or Z) in specific order. They are generally discouraged because they are not very stable and suffer from the infamous gimbal lock, but geometry aside, they still have the same problems. If my angles go from -180º to 180º (or -π to π in radians), values in the "frontier" will always cause instability.
Extensive rotation encodings. One stable way of encoding rotations is giving both sine and cosine values. So I could for example have both the sine and cosine of each Euler angle, or just the nine values of a 3D rotation matrix (which are not sines and cosines but are in a way derived from these, and also a stable representation). However this obviously increases the number of values to learn significantly, and I would be trying to learn independent values that actually have a relationship among them.


I haven't found relevant literature addressing this problem specifically, although maybe I am not searching with the right terms. Has anyone faced this issue before? Or has some idea or alternative that I could consider?
I found an intermediate solution between minimal representation and full rotation matrix, which is using two unit vectors indicating two orthogonal directions (e.g. forward and right) with respect to the orientation. These can be learned simply through squared differences and are simple to use (in my case the animation runtime already had functions to compute an orientation given two directions, taking into account that they may not be perfectly orthogonal). This representation takes 6 values per orientation which is not as good as 3 or 4 but not too terrible either, and in my experiments it seems to work just as fine, only without the unstable cases.