Is it possible to force Tensorflow to generate orthogonal matrix?
It should be possible.
I see two solutions.
If you don't care that the transformation is a perfect rotation, you can take the matrix, adjust it to what you think it's a good matrix (make it a perfect rotation) then compute the difference between the one you like and the original and add it as a loss.  With this approach you can push the model to do what you want but the model might not converge on a perfect rotation, especially if a perfect rotation is not a very good solution.
Another approach is to start with a proper rotation matrix and train the parameters of the rotation. In this case you would have the x,y,z of the rotation axis and the amount of rotation. So declare these as variables, compute the rotation matrix and use that in your model. The operations are derivable so gradient descent should work, but depending on the data itself you might depend on the start position (so try a few times).
Hard to say which one will work best in your case, so it might be worth trying both.
I'm using Tensorflow to generate a transformation matrix for a set of input vectors (X) to target vectors (Y). To minimize the error between the transformed input and the target vector samples I'm using a gradient descent algorithm. Later on I want to use the generated matrix to transform vectors coming from the same source as the training input vectors so that they look like the corresponding target vectors. Linear regression, pretty much, but with 3-dimensional targets. 
I can assume that the input and target vectors are in cartesian space. Thus, the transformation matrix should consist of a rotation and a translation. I'm working solely with unit vectors, so I can also safely assume that there's no translation, only rotation. 
So, in order to get a valid rotation matrix that I can turn into a rotation quaternion I understand that I have to make sure the matrix is orthogonal. 
Thus, the question is, is it possible to give Tensorflow some kind of constraint so that the matrix it tries to converge to is guaranteed to be orthogonal? Can be a parameter, a mathematical constraint, a specific optimizer, whatever. I just need to make sure the algorithm converges to a valid rotation matrix.