3D Rotation Using Fixed Point Arithmetic - Rotating Object is Deforming (and Shrinking)
Now that you know the problem, you can formulate a solution. 

Fixed point arithmetic is inaccurate. So is floating point, so that does not generally solve the problem, merely defer it. (In some respects strict IEE P754 is an exception to this in that certain properties of operations are guaranteed, but that's not a complete solution : read some of Professor Kahan's work for more...)

It sounds as though you are iteratively computing each frame from the previous frame, so that rounding errors are accumulating. Further, if the object is consistently shrinking, it suggests you are truncating rather than rounding.

Some common techniques may help here:

(1) Truncation rather than rounding introduces a systematic error tending towards 0. You can solve this by rounding properly anywhere you would truncate (narrow a wide result to fit a narrow word)

Example:

signal A,B,C   : unsigned(15 downto 0);
signal Product : unsigned(31 downto 0);
...
Product &lt;= A * B;
C &lt;= Product(31 downto 16);


You have thrown away bits 15 downto 0, or on average 0.5LSBs of output C. A correctly rounded calculation would use this information (or at least its LSB) to give either Product(31 downto 16) or Product(31 downto 16) + 1. 

For example

C &lt;= Product(31 downto 16) + Product(15 downto 15);


The maximum error in each operation is thereby halved, and distributed either side of 0. However this may still leave the object randomly distorting, though random growth or shrinkage should average out.

(2) Use transformations known to preserve invariants. For example, instead of calculating X,Y,Z iteratively from the previous step, calculate X and Y iteratively, and Z from the equation R^2 = X^2 + Y^2 + Z^2, as Z=sqrt(R^2 - X^2 - Y^2) so that errors in X and Y may introduce small errors in Z, but the invariant (R) is guaranteed correct - thus preserving the size of the object.

(3) Expanding on Ignacio's useful comment:

Keep the sample object intact. 

Track the rotation of a simple unit vector instead (length 1, any orientation) and having computed that for the current step (preserving its invariant i.e. R=1 as in step 2) rotate the original object by that unit vector. This guarantees that any errors do not persist beyond the current frame.