How does scaling work in Blender, *exactly*?
I wondered the same, and when I saw that there was no answer here, I looked it up in the source. Dealing with signs is complicated, but without considering signs, the method is as follows. The sources work with 3x3 matrices (position doesn't influence scale), so I'll do the same here.

Let $S$ be the original scaling matrix, and $R$ be the original rotation matrix. Now, to scale by a new matrix $N$, calculate $A = S\times R\times N$ and obtain the new sizes on each axis by calculating the Euclidean norm of each row of $A$.

In your case:


  Suppose that M is the transformation matrix of a Blender object (and therefore satisfied the conditions above). I now apply a scaling of factor 3 along the global x-axis. How is the resulting transformation matrix calculated?


Take the $3\times 3$ matrix $M' = S\times R$ by discarding the last row and column of $M$, then obtain $A = M'\times\left[\begin{matrix}3 &amp; 0 &amp; 0 \\ 0 &amp; 1 &amp; 0 \\ 0 &amp; 0 &amp; 1\end{matrix}\right]$. Now, the new X scale is $\sqrt{a_{11}^2 + a_{12}^2 + a_{13}^2}$, the new Y scale is $\sqrt{a_{21}^2 + a_{22}^2 + a_{23}^2}$, and the new Z scale is $\sqrt{a_{31}^2 + a_{32}^2 + a_{33}^2}$. Use these as the elements of the diagonal of the new scaling matrix, $S'$. The new $3\times3$ matrix is obtained as $S'\times R$ and the $4\times4$ matrix by adding back the last row and column of the original $4\times4$ matrix.

For your example:


  Start with the default cube. Rotate it by 30 degrees along the y-axis (ry30Enter). Now rescale by 3 in the x-direction (sx3Enter).


we have $S = I$ (no scaling on the cube to start with), $R = \left[\begin{matrix}\cos 30°&amp;0&amp;\sin 30°\\0&amp;1&amp;0\\-\sin 30°&amp;0&amp;\cos 30°\end{matrix}\right] = \left[\begin{matrix}\sqrt3/2&amp;0&amp;1/2\\0&amp;1&amp;0\\-1/2&amp;0&amp;\sqrt3/2\end{matrix}\right]$ and $N = \left[\begin{matrix}3 &amp; 0 &amp; 0 \\ 0 &amp; 1 &amp; 0 \\ 0 &amp; 0 &amp; 1\end{matrix}\right]$. Now $A = S\times R\times N = \left[\begin{matrix}3\sqrt3/2&amp;0&amp;1/2\\0&amp;1&amp;0\\-3/2&amp;0&amp;\sqrt3/2\end{matrix}\right]$ so the new scale on X is $\sqrt{(3\sqrt3/2)^2+(1/2)^2} = \sqrt7\approx 2.646$, on Y is 1, and on Z is $\sqrt{(-3/2)^2+(\sqrt3/2)^2} = \sqrt3\approx 1.732$.

The code can be found in source/blender/editors/transform/transform.c in function ElementResize(). Function TransMat3ToSize() extracts the norm of each row and deals with signs.