What is the mathematical difference between a transfer function and a state transition matrix?
A transfer function relates output variables to input variables. In the equation you have shown you only consider state variables (q) and inputs (u).

This model assumes that state variables are completely accessible from the outside.

A more comprehensive model would comprise an output equation such as:

$$
y(t) = C \cdot q(t) + D \cdot u(t)
$$

which transformed becomes:

$$
Y(s) = C \cdot Q(s) + D \cdot U(s)
$$

A transfer function (or, better, a transfer function matrix, since we are modeling MIMO systems, apparently) takes into account also the C and D matrices.