Transients in marginally stable system
I think your understanding is correct. The steady state response of an LTI system is the part of the response that is caused by a steady excitation at the input, like a sinusoidal signal, or any other periodic signal. The transient response is caused by changes at the input, like switching on a signal, or changing the parameters of a periodic input signal, such as amplitude, phase or frequency.
As an example of a marginally stable system, take an ideal integrator with impulse response $h(t)=u(t)$, where $u(t)$ is the unit step function. You can show that the response to a switched sinusoid at the input
$$x(t)=\sin(\omega_0t)u(t)\tag{1}$$
is
$$y(t)=\frac{1}{\omega_0}u(t)-\frac{1}{\omega_0}\cos(\omega_0t)u(t)\tag{2}$$
The first term on the right-hand side of $(2)$ is a non-decaying contribution caused by switching on the sinusoid at the input at $t=0$, and the second part is the steady state response to the sinusoidal input. The non-sinusoidal part of the output signal is what would be a transient if the system were a stable system, in which case it would decay. Since for this example it is non-decaying due to marginal stability, the term transient would of course be a misnomer.