Finding frequency respone of a differential/integral LTI system
Frequency response deals with the steady-state behavior. Differentiating with respect to $t$ yields:

$$\frac{d^2y}{dt^2}+2\frac{dy}{dt}+3y(t)=\frac{dx}{dt}-5\frac{d^2x}{dt^2}$$

The Laplace transform is $$(s^2+2s+3)Y(s)=(s-5s^2)X(s)$$
and we have

$$H(s)=\frac{Y(s)}{X(s)}=\frac{s-5s^2}{s^2+2s+3}$$
I think you can easily do the rest.