How can I find the transfer function of the following block diagram?
In such cases it can be useful to introduce auxiliary variables at the input of the integrators. For the given diagram you could use a signal $u(t)$ at the input of the second integrator. The equation for its Laplace transform $U(s)$ becomes
$$U(s)=\frac{1}{s}\big[X(s)-bY(s)\big]-aY(s)\tag{1}$$
You need another equation relating $U(s)$ to $Y(s)$, but that one is trivial. From those two equations you can express $Y(s)$ in terms of $X(s)$, and in this way you can obtain the transfer function $H(s)=Y(s)/X(s)$. The solution in your question is not correct. The correct solution must be a rational function, not a polynomial.