How to convert my transfer function to the frequency domain
Since this is not an LTI system, we cannot have $H(\omega) = K\Theta_1(\omega)$, where $K$ is a scalar complex number. But as OP mentioned in the comment he has knowledge of $\Theta_1(\omega)$, we can compute $\Theta_2(\omega)$ and $\Theta_3(\omega)$. As $x^2(t)=x(t)\times x(t)$,
$$
\Theta_2(\omega)= \Theta_1(\omega)*\Theta_1(\omega) =\int_{-\infty}^{+\infty} \Theta_1(\alpha)\Theta_1(\omega-\alpha)d\alpha
$$ 
where $*$ is the convolution operation. Similarly, $x^3(t)=x(t)\times x(t)\times x(t)$
$$
\Theta_3(\omega)= \Theta_1(\omega)*\Theta_1(\omega)*\Theta_1(\omega)=\Theta_1(\omega)*\Theta_2(\omega)
$$
So $H(\omega)=B\Theta_1(\omega)+a\Theta_2(\omega)+b\Theta_3(\omega)$, where $\Theta_2$ and $\Theta_3$ are computed from $\Theta_1$ above.