To find the filter Co-efficients of the following equation
You're overcomplicating things here. There's no need for sines and cosines and squares. Note that $\omega=0$ corresponds to $z=1$, and $\omega=\pi$ corresponds to $z=-1$. From the definition of the $\mathcal{Z}$-transform you should be able to figure out that the DC term of the transfer function $H(z)$ is given by

$$H(1)=h_1+h_2+h_3\tag{1}$$

and the value at Nyquist ($\omega=\pi$) is

$$H(-1)=h_1-h_2+h_3\tag{2}$$

Equating $(1)$ and $(2)$ with the desired values at these two frequencies gives you two equations for three unknowns. The third equation comes from the linear phase requirement. I'm sure you can take it from here.