Orthornormal Wavelet scaling function coefficient calculation (filter design)
Add twice 5.65 to 5.64, and let $j = h(0)+h(2)$ and $k = h(1)+h(3)$; then we get 

j + k = sqrt(2)
j*j + k*k = 1


which represents the intersection of a line with a circle, and we get

j = k = 1/sqrt(2)


Now subtract twice 5.65 from 5.64 and let $m = h(0)-h(2)$ and $n = h(1) - h(3)$; we get 

m*m + n*n = 1


so for same angle $b$ say we get $m = cos(b)$ and $n = sin(b)$. We now have, for example

h(0) + h(2) = 1/sqrt(2)
h(0) - h(2) = cos(b)


which we can solve for $h(0)$ and $h(2)$. Finally shifting $b$ to get a gives the required expressions.