Finding the Wiener filter transfer function I am trying to understand the mechanism of finding the transfer function for a Wiener function.

|Rxx(0)   Rxx(-1) ...... Rxx(1-N)| | h(0) |   | Ryx(0) |
|Rxx(1)   Rxx(0)  ...... Rxx(2-N)| | h(1) | = | Ryx(1) | 
|......   ......  ......  ...... | | .....|   |....... |
|Rxx(N-1) Rxx(N-2) ...... Rxx(0) | |h(N-1)|   |Ryx(N-1)|


Rxx is the autocorrelation of my input signal h is filter transfer function and Ryx is crosscorrelation of the input and output signal. My question is how to come up with the h. I mean i have two unknowns in this equation cause i dont know h and I dont know the y output signal therefore i cant calculate the crosscorelation Ryx.

Is there something that I need to assume in order to get the h??

Thanks in advance for the help.
What you've listed is a classical set of linear equations known as the Wiener-Hopf Equations. There are many methods to solve this set of equations. These equations generalize the Yule-Walker equations for Autoregressive (AR) modeling. In fact, if the Wiener-Hopf equations are used to solve for a linear predictor of your signal, the equations are identical to the Yule-Walker equations.

So in reality, there are many ways to solve these equations depending on how you're modeling your signal. This is where the concepts of Wiener filtering, Wiener smoothing, Wiener deconvolution arise. In each of these, they can be segmented further into causal/anti-causal realizations, FIR/IIR, etc. I recommend searching for the Wiener-Hopf equations and understand what they mean, then solving them comes secondary.

As an example, consider if you assume that you have a signal with additive noise that is uncorrelated with the signal. Then the cross-correlation terms have no dependence on the signal. Then you need to model your noise and plug in the resulting autocorrelation sequence on the right hand side of the equation.

Cheers