Finding the Wiener filter transfer function
What you've listed is a classical set of linear equations known as the Wiener-Hopf Equations. There are many methods to solve this set of equations. These equations generalize the Yule-Walker equations for Autoregressive (AR) modeling. In fact, if the Wiener-Hopf equations are used to solve for a linear predictor of your signal, the equations are identical to the Yule-Walker equations.

So in reality, there are many ways to solve these equations depending on how you're modeling your signal. This is where the concepts of Wiener filtering, Wiener smoothing, Wiener deconvolution arise. In each of these, they can be segmented further into causal/anti-causal realizations, FIR/IIR, etc. I recommend searching for the Wiener-Hopf equations and understand what they mean, then solving them comes secondary.

As an example, consider if you assume that you have a signal with additive noise that is uncorrelated with the signal. Then the cross-correlation terms have no dependence on the signal. Then you need to model your noise and plug in the resulting autocorrelation sequence on the right hand side of the equation.

Cheers