Finding the minimum phase h[n] and its Z transform
Since this looks like a homework problem I'll help you to solve it on your own, rather than just present you the solution.

The problem here is that you have a mistake in the first part of the solution. The evaluation of the first sum (from $n=-\infty$ to $n=-1$) is wrong. Redo the calculation and you should end up with

$$R(z)=\frac{1}{(1-az^{-1})(1-az)}\tag{1}$$

with some appropriate value for $a$. From $(1)$ it's immediately clear how you can factor $R(z)$ as

$$R(z)=H(z)H(z^{-1})\tag{2}$$

and how you have to choose $H(z)$ to get a minimum-phase systems.