Can an infinite universe explain the Fermi Paradox?
The true explanation to the Fermi paradox is not known. There is a very long list of proposed explanations, some of which are listed here:

https://en.wikipedia.org/wiki/Fermi_paradox#Hypothetical_explanations_for_the_paradox

What you're saying is literally the first one on the list, also known as the Rare Earth hypothesis.

https://en.wikipedia.org/wiki/Rare_Earth_hypothesis