How could we have so many past lives without reaching some degree of wisdom or enlightenment?
Infinities can be one-way. Consider the negative numbers.

What the Buddha said is that there is no "first point".


  SN15.19:1.3: No first point is found of sentient beings roaming and transmigrating, hindered by ignorance and fettered by craving.


And with the ending of ignorance, one does not return.


  AN2.36:4.7: When they pass away from there, they are non-returners, not coming back to this state of existence.


Non-returners (anāgāmi) and arahants do not return. They have left behind the negative infinity of suffering.