Using Infinite Scroll module with Views paging: Detrimental to Google SEO?
No it's not detrimental (well, not in the way you're thinking at least) - the standard pager links are hidden with infinite scroll, but still in the markup, so there's no problem from that perspective. Google will find and visit the subsequent pages as with a normal views pager.

The only question is whether Google apply some sort of penalty to the pages linked to from those hidden links. But that's something you'd need to get out of Google :)