Scrape an infinite-scroll page
I would want to understand how the infinite scroll is actually being applied.


Do you understand what javascript events actually trigger new items to be added?
Does it make more sense to simply trigger those events vs. worry about physically scrolling the browser?
Is the content being delivered via AJAX? Can you just query the AJAX endpoint more directly to get to the data you want to get?
Is there anything from the ajax response that you need to understand to know when you have reached the end of the list (no more items to be added)?


When you think through these you might find you have a better way to approach the problem.