Making a dynamic parser to collect data from a javascript enabled webpage
One of the potential disadvantages of HTML-parsing vs interacting with the page via Selenium API is that you would not be able to examine the current state of the page - e.g. determine which elements are visible, get only visible text of an element, click buttons and links, execute JavaScript. In other words, you would miss all the dynamic behaviors.

But, on the other hand, every selenium command is an HTTP request - JSON over HTTP which introduces tons of overhead. And, if your page is completely loaded and you just need to only extract the data itself, then it does make perfect sense to avoid going through selenium API.

I would only fix a couple of things in your current implementation:


I don't particularly like the ___3OWY9 and ___3BmxE in the class names - these parts look dynamically generated and it feels like this can change. I would use a CSS selector and a partial match on a class attribute:

With post.querySelector("[class^=ProductPrice__promo_price]")


and

With post.querySelector("[class^=ProductPrice__price]")

and I would also use a CSS selector for getting the product description:

With post.querySelector("h4 &gt; a")



Note that querySelector would return a single element - hence, you would not need (0) part after the item.