Making a dynamic parser to collect data from a javascript enabled webpage I've written a scraper in VBA in combination with selenium to parse product names and prices from a javascript enabled webpage. The thing is that I just used selenium to get the page source (as it was not possible to get the response text without opening the page with selenium) and as soon as I get it I reverted back to usual VBA method to complete the operation. 

Most importantly, it was hard to deal with parsing prices cause there are two types of class names for prices so if a particular class name is used in the script then it is noticed that, after scraping is done, some fields are blank in the price section in the spreadsheet. 

However, I have handled the two class names effectively in my parser to get prices flawlessly. Now, it is working great and way faster than selenium alone. If there is any improvement I can make in this script, I would be very happy to comply. Thanks in advance.

Here is what I've written:

Sub RedmartScraper()
  Dim driver As New ChromeDriver, html As New HTMLDocument
  Dim post As HTMLHtmlElement

  With driver
    .Get "https://redmart.com/bakery"
    html.body.innerHTML = .ExecuteScript("return document.body.innerHTML;")
    .Quit
  End With

  For Each post In html.getElementsByClassName("productDescriptionAndPrice")
    With post.getElementsByTagName("h4")(0).getElementsByTagName("a")
      If .Length Then i = i + 1: Cells(i, 1) = .item(0).innerText
    End With

    With post.getElementsByClassName("ProductPrice__promo_price___3OWY9")
        If .Length Then Cells(i, 2) = .item(0).innerText
    End With

    With post.getElementsByClassName("ProductPrice__price___3BmxE")
        If .Length Then Cells(i, 2) = .item(0).innerText
    End With
  Next post
End Sub


Btw, libraries to add in the reference library:


Microsoft Html Object Library
Selenium Type Library
One of the potential disadvantages of HTML-parsing vs interacting with the page via Selenium API is that you would not be able to examine the current state of the page - e.g. determine which elements are visible, get only visible text of an element, click buttons and links, execute JavaScript. In other words, you would miss all the dynamic behaviors.

But, on the other hand, every selenium command is an HTTP request - JSON over HTTP which introduces tons of overhead. And, if your page is completely loaded and you just need to only extract the data itself, then it does make perfect sense to avoid going through selenium API.

I would only fix a couple of things in your current implementation:


I don't particularly like the ___3OWY9 and ___3BmxE in the class names - these parts look dynamically generated and it feels like this can change. I would use a CSS selector and a partial match on a class attribute:

With post.querySelector("[class^=ProductPrice__promo_price]")


and

With post.querySelector("[class^=ProductPrice__price]")

and I would also use a CSS selector for getting the product description:

With post.querySelector("h4 &gt; a")



Note that querySelector would return a single element - hence, you would not need (0) part after the item.