Parsing HTML in Python - Some pages work and some don't...?
Question: Using the following script:

    
    
    from lxml import html
    import requests
    
    gameUrl = 'http://store.401games.ca/catalog/2415520/caylus'
    page = requests.get(gameUrl)
    tree = html.fromstring(page.content)
    
    stock = tree.xpath('//*[@id="stock"]/span[1]/div/*/text()')[0]
    
    print stock
    

It will correctly display the stock level listed on the page. (1 at this time)

    
    
    gameUrl = 'http://store.401games.ca/catalog/2415324/ticket-to-ride'
    

It displays the stock as 68, which is incorrect. (I have no idea where 68 is
even coming from).

I tried this with a LOT of pages from this site and 90% of them work correctly
using this script. But the other 10% fail and give random numbers...some are
quite different like 68 instead of 30. Or 1100 instead of 30. Some are closer,
like 12 instead of 9. I have no idea what is happening.

Does anyone have an idea of what may be the problem?


Answer: If you would open the page in the browser, you would see the `Quantity: 68`
flashing before it changes to `Quantity: 30`.

At first, I thought there is an XHR request that dynamically gets the product
availability from a certain endpoint after the page is loaded and almost
started to provide a usual answer about browser automation, but the problem
here is different.

If you would open the Network tab in browser developer tools, you may see the
`store.js` javascript file being loaded. At the beginning of the script, you
can see:

    
    
    if(stock>30) { $('div.availability span').text( "30" ); }
    var instock = $('div.availability').text();
    instock = instock.replace("In-Stock", "Quantity");
    

What it means is that, if the quantity is more than 30, it is "manually" set
to 30.

