Inconsistent lxml.etree.xpath element counting between Mac OS and Linux hosts
Question: This problem has puzzled me for a while now. I am trying to scrape tables from
financial websites. For example, from barchart.com ( finviz.com same problem )

    
    
    import lxml.html as lh
    import lxml, urllib2
    
    gList = ['gapup','gapdown']
    count = {}
    doc = {}
    for g in gList:
        doc[g] = lh.parse(urllib2.urlopen('http://www.barchart.com/stocks/{0}.php'.format(g)))
        count[g] = len(doc[g].xpath('/html/body//table[4]//tr'))
    

Basically, I am counting how many rows (stocks) there are in the table. On my
macbook (Python 2.7.5+ lxml 3.2.4). Works perfect:

    
    
    In [14]: count
    Out[14]: {'gapdown': 101, 'gapup': 34}
    

However, when I test the same code remotely on my dreamhost site. Won't work:

    
    
    In [7]: count
    Out[7]: {'gapdown': 0, 'gapup': 0}
    

Initially, I thought it's because Python 2.6 on dreamhost, and 2.6 is known to
have less xpath support. So I installed pyenv to use 2.7.5 today. Problem is
still there.

I tried to get around using this code (I call it xpath relay :) ):

    
    
    In [11]: len(doc['gapup'].xpath('/html/body//table')[3].xpath('//tr'))
    Out[11]: 43
    

As the number suggests (43 != 34), didn't work. Using `lh.tostring()` , I
realize it basically picked up all tr-s in the html file. No relay.

I am new to xpath. But why the same code won't work on same python 2.7 + lxml
3.2.4 setup on different machines? Thanks a lot!


Answer: I was able to reproduce the issue on an Ubuntu 12.04 host with Python 2.7.3.

For some reason, the table is the fifth table sometimes. Anyway, when parsing
HTML documents, better do not rely on counting elements, but use `@id`
attributes if given.

Using the XPath expression `//table[@id="dt1"]//tr` I could get reproducible
results, also on my Ubuntu host.

