title: Parsing HTML table with LXML in Python
body: <p>I need to parse html table of the following structure:</p>
  
  <pre><code>&lt;table class="table1" width="620" cellspacing="0" cellpadding="0" border="0"&gt;
   &lt;tbody&gt;
     &lt;tr width="620"&gt;
       &lt;th width="620"&gt;Smth1&lt;/th&gt;
       ...
     &lt;/tr&gt;
     &lt;tr bgcolor="ffffff" width="620"&gt;
       &lt;td width="620"&gt;Smth2&lt;/td&gt;
       ...
     &lt;/tr&gt;
     &lt;tr bgcolor="E4E4E4" width="620"&gt;
       &lt;td width="620"&gt;Smth3&lt;/td&gt;
       ...
     &lt;/tr&gt;
     &lt;tr bgcolor="ffffff" width="620"&gt;
       &lt;td width="620"&gt;Smth4&lt;/td&gt;
       ...
     &lt;/tr&gt;
   &lt;/tbody&gt;
  &lt;/table&gt;
  </code></pre>
  
  <p>Python code:</p>
  
  <pre><code>r = requests.post(url,data)
  html = lxml.html.document_fromstring(r.text)
  rows = html.xpath(xpath1)[0].findall("tr")
  #Getting Xpath with FireBug
  data = list()
  for row in rows:
      data.append([c.text for c in row.getchildren()])
  </code></pre>
  
  <p>But I get this on the third line:</p>
  
  <pre><code>IndexError: list index out of range
  </code></pre>
  
  <p>The task is to form python dict from this. Number of rows could be different.</p>
  
  <p><strong>UPD.</strong>
  Changed the way I'm getting html code to avoid possible problems with requests lib. Now it's a simple url:</p>
  
  <pre><code>html = lxml.html.parse(test_url)
  </code></pre>
  
  <p>This proves everyting is Ok with html:</p>
  
  <pre><code>lxml.html.open_in_browser(html)
  </code></pre>
  
  <p>But still the same problem:</p>
  
  <pre><code>rows = html.xpath(xpath1)[0].findall('tr')
  data = list()
  for row in rows:
      data.append([c.text for c in row.getchildren()])
  </code></pre>
  
  <p>Here is the xpath1:</p>
  
  <pre><code>'/html/body/table/tbody/tr[5]/td/table/tbody/tr/td[2]/table/tbody/tr/td/center/table'
  </code></pre>
  
  <p><strong>UPD2.</strong> It was found experimentally, that xpath crashes on:</p>
  
  <pre><code>xpath1 = '/html/body/table/tbody'
  print html.xpath(xpath1)
  #print returns []
  </code></pre>
  
  <p>If xpath1 is shorter, then it seeem to work well and returns <code>[&lt;Element table at 0x2cbadb0&gt;]</code> for <code>xpath1 = '/html/body/table'</code> </p>