title: Detecting header in HTML tables using beautifulsoup / lxml when table lacks thead element
body: <p>I'd like to detect the header of an HTML table when that table does not have <code>&lt;thead&gt;</code> elements. (MediaWiki, which drives Wikipedia, <a href="https://www.mediawiki.org/wiki/Help:Tables#Common_attributes_for_columns.2C_column_groups_and_row_groups" rel="nofollow noreferrer">does not support <code>&lt;thead&gt;</code> elements</a>.) I'd like to do this with python in both BeautifulSoup and lxml. Let's say I already have a <code>table</code> object and I'd like to get out of it a <code>thead</code> object, a <code>tbody</code> object, and a <code>tfoot</code> object.</p>
  
  <p>Currently, <code>parse_thead</code> does the following when the <code>&lt;thead&gt;</code> tag is present:</p>
  
  <ul>
  <li>In BeautifulSoup, I get table objects with <code>doc.find_all('table')</code> and I can use <code>table.find_all('thead')</code></li>
  <li>In lxml, I get table objects with <code>doc.xpath()</code> on an xpath_expr on <code>//table</code>, and I can use <code>table.xpath('.//thead')</code></li>
  </ul>
  
  <p>and <code>parse_tbody</code> and <code>parse_tfoot</code> work in the same way. (I did not write this code and I am not experienced with either BS or lxml.) However, without a <code>&lt;thead&gt;</code>, <code>parse_thead</code> returns nothing and <code>parse_tbody</code> returns the header and the body together.</p>
  
  <p>I append a <a href="https://en.wikipedia.org/wiki/List_of_England_Test_cricket_records#Most_extras_in_one_inning" rel="nofollow noreferrer">wikitable instance</a> below as an example. It lacks <code>&lt;thead&gt;</code> and <code>&lt;tbody&gt;</code>. Instead all rows, header or not, are enclosed in <code>&lt;tr&gt;...&lt;/tr&gt;</code>, but header rows have <code>&lt;th&gt;</code> elements and body rows have <code>&lt;td&gt;</code> elements. Without <code>&lt;thead&gt;</code>, it seems like the right criterion for identifying the header is "from the start, put rows into the header until you find a row that has an element that's not <code>&lt;th&gt;</code>".</p>
  
  <p>I'd appreciate suggestions on how I could write <code>parse_thead</code> and <code>parse_tbody</code>. Without much experience here, I would think I could either </p>
  
  <ul>
  <li>Dive into the table object and manually insert <code>thead</code> and <code>tbody</code> tags before parsing it (this seems nice because then I wouldn't have to change any other code that recognizes tables with <code>&lt;thead&gt;</code>), or alternately</li>
  <li>Change <code>parse_thead</code> and <code>parse_tbody</code> to recognize the table rows that have only <code>&lt;th&gt;</code> elements. (With either alternative, it seems like I really need to detect the head-body boundary in this way.)</li>
  </ul>
  
  <p>I don't know how to do either of those things and I'd appreciate advice on both which alternative is more sensible and how I might go about it.</p>
  
  <p>(Edit: Examples with <a href="https://en.wikipedia.org/wiki/List_of_Twenty20_International_records#Highest_innings_totals" rel="nofollow noreferrer">no header rows</a> and <a href="https://en.wikipedia.org/wiki/List_of_Cricket_World_Cup_records#One_tournament" rel="nofollow noreferrer">multiple header rows</a>. I can't assume it has only one header row.)</p>
  
  <pre><code>&lt;table class="wikitable"&gt;
  &lt;tr&gt;
  &lt;th&gt;Rank&lt;/th&gt;
  &lt;th&gt;Score&lt;/th&gt;
  &lt;th&gt;Overs&lt;/th&gt;
  &lt;th&gt;&lt;b&gt;Ext&lt;/b&gt;&lt;/th&gt;
  &lt;th&gt;b&lt;/th&gt;
  &lt;th&gt;lb&lt;/th&gt;
  &lt;th&gt;w&lt;/th&gt;
  &lt;th&gt;nb&lt;/th&gt;
  &lt;th&gt;Opposition&lt;/th&gt;
  &lt;th&gt;Ground&lt;/th&gt;
  &lt;th&gt;Match Date&lt;/th&gt;
  &lt;/tr&gt;
  &lt;tr&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;437&lt;/td&gt;
  &lt;td&gt;136.0&lt;/td&gt;
  &lt;td&gt;&lt;b&gt;64&lt;/b&gt;&lt;/td&gt;
  &lt;td&gt;18&lt;/td&gt;
  &lt;td&gt;11&lt;/td&gt;
  &lt;td&gt;1&lt;/td&gt;
  &lt;td&gt;34&lt;/td&gt;
  &lt;td&gt;v West Indies&lt;/td&gt;
  &lt;td&gt;Manchester&lt;/td&gt;
  &lt;td&gt;27 Jul 1995&lt;/td&gt;
  &lt;/tr&gt;
  &lt;/table&gt;
  </code></pre>