title: Parsing an HTML table to a list in python
body: <p>So I have a few strings that I am pulling from IMDb's award pages:</p>
  
  <pre><code>&lt;table&gt;&lt;tr&gt;&lt;td&gt;&lt;big&gt;Academy Awards, USA&lt;/big&gt;        &lt;/td&gt;      &lt;/tr&gt;      &lt;tr&gt;        &lt;th&gt;Year&lt;/th&gt;&lt;th&gt;Result&lt;/th&gt;&lt;th&gt;Award&lt;/th&gt;&lt;th&gt;Category/Recipient(s)&lt;/th&gt;      &lt;/tr&gt;            &lt;tr&gt;        &lt;td rowspan="11" align="center" valign="middle"&gt;          1978         &lt;/td&gt;                          &lt;td rowspan="7" align="center" valign="middle"&gt;&lt;b&gt;Won&lt;/b&gt;&lt;/td&gt;                                                      &lt;td rowspan="6" align="center" valign="middle"&gt;Oscar&lt;/td&gt;                                      &lt;td valign="top"&gt;          Best Art Direction-Set Decoration                                John Barry                                              Norman Reynolds                                              Leslie Dilley                                              Roger Christian                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Costume Design                                John Mollo                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Effects, Visual Effects                                John Stears                                              John Dykstra                                              Richard Edlund                                              Grant McCune                                              Robert Blalack                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Film Editing                                Paul Hirsch                                              Marcia Lucas                                              Richard Chew                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Music, Original Score                                John Williams                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Sound                                Don MacDougall                                              Ray West                                              Bob Minkler                                              Derek Ball                                                      &lt;small&gt;                                                                      Derek Ball was not present at the awards ceremony.          &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                                      &lt;tr&gt;                                            &lt;td rowspan="1" align="center" valign="middle"&gt;Special Achievement Award&lt;/td&gt;                                      &lt;td valign="top"&gt;                                          Ben Burtt             (as Benjamin Burtt Jr.)                                          &lt;small&gt;                                                                      For sound effects. (For the creation of the alien, creature and robot voices.)          &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                                            &lt;tr&gt;                  &lt;td rowspan="4" align="center" valign="middle"&gt;&lt;b&gt;Nominated&lt;/b&gt;&lt;/td&gt;                                                      &lt;td rowspan="4" align="center" valign="middle"&gt;Oscar&lt;/td&gt;                                      &lt;td valign="top"&gt;          Best Actor in a Supporting Role                                Alec Guinness                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Director                                George Lucas                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Picture                                Gary Kurtz                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                    &lt;tr&gt;                        &lt;td valign="top"&gt;          Best Writing, Screenplay Written Directly for the Screen                                George Lucas                                                      &lt;small&gt;                                                                                &lt;/small&gt;        &lt;/td&gt;      &lt;/tr&gt;                                                  &lt;tr&gt;        &lt;/tr&gt;&lt;/table&gt;
  </code></pre>
  
  <p>I want to pull the headers (Year, Result, Award, and Category/Recipient) to a list and then each of the columns to their own list, respectively. For example (using the Academy Award table)(Website for reference: <a href="http://www.imdb.com/title/tt0076759/awards" rel="nofollow">http://www.imdb.com/title/tt0076759/awards</a>):</p>
  
  <pre><code>Columns = {"Year", "Result", "Award", "Category/Recipient"}
  Years = {"1978", "1978", "1978", "1978", "1978", "1978", "1978"}
  Results = {"Oscar", "Oscar", "Oscar", "Oscar", "Oscar", "Oscar", "Special Achievement Award"}
  Categories/Recipients = {"Best Art Direction-Set Decoration (John Barry, Norman Reynolds, Leslie Dilley, Roger Christian)", "Best Costume Design (John Mollo)", "Best Effects, Visual Effects (John Stears, John Dykstra, Richard Edlund, Grant McCune, Robert Blalack)", Best Film Editing (Paul Hirsch, Marcia Lucas, Richard Chew)", "Best Music, Original Score (John Williams)", "Best Sound (Don MacDougall, Ray West, Bob Minkler, Derek Ball)", "(Ben Burtt (as Benjamin Burtt Jr.))"}
  </code></pre>
  
  <p>As you can see, I removed the unnecessary spacing from the table and put all the names in parenthesis. There are tags around all of the names, but I removed them (they can be kept in if it helps putting them in parenthesis easier). I also have the same number of items in each of the lists except for the Columns list.</p>
  
  <p>Here is my current script so you know how I manipulate it already:</p>
  
  <pre><code>import shutil
  import urllib2
  import re
  from lxml import etree
  
  award_usock = urllib2.urlopen('http://www.imdb.com/title/tt0076759' + '/awards')
  award_html = award_usock.read()
  award_usock.close()
  if "&lt;big&gt;" in award_html:
      for a_show in re.finditer('&lt;big&gt;',award_html):
          award_show_full_end = award_html.find('&lt;td colspan="4"&gt;&amp;nbsp;&lt;/td&gt;',a_show.end())
          award_show_full = award_html[a_show.start():award_show_full_end]
          award_show_full = award_show_full.replace('\n','')
          # award_show_full = award_show_full.replace('  ','')
          award_show_full = award_show_full.replace('&lt;/a&gt;','')
          award_show_full = award_show_full.replace('&lt;br /&gt;','')
          award_show_full = re.sub('&lt;a href="/name/[^&gt;]*&gt;',  '', award_show_full)
          award_show_full = re.sub('&lt;a href="/title/[^&gt;]*&gt;',  '', award_show_full)
          for a_s_title in re.finditer('&lt;a href="',award_show_full):
              award_title_loc = award_show_full.find('&lt;a href="')
              award_title_end = award_show_full.find('"&gt;',award_title_loc+10)
              award_title_del = award_show_full[award_title_loc:award_title_end+2]
              award_show_full = award_show_full.replace(award_title_del,'')
          award_show_full = '&lt;table&gt;&lt;tr&gt;&lt;td&gt;' + award_show_full.replace('&lt;br&gt;','') + '&lt;/tr&gt;&lt;/table&gt;'
          award_show_loc = award_html.find('&gt;',a_show.end())
          award_show_end = award_html.find('&lt;/a&gt;&lt;/big&gt;',a_show.end())
          award_show = award_html[award_show_loc+1:award_show_end]
          award_show_table = etree.XML(award_show_full)
          award_show_rows = iter(award_show_table)
          award_show_headers = [award_show_col.text for award_show_col in next(award_show_rows)]
          for award_show_row in award_show_rows:
              award_show_values = [award_show_col.text for award_show_col in award_show_row]
              print dict(zip(award_show_headers,award_show_values))
  </code></pre>
  
  <p>But this produces a result:</p>
  
  <pre><code>{None: 'Year'}
  {None: '          1978         '}
  {None: '          Best Costume Design                                John Mollo                                                      '}
  {None: '          Best Effects, Visual Effects                                John Stears                                              John Dykstra                                              Richard Edlund                                              Grant McCune                                              Robert Blalack                                                      '}
  {None: '          Best Film Editing                                Paul Hirsch                                              Marcia Lucas                                              Richard Chew                                                      '}
  {None: '          Best Music, Original Score                                John Williams                                                      '}
  {None: '          Best Sound                                Don MacDougall                                              Ray West                                              Bob Minkler                                              Derek Ball                                                      '}
  {None: 'Special Achievement Award'}
  {None: None}
  {None: '          Best Director                                George Lucas                                                      '}
  {None: '          Best Picture                                Gary Kurtz                                                      '}
  {None: '          Best Writing, Screenplay Written Directly for the Screen                                George Lucas                                                      '}
  {}
  </code></pre>