How to read html tables under multiple headers and combine them in a single pandas dataframe? I have an html file like this:

&lt;h1&gt;Group 1&lt;/h1&gt;
&lt;table&gt;
  &lt;tr&gt;
    &lt;td&gt;Col1&lt;/td&gt;
    &lt;td&gt;Col2&lt;/td&gt;
    &lt;td&gt;Col3&lt;/td&gt;
  &lt;/tr&gt;
  &lt;tr&gt;
    &lt;td&gt;ValA&lt;/td&gt;
    &lt;td&gt;ValB&lt;/td&gt;
    &lt;td&gt;ValC&lt;/td&gt;
  &lt;/tr&gt;
&lt;/table&gt;
&lt;h1&gt;Group 2&lt;/h1&gt;
&lt;table&gt;
  &lt;tr&gt;
    &lt;td&gt;Col1&lt;/td&gt;
    &lt;td&gt;Col2&lt;/td&gt;
    &lt;td&gt;Col3&lt;/td&gt;
  &lt;/tr&gt;
  &lt;tr&gt;
    &lt;td&gt;ValP&lt;/td&gt;
    &lt;td&gt;ValQ&lt;/td&gt;
    &lt;td&gt;ValR&lt;/td&gt;
  &lt;/tr&gt;
&lt;/table&gt;


I want to read it into Pandas as if it had this structure:

&lt;table&gt;
 &lt;tr&gt;
   &lt;td&gt;Caption&lt;/td&gt;
   &lt;td&gt;Col1&lt;/td&gt;
   &lt;td&gt;Col2&lt;/td&gt;
   &lt;td&gt;Col3&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
   &lt;td&gt;Group 1&lt;/td&gt;
   &lt;td&gt;ValA&lt;/td&gt;
   &lt;td&gt;ValB&lt;/td&gt;
   &lt;td&gt;ValC&lt;/td&gt;
 &lt;/tr&gt;
 &lt;tr&gt;
   &lt;td&gt;Group 2&lt;/td&gt;
   &lt;td&gt;ValP&lt;/td&gt;
   &lt;td&gt;ValQ&lt;/td&gt;
   &lt;td&gt;ValR&lt;/td&gt;
 &lt;/tr&gt;
&lt;/table&gt;


I can do it easily with PowerQuery the language for PowerBI:

let
    Source = Web.Page(File.Contents("multiple_tables.html")),
    #"Expanded Data" = Table.ExpandTableColumn(Source, "Data", {"Column1", "Column2", "Column3"}, {"Col1", "Col2", "Col3"}),
    #"Filtered Rows" = Table.SelectRows(#"Expanded Data", each ([Caption] &lt;&gt; "Document") and ([Col1] &lt;&gt; "Col1"))
in
    #"Filtered Rows"


Is there a way to achieve this effect in less than 10 lines of code using Python/Pandas plus some html parser opensource library? Or should I resign myself to writing lower level code to handle this?
With a bit of parsing via BeautifulSoup, we can get a pandas.Dataframe using pandas.read_html() like:

Code:

def get_tables(source):
    elems = iter(BeautifulSoup(source, 'lxml').find_all(['table', 'h1']))
    df = pd.DataFrame(
        pd.read_html(str(next(elems)), header=0)[0].iloc[0].rename(h1.text)
        for h1 in elems)
    df.index.names = ['Caption']
    return df


Test Code:

import pandas as pd
from bs4 import BeautifulSoup

with open('test.html', 'r') as f:
    print(get_tables(f))


Results:

         Col1  Col2  Col3
Caption                  
Group 1  ValA  ValB  ValC
Group 2  ValP  ValQ  ValR