<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
pd.read_html('https://www.ssa.gov/policy/docs/statcomps/supplement/2015/5h.html')[0]"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-s1">pd</span>.<span class="pl-en">read_html</span>(<span class="pl-s">'https://www.ssa.gov/policy/docs/statcomps/supplement/2015/5h.html'</span>)[<span class="pl-c1">0</span>]</pre></div>
<p dir="auto">This has complex table headings:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/976970/28495218-7db85530-6efb-11e7-9f64-8b56a61509d4.png"><img width="967" alt="annual_statistical_supplement__2015_-_beneficiary_families_with_oasdi_benefits_in_current-payment_status__5_h_" src="https://user-images.githubusercontent.com/976970/28495218-7db85530-6efb-11e7-9f64-8b56a61509d4.png" style="max-width: 100%;"></a></p>
<p dir="auto"><code class="notranslate">read_html</code> output begins with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="                                                 Year                            Retired-worker families        Survivor families             Disabled-worker families                                Unnamed: 4_level_0                              Unnamed: 5_level_0 Unnamed: 6_level_0 Unnamed: 7_level_0 Unnamed: 8_level_0 Unnamed: 9_level_0 Unnamed: 10_level_0 Unnamed: 11_level_0  \
                                          Worker only                                  Worker and wife?a  Non-disabled widow only        Widowed mother or father and?                                       Worker only                            Worker, wife,?b and?  Worker and spouse Unnamed: 7_level_1 Unnamed: 8_level_1 Unnamed: 9_level_1 Unnamed: 10_level_1 Unnamed: 11_level_1
                                                  All                                                Men                    Women                              1?child                                        2?children                              3 or more children                All                Men              Women            1?child  2 or more children Unnamed: 11_level_2
0                                                 NaN                                 Number?(thousands)                      NaN                                  NaN                                               NaN                                             NaN                NaN                NaN                NaN                NaN                 NaN                 NaN
1                                                1945                                                416                      338                                   78                                               181                                              95              86.00              48.00              24.00              .?.?.               .?.?.               .?.?."><pre class="notranslate"><code class="notranslate">                                                 Year                            Retired-worker families        Survivor families             Disabled-worker families                                Unnamed: 4_level_0                              Unnamed: 5_level_0 Unnamed: 6_level_0 Unnamed: 7_level_0 Unnamed: 8_level_0 Unnamed: 9_level_0 Unnamed: 10_level_0 Unnamed: 11_level_0  \
                                          Worker only                                  Worker and wife?a  Non-disabled widow only        Widowed mother or father and?                                       Worker only                            Worker, wife,?b and?  Worker and spouse Unnamed: 7_level_1 Unnamed: 8_level_1 Unnamed: 9_level_1 Unnamed: 10_level_1 Unnamed: 11_level_1
                                                  All                                                Men                    Women                              1?child                                        2?children                              3 or more children                All                Men              Women            1?child  2 or more children Unnamed: 11_level_2
0                                                 NaN                                 Number?(thousands)                      NaN                                  NaN                                               NaN                                             NaN                NaN                NaN                NaN                NaN                 NaN                 NaN
1                                                1945                                                416                      338                                   78                                               181                                              95              86.00              48.00              24.00              .?.?.               .?.?.               .?.?.
</code></pre></div>
<p dir="auto">(row 0 of the output is probably something one would have to manually eliminate)</p>
<h4 dir="auto">Problem description</h4>
<p dir="auto">For HTML headings with rowspan and colspan elements, <code class="notranslate">read_html</code> has undesirable behavior. Basically <code class="notranslate">read_html</code> packs all heading <code class="notranslate">&lt;th&gt;</code> elements in any particular row to the left, so any particular column no longer has any association with the <code class="notranslate">&lt;th&gt;</code> elements that are actually above it in the HTML table.</p>
<p dir="auto">Ample discussion here about the analogous pandas+Excel test case: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="18575759" data-permission-text="Title is private" data-url="https://github.com/pandas-dev/pandas/issues/4679" data-hovercard-type="issue" data-hovercard-url="/pandas-dev/pandas/issues/4679/hovercard" href="https://github.com/pandas-dev/pandas/issues/4679">#4679</a></p>
<p dir="auto">Relevant web discussions:</p>
<ul dir="auto">
<li><a href="https://stackoverflow.com/questions/23703638/bug-in-pandas-read-html-method" rel="nofollow">https://stackoverflow.com/questions/23703638/bug-in-pandas-read-html-method</a></li>
<li><a href="https://stackoverflow.com/questions/28763891/what-should-i-do-when-tr-has-rowspan" rel="nofollow">https://stackoverflow.com/questions/28763891/what-should-i-do-when-tr-has-rowspan</a></li>
<li><a href="https://johnricco.github.io/2017/04/04/python-html/" rel="nofollow">https://johnricco.github.io/2017/04/04/python-html/</a>   (using his example)</li>
</ul>
<p dir="auto">This may be an issue with the underlying parsers and cannot be solved well in pandas. This appears to be the behavior with both lxml and bs4/html5lib.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Each column should be associated with the <code class="notranslate">&lt;th&gt;</code> elements above it in the table. This might be a multi-row column name (as it is now) (a <code class="notranslate">MultiIndex</code>?) or a tuple (presumably if the argument <code class="notranslate">tupleize_cols</code> is set to <code class="notranslate">True</code>). Instead, currently, column <em>n</em> is associated with the <em>n</em> th <code class="notranslate">&lt;th&gt;</code> entry in the table row regardless of the settings of rowspan/colspan.</p>
<p dir="auto">It may be this is possible to do properly in current pandas in which case I apologize for filing the issue (but I'd be happy to know how to do it).</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.13.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.US-ASCII
LOCALE: None.None
<p dir="auto">pandas: 0.20.3<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 36.2.0<br>
Cython: 0.26<br>
numpy: 1.13.1<br>
scipy: 0.19.1<br>
xarray: None<br>
IPython: 5.3.0<br>
sphinx: 1.6.3<br>
patsy: None<br>
dateutil: 2.6.0<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.4.2<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: 2.4.7<br>
xlrd: 1.0.0<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: 3.7.3<br>
bs4: 4.5.3<br>
html5lib: 1.0b10<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>
<p dir="auto">Hi, guys! First of all, thanks for making this open source! =D Keep up the good work!</p>
<p dir="auto"><strong>SCRIPT</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import pandas as pd

dfs = pd.read_html('https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States')

print(dfs[0])"><pre class="notranslate"><code class="notranslate">import pandas as pd

dfs = pd.read_html('https://en.wikipedia.org/wiki/List_of_states_and_territories_of_the_United_States')

print(dfs[0])
</code></pre></div>
<p dir="auto">The output we found where  there is any colspan is considering only the first column, and the other values are pulled back in relation to the colspan amount, making the values on the tail's row as "NaN"</p>
<p dir="auto"><strong>#### Expected Output</strong><br>
In this web page, there is a colspan in the "capital" and "largest city" columns when they are the same, I think that in a general way we would expect that  the value would be duplicate on the following &lt;colspan's amount&gt; columns in the generated DataFrame</p>
<h4 dir="auto">output of <code class="notranslate">pd.show_versions()</code></h4>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.5.2.final.0<br>
python-bits: 64<br>
OS: Windows<br>
OS-release: 10<br>
machine: AMD64<br>
processor: Intel64 Family 6 Model 15 Stepping 13, GenuineIntel<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: None</p>
<p dir="auto">pandas: 0.18.1<br>
nose: None<br>
pip: 8.1.2<br>
setuptools: 27.2.0<br>
Cython: 0.24.1<br>
numpy: 1.11.1<br>
scipy: 0.18.0<br>
statsmodels: None<br>
xarray: None<br>
IPython: 5.1.0<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.5.3<br>
pytz: 2016.6.1<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
matplotlib: 1.5.1<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: 4.5.1<br>
html5lib: 0.999<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.8<br>
boto: None<br>
pandas_datareader: 0.2.1</p>