<h2 dir="auto">Minimal Verifiable Complete Exemple</h2>
<p dir="auto">Below a MVCE of the behavior:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
# Trial Data:
data = {
    'key1': list(range(6))*2
   ,'key2': [100, 100, 100, 100, 200, 200, 200, 300, 300, None, None, None]
   ,'data': ['a']*12
}
# Load Data:
df0 = pd.DataFrame(data)
# Index (Int64 upcasted to Float64, because of None converted into NaN)
df1 = df0.set_index('key2')
# MultiIndex containing NaN on second level:
df2 = df0.set_index(['key1', 'key2'])
# NaN values are replaced by last existing value:
idx = df2.index.remove_unused_levels()
# Then, Index are not equal:
idx.equals(df2.index) # False"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-c"># Trial Data:</span>
<span class="pl-s1">data</span> <span class="pl-c1">=</span> {
    <span class="pl-s">'key1'</span>: <span class="pl-en">list</span>(<span class="pl-en">range</span>(<span class="pl-c1">6</span>))<span class="pl-c1">*</span><span class="pl-c1">2</span>
   ,<span class="pl-s">'key2'</span>: [<span class="pl-c1">100</span>, <span class="pl-c1">100</span>, <span class="pl-c1">100</span>, <span class="pl-c1">100</span>, <span class="pl-c1">200</span>, <span class="pl-c1">200</span>, <span class="pl-c1">200</span>, <span class="pl-c1">300</span>, <span class="pl-c1">300</span>, <span class="pl-c1">None</span>, <span class="pl-c1">None</span>, <span class="pl-c1">None</span>]
   ,<span class="pl-s">'data'</span>: [<span class="pl-s">'a'</span>]<span class="pl-c1">*</span><span class="pl-c1">12</span>
}
<span class="pl-c"># Load Data:</span>
<span class="pl-s1">df0</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">data</span>)
<span class="pl-c"># Index (Int64 upcasted to Float64, because of None converted into NaN)</span>
<span class="pl-s1">df1</span> <span class="pl-c1">=</span> <span class="pl-s1">df0</span>.<span class="pl-en">set_index</span>(<span class="pl-s">'key2'</span>)
<span class="pl-c"># MultiIndex containing NaN on second level:</span>
<span class="pl-s1">df2</span> <span class="pl-c1">=</span> <span class="pl-s1">df0</span>.<span class="pl-en">set_index</span>([<span class="pl-s">'key1'</span>, <span class="pl-s">'key2'</span>])
<span class="pl-c"># NaN values are replaced by last existing value:</span>
<span class="pl-s1">idx</span> <span class="pl-c1">=</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-en">remove_unused_levels</span>()
<span class="pl-c"># Then, Index are not equal:</span>
<span class="pl-s1">idx</span>.<span class="pl-en">equals</span>(<span class="pl-s1">df2</span>.<span class="pl-s1">index</span>) <span class="pl-c"># False</span></pre></div>
<h2 dir="auto">Problem description</h2>
<p dir="auto">Using method <code class="notranslate">remove_unused_levels</code> on MultiIndex containing <code class="notranslate">NaN</code> create a new MultiIndex that is not equal to the original as documentation says:</p>
<blockquote>
<p dir="auto">The resulting MultiIndex will have the same outward appearance, meaning the same .values and ordering. It will also be .equals() to the original.</p>
</blockquote>
<p dir="auto">This is why I suspect it is a bug.</p>
<h3 dir="auto">Float Index</h3>
<p dir="auto">Single <code class="notranslate">Index</code> uses <code class="notranslate">NaN</code> as modality:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df1.index
Float64Index([100.0, 100.0, 100.0, 100.0, 200.0, 200.0, 200.0, 300.0, 300.0,
              nan, nan, nan],
             dtype='float64', name='key2')"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df1</span>.<span class="pl-s1">index</span>
<span class="pl-v">Float64Index</span>([<span class="pl-c1">100.0</span>, <span class="pl-c1">100.0</span>, <span class="pl-c1">100.0</span>, <span class="pl-c1">100.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">300.0</span>, <span class="pl-c1">300.0</span>,
              <span class="pl-s1">nan</span>, <span class="pl-s1">nan</span>, <span class="pl-s1">nan</span>],
             <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s">'float64'</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">'key2'</span>)</pre></div>
<p dir="auto">But, <code class="notranslate">MultiIndex</code> does not, it has negative modality index instead:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df2.index
MultiIndex(levels=[[0, 1, 2, 3, 4, 5], [100.0, 200.0, 300.0]],
           labels=[[0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5], [0, 0, 0, 0, 1, 1, 1, 2, 2, -1, -1, -1]],
           names=['key1', 'key2'])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>
<span class="pl-v">MultiIndex</span>(<span class="pl-s1">levels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">100.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">300.0</span>]],
           <span class="pl-s1">labels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-c1">-</span><span class="pl-c1">1</span>]],
           <span class="pl-s1">names</span><span class="pl-c1">=</span>[<span class="pl-s">'key1'</span>, <span class="pl-s">'key2'</span>])</pre></div>
<h3 dir="auto">MultiIndex corruption</h3>
<p dir="auto">When refreshed, <code class="notranslate">NaN</code> values point to a copy of the last <code class="notranslate">float</code> modality (here <code class="notranslate">300.0</code>) of the level, this lead to a kind of corrupted index because those auto-filled value do not have any meaning.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df2.index.remove_unused_levels()
MultiIndex(levels=[[0, 1, 2, 3, 4, 5], [100.0, 200.0, 300.0, 300.0]],
           labels=[[0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5], [0, 0, 0, 0, 1, 1, 1, 3, 3, 3, 3, 3]],
           names=['key1', 'key2'])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-en">remove_unused_levels</span>()
<span class="pl-v">MultiIndex</span>(<span class="pl-s1">levels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">100.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">300.0</span>, <span class="pl-c1">300.0</span>]],
           <span class="pl-s1">labels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>]],
           <span class="pl-s1">names</span><span class="pl-c1">=</span>[<span class="pl-s">'key1'</span>, <span class="pl-s">'key2'</span>])</pre></div>
<p dir="auto">As a consequence Index are not equal (which contradicts documentation):</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df2.index.remove_unused_levels().equals(df2.index)
False"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-en">remove_unused_levels</span>().<span class="pl-en">equals</span>(<span class="pl-s1">df2</span>.<span class="pl-s1">index</span>)
<span class="pl-c1">False</span></pre></div>
<p dir="auto">Even worse, original value (<code class="notranslate">300.0</code> is not referenced anymore), and then it is a unused value/modality in the newly generated index.</p>
<p dir="auto">To confirm it, lets apply the method twice, we get:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df2.index.remove_unused_levels().remove_unused_levels()
MultiIndex(levels=[[0, 1, 2, 3, 4, 5], [100.0, 200.0, 300.0]],
           labels=[[0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5], [0, 0, 0, 0, 1, 1, 1, 2, 2, 2, 2, 2]],
           names=['key1', 'key2'])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-en">remove_unused_levels</span>().<span class="pl-en">remove_unused_levels</span>()
<span class="pl-v">MultiIndex</span>(<span class="pl-s1">levels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">100.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">300.0</span>]],
           <span class="pl-s1">labels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>]],
           <span class="pl-s1">names</span><span class="pl-c1">=</span>[<span class="pl-s">'key1'</span>, <span class="pl-s">'key2'</span>])</pre></div>
<h2 dir="auto">Expected Output</h2>
<p dir="auto">I believe expected output of <code class="notranslate">set_index</code> and <code class="notranslate">remove_unused_levels</code> should be:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; df2.index.remove_unused_levels()
MultiIndex(levels=[[0, 1, 2, 3, 4, 5], [100.0, 200.0, 300.0, nan]],
           labels=[[0, 1, 2, 3, 4, 5, 0, 1, 2, 3, 4, 5], [0, 0, 0, 0, 1, 1, 1, 2, 2, 3, 3, 3]],
           names=['key1', 'key2'])"><pre class="notranslate"><span class="pl-c1">&gt;&gt;</span><span class="pl-c1">&gt;</span> <span class="pl-s1">df2</span>.<span class="pl-s1">index</span>.<span class="pl-en">remove_unused_levels</span>()
<span class="pl-v">MultiIndex</span>(<span class="pl-s1">levels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">100.0</span>, <span class="pl-c1">200.0</span>, <span class="pl-c1">300.0</span>, <span class="pl-s1">nan</span>]],
           <span class="pl-s1">labels</span><span class="pl-c1">=</span>[[<span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>], [<span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">1</span>, <span class="pl-c1">2</span>, <span class="pl-c1">2</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>, <span class="pl-c1">3</span>]],
           <span class="pl-s1">names</span><span class="pl-c1">=</span>[<span class="pl-s">'key1'</span>, <span class="pl-s">'key2'</span>])</pre></div>
<p dir="auto">The problem also occurs when rows are removed from the DataFrame, and then it makes sense to use the method <code class="notranslate">remove_unused_levels</code> to clean up index. Anyway, when building the MCVE I found it was working on the whole Index whatever the level order.</p>
<h3 dir="auto">Pandas Versions</h3>
<details>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-75-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 36.4.0
Cython: None
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.1.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None"><pre class="notranslate"><code class="notranslate">INSTALLED VERSIONS
------------------
commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-75-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.21.0
pytest: None
pip: 9.0.1
setuptools: 36.4.0
Cython: None
numpy: 1.13.3
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.1
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.1.9
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
</code></pre></div>
</details>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [2]: mi = pd.MultiIndex.from_tuples([('a', 'w'), ('b', 'x'), (np.nan, 'y'), ('d', 'z')])

In [3]: list(mi) # OK
Out[3]: [('a', 'w'), ('b', 'x'), (nan, 'y'), ('d', 'z')]

In [4]: list(mi.remove_unused_levels()) # not OK
Out[4]: [('a', 'w'), ('b', 'x'), ('d', 'y'), ('d', 'z')]

"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">mi</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">MultiIndex</span>.<span class="pl-en">from_tuples</span>([(<span class="pl-s">'a'</span>, <span class="pl-s">'w'</span>), (<span class="pl-s">'b'</span>, <span class="pl-s">'x'</span>), (<span class="pl-s1">np</span>.<span class="pl-s1">nan</span>, <span class="pl-s">'y'</span>), (<span class="pl-s">'d'</span>, <span class="pl-s">'z'</span>)])

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-en">list</span>(<span class="pl-s1">mi</span>) <span class="pl-c"># OK</span>
<span class="pl-v">Out</span>[<span class="pl-c1">3</span>]: [(<span class="pl-s">'a'</span>, <span class="pl-s">'w'</span>), (<span class="pl-s">'b'</span>, <span class="pl-s">'x'</span>), (<span class="pl-s1">nan</span>, <span class="pl-s">'y'</span>), (<span class="pl-s">'d'</span>, <span class="pl-s">'z'</span>)]

<span class="pl-v">In</span> [<span class="pl-c1">4</span>]: <span class="pl-en">list</span>(<span class="pl-s1">mi</span>.<span class="pl-en">remove_unused_levels</span>()) <span class="pl-c"># not OK</span>
<span class="pl-v">Out</span>[<span class="pl-c1">4</span>]: [(<span class="pl-s">'a'</span>, <span class="pl-s">'w'</span>), (<span class="pl-s">'b'</span>, <span class="pl-s">'x'</span>), (<span class="pl-s">'d'</span>, <span class="pl-s">'y'</span>), (<span class="pl-s">'d'</span>, <span class="pl-s">'z'</span>)]
</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">The missing value is replaced with another value.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto"><code class="notranslate">In [3]</code> and <code class="notranslate">In [4]</code> should return the same.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.5.3.final.0<br>
python-bits: 64<br>
OS: Linux<br>
OS-release: 4.9.0-4-amd64<br>
machine: x86_64<br>
processor:<br>
byteorder: little<br>
LC_ALL: None<br>
LANG: en_GB.UTF-8<br>
LOCALE: en_GB.UTF-8</p>
<p dir="auto">pandas: 0.22.0.dev0+163.g3b05a60e6.dirty<br>
pytest: 3.0.6<br>
pip: 9.0.1<br>
setuptools: 33.1.1<br>
Cython: 0.25.2<br>
numpy: 1.12.1<br>
scipy: 0.18.1<br>
pyarrow: None<br>
xarray: None<br>
IPython: 5.2.2<br>
sphinx: None<br>
patsy: 0.4.1+dev<br>
dateutil: 2.6.0<br>
pytz: 2016.10<br>
blosc: None<br>
bottleneck: 1.2.0<br>
tables: 3.3.0<br>
numexpr: 2.6.1<br>
feather: 0.3.1<br>
matplotlib: 2.0.0<br>
openpyxl: 2.3.0<br>
xlrd: 1.0.0<br>
xlwt: 1.2.0<br>
xlsxwriter: None<br>
lxml: 3.7.1<br>
bs4: 4.5.3<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.0.15<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.8<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>