<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="d=pd.DataFrame(
    [
        [1,2,3],
        [1,2,4],
        [2,3,5],
        [1,None,5],
    ],
    columns=['i','j','k']
)
print d.dtypes
d = d.set_index(['i','j'])['k']
#d = d.sort_index(level=[0,1])
d = d.sort_index()
print d.loc[(2,3)]   # PerformanceWarning even though sorted"><pre class="notranslate"><code class="notranslate">d=pd.DataFrame(
    [
        [1,2,3],
        [1,2,4],
        [2,3,5],
        [1,None,5],
    ],
    columns=['i','j','k']
)
print d.dtypes
d = d.set_index(['i','j'])['k']
#d = d.sort_index(level=[0,1])
d = d.sort_index()
print d.loc[(2,3)]   # PerformanceWarning even though sorted
</code></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">This behavior his is very confusing. The <a href="https://pandas.pydata.org/pandas-docs/stable/advanced.html" rel="nofollow">docs ask me to sort multiindexes</a> to avoid the performance warning,</p>
<p dir="auto">If I do it, I still get the PerformanceWarning. If I print the dataframe, I see that data was indeed sorted.</p>
<p dir="auto">This is resolved by explicitly specifying the levels to sort: <code class="notranslate">d.sort_index(level=[0,1])</code>.</p>
<p dir="auto">As a user, I'm confused. Wasn't <code class="notranslate">d.sort_index()</code> the same thing as <code class="notranslate">d.sort_index(level=...all levels...)</code>? Inspecting my dataframe, it appeared to be the, case (the index was sorted, and nested into a hierarchy).</p>
<p dir="auto">Shouldn't  <code class="notranslate">d.sort_index()</code> simply be the same as <code class="notranslate">d.sort_index(level=...all levels...)</code></p>
<p dir="auto">I'm surprised there's any behavioral difference to be?</p>
<p dir="auto">It seems due to:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="d = d.sort_index()
print d.index.is_lexsorted()  # False

d = d.sort_index(level=[0,1])
print d.index.is_lexsorted()  # True"><pre class="notranslate"><code class="notranslate">d = d.sort_index()
print d.index.is_lexsorted()  # False

d = d.sort_index(level=[0,1])
print d.index.is_lexsorted()  # True
</code></pre></div>
<p dir="auto">and then, they are only different if the index contains nulls.</p>
<p dir="auto">Why there's a difference doesn't seem to be documented.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">Either: it should be that <code class="notranslate">d.sort_index()</code>  === <code class="notranslate">d.sort_index(level=...all levels...)</code><br>
or, <a href="https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sort_index.html" rel="nofollow"> dataframe.sort_index documentation</a> could stand improvement and explain when the fully specified level parameter is different to not specifying level.</p>
<p dir="auto">I'd prefer:  <code class="notranslate">d.sort_index()  === d.sort_index(level=d.index.names)</code>, because:</p>
<ul dir="auto">
<li>this would allow functions function working on dataframes with vanilla indexes and calling sort_index(), also work optimally when provided dataframes with multindexes.</li>
<li>its cumbersome and error-prone to need to remember to specify the level names, since this is only needed in the multi-index case, and then also only in the case where the index contains NULLs.</li>
</ul>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None
<p dir="auto">pandas: 0.22.0<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 36.0.1<br>
Cython: 0.27.3<br>
numpy: 1.14.0<br>
scipy: 1.0.0<br>
pyarrow: None<br>
xarray: None<br>
IPython: 5.3.0<br>
sphinx: None<br>
patsy: 0.2.1<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: 3.1.1<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: 1.7.0<br>
xlrd: 0.9.2<br>
xlwt: 0.7.5<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.0.15<br>
pymysql: None<br>
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)<br>
jinja2: 2.9.6<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>
<h4 dir="auto">Code Sample, a copy-pastable example if possible</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: dates_range = pd.date_range('2016/10/1', periods=3).repeat(4)
In [4]: statuses = np.tile(['A', 'B', None, 'C'], 3)
In [5]: values = np.random.randn(12)
In [6]: df = pd.DataFrame({'date': dates_range, 'status': statuses, 'value': values})
In [7]: df
Out[7]:
         date status     value
0  2016-10-01      A -1.946876
1  2016-10-01      B  1.080243
2  2016-10-01   None  0.165715
3  2016-10-01      C -0.615913
4  2016-10-02      A  0.662645
5  2016-10-02      B  1.448593
6  2016-10-02   None -1.392233
7  2016-10-02      C  1.534083
8  2016-10-03      A  0.801988
9  2016-10-03      B -0.689987
10 2016-10-03   None -0.150036
11 2016-10-03      C -0.197410

In [8]: df.set_index(['date', 'status']).sort_index()
Out[8]:
                      value
date       status
2016-10-01 A      -1.946876
           B       1.080243
           C      -0.615913
           NaN     0.165715
2016-10-02 A       0.662645
           B       1.448593
           C       1.534083
           NaN    -1.392233
2016-10-03 A       0.801988
           B      -0.689987
           C      -0.197410
           NaN    -0.150036

In [9]: df.set_index(['date', 'status']).sort_index().index.is_lexsorted()
Out[9]: False

In [10]: df.set_index(['date', 'status']).sort_index(level=['date', 'status']).index.is_lexsorted()
Out[10]: True"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">dates_range</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">date_range</span>(<span class="pl-s">'2016/10/1'</span>, <span class="pl-s1">periods</span><span class="pl-c1">=</span><span class="pl-c1">3</span>).<span class="pl-en">repeat</span>(<span class="pl-c1">4</span>)
<span class="pl-v">In</span> [<span class="pl-c1">4</span>]: <span class="pl-s1">statuses</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-en">tile</span>([<span class="pl-s">'A'</span>, <span class="pl-s">'B'</span>, <span class="pl-c1">None</span>, <span class="pl-s">'C'</span>], <span class="pl-c1">3</span>)
<span class="pl-v">In</span> [<span class="pl-c1">5</span>]: <span class="pl-s1">values</span> <span class="pl-c1">=</span> <span class="pl-s1">np</span>.<span class="pl-s1">random</span>.<span class="pl-en">randn</span>(<span class="pl-c1">12</span>)
<span class="pl-v">In</span> [<span class="pl-c1">6</span>]: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>({<span class="pl-s">'date'</span>: <span class="pl-s1">dates_range</span>, <span class="pl-s">'status'</span>: <span class="pl-s1">statuses</span>, <span class="pl-s">'value'</span>: <span class="pl-s1">values</span>})
<span class="pl-v">In</span> [<span class="pl-c1">7</span>]: <span class="pl-s1">df</span>
<span class="pl-v">Out</span>[<span class="pl-c1">7</span>]:
         <span class="pl-s1">date</span> <span class="pl-s1">status</span>     <span class="pl-s1">value</span>
<span class="pl-c1">0</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span>      <span class="pl-v">A</span> <span class="pl-c1">-</span><span class="pl-c1">1.946876</span>
<span class="pl-c1">1</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span>      <span class="pl-v">B</span>  <span class="pl-c1">1.080243</span>
<span class="pl-c1">2</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span>   <span class="pl-v">None</span>  <span class="pl-c1">0.165715</span>
<span class="pl-c1">3</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span>      <span class="pl-v">C</span> <span class="pl-c1">-</span><span class="pl-c1">0.615913</span>
<span class="pl-c1">4</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">02</span>      <span class="pl-v">A</span>  <span class="pl-c1">0.662645</span>
<span class="pl-c1">5</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">02</span>      <span class="pl-v">B</span>  <span class="pl-c1">1.448593</span>
<span class="pl-c1">6</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">02</span>   <span class="pl-c1">None</span> <span class="pl-c1">-</span><span class="pl-c1">1.392233</span>
<span class="pl-c1">7</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">02</span>      <span class="pl-v">C</span>  <span class="pl-c1">1.534083</span>
<span class="pl-c1">8</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">03</span>      <span class="pl-v">A</span>  <span class="pl-c1">0.801988</span>
<span class="pl-c1">9</span>  <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">03</span>      <span class="pl-v">B</span> <span class="pl-c1">-</span><span class="pl-c1">0.689987</span>
<span class="pl-c1">10</span> <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">03</span>   <span class="pl-c1">None</span> <span class="pl-c1">-</span><span class="pl-c1">0.150036</span>
<span class="pl-c1">11</span> <span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">03</span>      <span class="pl-v">C</span> <span class="pl-c1">-</span><span class="pl-c1">0.197410</span>

<span class="pl-v">In</span> [<span class="pl-c1">8</span>]: <span class="pl-s1">df</span>.<span class="pl-en">set_index</span>([<span class="pl-s">'date'</span>, <span class="pl-s">'status'</span>]).<span class="pl-en">sort_index</span>()
<span class="pl-v">Out</span>[<span class="pl-c1">8</span>]:
                      <span class="pl-s1">value</span>
<span class="pl-s1">date</span>       <span class="pl-s1">status</span>
<span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-v">A</span>      <span class="pl-c1">-</span><span class="pl-c1">1.946876</span>
           <span class="pl-v">B</span>       <span class="pl-c1">1.080243</span>
           <span class="pl-v">C</span>      <span class="pl-c1">-</span><span class="pl-c1">0.615913</span>
           <span class="pl-v">NaN</span>     <span class="pl-c1">0.165715</span>
<span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">02</span> <span class="pl-v">A</span>       <span class="pl-c1">0.662645</span>
           <span class="pl-v">B</span>       <span class="pl-c1">1.448593</span>
           <span class="pl-v">C</span>       <span class="pl-c1">1.534083</span>
           <span class="pl-v">NaN</span>    <span class="pl-c1">-</span><span class="pl-c1">1.392233</span>
<span class="pl-c1">2016</span><span class="pl-c1">-</span><span class="pl-c1">10</span><span class="pl-c1">-</span><span class="pl-c1">03</span> <span class="pl-v">A</span>       <span class="pl-c1">0.801988</span>
           <span class="pl-v">B</span>      <span class="pl-c1">-</span><span class="pl-c1">0.689987</span>
           <span class="pl-v">C</span>      <span class="pl-c1">-</span><span class="pl-c1">0.197410</span>
           <span class="pl-v">NaN</span>    <span class="pl-c1">-</span><span class="pl-c1">0.150036</span>

<span class="pl-v">In</span> [<span class="pl-c1">9</span>]: <span class="pl-s1">df</span>.<span class="pl-en">set_index</span>([<span class="pl-s">'date'</span>, <span class="pl-s">'status'</span>]).<span class="pl-en">sort_index</span>().<span class="pl-s1">index</span>.<span class="pl-en">is_lexsorted</span>()
<span class="pl-v">Out</span>[<span class="pl-c1">9</span>]: <span class="pl-c1">False</span>

<span class="pl-v">In</span> [<span class="pl-c1">10</span>]: <span class="pl-s1">df</span>.<span class="pl-en">set_index</span>([<span class="pl-s">'date'</span>, <span class="pl-s">'status'</span>]).<span class="pl-en">sort_index</span>(<span class="pl-s1">level</span><span class="pl-c1">=</span>[<span class="pl-s">'date'</span>, <span class="pl-s">'status'</span>]).<span class="pl-s1">index</span>.<span class="pl-en">is_lexsorted</span>()
<span class="pl-v">Out</span>[<span class="pl-c1">10</span>]: <span class="pl-c1">True</span></pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">By giving <code class="notranslate">sort_index()</code> no <code class="notranslate">level</code> argument I expect to have the dataframe sorted on all the levels, as indeed it happens. However the <code class="notranslate">is_lexsorted()</code> method afterwards reports that it is not. The problem is fixed if I explicitly pass the <code class="notranslate">level</code> argument.</p>
<p dir="auto">The behavior is particularly problematic when the <code class="notranslate">MultiIndex</code> has many levels, say <code class="notranslate">n</code>, and the <code class="notranslate">NaN</code>s appear at level <code class="notranslate">m</code> &lt; <code class="notranslate">n</code>. Then slicing the <code class="notranslate">MultiIndex</code> for levels up to <code class="notranslate">m</code> will work, while for levels from <code class="notranslate">m</code> and above it will crash.</p>
<p dir="auto">Note that this was not so in previous versions. Unfortunately not sure which ones; it worked OK in spring 2017, I came upon the issue by re-running code from that time.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">The expected behavior is the one that I would get with passing the <code class="notranslate">level</code> argument with all index levels, as in the transcript above.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None<br>
python: 3.6.3.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 17.0.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: en_US.UTF-8<br>
LANG: en_US.UTF-8<br>
LOCALE: en_US.UTF-8</p>
<p dir="auto">pandas: 0.20.3<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 36.5.0<br>
Cython: None<br>
numpy: 1.13.3<br>
scipy: 0.19.1<br>
xarray: None<br>
IPython: 6.2.0<br>
sphinx: None<br>
patsy: 0.4.1<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: None<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: None<br>
xlrd: 1.0.0<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: 4.6.0<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.1.14<br>
pymysql: 0.7.11.None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>