<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="In [1]: import pandas as pd

In [2]: df = pd.DataFrame([[1, 0, 0],[0,0,0]])

# non sparse
In [19]: %time df.loc[0]
CPU times: user 367 µs, sys: 14 µs, total: 381 µs
Wall time: 374 µs
Out[19]:
0    1
1    0
2    0
Name: 0, dtype: int64

In [20]: %time df.loc[[0]]
CPU times: user 808 µs, sys: 57 µs, total: 865 µs
Wall time: 827 µs
Out[20]:
   0  1  2
0  1  0  0

In [3]: df = df.to_sparse()

# sparse
In [5]: %time df.loc[0]
CPU times: user 429 µs, sys: 14 µs, total: 443 µs
Wall time: 436 µs
Out[5]:
0    1.0
1    0.0
2    0.0
Name: 0, dtype: float64
BlockIndex
Block locations: array([0], dtype=int32)
Block lengths: array([3], dtype=int32)

In [6]: %time df.loc[[0]]
CPU times: user 4.07 ms, sys: 406 µs, total: 4.48 ms
Wall time: 6.16 ms
Out[6]:
     0    1    2
0  1.0  0.0  0.0
"><pre class="notranslate"><span class="pl-v">In</span> [<span class="pl-c1">1</span>]: <span class="pl-s1">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>

<span class="pl-v">In</span> [<span class="pl-c1">2</span>]: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>([[<span class="pl-c1">1</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>],[<span class="pl-c1">0</span>,<span class="pl-c1">0</span>,<span class="pl-c1">0</span>]])

<span class="pl-c"># non sparse</span>
<span class="pl-v">In</span> [<span class="pl-c1">19</span>]: <span class="pl-c1">%</span><span class="pl-s1">time</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-c1">0</span>]
<span class="pl-v">CPU</span> <span class="pl-s1">times</span>: <span class="pl-s1">user</span> <span class="pl-c1">367</span> <span class="pl-s1">µs</span>, <span class="pl-s1">sys</span>: <span class="pl-c1">14</span> <span class="pl-s1">µs</span>, <span class="pl-s1">total</span>: <span class="pl-c1">381</span> <span class="pl-s1">µs</span>
<span class="pl-v">Wall</span> <span class="pl-s1">time</span>: <span class="pl-c1">374</span> <span class="pl-s1">µs</span>
<span class="pl-v">Out</span>[<span class="pl-c1">19</span>]:
<span class="pl-c1">0</span>    <span class="pl-c1">1</span>
<span class="pl-c1">1</span>    <span class="pl-c1">0</span>
<span class="pl-c1">2</span>    <span class="pl-c1">0</span>
<span class="pl-v">Name</span>: <span class="pl-c1">0</span>, <span class="pl-s1">dtype</span>: <span class="pl-s1">int64</span>

<span class="pl-v">In</span> [<span class="pl-c1">20</span>]: <span class="pl-c1">%</span><span class="pl-s1">time</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[[<span class="pl-c1">0</span>]]
<span class="pl-v">CPU</span> <span class="pl-s1">times</span>: <span class="pl-s1">user</span> <span class="pl-c1">808</span> <span class="pl-s1">µs</span>, <span class="pl-s1">sys</span>: <span class="pl-c1">57</span> <span class="pl-s1">µs</span>, <span class="pl-s1">total</span>: <span class="pl-c1">865</span> <span class="pl-s1">µs</span>
<span class="pl-v">Wall</span> <span class="pl-s1">time</span>: <span class="pl-c1">827</span> <span class="pl-s1">µs</span>
<span class="pl-v">Out</span>[<span class="pl-c1">20</span>]:
   <span class="pl-c1">0</span>  <span class="pl-c1">1</span>  <span class="pl-c1">2</span>
<span class="pl-c1">0</span>  <span class="pl-c1">1</span>  <span class="pl-c1">0</span>  <span class="pl-c1">0</span>

<span class="pl-v">In</span> [<span class="pl-c1">3</span>]: <span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">df</span>.<span class="pl-en">to_sparse</span>()

<span class="pl-c"># sparse</span>
<span class="pl-v">In</span> [<span class="pl-c1">5</span>]: <span class="pl-c1">%</span><span class="pl-s1">time</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[<span class="pl-c1">0</span>]
<span class="pl-v">CPU</span> <span class="pl-s1">times</span>: <span class="pl-s1">user</span> <span class="pl-c1">429</span> <span class="pl-s1">µs</span>, <span class="pl-s1">sys</span>: <span class="pl-c1">14</span> <span class="pl-s1">µs</span>, <span class="pl-s1">total</span>: <span class="pl-c1">443</span> <span class="pl-s1">µs</span>
<span class="pl-v">Wall</span> <span class="pl-s1">time</span>: <span class="pl-c1">436</span> <span class="pl-s1">µs</span>
<span class="pl-v">Out</span>[<span class="pl-c1">5</span>]:
<span class="pl-c1">0</span>    <span class="pl-c1">1.0</span>
<span class="pl-c1">1</span>    <span class="pl-c1">0.0</span>
<span class="pl-c1">2</span>    <span class="pl-c1">0.0</span>
<span class="pl-v">Name</span>: <span class="pl-c1">0</span>, <span class="pl-s1">dtype</span>: <span class="pl-s1">float64</span>
<span class="pl-v">BlockIndex</span>
<span class="pl-v">Block</span> <span class="pl-s1">locations</span>: <span class="pl-en">array</span>([<span class="pl-c1">0</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">int32</span>)
<span class="pl-v">Block</span> <span class="pl-s1">lengths</span>: <span class="pl-en">array</span>([<span class="pl-c1">3</span>], <span class="pl-s1">dtype</span><span class="pl-c1">=</span><span class="pl-s1">int32</span>)

<span class="pl-v">In</span> [<span class="pl-c1">6</span>]: <span class="pl-c1">%</span><span class="pl-s1">time</span> <span class="pl-s1">df</span>.<span class="pl-s1">loc</span>[[<span class="pl-c1">0</span>]]
<span class="pl-v">CPU</span> <span class="pl-s1">times</span>: <span class="pl-s1">user</span> <span class="pl-c1">4.07</span> <span class="pl-s1">ms</span>, <span class="pl-s1">sys</span>: <span class="pl-c1">406</span> <span class="pl-s1">µs</span>, <span class="pl-s1">total</span>: <span class="pl-c1">4.48</span> <span class="pl-s1">ms</span>
<span class="pl-v">Wall</span> <span class="pl-s1">time</span>: <span class="pl-c1">6.16</span> <span class="pl-s1">ms</span>
<span class="pl-v">Out</span>[<span class="pl-c1">6</span>]:
     <span class="pl-c1">0</span>    <span class="pl-c1">1</span>    <span class="pl-c1">2</span>
<span class="pl-c1">0</span>  <span class="pl-c1">1.0</span>  <span class="pl-c1">0.0</span>  <span class="pl-c1">0.0</span></pre></div>
<p dir="auto">It seems in SparseDataFrame, selecting as dataframe, the time increased much more than it does in DataFrame, I would expect it runs faster(on my 1m x 500 SparseDataFrame it takes seconds for the loc operation)</p>
<details>
## INSTALLED VERSIONS
<p dir="auto">commit: None<br>
python: 3.5.1.final.0<br>
python-bits: 64<br>
OS: Darwin<br>
OS-release: 15.6.0<br>
machine: x86_64<br>
processor: i386<br>
byteorder: little<br>
LC_ALL: en_GB.UTF-8<br>
LANG: en_GB.UTF-8</p>
<p dir="auto">pandas: 0.18.1<br>
nose: None<br>
pip: 8.1.2<br>
setuptools: 24.0.2<br>
Cython: None<br>
numpy: 1.11.0<br>
scipy: 0.17.1<br>
statsmodels: None<br>
xarray: None<br>
IPython: 5.0.0<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.5.3<br>
pytz: 2016.4<br>
blosc: None<br>
bottleneck: None<br>
tables: None<br>
numexpr: 2.6.1<br>
matplotlib: 1.5.1<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: None<br>
html5lib: None<br>
httplib2: None<br>
apiclient: None<br>
sqlalchemy: 1.0.13<br>
pymysql: None<br>
psycopg2: 2.6.1 (dt dec pq3 ext lo64)<br>
jinja2: None<br>
boto: None<br>
pandas_datareader: None</p>
</details>
<h4 dir="auto">Code Sample</h4>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="df = Dataframe(numpy.zeros(10000,10000))
random_fill_df(df, num_elements=20)
df = df.to_sparse(fill_value=0)
timeit.timeit('df.loc[[23, 45, 65, 67],:]', globals=globals(), number=10)"><pre class="notranslate"><span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-v">Dataframe</span>(<span class="pl-s1">numpy</span>.<span class="pl-en">zeros</span>(<span class="pl-c1">10000</span>,<span class="pl-c1">10000</span>))
<span class="pl-en">random_fill_df</span>(<span class="pl-s1">df</span>, <span class="pl-s1">num_elements</span><span class="pl-c1">=</span><span class="pl-c1">20</span>)
<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">df</span>.<span class="pl-en">to_sparse</span>(<span class="pl-s1">fill_value</span><span class="pl-c1">=</span><span class="pl-c1">0</span>)
<span class="pl-s1">timeit</span>.<span class="pl-en">timeit</span>(<span class="pl-s">'df.loc[[23, 45, 65, 67],:]'</span>, <span class="pl-s1">globals</span><span class="pl-c1">=</span><span class="pl-en">globals</span>(), <span class="pl-s1">number</span><span class="pl-c1">=</span><span class="pl-c1">10</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">The reason why row slicing takes so long is because a sparse dataframe a bunch of sparse series. Column slicing is several order of magnitude faster but row slicing is very poor. The sparse dataframe doesn't take advantage of the scipy sparse matrix library which is even faster (both column and row).</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">In case data is stored as a scipy sparse matrix (as well) inside dataframe object, the slicing operations can be improved, by several orders of magnitude.</p>
<p dir="auto">I propose that data be stored as a sparse matrix as well in the dataframe object.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<p dir="auto">pandas: 0.20.3<br>
pytest: None<br>
pip: 9.0.1<br>
setuptools: 36.2.0<br>
Cython: None<br>
numpy: 1.13.1<br>
scipy: 0.19.1<br>
xarray: None<br>
IPython: None<br>
sphinx: None<br>
patsy: None<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: None<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.0.2<br>
openpyxl: None<br>
xlrd: None<br>
xlwt: None<br>
xlsxwriter: None<br>
lxml: None<br>
bs4: 4.6.0<br>
html5lib: 0.999999999<br>
sqlalchemy: None<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: None<br>
s3fs: None<br>
pandas_gbq: None<br>
pandas_datareader: None</p>
</details>