<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="import pandas as pd
import numpy as np
from sklearn.datasets import load_iris

iris = load_iris()
data = iris.data
column_names = iris.feature_names

df = pd.DataFrame(iris.data, columns = column_names)

print(df.head(n=5))

df_dummy = pd.get_dummies(df, sparse = True, columns = ['petal length (cm)'] )

print(list(df), df_dummy.shape)

X = df_dummy.drop(['sepal length (cm)'], axis = 1)"><pre class="notranslate"><span class="pl-k">import</span> <span class="pl-s1">pandas</span> <span class="pl-k">as</span> <span class="pl-s1">pd</span>
<span class="pl-k">import</span> <span class="pl-s1">numpy</span> <span class="pl-k">as</span> <span class="pl-s1">np</span>
<span class="pl-k">from</span> <span class="pl-s1">sklearn</span>.<span class="pl-s1">datasets</span> <span class="pl-k">import</span> <span class="pl-s1">load_iris</span>

<span class="pl-s1">iris</span> <span class="pl-c1">=</span> <span class="pl-en">load_iris</span>()
<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">iris</span>.<span class="pl-s1">data</span>
<span class="pl-s1">column_names</span> <span class="pl-c1">=</span> <span class="pl-s1">iris</span>.<span class="pl-s1">feature_names</span>

<span class="pl-s1">df</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-v">DataFrame</span>(<span class="pl-s1">iris</span>.<span class="pl-s1">data</span>, <span class="pl-s1">columns</span> <span class="pl-c1">=</span> <span class="pl-s1">column_names</span>)

<span class="pl-en">print</span>(<span class="pl-s1">df</span>.<span class="pl-en">head</span>(<span class="pl-s1">n</span><span class="pl-c1">=</span><span class="pl-c1">5</span>))

<span class="pl-s1">df_dummy</span> <span class="pl-c1">=</span> <span class="pl-s1">pd</span>.<span class="pl-en">get_dummies</span>(<span class="pl-s1">df</span>, <span class="pl-s1">sparse</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>, <span class="pl-s1">columns</span> <span class="pl-c1">=</span> [<span class="pl-s">'petal length (cm)'</span>] )

<span class="pl-en">print</span>(<span class="pl-en">list</span>(<span class="pl-s1">df</span>), <span class="pl-s1">df_dummy</span>.<span class="pl-s1">shape</span>)

<span class="pl-v">X</span> <span class="pl-c1">=</span> <span class="pl-s1">df_dummy</span>.<span class="pl-en">drop</span>([<span class="pl-s">'sepal length (cm)'</span>], <span class="pl-s1">axis</span> <span class="pl-c1">=</span> <span class="pl-c1">1</span>)</pre></div>
<h4 dir="auto">Problem description</h4>
<p dir="auto">I can't do anything with the dataframe based on col names when using Sparse = True - such as dropping a variable, which is replicated above. The full error print is under details.</p>
<h4 dir="auto">Expected Output</h4>
<p dir="auto">The expected output can be seen by removing <code class="notranslate">sparse = True</code> from <code class="notranslate">df_dummy = pd.get_dummies(df, sparse = True, columns = ['petal length (cm)'] )</code>, and then running the subsequent lines.</p>
<h4 dir="auto">Output of <code class="notranslate">pd.show_versions()</code></h4>
<details>
<h2 dir="auto">INSTALLED VERSIONS</h2>
<p dir="auto">commit: None</p>
<p dir="auto">pandas: 0.21.0<br>
pytest: 3.2.1<br>
pip: 9.0.1<br>
setuptools: 36.5.0.post20170921<br>
Cython: 0.26.1<br>
numpy: 1.12.1<br>
scipy: 1.0.0<br>
pyarrow: None<br>
xarray: None<br>
IPython: 6.1.0<br>
sphinx: 1.6.3<br>
patsy: 0.4.1<br>
dateutil: 2.6.1<br>
pytz: 2017.2<br>
blosc: None<br>
bottleneck: 1.2.1<br>
tables: 3.4.2<br>
numexpr: 2.6.2<br>
feather: None<br>
matplotlib: 2.1.0<br>
openpyxl: 2.4.8<br>
xlrd: 1.1.0<br>
xlwt: 1.2.0<br>
xlsxwriter: 1.0.2<br>
lxml: 4.1.0<br>
bs4: 4.6.0<br>
html5lib: 0.999999999<br>
sqlalchemy: 1.1.13<br>
pymysql: None<br>
psycopg2: None<br>
jinja2: 2.9.6<br>
s3fs: None<br>
fastparquet: None<br>
pandas_gbq: None<br>
pandas_datareader: None<br>
In [ ]:</p>
<h2 dir="auto">COMPLETE ERROR PRINT</h2>
<hr>
<p dir="auto">TypeError                                 Traceback (most recent call last)<br>
 in ()<br>
15 print(list(df), df_dummy.shape)<br>
16<br>
---&gt; 17 X = df_dummy.drop(['sepal length (cm)'], axis = 1)</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py in drop(self, labels, axis, index, columns, level, inplace, errors)<br>
2528         for axis, labels in axes.items():<br>
2529             if labels is not None:<br>
-&gt; 2530                 obj = obj._drop_axis(labels, axis, level=level, errors=errors)<br>
2531<br>
2532         if inplace:</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py in <em>drop_axis(self, labels, axis, level, errors)<br>
2561             else:<br>
2562                 new_axis = axis.drop(labels, errors=errors)<br>
-&gt; 2563             dropped = self.reindex(**{axis_name: new_axis})<br>
2564             try:<br>
2565                 dropped.axes[axis</em>].set_names(axis.names, inplace=True)</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/util/_decorators.py in wrapper(*args, **kwargs)<br>
125         <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wraps/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wraps">@wraps</a>(func)<br>
126         def wrapper(*args, **kwargs):<br>
--&gt; 127             return func(*args, **kwargs)<br>
128<br>
129         if not PY2:</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in reindex(self, *args, **kwargs)<br>
2933         kwargs.pop('axis', None)<br>
2934         kwargs.pop('labels', None)<br>
-&gt; 2935         return super(DataFrame, self).reindex(**kwargs)<br>
2936<br>
2937     <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/appender/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/appender">@appender</a>(_shared_docs['reindex_axis'] % _shared_doc_kwargs)</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/generic.py in reindex(self, *args, **kwargs)<br>
3021         # perform the reindex on the axes<br>
3022         return self._reindex_axes(axes, level, limit, tolerance, method,<br>
-&gt; 3023                                   fill_value, copy).<strong>finalize</strong>(self)<br>
3024<br>
3025     def _reindex_axes(self, axes, level, limit, tolerance, method, fill_value,</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/frame.py in _reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)<br>
2863         if columns is not None:<br>
2864             frame = frame._reindex_columns(columns, method, copy, level,<br>
-&gt; 2865                                            fill_value, limit, tolerance)<br>
2866<br>
2867         index = axes['index']</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/sparse/frame.py in _reindex_columns(self, columns, method, copy, level, fill_value, limit, takeable)<br>
701         return self._constructor(<br>
702             sdict, index=self.index, columns=columns,<br>
--&gt; 703             default_fill_value=self._default_fill_value).<strong>finalize</strong>(self)<br>
704<br>
705     def _reindex_with_indexers(self, reindexers, method=None, fill_value=None,</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/sparse/frame.py in <strong>init</strong>(self, data, index, columns, default_kind, default_fill_value, dtype, copy)<br>
89                                       fill_value=default_fill_value)<br>
90         elif isinstance(data, dict):<br>
---&gt; 91             mgr = self._init_dict(data, index, columns, dtype=dtype)<br>
92         elif isinstance(data, (np.ndarray, list)):<br>
93             mgr = self._init_matrix(data, index, columns, dtype=dtype)</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/sparse/frame.py in _init_dict(self, data, index, columns, dtype)<br>
168         sdict.update((c, nan_arr) for c in columns if c not in sdict)<br>
169<br>
--&gt; 170         return to_manager(sdict, columns, index)<br>
171<br>
172     def _init_matrix(self, data, index, columns, dtype=None):</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/sparse/frame.py in to_manager(sdf, columns, index)<br>
896<br>
897     return create_block_manager_from_arrays(<br>
--&gt; 898         [sdf[c] for c in columns], columns, axes)<br>
899<br>
900</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in create_block_manager_from_arrays(arrays, names, axes)<br>
4630<br>
4631     try:<br>
-&gt; 4632         blocks = form_blocks(arrays, names, axes)<br>
4633         mgr = BlockManager(blocks, axes)<br>
4634         mgr._consolidate_inplace()</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in form_blocks(arrays, names, axes)<br>
4726<br>
4727     if len(sparse_items) &gt; 0:<br>
-&gt; 4728         sparse_blocks = _sparse_blockify(sparse_items)<br>
4729         blocks.extend(sparse_blocks)<br>
4730</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in _sparse_blockify(tuples, dtype)<br>
4788         array = _maybe_to_sparse(array)<br>
4789         block = make_block(array, klass=SparseBlock, fastpath=True,<br>
-&gt; 4790                            placement=[i])<br>
4791         new_blocks.append(block)<br>
4792</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in make_block(values, placement, klass, ndim, dtype, fastpath)<br>
2950                      placement=placement, dtype=dtype)<br>
2951<br>
-&gt; 2952     return klass(values, ndim=ndim, fastpath=fastpath, placement=placement)<br>
2953<br>
2954 # TODO: flexible with index=None and/or items=None</p>
<p dir="auto">/anaconda3/lib/python3.6/site-packages/pandas/core/internals.py in <strong>init</strong>(self, values, placement, ndim, fastpath, **kwargs)<br>
1701<br>
1702         if not isinstance(values, self._holder):<br>
-&gt; 1703             raise TypeError("values must be {0}".format(self._holder.<strong>name</strong>))<br>
1704<br>
1705         self.values = values</p>
</details>
<p dir="auto">I got the error<br>
<b> AttributeError: 'IntBlock' object has no attribute 'sp_index' </b></p>
<p dir="auto">when converting a SparseDataFrame to Scipy csr_matrix using the following code:</p>
<p dir="auto"><i>dfTotalCat = get_dummies(dfTotalCat, sparse=True)</i></p><i>
</i><p dir="auto"><i>XTotalCat = csr_matrix(dfTotalCat.to_coo()) </i></p>
<p dir="auto">The SparseDataFrame is obtained from get_dummies.</p>
<p dir="auto">Following is the exact error trace:</p>
<p dir="auto">Traceback (most recent call last):<br>
File "pandaSrc.py", line 76, in <br>
XTotalCat = csr_matrix(dfTotalCat.to_coo())<br>
File "C:\Users\nagabhushan.s\AppData\Local\Programs\Python\Python36\lib\site-p<br>
ackages\pandas\core\sparse\frame.py", line 255, in to_coo<br>
row = s.sp_index.to_int_index().indices<br>
File "C:\Users\nagabhushan.s\AppData\Local\Programs\Python\Python36\lib\site-p<br>
ackages\pandas\core\generic.py", line 3614, in <strong>getattr</strong><br>
return object.<strong>getattribute</strong>(self, name)<br>
File "C:\Users\nagabhushan.s\AppData\Local\Programs\Python\Python36\lib\site-p<br>
ackages\pandas\core\sparse\series.py", line 245, in sp_index<br>
return self.block.sp_index<br>
AttributeError: 'IntBlock' object has no attribute 'sp_index'</p>