title: Why are simple operations on pandas.DataFrames so slow compared to the same operations on numpy.ndarrays?
body: <p>Why are operations on <code>pandas.DataFrame</code>s so slow?! Look at the following examples. </p>
  
  <h3>Measurement:</h3>
  
  <ul>
  <li>Create a <code>numpy.ndarray</code> populated with random floating point numbers</li>
  <li>Create a <code>pandas.DataFrame</code> populated with the same numpy array</li>
  </ul>
  
  <p>The I measure the time of the following operations</p>
  
  <ol>
  <li><p>For the <code>numpy.ndarray</code></p>
  
  <ul>
  <li>Take the sum along the 0-axis</li>
  <li>Take the sum along the 1-axis</li>
  </ul></li>
  <li><p>For the <code>pandas.DataFrame</code></p>
  
  <ul>
  <li>Take the sum along the 0-axis</li>
  <li>Take the sum along the 1-axis</li>
  </ul></li>
  <li><p>For the <code>pandas.DataFrame.values -&gt; np.ndarray</code></p>
  
  <ul>
  <li>Take the sum along the 0-axis</li>
  <li>Take the sum along the 1-axis</li>
  </ul></li>
  </ol>
  
  <h3>Observations</h3>
  
  <ul>
  <li>Summing over <code>numpy.ndarrays' is much faster then operating on</code>pandas.DataFrames`. </li>
  <li>This is even true, if the <code>pd.DataFrame</code> does not contain only floating point numbers and has nothing special attached (MultiIndex or whatever).</li>
  <li>Operations on <code>numpy.ndarray</code> are about 7 to 10 times faster.  </li>
  </ul>
  
  <h3>Questions</h3>
  
  <ol>
  <li>Why does this happen?</li>
  <li>How can this be optimized?</li>
  <li>Is <code>pandas</code> not able to call or pass through <code>numpy</code>s' operations?</li>
  </ol>
  
  <pre class="lang-py prettyprint-override"><code>import numpy as np
  import pandas as pd
  
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>n = 50000
  m = 5000
  array = np.random.uniform(0, 1, (n, m))
  dataframe = pd.DataFrame(array)
  </code></pre>
  
  <h3>Numpy</h3>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  array.sum(axis=0)
  </code></pre>
  
  <pre><code>206 ms ± 3.78 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  array.sum(axis=1)
  </code></pre>
  
  <pre><code>233 ms ± 33.9 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
  </code></pre>
  
  <h3>Pandas</h3>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  dataframe.sum(axis=0)
  </code></pre>
  
  <pre><code>1.65 s ± 14.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  dataframe.sum(axis=1)
  </code></pre>
  
  <pre><code>1.74 s ± 15.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <h3>Pandas without Pandas</h3>
  
  <p>Let's operate on the values alone ...</p>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  dataframe.values.sum(axis=0)
  </code></pre>
  
  <pre><code>206 ms ± 7.13 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  </code></pre>
  
  <pre class="lang-py prettyprint-override"><code>%%timeit
  dataframe.values.sum(axis=1)
  </code></pre>
  
  <pre><code>181 ms ± 1.66 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
  </code></pre>