title: Performance of Pandas apply vs np.vectorize to create new column from existing columns
body: <p>I am using Pandas dataframes and want to create a new column as a function of existing columns. I have not seen a good discussion of the speed difference between <code>df.apply()</code> and <code>np.vectorize()</code>, so I thought I would ask here. </p>
  
  <p>The Pandas <code>apply()</code> function is slow. From what I measured (shown below in some experiments), using <code>np.vectorize()</code> is 25x faster (or more) than using the DataFrame function <code>apply()</code> , at least on my 2016 MacBook Pro. <strong>Is this an expected result, and why?</strong> </p>
  
  <p>For example, suppose I have the following dataframe with <code>N</code> rows:</p>
  
  <pre><code>N = 10
  A_list = np.random.randint(1, 100, N)
  B_list = np.random.randint(1, 100, N)
  df = pd.DataFrame({'A': A_list, 'B': B_list})
  df.head()
  #     A   B
  # 0  78  50
  # 1  23  91
  # 2  55  62
  # 3  82  64
  # 4  99  80
  </code></pre>
  
  <p>Suppose further that I want to create a new column as a function of the two columns <code>A</code> and <code>B</code>. In the example below, I'll use a simple function <code>divide()</code>. To apply the function, I can use either <code>df.apply()</code> or <code>np.vectorize()</code>:</p>
  
  <pre><code>def divide(a, b):
      if b == 0:
          return 0.0
      return float(a)/b
  
  df['result'] = df.apply(lambda row: divide(row['A'], row['B']), axis=1)
  
  df['result2'] = np.vectorize(divide)(df['A'], df['B'])
  
  df.head()
  #     A   B    result   result2
  # 0  78  50  1.560000  1.560000
  # 1  23  91  0.252747  0.252747
  # 2  55  62  0.887097  0.887097
  # 3  82  64  1.281250  1.281250
  # 4  99  80  1.237500  1.237500
  </code></pre>
  
  <p>If I increase <code>N</code> to real-world sizes like 1 million or more, then I observe that <code>np.vectorize()</code> is 25x faster or more than <code>df.apply()</code>.</p>
  
  <p>Below is some complete benchmarking code:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import time
  
  def divide(a, b):
      if b == 0:
          return 0.0
      return float(a)/b
  
  for N in [1000, 10000, 100000, 1000000, 10000000]:    
  
      print ''
      A_list = np.random.randint(1, 100, N)
      B_list = np.random.randint(1, 100, N)
      df = pd.DataFrame({'A': A_list, 'B': B_list})
  
      start_epoch_sec = int(time.time())
      df['result'] = df.apply(lambda row: divide(row['A'], row['B']), axis=1)
      end_epoch_sec = int(time.time())
      result_apply = end_epoch_sec - start_epoch_sec
  
      start_epoch_sec = int(time.time())
      df['result2'] = np.vectorize(divide)(df['A'], df['B'])
      end_epoch_sec = int(time.time())
      result_vectorize = end_epoch_sec - start_epoch_sec
  
  
      print 'N=%d, df.apply: %d sec, np.vectorize: %d sec' % \
              (N, result_apply, result_vectorize)
  
      # Make sure results from df.apply and np.vectorize match.
      assert(df['result'].equals(df['result2']))
  </code></pre>
  
  <p>The results are shown below:</p>
  
  <pre><code>N=1000, df.apply: 0 sec, np.vectorize: 0 sec
  
  N=10000, df.apply: 1 sec, np.vectorize: 0 sec
  
  N=100000, df.apply: 2 sec, np.vectorize: 0 sec
  
  N=1000000, df.apply: 24 sec, np.vectorize: 1 sec
  
  N=10000000, df.apply: 262 sec, np.vectorize: 4 sec
  </code></pre>
  
  <p>If <code>np.vectorize()</code> is in general always faster than <code>df.apply()</code>, then why is <code>np.vectorize()</code> not mentioned more? I only ever see StackOverflow posts related to <code>df.apply()</code>, such as:</p>
  
  <p><a href="https://stackoverflow.com/questions/26886653/pandas-create-new-column-based-on-values-from-other-columns">pandas create new column based on values from other columns</a></p>
  
  <p><a href="https://stackoverflow.com/questions/16353729/how-do-i-use-pandas-apply-function-to-multiple-columns">How do I use Pandas &#39;apply&#39; function to multiple columns?</a></p>
  
  <p><a href="https://stackoverflow.com/questions/13331698/how-to-apply-a-function-to-two-columns-of-pandas-dataframe">How to apply a function to two columns of Pandas dataframe</a></p>