title: Pandas Dataframe performance vs list performance
body: <p>I'm comparing two dataframes to determine if rows in df1 begin any row in df2. df1 is on the order of a thousand entries, df2 is in the millions.</p>
  
  <p>This does the job but is rather slow. </p>
  
  <pre><code>df1['name'].map(lambda x: any(df2['name'].str.startswith(x)))
  </code></pre>
  
  <p>When run on a subset of df1 (10 items), this is the result:</p>
  
  <pre><code>35243     True
  39980    False
  40641    False
  45974    False
  53788    False
  59895     True
  61856    False
  81083     True
  83054     True
  87717    False
  Name: name, dtype: bool
  Time: 57.8873581886 secs
  </code></pre>
  
  <p>When I converted df2 to a list, it runs much faster:</p>
  
  <pre><code>df2_list = df2['name'].tolist()
  
  df1['name'].map(lambda x: any(item.startswith(x + ' ') for item in df2_list))
  
  35243     True
  39980    False
  40641    False
  45974    False
  53788    False
  59895     True
  61856    False
  81083     True
  83054     True
  87717    False
  Name: name, dtype: bool
  Time: 33.0746209621 secs
  </code></pre>
  
  <p>Why is it quicker to iterate through a list than a Series?</p>