title: Make piece of code efficient for big data
body: <p>I have the following code:</p>
  <pre><code>new_df = pd.DataFrame(columns=df.columns)
  for i in list:
      temp = df[df[&quot;customer id&quot;]==i]
      new_df = new_df.append(temp)
  </code></pre>
  <p>where <code>list</code> is a list of customer id's for the customers that meet a criteria chosen before. I use the <code>temp</code> dataframe because there are multiple rows for the same customer.</p>
  <p>I consider that I know how to code, but I have never learnt how to code for big data efficiency. In this case, the <code>df</code> has around 3 million rows and <code>list</code> contains around 100,000 items. This code ran for more than 24h and it was still not done, so I need to ask, am I doing something terribly wrong? Is there a way to make this code more efficient?</p>