title: Is there a way to speed up the following pandas for loop?
body: <p>My <code>data</code> frame contains 10,000,000 rows! After group by, ~ 9,000,000 sub-frames  remain to loop through.</p>
  
  <p>The code is:</p>
  
  <pre><code>data = read.csv('big.csv')
  for id, new_df in data.groupby(level=0): # look at mini df and do some analysis
      # some code for each of the small data frames
  </code></pre>
  
  <p>This is super inefficient, and the code has been running for 10+ hours now. </p>
  
  <p>Is there a way to speed it up?</p>
  
  <p><strong>Full Code:</strong></p>
  
  <pre><code>d = pd.DataFrame() # new df to populate
  print 'Start of the loop'
  for id, new_df in data.groupby(level=0):
      c = [new_df.iloc[i:] for i in range(len(new_df.index))]
      x = pd.concat(c, keys=new_df.index).reset_index(level=(2,3), drop=True).reset_index()
      x = x.set_index(['level_0','level_1', x.groupby(['level_0','level_1']).cumcount()])
      d = pd.concat([d, x])
  </code></pre>
  
  <p><strong>To get the data:</strong></p>
  
  <pre><code>data = pd.read_csv('https://raw.githubusercontent.com/skiler07/data/master/so_data.csv', index_col=0).set_index(['id','date'])
  </code></pre>
  
  <p><strong>Note:</strong></p>
  
  <p>Most of id's will only have 1 date. This indicates only 1 visit. For id's with more visits, I would like to structure them in a 3d format e.g. store all of their visits in the 2nd dimension out of 3. The output is <em>(id, visits, features)</em></p>