title: Fastet way to shuffle a dataframe to calculate P value in pandas
body: <p>I have the following data frame (actual length of dataframe is much larger) :</p>
  
  <pre><code>spend_df:
  
  var   spend
   1     120
   2     200
   1     111
   1     143
   1     230
   2     180
   2     190
   2     94
   2     100
   1     278
  </code></pre>
  
  <p>here ´var´ column tell which group of the test row belongs, i need to shuffle the df 1000 times to assign each rows randomly to different sides of test randomly then groupby 'var' and calculate the difference fro every iteration. At  the end calculate how many time the difference between groups sum was larger that actual difference:</p>
  
  <p>i take the following approach:</p>
  
  <pre><code>def simulate(df):
      simul_df = df.copy()
  
      shuffled_var = simul_df.sample(frac=1)
      shuffled_var.index = simul_df.index # replace shuffled series' index with the real one
  
      simul_df['var'] = shuffled_var
      simul_grouped = simul_df.groupby('var').sum()['spend']
      simul_difference = simul_grouped.iloc[1] - simul_grouped.iloc[0]
      return simul_difference
  
  
  simulate(spend_df)
  </code></pre>
  
  <p>and this is how i calculate how many times simulated difference was larger that actual difference:</p>
  
  <pre><code>simulations = np.array([simulate(spend_df) for i in range(1000)])
  (simulations &gt; real_difference).mean()
  </code></pre>
  
  <p>here real_difference is the difference between sums of two groups spend(group by 'var'):</p>
  
  <pre><code>grouped=df.groupby('var').sum().reset_index()
  real_diff=grouped.iloc[1] - grouped.iloc[0]
  </code></pre>
  
  <p>in this case it is : 882-764= 118</p>
  
  <p>whenever i shuffle the dataframe i need to calculate the difference for new groups like above then i need to find out how many times this simulated differences where more than real_diff</p>
  
  <p>This solution seems a bit hacky to me is there any faster or more organized way to this simulation task for calculating P value? without going deep to statistical methods.</p>