title: identifying "similar" datasets by their curves with pandas
body: <p>Consider the following </p>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.DataFrame({'y':np.sin(np.linspace(1,100,10000)),'x':pandas.date_range('2019-01-01','2020-01-01',periods=10000})
  df.index = df['x']
  </code></pre>
  
  <p>now in aggregate form 1</p>
  
  <pre><code>#this cut might not be exactly right ... but i dont have the actual code in front of me at the minute... the idea should be clear however
  df['y'].groupby(pd.cut(df.index,1400)).agg('mean') # cut into 1400 equalish bins
  </code></pre>
  
  <p>and in version 2</p>
  
  <pre><code>df['y'].resample("3H").agg('mean') # cut into 3 hour bins
  </code></pre>
  
  <p>these two are really the same dataset, and the curves should be the same ... but whats a good method to validate that the curves are the same(ish) without plotting them and visually examining them (current method)</p>
  
  <p>you can visually identify the sameness in the following graph (the blue line is one aggregation the yellow is another)</p>
  
  <p><a href="https://i.stack.imgur.com/RL6uW.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RL6uW.png" alt="enter image description here"></a></p>
  
  <p>the motivation here is that the developer may mistakenly use the wrong aggregation (sum instead of mean, or maybe the wrong precision rounding)</p>
  
  <p>and being able to validate that the outputs are roughly equivalent programmatically would be a huge boon</p>
  
  <p>(see also <a href="https://repl.it/@JoranBeasley/EvilAwfulSimulation" rel="nofollow noreferrer">https://repl.it/@JoranBeasley/EvilAwfulSimulation</a> for a runnable example)</p>
  
  <p>one idea I had is to try and segment up the chart into N pieces and use np.trapz on each subset of the x axis and compare the values </p>
  
  <p>another idea might be to check the normal distribution of each subsection of the x axis... so far I have not had much success with either method</p>