title: Aligning two data sets in Python
body: <p>I want to develop some python code to align datasets obtained by different instruments recording the same event.</p>
  
  <p>As an example, say I have two sets of measurements:</p>
  
  <pre><code>import numpy as np
  import pandas as pd
  import matplotlib.pyplot as plt
  
  # Define some data
  data1 = pd.DataFrame({'TIME':[1.1, 2.4, 3.2, 4.1, 5.3],\
                        'VALUE':[10.3, 10.5, 11.0, 10.9, 10.7],\
                'ERROR':[0.2, 0.1, 0.4, 0.3, 0.2]})
  
  data2 = pd.DataFrame({'TIME':[0.9, 2.1, 2.9, 4.2],\
                        'VALUE':[18.4, 18.7, 18.9, 18.8],\
                'ERROR':[0.3, 0.2, 0.5, 0.4]})
  
  # Plot the data      
  plt.errorbar(data1.TIME, data1.VALUE, yerr=data1.ERROR, fmt='ro')
  plt.errorbar(data2.TIME, data2.VALUE, yerr=data2.ERROR, fmt='bo')
  plt.show()
  </code></pre>
  
  <p>The result is plotted here:
  <img src="https://i.stack.imgur.com/jnoFd.png" alt="enter image description here"></p>
  
  <p>What I would like to do now is to align the second dataset (data2) to the first one (data1). i.e. to get this:
  <img src="https://i.stack.imgur.com/8ulwP.png" alt="enter image description here"></p>
  
  <p>The second dataset must be shifted to match the first one by subtracting a constant (to be determined) from all its values. All I know is that the datasets are correlated since the two instruments are measuring the same event but with different sampling rates.</p>
  
  <p>At this stage I do not want to make any assumptions about what function best describes the data (fitting will be done after alignment).  </p>
  
  <p>I am cautious about using means to perform shifts since it may produce bad results, depending on how the data is sampled. I was considering taking each data2[TIME_i] and working out the shortest distance to data1[~TIME_i]. Then minimizing the sum of those. But I am not sure that would work well either.</p>
  
  <p>Does anyone have any suggestions on a good method to use? I looked at <a href="http://mlpy.sourceforge.net/" rel="nofollow noreferrer">mlpy</a> but it seems to only work on 1D arrays. </p>
  
  <p>Thanks.</p>