Calculating correlation of slightly out of sync data I am trying to do some analysis on some data that comes from special glasses that track a few things including pupil size and gaze velocity. I  would like to calculate the correlation between two glasses on two different people. At the moment I cannot use df.correlate() because the timestamps are not identical and therefore the data looks something like this:

index | ts | r_person | l_person
-----------------------
0     | 23 | 3.0      | NAN
1     | 25 | NAN      | 3.2
2     | 28 | 3.1      | NAN
3     | 32 | 3.0      | NAN


I was wondering if there was still any way to calculate a correlation directly.

At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.

This would be less trivial than it seems because even at the start it wasn't always one data point R one data point L and after cleaning the data it has become less so. In other words multiple NAN values might appear in the same column as you can see in the example. I can still deal with that by just spreading out the average. My second technique was going to try and merge the values together if they were close enough. Bearing in mind the data was collected at 50hz. 

My question is whether anyone has a quicker or better way of aligning the data without losing it or changing it too much?
At the moment I was thinking of possible filling the NAN values with the averages of the data points above and below. For example row 2 column r_person would become $3.05$.


The problem here is that the missing data point isn't necessarily associated with a timestamp that is the average of the timestamps above and below. If it were, the procedure you suggested would be equivalent to a linear interpolation, i.e. drawing a line between the points before and after the missing timestamp, and using that line to construct a local prediction for the missing observation. Which is what you should actually do instead. 

For example in the case above, the line for the point your trying to interpolate is $\text{r_person}(ts) = 3.0 +\frac{.1}{5}(ts-23)$, so $\text{r_person}(25) = 3.04$ (I think... doing this in my head right now).