How to calculate correlation on large number of records?
There is nothing special about correlation of large data sets. All you need to do is some simple aggregation.
If you want to improve your numerical precision (remember that floating point math is lossy) you can use Kahan summation and similar techniques, in particular for values close to 0.
But maybe your data justt doesn't have strong correlation?
Try visualizing a sample!
I am trying to calculate correlation amongst three columns in a dataset. The dataset is relatively large (4 GB in size). When I calculate correlation among the columns of interest, I get small values like 0.0024, -0.0067 etc. I am not sure this result makes any sense or not. Should I sample the data and then try calculating correlation?
Any thoughts/experience on this topic would be appreciated.