Finding Pearson correlation for millions of records
You can use numpy or pandas (vectorised version instead of iterating over all records).
I am doing an NLP task, where I have a distribution of word occurrence in the texts grouped by day. I want to find Pearson correlation between a given word and all other words in the database. Is there possibly some algorithmic solution rather than just manually iterating over all other words in database and calculating correlation. May be something to do with graphs?