title: Problem getting correlation matrix from large csv file with Python / Pandas
body: <p>First of all: I'm a beginner with python and data analytics BUT I'm confident I understand the concepts enough so you don't have to over-simplify your answers.</p>
  
  <p>My challenge is that I have to analyze huge chunks of machine data (timeseries over two years; 24 structure-identical csv-files, each with 170 columns, ~ 2.5 million rows, ~ 2.6gb size).</p>
  
  <p>This data has to be analyzed in regard to correlations. The initally desired output is an 170x170 correlation matrix. Further analysis (lag, an asymetrical correlation matrix Input x Output) shall be postponed to the next step and is not primarily to be considered for your answer.</p>
  
  <p>I've been able to read one of the files into a dataframe (using the IPython-Console of Spyder; for the cost of a lot of my 16gb memory).</p>
  
  <pre><code>import pandas as pd
  
  df = pd.read_csv(r"C:\MyFilePath\...\TestData.csv", sep=';', encoding='iso-8859-1')
  
  In[]: len(df.columns)
  Out[]: 170
  
  In[]: len(df)
  Out[]: 2678401
  </code></pre>
  
  <p>But from there on I'm stuck...</p>
  
  <p>The pandas.DataFrame.corr method does not work properly and returns (if it works) only a 10 x 10 Matrix with a lot of NaN values (which are in my understanding just a display for a non existent pearson correlation (close to or equal to zero)).</p>
  
  <p>I have found several descriptions how to load data into my dataframe, which exceeds my RAM. Yet I was not able to fully understand the concept of loading chunks, especially in combination with my time series.</p>
  
  <p>I would really appreciate, if you could provide me with a proper hint or snippet, so that I can solve this problem.</p>
  
  <p>Ideally the result is, that I can run over all the csv-files and get the desired correlation matrix for all parameters.</p>
  
  <p><strong>Note:</strong>
  I am not bound to pandas. If you suggest another library which serves this problem in a better way, I'm happy to hear your solution.
  But due to the security policy of my company I am obliged to not download any additional software (or to be more precise: it is complicated...)
  The only other option I have at hand is MATLAB R2018.a</p>