title: How to manage large a dataset for regression?
body: <p>my question has to do with a very large dataset I'm running a regression on in Python. I have categorical data (gender, industry, region, salary groupings, etc.) that I would like to run a regression on with statsmodels. The whole dataframe comes out to be about 83 columns in width after using pd.getdummies() on roughly 5 million lines.</p>
  
  <p>Code:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  import statsmodels.api as sm
  import statsmodels.formula.api as smf
  from datetime import datetime as dt
  
  
  #Start time
  print('Start Time: ', dt.now())
  
  #Variables
  groups = ['sex', 'central_age', 'group_size', 'industry', 'region', 'salary']
  base_cases = ['sex_Male', 'central_age_47.0', 'group_size_F. 100-249', 'salary_A. &lt; 25',
                'industry_H. Manufacturing - heavy, steel etc.', 'region_C. Division 3: East North Central']
  aggregates = ['death_amount_exposed', 'death_claim_amount']
  
  
  #Read/ format data to transform data into categorical variables
  df = pd.read_pickle(r'./Life_Mortality_Data.pkl')
  df = df[df['death_amount_exposed']!=0]
  df['central_age'] = df['central_age'].apply(str)
  final = pd.get_dummies(df[groups]).join(df[aggregates]).astype(float)
  final.drop(base_cases, axis=1, inplace=True)
  
  
  #Prepare sting of variables to regress on in next step
  var_columns = list(final.columns)
  for i in aggregates:
      var_columns.remove(i)
  variables = '+'.join('Q("' + i + '")' for i in var_columns)
  
  
  #Training and testing with Poisson model
  print('Regression Time: ', dt.now(), '\n')
  res1 = smf.glm(formula='death_claim_amount ~'+variables, data=final, offset=np.log(final['death_amount_exposed']), family=sm.families.Poisson(sm.families.links.log())).fit()
  
  
  #Print stats summary, base cases, and multiplicative factors
  print(res1.summary())
  print('Base Cases:')
  for case in base_cases:
      print(case)
  print('\nParameters:\n', np.exp(res1.params))
  
  
  #This takes the result of a statsmodel results table and transforms it into a dataframe
  def results_summary_to_dataframe(results):
      pvals = results.pvalues
      coeff = results.params
      std_err = results.bse
      conf_lower = results.conf_int()[0]
      conf_higher = results.conf_int()[1]
  
      results_df = pd.DataFrame({"pvals":pvals,
                                 "coeff":coeff,
                                 "std_error":std_err,
                                 "conf_lower":conf_lower,
                                 "conf_higher":conf_higher
                                  })
  
      #Reordering columns
      results_df = results_df[["coeff","std_error","pvals","conf_lower","conf_higher"]]
      return results_df
  
  
  #Write data to excel
  results_summary_to_dataframe(res1).to_excel(r'./All_Regression_Amounts_v1.xlsx')
  
  
  #End time
  print('\nEnd Time: ', dt.now())
  </code></pre>
  
  <p>The problem I'm having is that I run out of memory at the point where the statsmodels regression is run. I am using the 64-bit version of Python on Windows and have 32 GB of memory which I thought would be more than enough to handle this kind of computation but am not sure if I'm not using all available memory or if something may be wrong with my code. I'm very new to this kind of analysis and handling this much data. I'd really appreciate any help on what I can do to resolve this error</p>