title: Python MemoryError w/ Plenty of RAM (64-bit)
body: <p>I am currently working with a large data file of dimension 15,883,912x105.  The storage size of this data set is just a shade under 10 GBs.  It is a fairly large set, but I am working on a box with 24 GB of RAM.  </p>
  
  <p>I am working on a 64-bit Windows machine (<a href="http://www.dell.com/us/business/p/precision-t3500/pd" rel="nofollow">Dell Precision T3500</a> to be exact).  Further, this is all being done with a 64-bit Python install (v2.7.6 from <a href="http://docs.continuum.io/anaconda/" rel="nofollow">Anaconda</a> 2.0.0).  </p>
  
  <p>The file actually reads in just fine, but when I subset to a two variables with under 500,000 rows, I am  encountering a <code>MemoryError</code>.  Specifically, I am trying to calculate the Gini coefficient (trapezoidal approximation) from a vector of data with the following code:</p>
  
  <pre><code>#Isolate test set
  gini_test=gini.ix[1979]['adj_market_income']
  
  #Define Gini calculator
  def gini_trap(df,var,wt):
      '''Function takes incomes and weights, and returns the *weighted* Gini Coefficient.'''
      #Sort the df by income
      df.sort(columns=var,inplace=True)
      #Generate non-negative version
      inc=df[var].apply(lambda x: max(x,0.))
      #Calculate total weight
      tot_wt=df[wt].sum()
      #Calculate total weighted income
      tot_inc=(df[wt]*inc).sum()
      #Initialize share and cumulative variables
      wt_share=0
      inc_share=0
      cum_wt=0
      cum_inc=0
      #Initialize Gini value
      gini=0
      #For each record...
      for i,val in enumerate(inc):
          #...calculate the current records share of income and population...
          wt_share=df[wt].iloc[i]/tot_wt
          inc_share=(df[wt].iloc[i]*val)/tot_inc
          #...and augment the cumulative measures for income and population/update the gini value...
          cum_wt+=wt_share
          gini+=wt_share*(cum_inc+cum_inc+inc_share)
          cum_inc+=inc_share
      print cum_inc,cum_wt
      return 1-gini
  
  print gini_trap(gini.ix[1979],'adj_market_income','wt')
  </code></pre>
  
  <p>There are a number of <code>MemoryError</code> issues here on Stack, but the common theme I have seen is to make sure one uses a 64-bit machine and has plenty of RAM (before getting into chunking and whatnot).  Any thoughts on what is happening here?</p>