Why does this Python logical indexing take up so much memory
Question: I have a 1-D time series that is 1e8 long (100,000,000 elements).
[Here](https://www.dropbox.com/s/3eb6ywjzoo468rd/ch008.ddt) is a link to the
data I am using on Dropbox. (The file is 382 MB.)

**Update**

Based on memory_profiling, the error occurs in the line

    
    
    data[absolute(data-dc)< m*std(data)]=dc.
    

More specifically, the operation `absolute(data-dc)` eats up all the memory.
`Data` is as described above and `dc` is a constant. Perhaps this is a subtle
syntax error?

I want to remove outliers and artifacts from it and replace those values with
the median. I attempt to do that with the following function.

    
    
     from numpy import *
    
     from sys import argv
    
     from scipy.io import savemat
     from scipy.stats import scoreatpercentile
    
     def reject_outliers(data,dc,m=3):
          data[data==0] = dc
          data[bp.absolute(data-dc) < m*np.std(data)] = dc
          return data
    
     def butter_bandpass(lowcut,highcut,fs,order=8):
        nyq = 0.5*fs
        low = lowcut/nyq
        high = highcut/nyq
    
        b,a= butter(order, [low, high], btype='band')
        return b,a
    
     def butter_bandpass_filter(data,lowcut,highcut,fs,order=8):
        b,a = butter_bandpass(lowcut,highcut,fs,order=order)
        return lfilter(b,a,data) 
    
     OFFSET = 432
     filename = argv[1]
     outname = argv[2]  
    
     print 'Opening '+ filename
     with open(filename,'rb') as stream:
          stream.seek(OFFSET)
          data=fromfile(stream,dtype='int16')
     print 'Removing Artifacts, accounting for zero-filling'
     dc = median(data)
     data = reject_outliers(data,dc)
    
     threshold = scoreatpercentile(absolute(data),85)   
     print 'Filtering and Detrending'
     data = butter_bandpass_filter(data,300,7000,20000)
     savemat(outname+'.mat',mdict={'data':data})
    

Calling this on one file eats up 4 GB of RAM and 3 GB of virtual memory. I'm
sure it's the second line of this function because I stepped through the
script I wrote and it always hangs on this part. I can even see (in Finder on
OS X) the available hard drive space plummeting by the second.

The time series is not long enough to explain it. What is wrong with the
second line of `reject-outliers`?


Answer: I just generated 100,000,000 random floats and did the same indexing you
describe. Memory usage was well under a gigabyte throughout. What else is your
code doing that you are not telling us about? Try running your code through
the excellent [memory_profiler](http://pypi.python.org/pypi/memory_profiler).

* * *

Edit: Added code and output of memory_profiler:

    
    
    from numpy.random import uniform
    import numpy
    
    @profile
    def go(m=3):
        data = uniform(size=100000000)
        dc = numpy.median(data)
        data[numpy.absolute(data-dc) < m*numpy.std(data)] = dc
        return data
    
    if __name__ == '__main__':
        go()
    

Output:

    
    
    Filename: example.py
    
    Line #    Mem usage    Increment   Line Contents
    ================================================
         3                             @profile
         4     15.89 MB      0.00 MB   def go(m=3):
         5    778.84 MB    762.95 MB    data = uniform(size=100000000)
         6    778.91 MB      0.06 MB    dc = numpy.median(data)
         7    874.34 MB     95.44 MB    data[numpy.absolute(data-dc) < m*numpy.std(data)] = dc
         8    874.34 MB      0.00 MB    return data
    

As you can see, 100M floats do not use up that much memory.

