pcolormesh use of memory
Question: I've been struggling with this for a while. I have a set of images, I perform
some math on the X, Y coordinates of these images and then plot the new images
using pcolormesh. All the calculations I've already done, all I do is load the
new X's and new Y's and use the colors from the image in pcolormesh.

The images are 2048x2448 pixels (say approx 5mp), first image goes pretty fast
and every image after that the script gets slower and eats more memory. I have
tried some garbage collection but it doesn't work.

My script:

    
    
    import numpy as np
    from PIL import Image
    import cPickle as pickle
    import matplotlib.pyplot as plt
    import os
    
    # TRY forced garbage collection!
    import gc
    
    def rectify(cam_files, cam_rec_files, rec_file_location):
        ''' cam_files is a dictionary that contains the filenames with the camera-names as index
                example: {'KDXX04C' : C:\Users\Yorian\Desktop\TU\Stage Shore\python_files\Rectify, metadata and upload\v3\archive\KDXXXXX\original\snap\1381383000\{filename}.jpg }
    
            cam_rec_files_dir contains a dictionary, cameraName : fileLocation
                example: {'KDXX04C' : C:\Users\Yorian\Desktop\TU\Stage Shore\python_files\Rectify, metadata and upload\v3\camdata\KDXXXXX\KDXX04C.pkl }
    
            rec_file_location is a string that shows where the new rectification needs to be saved '''
        fig, ax = plt.subplots(1, 1, figsize=(60,90))
    
        for camname in cam_files:
            img = Image.open(cam_files[camname])
            img = np.asarray(img, dtype=np.uint8)
            height, width, channels = img.shape
    
            # load plot data
            fh = open(cam_rec_files[camname], 'rb')
            X = pickle.load(fh)
            Y = pickle.load(fh)
    
            masks = [X<=0, X>1500, Y>4000, Y<-4000]
            total_mask = masks[0] | masks[1] | masks[2] | masks[3]
            first_false = np.argwhere(total_mask == 0)
            start = int(first_false[0]/width)
    
            rgb = img.reshape((-1,3))/255.0
            rgba = np.concatenate((rgb, np.ones((rgb.shape[0],1), dtype=np.uint8)), axis=1)
            rgba[total_mask,3] = 0
            rgba = rgba.reshape((height,width,4))[:,:-1,:]
            rgba = rgba.reshape((-1,4))
    
            plotimg = ax.pcolormesh(X.reshape((height, width))[start:,:], Y.reshape((height, width))[start:,:], img.mean(-1)[start:,:], cmap='Greys') # img.mean(-1)
    
            plotimg.set_array(None)
            plotimg.set_edgecolor('none')
            plotimg.set_facecolor(rgba[(start*(width-1)):,:])
    
            fh.close()
    
        plt.savefig(rec_file_location)
        gc.collect()
    

It works until six images, but when I try eight for example I have
insufficient memory (I use python 64bit and have 12gb of memory on my computer
which I imagined to be enough).

Does anybody have an idea on how to solve this problem?


Answer: In a nutshell, call `plt.close(fig)` when you're through with it if you're
using the `pyplot` interface and want to generate lots of figures without
displaying them.

Each time you call your `rectify` function, you're making a new (very large!!)
figure and then _keeping it in memory_. `pyplot` keeps a reference to the
figure so it can be displayed when you call `plt.show()`. Either call
`plt.close(fig)` or create the figures without using the `pyplot` state
machine. (`fig.clf()` will also work, but will keep references to a blank
figures around.)

Also, given that you're reading in image files, your values are presumably on
a regular x and y grid. If so, use `imshow` instead of `pcolormesh`. It's much
faster and more memory efficient.

* * *

As an example of the first issue, your `rectify` function basically does
something like this, and you're presumably calling it repeatedly (as the loop
below does):

    
    
    import numpy as np
    import matplotlib.pyplot as plt
    
    def rectify():
        fig, ax = plt.subplots()
        ax.pcolormesh(np.random.random((10,10)))
        fig.savefig('blah.png')
    
    for _ in range(10):
        rectify()
    
    plt.show()
    

Notice that we'll get 10 figures popping up. `pyplot` holds on to a reference
to the figure so that it can be displayed with `show`.

If you want to remove the figure from the `pyplot` state machine, call
`plt.close(fig)`.

For example, no figures will be displayed if you do this: (each figure will be
garbage collected as you'd expect after you remove the figure from pyplot's
figure manager by calling `plt.close(fig)`.)

    
    
    import numpy as np
    import matplotlib.pyplot as plt
    
    def rectify():
        fig, ax = plt.subplots()
        ax.pcolormesh(np.random.random((10,10)))
        fig.savefig('blah.png')
        plt.close(fig)
    
    for _ in range(10):
        rectify()
    
    plt.show()
    

Alternately, you can bypass `pyplot` and make the figure and canvas directly.
Pyplot's figure manager won't be involved, and the figure instance will be
garbage collected as you'd expect. However, this method is rather verbose, and
assumes you know a bit more about how matplotlib works behind-the-scenes:

    
    
    import numpy as np
    from matplotlib.figure import Figure
    from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
    
    # Don't need this, but just to demonstrate that `show()` has no effect...
    import matplotlib.pyplot as plt
    
    def rectify():
        fig = Figure()
        FigureCanvas(fig)
        ax = fig.add_subplot(1,1,1)
        ax.pcolormesh(np.random.random((10,10)))
        fig.savefig('blah.png')
    
    for _ in range(10):
        rectify()
    
    plt.show()
    

