Memory Leak caused by large objects in Python
<blockquote>
  <p><strong>Possible Duplicate:</strong><br>
  <a href="http://stackoverflow.com/questions/5722872/possible-fix-to-deal-with-python-memory-usage">Possible Fix to deal with Python Memory usage.</a>  </p>
</blockquote>



<p>Hi all,</p>

<p>I am currently retrieving data from a program with their API using Python. I am storing the data into the following class:</p>

<pre><code>class c1:

STORE = []
ITEMS = []
PRICES = []

def __init__(self, STORE, ITEMS,PRICES):
    self.STORE = STORE
    self.ITEMS = ITEMS
    self.PRICES = PRICES
</code></pre>

<p>The list STORE contains a list of STORES which items under ITEMS which might be available at. The lists should look like the following:</p>

<p>STORES - ['S1','S2',...,'S30000']</p>

<p>ITEMS - ['item1','item2',...,'item800']</p>

<p>The list PRICES is a multidimensional list which contains the prices of items through the stores (ex. PRICES[0][0] will get you the price of 'item1' at 'S1', and PRICES[20][4] will get you the price of 'item20' at 'S4').</p>

<p>I am creating multiple objects with this class, up to 6. Which each object will contain up too 30,000 Stores, up too 800 Items, and a Price matrix of 800 by 30,000.</p>

<p>The purpose of this is to create a report which will show different metrics concerning store overpricing, price based on location, etc.</p>

<p>I am running into issues where I get memory leaks, and looking at the Windows Task Manager it seems that I am using up to 1.8 GB of memory.</p>

<p>All of you might suggest for me to use the API to read each price, item, and store individually and not storing anything in memory, but that is out of the question. I already have that code developed and it takes approximately about 3 hours to run, which for the current needs is not an option.</p>

<p>So I have 2 very simple questions:</p>

<ol>
<li><p>I am using gc, but it seems that my code doesn't 'free' up the memory after gc. Is there a brute force method of 'freeing' up memory? Or extending the allowed memory for python to use?</p></li>
<li><p>Are there other data structures which could be more lightweight than lists? The instances under STORE and ITEMS are strings, and the instances under PRICES are doubles. Already looked at Numpy but I am forced to use Python 2.3 which Numpy is not compatible with.</p></li>
</ol>

<p>In advance thank you.</p>
