title: Celery - minimize memory consumption
body: <p>We have ~300 celeryd processes running under Ubuntu 10.4 64-bit , in idle every process takes ~19mb RES, ~174mb VIRT, thus - it's around 6GB of RAM in idle for all processes.
  In active state - process takes up to 100mb of RES and ~300mb VIRT</p>
  
  <p>Every process uses minidom(xml files are &lt; 500kb, simple structure) and urllib. </p>
  
  <p>Quetions is - how can we decrease RAM consuption - at least for idle workers, probably some celery or python options may help?
  How to determine which part takes most of memory?</p>
  
  <p><strong>UPD:</strong> thats flight search agents, one worker for one agency/date. We have 10 agencies, one user search == 9 dates, thus we have 10*9 agents per one user search. </p>
  
  <p>Is it possible start celeryd processes on demand to avoid idle workers(something like MaxSpareServers on apache)?</p>
  
  <p><strong>UPD2:</strong> Agent lifecycle is - send HTTP request, wait for response ~10-20 sec, parse xml( takes less then 0.02s), save result to MySQL</p>