Python - Process Disappearing and Freezing when working with large Dict?
Question: I'm in the process of dealing with XML files that are about 85MB in size.
Currently I'm just trying to process one. What I'm doing is download a zip,
extract to disk, then convert the XML to a Python Dict, change a couple
things, then save the Dict, and send it to a MongoDB. Except when I get to
converting to a Python dict, process freezes/disappears..

I'm running this script on a VM with Ubuntu 13.04 Server, 4 cores at 2.6, 16GB
of ram, and a 1TB 15,000RPM. I'm monitoring the script as it runs, essentially
python takes up 12% of RAM over 7 minutes, and then gone, process falls off my
high usage list and my pipe from the terminal just doesn't move. I kill it
with CTRL+Z and get returned _"Write failed: Broken pipe"_.

The last thing printed on terminal is **"Converting
dailyprice_0505_eur.xml.zip"** , which makes me suspect something maybe with
xmltodict, but I'm honestly stuck. The example code, with data, should work
for anyone who willing to help me test this out. Any help is appreciated!
Thanks.

    
    
    #Importing
    import urllib, xmltodict, os
    from zipfile import ZipFile
    
    #Getting Working Dir
    abspath = os.path.abspath(__file__)
    root = os.path.dirname(abspath) + "/"
    print "Current Working Directory: " + root
    
    #Defining
    urlAuth = 'https://dl.dropboxusercontent.com/u/9235267/'
    dailypriceFL = ['dailyprice_0505_eur.xml.zip']
    dailyPriceDict = {}
    
    for x in dailyPriceFL:
        print '  * Downloading',x
        urllib.urlretrieve(urlAuth+x, x)
    
        print '  * Extracting',x
        with ZipFile(x, "r") as z:
            z.extractall(root)
    
        print '  * Converting',x
        f = open(root+x.replace(".zip",""))
        data = xmltodict.parse(f.read())
        f.close()
    
        print '  * Adding Currency to Dict',x
        for y in data['prices']['price']:
            y.update({"currency": x[-7:].replace(".xml","").upper()})
    
        print '  * Ammending',x
        dailyPriceDict.update(data)
    
        print '  * Deleting',x
        os.remove(root+x)
        os.remove(root+x.replace(".zip",""))
        print '  * Finished',x
    


Answer: I wonder if it's not a memory error despite the specs you listed. When I
attempt to load your XML just in `xml.dom.minidom` I get this:

    
    
    Traceback (most recent call last):
      File "c:\python27\lib\xml\dom\minidom.py", line 1930, in parseString
        return expatbuilder.parseString(string)
      File "c:\python27\lib\xml\dom\expatbuilder.py", line 940, in parseString
        return builder.parseString(string)
      File "c:\python27\lib\xml\dom\expatbuilder.py", line 223, in parseString
        parser.Parse(string, True)
      File "c:\python27\lib\xml\dom\expatbuilder.py", line 751, in start_element_handler
        node = minidom.Element(qname, uri, prefix, localname)
      File "c:\python27\lib\xml\dom\minidom.py", line 653, in __init__
        self._attrs = {}   # attributes are double-indexed:
    MemoryError
    

This does seem to work on my side of things though:

    
    
    >>> import xmltodict, os
    >>> data = open('price.xml').read()
    >>> xml = xmltodict.parse(data)
    >>> xml['prices']['price'][0]
    OrderedDict([(u'code', u'AD1550.301.1'), 
                 (u'startdate', u'2013-08-24'), 
                 (u'enddate', u'2013-09-30'), 
                 (u'rentalprice', u'126.00'), 
                 (u'midweekrentalprice', u'0.00'), 
                 (u'weekendrentalprice', u'0.00'), 
                 (u'fixprice', u'0.00')])
    

That does take nearly 800Mb on my system, but I don't seem to get errors.
However, if I try to parse it into xmltodict _after_ I've attempted to use
minidom, I get something more like this:

    
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "c:\python27\lib\site-packages\xmltodict.py", line 228, in parse
        parser.Parse(xml_input, True)
    xml.parsers.expat.ExpatError: out of memory: line 1, column 0
    

Which suggests rather directly that, because both are based on `expat`,
something is not getting freed up between runs of your script (or iterations
of your loop), and you'll need to refactor somehow.

