title: python memoryerror - large loop xml to mongodb
body: <p>I downloaded a zip file from <a href="https://clinicaltrials.gov/AllPublicXML.zip" rel="nofollow noreferrer">https://clinicaltrials.gov/AllPublicXML.zip</a>, which contains over 200k xml files (most are &lt; 10 kb in size), to a directory (see 'dirpath_zip' in the CODE) I created in ubuntu 16.04 (using DigitalOcean).  What I'm trying to accomplish is loading all of these into MongoDB (also installed in the same location as the zip file).  </p>
  
  <p>I ran the CODE below twice and consistently failed when processing the 15988th file.  </p>
  
  <p>I've googled around and tried reading other posts regarding this particular error, but couldn't find a way to solve this particular issue.  Actually, I'm not really sure what problem really is... any help is much appreciated!!</p>
  
  <p>CODE:</p>
  
  <pre><code>import re
  import json
  import zipfile
  import pymongo
  import datetime
  import xmltodict
  from bs4 import BeautifulSoup
  from pprint import pprint as ppt
  
  
  def timestamper(stamp_type="regular"):
      if stamp_type == "regular":
          timestamp = str(datetime.datetime.now())
      elif stamp_type == "filename":
          timestamp = str(datetime.datetime.now()).replace("-", "").replace(":", "").replace(" ", "_")[:15]
      else:
          sys.exit("ERROR [timestamper()]: unexpected 'stamp_type' (parameter) encountered")
      return timestamp
  
  
  client = pymongo.MongoClient()
  db = client['ctgov']
  coll_name = "ts_"+timestamper(stamp_type="filename")
  coll = db[coll_name]
  
  dirpath_zip = '/glbdat/ctgov/all/alltrials_20180402.zip'
  z = zipfile.ZipFile(dirpath_zip, 'r')
  i = 0
  for xmlfile in z.namelist():
      print(i, 'parsing:', xmlfile)
      if xmlfile == 'Contents.txt':
          print(xmlfile, '==&gt; entering "continue"')
          continue
      else:
          soup = BeautifulSoup(z.read(xmlfile), 'lxml')
  
          json_study = json.loads(re.sub('\s', ' ', json.dumps(xmltodict.parse(str(soup.find('clinical_study'))))).strip())
  
          coll.insert_one(json_study)
  
          i+=1
  </code></pre>
  
  <p>ERROR MESSAGE:</p>
  
  <pre><code>Traceback (most recent call last):
    File "zip_to_mongo_alltrials.py", line 38, in &lt;module&gt;
      soup = BeautifulSoup(z.read(xmlfile), 'lxml')
    File "/usr/local/lib/python3.5/dist-packages/bs4/__init__.py", line 225, in __init__
      markup, from_encoding, exclude_encodings=exclude_encodings)):
    File "/usr/local/lib/python3.5/dist-packages/bs4/builder/_lxml.py", line 118, in prepare_markup
      for encoding in detector.encodings:
    File "/usr/local/lib/python3.5/dist-packages/bs4/dammit.py", line 264, in encodings
      self.chardet_encoding = chardet_dammit(self.markup)
    File "/usr/local/lib/python3.5/dist-packages/bs4/dammit.py", line 34, in chardet_dammit
      return chardet.detect(s)['encoding']
    File "/usr/lib/python3/dist-packages/chardet/__init__.py", line 30, in detect
      u.feed(aBuf)
    File "/usr/lib/python3/dist-packages/chardet/universaldetector.py", line 128, in feed
      if prober.feed(aBuf) == constants.eFoundIt:
    File "/usr/lib/python3/dist-packages/chardet/charsetgroupprober.py", line 64, in feed
      st = prober.feed(aBuf)
    File "/usr/lib/python3/dist-packages/chardet/hebrewprober.py", line 224, in feed
      aBuf = self.filter_high_bit_only(aBuf)
    File "/usr/lib/python3/dist-packages/chardet/charsetprober.py", line 53, in filter_high_bit_only
      aBuf = re.sub(b'([\x00-\x7F])+', b' ', aBuf)
    File "/usr/lib/python3.5/re.py", line 182, in sub
      return _compile(pattern, flags).sub(repl, string, count)
  MemoryError
  </code></pre>