title: Python multiprocessing pool hangs on map call
body: <p>I have a function that parses a file and inserts the data into MySQL using SQLAlchemy. I've been running the function sequentially on the result of <code>os.listdir()</code> and everything works perfectly.</p>
  
  <p>Because most of the time is spent reading the file and writing to the DB, I wanted to use multiprocessing to speed things up. Here is my pseduocode as the actual code is too long:</p>
  
  <pre><code>def parse_file(filename):
      f = open(filename, 'rb')
      data = f.read()
      f.close()
  
      soup = BeautifulSoup(data,features="lxml", from_encoding='utf-8')
  
      # parse file here
  
      db_record = MyDBRecord(parsed_data)
  
      session.add(db_record)
      session.commit()
  
  pool = mp.Pool(processes=8)
  
  pool.map(parse_file, ['my_dir/' + filename for filename in os.listdir("my_dir")])
  </code></pre>
  
  <p>The problem I'm seeing is that the script hangs and never finishes. I usually get 48 of 63 records into the database. Sometimes it's more, sometimes it's less.</p>
  
  <p>I've tried using <code>pool.close()</code> and in combination with <code>pool.join()</code> and neither seems to help.</p>
  
  <p>How do I get this script to complete? What am I doing wrong? I'm using Python 2.7.8 on a Linux box.</p>