title: Python multiprocessing passing database cursor object
body: <p>I have a Queue that gets populated with <code>getq()</code> method. The data populated in this queue needs to be inserted into database. </p>
  
  <p>I'm trying to spawn few Processes and insert the data into database. Since total data is around 100MB, doing parallel processing saves in-memory!</p>
  
  <p>I'm using SQL Server, with <code>pymssql</code> library on Python 2.7.</p>
  
  <p>The below is my code</p>
  
  <pre><code>def worker(queue, db_queue):
      db = get_db()
      cursor = db.cursor()
      db_queue.put(db)
      while True:
          data = queue.get()
          # perform database operations inside a function
          queue.task_done()
  
  def main():
      queue = JoinableQueue()
      db_queue = Queue()
      for _ in range(10):
          p = Process(target=worker, args=(queue, db_queue))
          p.daemon = True
          p.start()
      getq(q)
      queue.join()
  
      while not db_queue.empty():
            db_queue.get().commit()
  
  if __name__ == "__main__":
     main()
  </code></pre>
  
  <p>When I am putting <code>db</code> aka connection objects into a <code>multiprocessing.Queue</code>, and getting it back in the main process, the connection is getting closed!</p>
  
  <p>My basic intention is to insert all the data into database using several processes. Only after successful insertions in all processes, commit them. If even one fails to commit, rollback everything!</p>
  
  <p>I'm not sure where things are going wrong.
  Can anyone please help?</p>