title: How do I get improved pymongo performance using threading?
body: <p>I'm trying to see performance improvements on pymongo, but I'm not observing any.</p>
  
  <p>My sample db has 400,000 records.  Essentially I'm seeing threaded and single threaded performance be equal - and the only performance gain coming from multiple process execution.</p>
  
  <p>Does pymongo not release the GIL during queries?</p>
  
  <p>Single Perf: real    0m0.618s</p>
  
  <p>Multiproc:real    0m0.144s</p>
  
  <p>Multithread:real    0m0.656s </p>
  
  <p>Regular code:</p>
  
  <pre><code>choices = ['foo','bar','baz']
  
  
  def regular_read(db, sample_choice):
      rows = db.test_samples.find({'choice':sample_choice})
      return 42  # done to remove calculations from the picture
  
  def main():
      client = MongoClient('localhost', 27017)
      db = client['test-async']
      for sample_choice in choices:
          regular_read(db, sample_choice)
  
  if __name__ == '__main__':
      main()
  
  $ time python3 mongotest_read.py 
  
  real    0m0.618s
  user    0m0.085s
  sys 0m0.018s
  </code></pre>
  
  <p>Now when I use multiprocessing I can see some improvement.</p>
  
  <pre><code>from random import randint, choice
  
  import functools
  from pymongo import MongoClient
  from concurrent import futures
  
  choices = ['foo','bar','baz']
  MAX_WORKERS = 4
  
  def regular_read(sample_choice):
      client = MongoClient('localhost', 27017,connect=False)
      db = client['test-async']
      rows = db.test_samples.find({'choice':sample_choice})
      #return sum(r['data'] for r in rows)
      return 42
  
  def main():
      f = functools.partial(regular_read)
      with futures.ProcessPoolExecutor(MAX_WORKERS) as executor:
          res = executor.map(f, choices)
  
      print(list(res))
      return len(list(res))
  
  if __name__ == '__main__':
      main()
  
  $ time python3 mongotest_proc_read.py 
  [42, 42, 42]
  
  real    0m0.144s
  user    0m0.106s
  sys 0m0.041s
  </code></pre>
  
  <p>But when you switch from ProcessPoolExecutor to ThreadPoolExecutor the speed drops back to single threaded mode.</p>
  
  <p>...</p>
  
  <pre><code>def main():
      client = MongoClient('localhost', 27017,connect=False)
      f = functools.partial(regular_read, client)
      with futures.ThreadPoolExecutor(MAX_WORKERS) as executor:
          res = executor.map(f, choices)
  
      print(list(res))
      return len(list(res))
  
  $ time python3 mongotest_thread_read.py 
  [42, 42, 42]
  
  real    0m0.656s 
  user    0m0.111s
  sys 0m0.024s
  </code></pre>
  
  <p>...</p>