title: Should I call my async function in another process/thread?
body: <p>I am screenshotting several thousand web pages with pyppeteer. I discovered by accident, that running the same script in 2 open terminals doubles the output I get. I tested this by opening up to 6 terminals and running the script and I was able to get up to 6 times the performance.</p>
  
  <p>I am considering using <code>loop.run_in_executor</code> to run the script in multiple processes or threads from a main program. </p>
  
  <p>Is this the right call or is am I hitting some IO/CPU limit in my script?</p>
  
  <p>Here is how I'm thinking of doing it. I don't know if this is the right thing to do.</p>
  
  <pre><code>import asyncio
  import concurrent.futures
  
  async def blocking_io():
      # File operations (such as logging) can block the
      # event loop: run them in a thread pool.
      with open('/dev/urandom', 'rb') as f:
          return f.read(100)
  
  async def cpu_bound():
      # CPU-bound operations will block the event loop:
      # in general it is preferable to run them in a
      # process pool.
      return sum(i * i for i in range(10 ** 7))
  
  def wrap_blocking_io():
      return asyncio.run(blocking_io())
  
  def wrap_cpu_bound():
      return asyncio.run(cpu_bound())
  
  async def main():
      loop = asyncio.get_running_loop()
      # Options:
      # 1. Run in the default loop's executor:
      result = await loop.run_in_executor(
          None, wrap_blocking_io)
      print('default thread pool', result)
      # 2. Run in a custom thread pool:
      with concurrent.futures.ThreadPoolExecutor(max_workers=6) as pool:
          result = await loop.run_in_executor(
              pool, wrap_blocking_io)
          print('custom thread pool', result)
      # 3. Run in a custom process pool:
      with concurrent.futures.ProcessPoolExecutor(max_workers=6) as pool:
          result = await loop.run_in_executor(
              pool, wrap_cpu_bound)
          print('custom process pool', result)
  
  asyncio.run(main())
  </code></pre>