title: CPU Bound Task - Multiprocessing Approach Performance Way Worse Than Synchronous Approach -Why?
body: <p>I just get start with asynchronous programming, and I have one questions regarding CPU bound task with multiprocessing. In short, why multiprocessing generated way worse time performance than Synchronous approach? Did I do anything wrong with my code in asynchronous version? Any suggestions are welcome!</p>
  <p><em><strong>1: Task description</strong></em></p>
  <p>I want use one of <a href="http://storage.googleapis.com/books/ngrams/books/datasetsv2.html" rel="nofollow noreferrer">the Google's Ngram datasets</a> as input, and create a huge dictionary includes each words and corresponding words count.</p>
  <p>Each Record in the dataset looks like follow :</p>
  <p>&quot;corpus\tyear\tWord_Count\t\Number_of_Book_Corpus_Showup&quot;</p>
  <p>Example:</p>
  <p>&quot;A'Aang_NOUN\t1879\t45\t5\n&quot;</p>
  <p><em><strong>2: Hardware Information:</strong></em>
  Intel Core i5-5300U CPU @ 2.30 GHz 8GB RAM</p>
  <p><em><strong>3: Synchronous Version - Time Spent 170.6280147 sec</strong></em></p>
  <pre><code>import time
  
  with open(r&quot;.\googlebooks-eng-all-1gram-20120701-a.gz&quot;,encoding='utf-8') as file:
      start = time.perf_counter()
  
      all_input = file.readlines()
      word_count_dict = {}
  
      for line in all_input:
          temp = line.replace('\n','').split('\t')
          if temp[0] not in word_count_dict.keys():
              word_count_dict[temp[0]] = temp[2]
          else:
              word_count_dict[temp[0]] += temp[2]
  
      print(f'total time used for sync version as {time.perf_counter() - start}')
  </code></pre>
  <p><em><strong>4: Asynchronous Version - Time Spent 611.5669237 sec</strong></em></p>
  <pre><code>import asyncio
  from concurrent.futures import ProcessPoolExecutor
  import functools
  import time
  
  def data_spliting(input_data,chunk_size): # todo see next part for how to set chunk size
      for x in range(0,len(input_data),chunk_size):
          yield input_data[x:x+chunk_size]
  
  def single_chunk_dict(chunk):
      result = {}
      for line in chunk:
          temp = line.replace('\n','').split('\t')
          if temp[0] not in result.keys():
              result[temp[0]] = temp[2]
          else:
              result[temp[0]] += temp[2]
  
      return result
  
  def word_reduce(first_dict,second_dict):
      result = {}
      for map in [first_dict,second_dict]:
          for key, value in map.items():
              if key not in result.keys():
                  result[key] = value
              else:
                  result[key] += value
      return result
  
  async def main():
      with open(r&quot;.\googlebooks-eng-all-1gram-20120701-a.gz&quot;,encoding='utf-8') as file:
          test = file.readlines()
  
      with ProcessPoolExecutor() as process_pool:
          loop = asyncio.get_running_loop()
          tasks = [functools.partial(single_chunk_dict,ch) for ch in data_spliting(test,21654626)]
          result = [loop.run_in_executor(process_pool,x) for x in tasks]
          result = await asyncio.gather(*result)
  
          output = functools.reduce(lambda x,y: word_reduce(x,y),result)
          print(f'output total keys = {len(output.keys())}')
  
  
  if __name__ == '__main__':
      start = time.perf_counter()
      asyncio.run(main())
      print(f'Total Time for Completion as {time.perf_counter() - start}')
  </code></pre>
  <hr />
  <p><strong>Further Questions Regarding Mr. Peters' Answer.</strong></p>
  <p>I cannot believe my question get answered by Tim Peters. This is so cool!</p>
  <p>1: In the book that I am reading, the author uses this task to demonstrate MapReduce. I am wondering if MapReduce is a good candidate for multiprocessing.</p>
  <p>2: In the book, the author suggests hooking ProcessPoolExecutor with asyncio event loop let us use the API function such as gather() and as_complete(). Is mixing ProcessPoolExecutor with asyncio a good practice? Or should I stick with the map() function in ProcessPoolExecutor?</p>
  <p>3: &quot;An &quot;ideal&quot; candidate for coarse-grained parallelism does a whole lot of computation per byte that needs to be transferred between processes, and not need much inter-process communication at all.&quot;</p>
  <p>What do &quot;transferred between processes&quot; and &quot;inter-process communication&quot; mean?</p>