title: Update a DataFrame in different python processes realtime
body: <p>So lets say You have a Python process which is collecting data realtime with around 500 rows per second (this can be further parallelized to reduce to around 50 p.s.) from a queueing system and appending it to a <code>DataFrame</code>:</p>
  
  <pre><code>rq = MyRedisQueue(..)
  df = pd.DataFrame()
  while 1:
      recv = rq.get(block=True)
      # some converting
      df.append(recv, ignore_index = True)
  </code></pre>
  
  <p>Now the question is: <strong>How to utilize the CPUs based on this data?</strong> So I am fully aware of the limitations of the <a href="https://wiki.python.org/moin/GlobalInterpreterLock" rel="noreferrer">GIL</a>, and looked into <a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.managers" rel="noreferrer">multiprocessing Manager</a> <a href="https://stackoverflow.com/a/22487898/940592">namespace</a>, <a href="https://stackoverflow.com/a/6832693/940592">here too</a>, but it looks like there are <a href="https://stackoverflow.com/questions/22487296/multiprocessing-in-python-sharing-large-object-e-g-pandas-dataframe-between/22487898#comment48387111_22487898">some drawbacks with regard to latency on the centerally hold dataframe</a>. Before digging into it, I also tried <code>pool.map</code> which I <a href="https://stackoverflow.com/a/38135787/940592">than recognized</a> to apply <code>pickle</code> between the processes, which is way to slow and has too much overhead.</p>
  
  <p><strong>So after all of this I finally wonder, how (if) an insert of 500 rows per second (or even 50 rows per second) can be transfered to different processes with some CPU time left for applying statistics and heuristics on the data in the child processes?</strong> </p>
  
  <p>Maybe it would be better to implement a custom tcp socket or queueing system between the two processes? Or are there some implementations in <code>pandas</code> or other libaries <strong>to really allow a fast access to the one big dataframe in the parent process</strong>? I love pandas!</p>