title: How to use python Ray to parallelise over a large list?
body: <p>I want to parallelise the operation of a function on each element of a list using <a href="https://github.com/ray-project/ray" rel="nofollow noreferrer">ray</a>. A simplified snippet is below</p>
  <pre><code>import numpy as np
  import time
  
  import ray
  import psutil
  num_cpus = psutil.cpu_count(logical=False)
  ray.init(num_cpus=num_cpus)
  
  
  @ray.remote
  def f(a, b, c):
      return a * b - c
  
  
  def g(a, b, c):
      return a * b - c
  
  
  def my_func_par(large_list):
      # arguments a and b are constant just to illustrate
      # argument c is is each element of a list large_list
      [f.remote(1.5, 2, i) for i in large_list]
  
  
  def my_func_seq(large_list):
      # arguments a anf b are constant just to illustrate
      # argument c is is each element of a list large_list
      [g(1.5, 2, i) for i in large_list]
  
  my_list = np.arange(1, 10000)
  
  s = time.time()
  my_func_par(my_list)
  print(time.time() - s)
  &gt;&gt;&gt; 2.007
  
  s = time.time()
  my_func_seq(my_list)
  print(time.time() - s)
  &gt;&gt;&gt; 0.0372
  </code></pre>
  <p>The problem is, when I time <code>my_func_par</code>, it is much slower (~54x as can be seen above) than <code>my_func_seq</code>. One of the authors of ray does answer a comment on <a href="https://medium.com/@robertnishihara/the-reason-is-most-likely-the-following-a446c4017711" rel="nofollow noreferrer">this blog</a> that seems to explain what I am doing is setting up <code>len(large_list)</code> different tasks, which is incorrect.</p>
  <p>How do I use ray and modify the code above to run it in parallel? (maybe by splitting <code>large_list</code> into chunks with the number of chunks being equal to the number of cpus)</p>
  <p><strong>EDIT:</strong> There are two important criteria in this question</p>
  <ul>
  <li>The function <code>f</code> needs to accept multiple arguments</li>
  <li>It may be necessarry to use <code>ray.put(large_list)</code> so that the <code>larg_list</code> variable can be stored in shared memory rather than copied to each processor</li>
  </ul>