title: Parallel for loop over entries of an array without sharing it with all the workers
body: <p>I am working with big data arrays, something of order of 10^10 elements. I fill the entries of these arrays by calling a certain function. All entries are independent so I would like to make use of this and fill the array simultaneously by a parallel for loop that runs over a set of indices and calls the function. I know about SharedArrays and this how I usually implement such thing but because I am using huge arrays, I don't want to share them over all the workers. I want to keep my array only on the main worker and then execute a parallel for loops and transferring the result of each loop to the main worker to be stored in the array. </p>
  
  <p>For example, this is what I normally do for small arrays.</p>
  
  <pre><code>H = SharedArray{ComplexF64}(n,n) #creates a shared array of size n*n
  @sync @distributed for i in 1:q 
      H[i] = f(i) #f is a function defined on every worker
  end
  </code></pre>
  
  <p>The problem with such construction is that if the size of the array n is too big, sharing it with all the workers is not very efficient. Is there a way of getting around this? I realize my question might be very naive and I apologize for this.</p>