title: How to process massive amounts of data in parallel without using up memory with Python Ray?
body: <p>I am considering using Ray for a simple implementation of parallel processing of data:</p>
  
  <ul>
  <li>there are massive amounts of data items to be processed which become available through a stream / iterator. Each item is of significant size</li>
  <li>a function should be run on each of the items and will produce a result of significant size</li>
  <li>the processed data should get passed on in a stream or get stored in some kind of sink that can only accept a certain amount of data within some period of time</li>
  </ul>
  
  <p>I want to find out if this is something that can be done in Ray. </p>
  
  <p>Currently I have the following simple implementation based on pythons multiprocessing library:</p>
  
  <ul>
  <li>one process reads the stream and passes items on to a queue which will block after k items (so that the memory needed for the queue will not exceed some limit)</li>
  <li>there are several worker processes which will read from the input queue and process the items. The processed items are passed on to a result queue, which is again of limited size</li>
  <li>another process reads the result queue to pass on the items </li>
  </ul>
  
  <p>With this, as soon as the workers cannot process any more items, the queue will block and no attempt is made to pass more work on to the workers. 
  If the sink process cannot store more items, the result queue will block which will in turn block the workers which will in turn block the input queue until the writer process can write more results again.</p>
  
  <p>So does Ray have abstractions to do something like this? How would I make sure that only a certain amount of work can be passed on to the workers and how can I have something like the single-process output function and make sure that the workers cannot flood that function with so many results that the memory/storage is exhausted? </p>