title: Multiprocessing where new process starts hafway through other process
body: <p>I have a Python script that does two things; 1) it downloads a large file by making an API call, and 2) preprocess that large file. I want to use Multiprocessing to run my script. Each individual part (1 and 2) takes quite long. Everything happens in-memory due to the large size of the files, so ideally a single core would do both (1) and (2) consecutively.  I have a large amount of cores available (100+), but I can only have 4 API calls running at the same time (limitation set by the API developers). So what I want to do is spawn 4 cores that start downloading by making an API-call, and as soon as one of those cores is done downloading and starts preprocessing I want a new core to start the whole process as well. This so there's always 4 cores downloading, and as many cores as needed doing the pre-processing. I do not know however how to have a new core spawn as soon as another core is finished with the first part of the script.</p>
  <p>My actual code is way too complex to just dump here, but let's say I have the following two functions:</p>
  <pre><code>import requests 
  
  def make_api_call(val):
      &quot;&quot;&quot;Function that does part 1); makes an API call, stores it in memory and returns a large 
         satellite GeoTIFF
      &quot;&quot;&quot;
      large_image = requests.get(val)
      return(large_image)
  
  def preprocess_large_image(large_image):
      &quot;&quot;&quot;Function that does part 2); preprocesses a large image, and returns the relevant data
      &quot;&quot;&quot;
      results = preprocess(large_image)
      return(results)
  </code></pre>
  <p>how then can I make sure that as soon as a single core/process is finished with 'make_api_call' and starts with 'preprocess_large_image', another core spawns and starts the entire process as well? This so there is always 4 images downloading side-by-side. Thank you in advance for the help!</p>