title: Combine framework: how to process each element of array asynchronously before proceeding
body: <p>I'm having a bit of a mental block using the iOS Combine framework.</p>
  
  <p>I'm converting some code from "manual" fetching from a remote API to using Combine. Basically, the API is SQL and REST (in actual fact it's Salesforce, but that's irrelevant to the question). What the code used to do is call a REST query method that takes a completion handler. What I'm doing is replacing this everywhere with a Combine Future. So far, so good.</p>
  
  <p>The problem arises when the following scenario happens (and it happens a lot):</p>
  
  <ol>
  <li><p>We do a REST query and get back an array of "objects".</p></li>
  <li><p>But these "objects" are not completely populated. Each one of them needs additional data from some related object. So for each "object", we do another REST query using information from that "object", thus giving us <em>another</em> array of "objects".</p></li>
  <li><p>This might or might not allow us to finish populating the first "objects" — or else, we might have to do <em>another</em> REST query using information from each of the <em>second</em> "object", and so on.</p></li>
  </ol>
  
  <p>The result was a lot of code structured like this (this is pseudocode):</p>
  
  <pre><code>func fetchObjects(completion: @escaping ([Object] -&gt; Void) {
      let restQuery = ...
      RESTClient.performQuery(restQuery) { results in
          let partialObjects = results.map { ... }
          let group = DispatchGroup()
          for partialObject in partialObjects {
              let restQuery = ... // something based on partialObject
              group.enter()
              RESTClient.performQuery(restQuery) { results in
                  group.leave()
                  let partialObjects2 = results.map { ... }
                  partialObject.property1 = // something from partialObjects2
                  partialObject.property2 = // something from partialObjects2
                  // and we could go down yet _another_ level in some cases
              }
          }
          group.notify {
              completion([partialObjects])
          }
      }
  }
  </code></pre>
  
  <p>Every time I say <code>results in</code> in the pseudocode, that's the completion handler of an asynchronous networking call.</p>
  
  <p>Okay, well, I see well enough how to chain asynchronous calls in Combine, for example by using Futures and <code>flatMap</code> (pseudocode again):</p>
  
  <pre><code>let future1 = Future...
  future1.map {
      // do something
  }.flatMap {
      let future2 = Future...
      return future2.map {
          // do something
      }
  }
  // ...
  </code></pre>
  
  <p>In that code, the way we form <code>future2</code> can depend upon the value we received from the execution of <code>future1</code>, and in the <code>map</code> on <code>future2</code> we can modify what we received from upstream before it gets passed on down the pipeline. No problem. It's all quite beautiful.</p>
  
  <p>But that doesn't give me what I was doing in the pre-Combine code, namely <em>the loop</em>. Here I was, doing <em>multiple</em> asynchronous calls in a <em>loop</em>, held in place by a DispatchGroup before proceeding. The question is:</p>
  
  <p><em>What is the Combine pattern for doing that?</em></p>
  
  <p>Remember the situation. I've got an <em>array</em> of some object. I want to <em>loop</em> through that array, doing an asynchronous call for <em>each</em> object in the loop, fetching new info asynchronously and modifying that object on that basis, before proceeding on down the pipeline. And each loop might involve a further <em>nested</em> loop gathering even <em>more</em> information asynchronously:</p>
  
  <pre><code>Fetch info from online database, it's an array
     |
     V
  For each element in the array, fetch _more_ info, _that's_ an array
     |
     V
  For each element in _that_ array, fetch _more_ info
     |
     V
  Loop thru the accumulated info and populate that element of the original array 
  </code></pre>
  
  <p>The old code for doing this was horrible-looking, full of nested completion handlers and loops held in place by DispatchGroup <code>enter</code>/<code>leave</code>/<code>notify</code>. <em>But it worked</em>. I can't get my Combine code to work the same way. How do I do it? Basically my pipeline output is an array of something, I feel like I need to split up that array into individual elements, do something <em>asynchronously</em> to each element, and put the elements back together into an array. How?</p>
  
  <hr>
  
  <p>The way I've been solving this works, but doesn't scale, especially when an asynchronous call needs information that arrived several steps <em>back</em> in the pipeline chain. I've been doing something like this (I got this idea from <a href="https://stackoverflow.com/a/58708381/341994">https://stackoverflow.com/a/58708381/341994</a>):</p>
  
  <ol>
  <li><p>An array of objects arrives from upstream.</p></li>
  <li><p>I enter a <code>flatMap</code> and <code>map</code> the array to an array of publishers, each headed by a Future that fetches further online stuff related to <em>one</em> object, and followed by a pipeline that produces the <em>modified</em> object.</p></li>
  <li><p>Now I have an array of pipelines, each producing a single object. I <code>merge</code> that array and produce that publisher (a MergeMany) from the <code>flatMap</code>.</p></li>
  <li><p>I <code>collect</code> the resulting values back into an array.</p></li>
  </ol>
  
  <p>But this still seems like a lot of work, and even worse, it doesn't scale when each sub-pipeline itself needs to spawn an array of sub-pipelines. It all becomes incomprehensible, and information that used to arrive easily into a completion block (because of Swift's scoping rules) no longer arrives into a subsequent step in the main pipeline (or arrives only with difficulty because I pass bigger and bigger tuples down the pipeline).</p>
  
  <p>There must be some simple Combine pattern for doing this, but I'm completely missing it. Please tell me what it is.</p>