title: Synchronize write to file from heavy operations in different threads
body: <p>I need to elaborate a file (potentially a big file) one block at a time and write the result to a new file. 
  To put it simply, I have the basic function to elaborate a block:</p>
  
  <p><code>func elaborateBlock(block []byte) []byte { ... }</code></p>
  
  <p>Every block needs to be elaborated and then written to the output file sequentially (preserving original order).</p>
  
  <p>The one-thread implementation is trivial:</p>
  
  <pre><code>for {
          buffer := make([]byte, BlockSize)
          _, err := inputFile.Read(buffer)
  
          if err == io.EOF {
              break
          }
          processedData := elaborateBlock(buffer)
          outputFile.Write(processedData)
  }
  </code></pre>
  
  <p>But the elaboration can be heavy and every block can be processed separately, so a multi-threaded implementation is the natural evolution.</p>
  
  <p>The solution I came up with is to create an array of channels, compute every block in a different thread and sync the final write by looping the channel array:</p>
  
  <p>Utility function:</p>
  
  <pre><code>func blockThread(channel chan []byte, block []byte) {
      channel &lt;- elaborateBlock(block)
  }
  </code></pre>
  
  <p>In the main program:</p>
  
  <pre><code>chans = []chan []byte {}
  
  for {
      buffer := make([]byte, BlockSize)
      _, err := inputFile.Read(buffer)
  
      if err == io.EOF {
          break
      }
  
      channel := make(chan []byte)
      chans = append(chans, channel)
  
      go blockThread(channel, buffer)
  }
  
  for i := range chans {
      data := &lt;- chans[i]
      outputFile.Write(data)
  }
  </code></pre>
  
  <p>This approach works but can be problematic with large files because it requires to load the whole file in memory before starting writing the output.</p>
  
  <p>Do you think there can be a better solution, with also better performance overall?</p>