title: How to perform concurrent downloads in Go
body: <p>We have a process whereby users request files that we need to get from our source. This source isn't the most reliable so we implemented a queue using Amazon SQS. We put the download URL into the queue and then we poll it with a small app that we wrote in Go. This app simply retrieves the messages, downloads the file and then pushes it to S3 where we store it. Once all of this is complete it calls back a service which will email the user to let them know that the file is ready.</p>
  
  <p>Originally I wrote this to create <em>n</em> channels and then attached 1 go-routine to each and had the go-routine in an infinite loop. This way I could ensure that I was only ever processing a fixed number of downloads at a time.</p>
  
  <p>I realised that this isn't the way that channels are supposed to be used and, if I'm understanding correctly now, there should actually be one channel with <em>n</em> go-routines receiving on that channel. Each go-routine is in an infinite loop, waiting on a message and when it receives it will process the data, do everything that it's supposed to and when it's done it will wait on the next message. This allows me to ensure that I'm only ever processing <em>n</em> files at a time. I think this is the right way to do it. I believe this is <em>fan-out</em>, right?</p>
  
  <p>What I <em>don't</em> need to do, is to merge these processes back together. Once the download is done it is calling back a remote service so that handles the remainder of the process. There is nothing else that the app needs to do.</p>
  
  <p>OK, so some code:</p>
  
  <pre><code>func main() {
      queue, err := ConnectToQueue() // This works fine...
      if err != nil {
          log.Fatalf("Could not connect to queue: %s\n", err)
      }
  
      msgChannel := make(chan sqs.Message, 10)
  
      for i := 0; i &lt; MAX_CONCURRENT_ROUTINES; i++ {
          go processMessage(msgChannel, queue)
      }
  
      for {
          response, _ := queue.ReceiveMessage(MAX_SQS_MESSAGES)
  
          for _, m := range response.Messages {
              msgChannel &lt;- m
          }
      }
  }
  
  func processMessage(ch &lt;-chan sqs.Message, queue *sqs.Queue) {
      for {
          m := &lt;-ch
          // Do something with message m
  
          // Delete message from queue when we're done
          queue.DeleteMessage(&amp;m)
      }
  }
  </code></pre>
  
  <p>Am I anywhere close here? I have <em>n</em> running go-routines (where <code>MAX_CONCURRENT_ROUTINES</code> = <em>n</em>) and in the loop we will keep passing messages in to the single channel. Is this the right way to do it? Do I need to close anything or can I just leave this running indefinitely?</p>
  
  <p>One thing that I'm noticing is that SQS is returning messages but once I've had 10 messages passed into <code>processMessage()</code> (10 being the size of the channel buffer) that no further messages are actually processed.</p>
  
  <p>Thanks all</p>