Recently Reverb began using NSQ as a distributed message queue. After establishing producers in our Rails app, we wrote a simple Go consumer for some of our services to use the new queue.

What the NSQ?

Before detailing our consumer, let’s talk a bit about queues. While there are a number of open source options, we chose NSQ for its ease of deployment, its simple local broker strategy, its resiliency, and speed.

NSQ’s architecture is made up of nsqd instances and nsqlookupd instances for topic discovery. With no single point of failure consumers need to query the nsqlookupd instances to discover topics and their producers. At Reverb we have nsqd instances deployed to every box where we produce messages. This makes our system resilient to failure while allowing us to scale simply.

nsq’s messaging architecture also allows each queue to have either topic or queue like semantics. This allows us to choose whether we round robin messages to a number of consumers on a shared channel or replicate messages to a number of different consumers.

nsq provides both a TCP and a simple HTTP interface, while we will be using the TCP protocol in our consumer, the HTTP interface provides a simple way to interact with nsq.

Writing a consumer in Go is simple with the official nsq go package. While light on documentation the official client has a number of examples that ship along with nsq.

Our Consumer

One of the benefits of using NSQ’s official package is that replaced a large amount of worker boilerplate code that we were using with a previous messaging architecture. Go is great at this worker pattern, but the package provides everything you’d normally need.

We want a single consumer to concurrently consume a number of messages from a shared topic. We will create a channel specific to our application called “metrics” and log any messages to STDOUT. We’ll also ensure that we shutdown properly without any messages still in flight.

Have any questions? Feel free to leave us a comment. Like writing Go and have a passion for music gear? We’re hiring.

Erik Benoist

Joe Levering