This article is an excerpt from The SaaS Guidebook. It’s just one of the topics the book will cover. If you’re interested in building and running solid SaaS applications, please sign up for updates on the book.
Building specialized integrations with other tools is one way to approach this. However, wouldn’t it be nice if your application had a generic way for others to be notified about things that happen on your side so they can work with that information?
When it comes to passing data around in (near) realtime, webhooks are the way to go. There is no real standard for them, but most services are working with simple POST requests that send JSON as their request body. Webhooks are triggered for different events and transmit relevant data to other parties.
In the following, we’ll be talking about how to allow other services to integrate with your application, by allowing them to register webhook endpoints and notifying them when something interesting happens.
In order to send webhooks, you have to know where to send them to. You have to provide your users with a way to set up new webhook endpoints. This can be done via user interface. In it’s simplest form, it would just be a textfield where users can enter a URL. In more advanced scenarios, you might allow them to specify multiple different endpoints and the types of events that should send a webhook to these endpoints.
To make things easier for your users, you can also decide to allow webhook endpoint registration via your API. This way, other services can just setup the endpoints they need, making things a lot easier for your users. Add authentication via OAuth on top, and it can be as easy as a single click of a button.
We start off by implementing a Webhook::Endpoint model. Each endpoint needs a target url, a set of events that it cares about, and an account this endpoint belongs to. To make things a little easier for ourselves, we use PostgreSQL’s array columns to store the events as a simple list.
This example uses namespaces to keep things a bit more organized. We put all webhooks related code into a webhook directory. In the database we prefix all webhook related tables with webhook_.
To make the events array column work, we have to tell Rails about it by adding an attribute macro to the model.
On the validation side, we only want valid http urls as target url. Luckily Ruby comes with a handy URI regular expression in it’s standard library, ready for us to use. We also add a presence validation for the events, so every endpoint is triggered by at least one event.
In order to quickly retrieve endpoints for one (or more) events later on, we implement a custom scope. It uses PostgreSQL’s contains operator (@>) to just return the endpoints interested in a particular event.
Notice that we’re using Array(events) to convert the methods arguments into an array, or just leave it untouched if it already is one. This allows us to call the scope with one event name, or an array of multiple ones.
Next, we add some normalization and sanitization logic to the events setter method. This can be done, by simply overwriting the method, doing the necessary changes, and then calling super.
In this case we’re normalizing the events into underscore strings. Afterwards we intersect the resulting array with a predefined array of event names (more on that later), to make sure all the events name are actually valid.
Finally, we add a method to deliver an event to this endpoint. It’s empty on purpose for now.
With the Webhook::Endpoint model in place, we need a controller, the view layer and most likely an API representation for it. There’s nothing special about the implementation of all of this, so we won’t cover it here.
As mentioned before there is no standard on how webhook payloads should look like. In this example we use them to represent events that are triggered within your application. Assuming the application somehow works with a Project model, here’s an example of an event for a newly created project.
The webhook payload follows a simple structure. It always has an event_name and a set of embedded models with relevant data. You’re of course free to settle on a different format. Instead of embedding the event name in the payload itself, you might choose to send an X-Event-Name header with your request. Whatever you do, just stick to a pattern for all the webhooks you’re sending and document it. All the developers building integrations with your application will thank you for it.
Let’s implement a Webhook::Event model to represent a single event. This can be done as “plain old ruby object”. If you want to keep track of the state of every single event, you can store all events in the database as well. For the purpose of this example, we’ll keep it simple.
The implementation of Webhook::Event is straight-forward. It’s an object with two instance variables. One for the event’s name, and another one for its (optional) payload. To keep track of valid events, we add the EVENT_TYPES constant used earlier. Whenever you add a new event, add it to this list.
As we want events to be delivered as JSON in the HTTP request’s body, we have to implement serialization for this event model. This can be as simple as taking a copy of the payload and adding the event_name to it.
However, we want to keep our code clean and not come up with the entire payload for every single event. If you’re using ActiveModel::Serializers to handle serialization of your API resources, you can leverage it to properly serialize the payload for you.
This way, you can create events and simply pass your model instances as payload.
With both Webhook::Endpoint and Webhook::Event in place, it’s time to actually trigger events. To keep things organized and DRY, we use a module to implement this. Whenever you want to trigger events, you just include that module in your model and call one of the helper methods.
We start off by adding a webhook_scope method. It’s just a stub, that has to be overwritten by each model to return the scope of the webhooks. If you’re getting a bit confused by that, don’t worry. It’ll get clear once you see the module used in an actual example in a few moments.
Next, we add a method to trigger events. It just takes two arguments. One for the event name, and one for its payload. It then constructs an Webhook::Event object from those and asks every endpoint interested in the event to deliver it.
To make things even simpler, we implement a default webhook_payload method, as well as a deliver_webhook method. The latter relies on a naming convention to generate an event for the current model based on an action name.
As most events revolve around creating, updating, and removing models in your database, we can also build a simple module to handle those events automatically.
Please notice that the module is using after_commit callbacks. It’s important to only send webhooks after the data is actually present in your database. This is to prevent weird “not found” errors in the occasions where webhooks are delivered faster than your database takes to commit the transactions or it’s part of a larger transaction that fails shortly afterwards.
To deliver webhooks, I strongly recommend using a background job queue. As delivery depends on 3rd party services, you should expect them to be slow or even unreachable. As your application might send webhooks to multiple endpoints per event, you definitely don’t want your users having to wait until all of them are delivered. Using a queue also allows for retries when delivery fails for some reason.
In this example, we’re using Sidekiq as background job queue. This is not a requirement, though. ActiveJob with any backend you like will to this job as well.
The implementation will raise an exception for all HTTP status codes that indicate some sort of error. Sidekiq will catch the exception and just retry the delivery a few moments later. Please know that HTTP status code 410 Gone is an exception. It’s a small idea picked up from Zapier. Whenever the endpoint returns a 410, we consider it no longer valid and just remove it from the database.
The actual HTTP request is nothing fancy. As a result, we just use Net::HTTP from Ruby’s standard library.
Finally, we add the implementation of Webhook::Endpoint#deliver we skipped earlier. It enqueues a new Webhook::DeliveryWorker for the given event.
It’s important to serialize the event model in the very moment the delivery is triggered. Otherwise the payload might have changed until the delivery takes place, resulting in weird data and confused developers.
While the implementation shown here covers the most important parts to get webhooks implemented in your application, there are a few more things you should consider.
As webhooks are usually delivered to more or less public endpoints it’s best to add some way that allows the receiving application to ensure the data is actually coming from you. One common way to do this is to add a signature somewhere in the payload. This can either be done in the JSON body or in a custom HTTP header. The signature should be based on the payload itself and a secret only known to the sending and the receiving application.
Based on the details of your application, it might happen that you send a lot of requests to the endpoints. Some of them might not be able to handle the load. With retries in place, the number of requests sent to the endpoint will only increase, making things worse. It’s a good idea to have some sort of rate limiting in place or even temporarily disable an endpoint that is unable to process requests right now.
Webhooks are a great way to allow applications to integrate with yours. At their core, they’re simple and easy to understand HTTP requests. With just a little bit of code, they allow your users to use your application in ways that you didn’t think of before.
You can get the source code of everything explained in this article at GitHub’s Gist. You should be able to use it in your existing applications, with just minor modifications.