Keep a Changelog, Ya Filthy Animal!
Keep a changelog, ya filthy animal. - Ryan H - Medium
You’ve read about the importance of keeping a changelog for end users. “What changed!?” they ask. So, you tell them — by means of a nicely formatted list of features and functions that have changed since the last version. Makes sense, right?But let’s forget about your end users for a moment (sorry end users). What about all the changes that happen behind the scenes to keep your ever-complex infrastructure growing and running smoothly?Users don’t care that a server’s configuration changed or that a new API was deployed or that indexing was added to a database.But engineering teams do care.An infrastructure changelog is used to track all of the production changes that are happening within your stack. At a larger scale, it tracks and communicates the state of the environment at a macro level.What changes should you log?Generally, any changes that affect your production infrastructure:Configuration Changes (e.g. cluster scaling, individual server settings, library/package updates)Code Deployments (CI/CD)Database ChangesAny other ad-hoc production changesA changelog should be treated much differently than standard logging. Logging systems usually involve aggregating lots of noisy system signals (e.g. http requests, events, errors, user actions). A changelog should not be noise — it should be brief, contextual and high-level.Why should you keep a changelog?1. Incident resolution“99% of all outages are caused by a change.”Over the years, we’ve found that a majority of our system outages were caused by changes that we made and not some exogenous evil force. It’s good to know what changed, especially when rapidly investigating outages.2. Peer Review/Team TransparencyIt serves as a “newsfeed of changes” where everyone has visibility into the changes that are happening to the stack. No longer does the DBA team need to wonder why the queries per second on a particular schema just jumped by 25%. They can look at the changelog and see that the API team just released a new version of the backend that makes heavier use of that schema.If done well, this can also act as a peer reviewing system where poor patterns are stamped out and replaced with better approaches in the long run.3. Communicating to non-engineersSo often, we hear that engineering teams are isolated from the other parts of the business like sales, customer support, even management. A changelog can also be used to communicate the changes that may affect others (e.g. new features rolling out, performance updates, improved security).How do you build a changelog?There is certainly a range of practice here. An important component is adequate integration and automation to limit manual maintenance. Otherwise, the cost may outweigh the benefits. I will outline a few approaches encountered.Email/Slack Channels: Automated and manual posts to email or Slack channels for all to see. This can work reasonably well. While easy to hack together, email and slack channels have limitations when it comes to tagging, editing, filtering and visualizing changes.Repurpose existing tools: Tools like Jira or Datadog, while not designed for this purpose, can be used for change logging.Build your own tool: While more expensive, some organizations build a homegrown changelog tool with features like notifications, full-text search, and visualizations.