To this day, I am surprised at the number of people who complain about the Journal's binary storage format. Having spent years working as a system administrator, and after years of working with and on syslog-ng - in the capacity of maintainer of the Open Source Edition for more than a year -, I am increasingly puzzled about all the hostility towards non-text storage formats. I am even more puzzled about the arguments against it. Maybe I'm living in a different world, but there are very, very few reasons I see for using text based storage, when there is a better option available. I've been asked numerous times about my logging needs, and why I am so vigilantly against text based log storage, and this is my attempt at answering those questions.

Please note that this is not a journal-apology: I've been deploying binary log storage way before the Journal happened, and my recommended solution is not the Journal either. (Truth be told, I like the idea of the Journal, and am enjoying its features on some machines of mine, but it is not a solution to my needs, and never will be.)

(As a side-note, in reply to some of the reactions I've seen, I wrote a followup post.)

Table of Contents

Requirements

To start off, I'll describe my systems and my requirements. I have two setups, with different needs, and I originally used different solutions, but converged to a single one later.

The Small One

The easier setup is my personal setup: my own servers, computers and gadgets. It consists of two VPS machines, a PC at home, two laptops (all of these run Debian) and a Raspberry Pi B (running either OpenELEC, whose logs I do not collect or care about; or Raspbian, which is tied in to the above infrastructure). I used to have a router with custom firmware, but I no longer have that.

On average, I have a few hundred messages incoming a second.

My requirements here are simple:

I need one central place to store my logs. I do not care - nor store - logs on each computer. They all send to a central server, and only hold logs themselves if the central is unreachable. If I lose a few messages here and there, that is no problem.

The central must be easy to change. While on the go, possibly without internet access, I do not want my laptops to even try sending to central. So I want to be able to pull up a dummy node locally. I don't care whether the temporary local collector node and the central one are mergeable. If need be, I can export my logs from the local one, and import it into central, but I rarely do that.

I want to preserve all logs in an efficient way. I need historic data for experiments and some toy projects of mine.

I post-process data, and store a structured, processed version only. I take all my logs, may they come from syslog, the Journal or any other source, and post-process them. I extract out key fields, correlate messages, and so on. I'm only interested in this part of the data, the original messages are discarded.

I want to do queries that span programs and machines. One thing I reasonably frequently do, is follow the life of e-mail I send: the logs from Gnus that composed the message from my PC, through msmtpd on the same machine, through postfix on the raspberry pi, then postfix on my remote server. That spans three hosts and four programs. I want to ask my system this: "Show me all the logs for the e-mail with message-id X !", or "Show me all the logs of e-mails I sent today that were delayed more than an hour!", and a number of similar questions.

I want reasonably fast and efficient ad-hoc queries. While I post-process my logs, I want to be able to do queries that I came up with on the spot, that work on historic data without having to re-process past logs. Of course, this only has to work for fields that I actually extracted. If I want to introduce a new field, I'll either re-process old data, or only care about new logs.

The Big One

A more complex setup is a tiny bit bigger. The bulk of it is a five-node cluster of some beefy machines, which collect about 100Gb of raw logs per node a day from the main application-group running on them.

The requirements are similar:

We need one central place to store the logs. We do not care - nor store - logs on each computer. They all send to a central server, and only hold logs themselves if the central is unreachable. The central storage is redundant, and the local machines can buffer half