Writing append only gzipped log files in Python
Question: I am building a service where I log plain text format logs from several
sources (one file per source). I do not intend to rotate these logs as they
must be around forever.

To make these forever around files smaller I hope I could gzip them in fly. As
they are log data, the files compress very well.

What is a good approach in Python to write append-only gzipped text files, so
that the writing can be later resumed when service goes on and off? I am not
that worried about losing few lines, but if gzip container itself breaks down
and the file becomes unreadable that's no no.

Also, if it's no go, I can simply write them in as plain text without gzipping
if it's not worth of the hassle.


Answer: Note: On unix systems you should seriously consider using an external program,
written for this exact task:

  * [`logrotate`](http://linuxcommand.org/man_pages/logrotate8.html) (rotates, compresses, and mails system logs)

You can set the number of _rotations_ so high, that the first file would be
deleted [in 100 years](http://serverfault.com/a/50185/4346) or so.

* * *

In Python 2, `logging.FileHandler` takes an keyword argument `encoding` that
can be set to `bz2` or `zlib`.

This is because `logging`
[uses](http://hg.python.org/cpython/file/8ee6d96a1019/Lib/logging/__init__.py#l919)
the `codecs` module, which in turn treats `bz2` (or `zlib`) as _encoding_ :

    
    
    >>> import codecs
    >>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2") as fh:
    ...     fh.write("Hello World\n")
    
    $ bzcat on-the-fly-compressed.txt.bz2 
    Hello World
    

* * *

Python 3 version (although the docs
[mention](http://docs.python.org/3.3/library/codecs.html#standard-encodings)
`bz2` as alias, you'll actually have to use `bz2_codec` \- at least w/ 3.2.3):

    
    
    >>> import codecs
    >>> with codecs.open("on-the-fly-compressed.txt.bz2", "w", "bz2_codec") as fh:
    ...     fh.write(b"Hello World\n")
    
    $ bzcat on-the-fly-compressed.txt.bz2 
    Hello World
    

