To commemorate the third annual GopherCon US in Denver this week, we’re releasing cgo bindings to two compression libraries that we’ve been using in production at Datadog for a while now: czlib and zstd.

czlib started as a fork of the vitess project’s cgzip package. Our primary data pipeline uses zlib compressed messages, but the standard library’s pure Go implementation can be significantly slower than the C zlib library. In order to address this gap, we modified a few flags in cgzip to make it encode and decode with zlib wrapping rather than with gzip headers.

We’ve detailed some of the other more novel design decisions in czlib , including its batch interfaces, in our general blog on performance in Go a couple of years ago. Performance varies quite a bit among the various interfaces, so it pays to benchmark using a message that is typical for your system by running the czlib benchmark suite with PAYLOAD=path_to_message go test -run=NONE -bench .

Here are modern benchmark results running go1.7beta2 for compression and decompression using the non-streaming interface in czlib, the streaming interface, and the standard library’s compress/zlib that show the variance in performance:

# using a 2kb plaintext message BenchmarkCompress-4 30000 47415 ns/op 44.42 MB/s BenchmarkCompressStream-4 20000 61732 ns/op 34.11 MB/s BenchmarkCompressStdZlib-4 5000 227182 ns/op 9.27 MB/s BenchmarkDecompress-4 200000 8238 ns/op 255.62 MB/s BenchmarkDecompressStream-4 100000 18352 ns/op 114.75 MB/s BenchmarkDecompressStdZlib-4 50000 31565 ns/op 66.72 MB/s # using a 1.7MB plaintext message BenchmarkCompress-4 20 69808144 ns/op 24.70 MB/s BenchmarkCompressStream-4 20 73170819 ns/op 23.56 MB/s BenchmarkCompressStdZlib-4 20 70498763 ns/op 24.46 MB/s BenchmarkDecompress-4 200 6709252 ns/op 256.98 MB/s BenchmarkDecompressStream-4 200 6891833 ns/op 250.18 MB/s BenchmarkDecompressStdZlib-4 100 14256445 ns/op 120.94 MB/s

zstd, pronounced Zstandard, is a relatively new fast compression library from Yann Collet, the author of lz4. It has recently finalized its format, and a 1.0 release is pending. It compresses slightly faster than zlib at level 6 at a slightly better ratio, and decompresses much faster, making it a great general purpose zlib replacement.

The zstd library supports some interfaces that are common in more advanced compression libraries like stream compression, compression levels and pre-computed dictionaries. These are all exposed by our zstd Go binding, with the dictionary builder available in the upstream repos. The binding intentionally mimics the zlib interface, and aside from a few functions that do not return error in zstd, it is functionally a drop-in replacement. It also exposes a fixed-length batch compression interface present in the underlying library, very similar to the lz4 interface.