Can I save blend files with lzma instead of gzip? I compiled blender with Lzma on, but when I check "compress file" while saving, it currently compresses with gzip (zlib).

Is there a special compile option to change it? Or can I do a save script in python that compress/decompress to/from lzma (without writing to the HDD).

I'm working on very big files and Lzma2 (tested by compressing to .xz with linux) is between 2x and 3x smaller and not much slower because multi threaded.
The GZIP compression is currently the only compression format considered when writing or reading a .blend file. A single reference to LZMA in Blender's code is for compressing point-cache files, with accompanying caveat:

"Effective but slow compression"


... something we wouldn't consider as primary choice, for dealing with potentially large .blend files.

I think your best bet is to write export/import operators, utilizing external LZMA (de)compressor.