Can I save blend files with lzma instead of gzip?
The GZIP compression is currently the only compression format considered when writing or reading a .blend file. A single reference to LZMA in Blender's code is for compressing point-cache files, with accompanying caveat:

"Effective but slow compression"


... something we wouldn't consider as primary choice, for dealing with potentially large .blend files.

I think your best bet is to write export/import operators, utilizing external LZMA (de)compressor.