Why does blender delete the headlessly baked cache on startup?
After quite some digging around and wading through the source code, I found that upon initialising the ptcache for the smoke simulation, blender checks a flag PTCACHE_BAKED. This flag appears to be saved with the blend file. If the flag is not set, the cache directory is flushed.

To prevent this from happening, the file must be saved after baking. For this, a modified version of bake-all.py works:

import bpy

try:
    for scene in bpy.data.scenes:
        bpy.ops.ptcache.free_bake_all()
        bpy.ops.ptcache.bake_all({'scene': scene,})
finally:
    bpy.ops.wm.save_mainfile()


Two additions here:


A try-finally block to save the mainfile, even if the bake is aborted with Ctrl-C.
A call to free_bake_all(); this is required for re-bakes, otherwise bake_all will not do anything, for the same reason as blender will not delete the smokecache when starting (it assumes the bake is already finished).