Using version control/deployments with Context
OK; so I swear I did look reasonably hard for this before I posted the question, but I have done the thing where I've genuinely found the answer 15 minutes after posting. 

http://www.mcdruid.co.uk/content/default-drupal-contexts-like-default-views was the blogpost I needed - he's created an issue to get this information into the module, but it looks like it won't happened.

This seems to work really well, and I've created a separate submodule just for contexts.