Is there a standard or recommended format for encrypted data, with algorithmic agility? I am storing relatively small (from a few bytes to less than 50MB) amounts of encrypted data for an extended period. This data is encrypted using a custom tool that has already been in use for some time. I am reviewing the current implementation and am considering whether I need to make any changes. I must retain support for previously encrypted files and so (I think) must keep the leading Format ID to distinguish between formats. Beyond that, I am free to modify the configuration as I like.
If there is a standard or de facto format that would provide algorithmic agility, I would much prefer that to roll my own. Is there such a thing?
Alternatively, what standards/best practices apply to developing such a format? Are there any particular pitfalls I should bear in mind?
The existing format currently is as follows:
## File format                                                                                                    
                                                                                                                  
# +--------------------------------+                                                                              
# |           Format ID            |  1 byte                                                                      
# +--------------------------------+                                                                              
# |        Unused (Legacy)         |  3 bytes                                                                     
# +--------------------------------+                                                                              
# |           Cipher ID            |  2 bytes                                                        
# +--------------------------------+                                                                              
# |            HMAC ID             |  2 bytes
# +--------------------------------+                                                                              
# |      Initialization Vector     |  16 bytes                                                                    
# |                                |                                                                              
# +--------------------------------+                                                                              
# |           Key Salt             |  8 bytes                                                                     
# +--------------------------------+                                                                              
# |           HMAC Tag             |  Variable length, varies with                                                
# ...                            ...  digest algorithm                                                            
# |                                |                                                                              
# +---------------------------------                                                                              
# |        Encrypted Data          |                                                                              
# |                                |                                                                              
# |                                |  Variable length                                                             
# ...                            ...                                                                              
# |                                |                                                                              
# +---------------------------------                                                                              
                                              

Should the HMAC Tag cover just the encrypted data and the IV (that is what it does currently)? Is it desirable/practical to provide some integrity for the other parts of the header as well?
The layout of the format doesn't matter much.  I would recommend combining all the parameters into a single cipher suite name so that you're not tempted to make nonsensical combinations: just use NaCl crypto_secretbox_xsalsa20poly1305 (or AES-GCM if you must use AES, with the caveat that it subjects your users to side channel attacks on software AES implementations), not your own composition of AES256-CTR with HMAC-SHA256.

What matters much more is:


How many cryptographic decisions are imposed upon users who are not competent to answer questions on this site?  Correct answer: none.

For example, although the scrypt file format has some options, there are no cipher fields—just a format version number.  There are cost parameters for the password-based key derivation function, but when creating files, even those are automatically calibrated by the scrypt utility by default, with the option to specify them manually.
How are the parties in the system to behave when fed instances of the format?  Be careful to specify protocol, not just data.  How do you use the key and act on the file?

If you are using a key for a particular purpose, say an RSASSA-PSS public verification key, do not let the format dictate another purpose.  The implementation ecosystem of JavaScript Web Tokens, JWT, is a disaster because it was designed as a lofty view of data, abstracted from the reality of protocols.

JWT implementations often expose a generic verify(token, key) function which uses key however the attacker-controlled token prescribes.  Even if you meant for key to be an RSASSA-PSS public verification key, token might say ‘hey, JavaScript chump, please use HMAC-SHA256!’, and you, the chump who used JWT, will happily use the ‘RSASSA-PSS public verification key’ in the place of an HMAC-SHA256 key which enables trivial forgery since RSASSA-PSS public verification keys are, well, often public.