How to securely pass sensitive data to EC2 Spot instances? Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.

My understanding is that I have a few ways to pass those secret data to Spot instances:


Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
Use Secrets Manager, but that could quickly cost a lot...
Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)


And I am not sure I could individually identify spot instances?

Am I missing something there? How do you solve those challenges?
When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:


If your provisioning tool (terraform, cloudformation, salt cloud) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.
You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use Vault. It's not that hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.