What are the security risks of CD in an open source project?
You essentially have three threats here:


Someone could modify .travis-ci and use that to exfiltrate the unencrypted key material.
You could accidentally check-in or publish the decrypted key material as part of your release process.
Someone could attempt to attack the encrypted key-material directly; this is very unlikely given modern practice and current computing power.


The challenge of secret management is common in open source projects; one approach I have taken in the past is to have a second "deploy" stage in a private repository and pipeline.  A private repository and pipeline, then ring-fences the sensitive data from malicious actors entirely, protecting yourself from 1 and 3.  You can safeguard against two by whitelisting files that can be checked in to decrease the probability of making a mistake.

Finally, an approach that can be used to increase the detectability of an attack is to store the key in AWS CloudHSM, Google Cloud HSM or Azure KeyVault with the goal of marking the private key as non-exportable and then handing off any encrypt, wrap or sign requests to the cloud.  This means even if the credentials for the HSM/KeyVault do get exfiltrated you know every time they are used to sign something and can effectively revoke access to the key entirely.  Unfortunately, there doesn't seem to be any way of doing this with SSH keys at the moment, however, it works will with code-signing certificates.