Third Base — Shared and secured access AWS keys

Once we had the AWS key distribution to CI/CD pipelines automated we stopped having broken builds due to incorrect or otherwise missing AWS keys. However, sometimes CI/CD is not available and we need to run our automation pipelines manually (i.e. from our laptops). Given the way we setup our CI/CD that shouldn’t be a problem since we can easily execute the same script that the jobs execute. Except, CI/CD variables have the latest and correct AWS keys while our laptops did not!

Now, what if our laptops (automatically) had access to the exact same AWS keys as the CI/CD pipelines do! In other words, what if we could distribute the same AWS keys to both the CI/CD pipeline and the developers? That way we would be absolutely sure they do not diverge.

Enter Keybase. Keybase is a nice up and coming project, “A Slack but for the whole world.” as they say on their site. Everything is end-to-end encrypted based on asymmetric key cryptography. We learned about Keybase while using Terraform because Terraform can use Keybase to encrypt AWS keys held in its state using a Keybase user handle. This protects the key that is stored in a shared state file to ensure only its owner can actually use it (by using Keybase to decrypt it). Furthermore, Keybase provides a user-space mount point (usually at /keybase ) that simulates a filesystem tree containing 3 main directories:

public: holds files that the user wants to make publicly available.

private: holds files that the user wants to keep private.

team: holds sub-directories per team the user is member of with private files that are available to the members of each team.

What is so special about this filesystem tree is that the files and directories are not actually on the filesystems, instead each time we access them the Keybase client makes a call to the Keybase APIs to retrieve them. For us the implementation that made most sense is the one that leverages Keybase teams to store shared AWS keys. Our personal user accounts to not have AWS keys, instead the AWS keys we use belong to functional/system/bot accounts that the people on our team share. This is why it is so important for us to rotate all these keys often, and especially whenever someone leaves the team.

Now instead of distributing the AWS keys to Gitlab CI/CD variables we started to distribute them to AWS credentials files on Keybase teams. We can of course use these files from anywhere by setting the appropriate environment variables. All we had to make sure is that all our team members are on the right Keybase teams and that our bot (who runs the CI/CD pipelines for us) is also member of those teams. In reality we also had to make sure the bot running CI/CD pipelines could login to Keybase via the Keybase client and that the containers running our builds were able to properly interact with the Keybase filesystem via filesystem in userspace (FUSE). We did face some normal automation issues as we had to wait for Keybase client to be ready before we can access the team filesystem, as well as making sure that updates have been flushed out to Keybase servers before terminating the containers that run the pipeline jobs.

To complete the picture the same AWS keys also need to be stored in 1Password since that’s our source of truth for all our secrets. Luckily 1Password has the ‘op’ command line client that can talk to its API so we can automate that away as well.

With some scripting we are able to get the AWS keys out of the terraform state and push them to both the right aws-credentials file and the right 1Password Vault. The script basically does 2 things:

Asks Terraform for the AWS keys in the state; Every AWS key gets stored in a aws-credentials file and in 1Password.

When we now rotate keys (either via CI/CD pipeline or locally on a laptop) the same files are updated and almost instantly available to anyone that has access to the Keybase team (be it a developer or a bot running CI/CD pipelines).