<p dir="auto">Is it possible to encrypt snapshots before it leaves the node ?<br>
So that data is encrypted in flight ? May be a config variable to<br>
set up password for encryption and decryption ?</p>
<p dir="auto">This concerns the encryption of snapshot data before it leaves the nodes.</p>
<p dir="auto">We have 3 types of cloud snapshot repository types: Google Cloud Storage, Azure Storage and Amazon S3. <a href="https://aws.amazon.com/articles/client-side-data-encryption-with-the-aws-sdk-for-java-and-amazon-s3/" rel="nofollow">Amazon</a> and <a href="https://docs.microsoft.com/en-us/azure/storage/common/storage-client-side-encryption-java" rel="nofollow">Azure</a> support client side encryption for their java clients, but <a href="https://cloud.google.com/storage/docs/encryption/client-side-keys" rel="nofollow">Google</a> does <strong>not</strong>.</p>
<p dir="auto">Amazon and Azure, which support client side encryption, allow the keys to be managed by the client (us) or by their <em>Key Management Service</em> (Vault-like). They both use the <a href="https://cloud.google.com/kms/docs/envelope-encryption" rel="nofollow">Envelope Encryption</a> method; each blob is individually AES-256 encrypted with a randomly generated (locally) key, and this key (Data/Content Encryption Key) is also encrypted with another <em>Master Key</em> (locally or by the <em>Vault</em> service) and then stored alongside the blob in its metadata. The envelope encryption facilitates <em>Master Key</em> rotation because only the small <em>(D/C)EK</em> key has to be re-encrypted, rather than the complete blob.</p>
<p dir="auto">On the ES side we discussed on having a <em>single fixed URN key handler</em> at the repository settings level.<br>
This URN identifies the <em>Master Key</em>; for example this could point to a key on the Amazon Vault Service or the Keys on each node's keystore. In this alternative it is not possible to rotate the keys via the repository API (it might be possible to do it outside ES, which is obviously preferable, but see below).</p>
<p dir="auto">I believe this is the rough picture of the puzzle that we need to put together.</p>
<p dir="auto">We oscillated between implementation alternatives, and I will lay out the one which I think is favorable. Whatever solution we initially implement, given that the <em>Master Key</em> identifier is an URN we can multiplex multiple implementations for the same repository type.</p>
<p dir="auto">We mirror the <em>Envelope Encryption</em> algorithm, employed by Amazon and Azure, at the <code class="notranslate">BlobContainer</code> level. The key is stored on each node's keystore (and is pointed to by the repository level URN reference).</p>
<p dir="auto">Advantages:</p>
<ul dir="auto">
<li>Implement once for all cloud repository types and it will even work for the file system repository type!</li>
<li>Testing! We can have unit tests for the base implementation of the ~<code class="notranslate">EncryptedBlobContainer</code> and end-to-end integration tests in only one of the implementation (Amazon or FS), where we can decrypt the data on the service fixture.</li>
</ul>
<p dir="auto">Disadvantages:</p>
<ul dir="auto">
<li>Duplicates code in one SDK</li>
<li>We're in the open with Key Rotation. We might need to implement our own cmd line tool to rotate keys (download objects metadata, decrypt the key and re-encrypt it). Tool will be "easy" to implement.</li>
<li>Does not support <em>Vault</em> keys.</li>
</ul>
<p dir="auto">In the opposite corner, there could be this alternative:<br>
We use the AWS cloud library facility to implement it only for the S3 repository type. The key is stored either on the node's keystore or on the AWS Key Management Service.</p>
<p dir="auto">Advantages:</p>
<ul dir="auto">
<li>Easiest to implement</li>
<li>Supports AWS's Vault Service</li>
<li>We <em>might</em> have support for key rotation, using amazon's <a href="https://aws.amazon.com/blogs/security/how-to-encrypt-and-decrypt-your-data-with-the-aws-encryption-cli/" rel="nofollow">command line tool</a></li>
</ul>
<p dir="auto">Disadvantages:</p>
<ul dir="auto">
<li>Only S3 repository type is supported</li>
<li>Testing. We either mock the client and check that the code indeed calls the "crypto" APIs or we do an end-to-end integration test, where we decrypt the data on the fixture. Either way we kinda "test the library" rather than our code. This is pointless and brittle, but we need testing because the risks are too great.</li>
</ul>
<p dir="auto">Relates <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="370145189" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/34454" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/34454/hovercard" href="https://github.com/elastic/elasticsearch/issues/34454">#34454</a> <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="424973630" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/40416" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/40416/hovercard" href="https://github.com/elastic/elasticsearch/pull/40416">#40416</a></p>
<hr>
<p dir="auto"><strong>EDITED 28.01.2021 Backlog:</strong></p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Add the new API that changes the password for a given encrypted repository.<br>
The encrypted repository must already be configured with the correct password. The API iterates over all the associated<br>
wrapped DEKs (contained inside a dedicated blob container under the repository's base path), and proceeds to unwrap and re-wrap all the DEKs with the new password.<br>
Finally, the old DEKs are removed, so that the old password cannot be used any longer.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> implement searchable and encrypted repositories.<br>
This mainly requires implementing the AbstractBlobContainer#readBlob interface. This is slightly problematic because the association id between an encrypted blob and its DEK is prepended at the beginning of the blob, so that decryption at an internal position currently requires a seek at the beginning. Double check that the definition structure is reasonable (ie. is it searchable and encrypted or vice-versa, ping David about this).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> ensure compressed &amp; encrypted repositories work</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> investigate metered and encrypted repositories</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> thoroughly test failure scenarios where IOExceptions are thrown. Generally speaking, (although nor really true in practice) reads and writes contain<br>
two operations that can fail independently. Make sure testing covers this.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> create benchmarks (distributed team is working on something that measures the throughput of a repository)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> permit (and test) encrypted HDFS repositories (it should work)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> double check that AbstractBlobContainer#blobExists and EncryptedBlobContainer#listBlobs/listBlobsByPrefix for EncryptedBlobContainer can return <code class="notranslate">true</code> and then reading to fail because of decryption problems</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> double check that we're not relying on system encoding (that strings are always written UTF-8 encoded, and reads are always decoded with the same UTF-8)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> ensure that there's no problem that EncryptedBlobContainer#writeBlobAtomic is not atomic (in general, it cann't be because a write might also generate and write the DEK, so there are two operations that can fail independently)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> ensure that it's alright for EncryptedBlobContainer#listBlobs/listBlobsByPrefix to return the encrypted blob size (which is larger), instead of the expected blob size of the decrypted blob</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> think about possibly renaming "password name" to "password label"</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> investigate repository password situation on cloud.<br>
On-premise repository passwords are cached in memory when the node starts, usually requiring a node restart when configuring a new snapshot repository. The security settings implementation on cloud is different, so that maybe we can read the repository passwords immediately after they've been added, changed, without requiring a restart.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> repository password min-length limit (an encrypted repository with a short password )</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> make KDF parameters configurable</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> investigate the naming of the delegating and delegated repositories (they are the same currently, is this a problem?)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> make crypto provider selectable for operations on the client side-encrypted repo</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> test that encrypted repos can share the bucker (but different base path, otherwise we already test that the passwords must be the same) and can also share the repository client</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Revisit definition of password name in repository settings (see: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="578762432" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/53352" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/53352/hovercard?comment_id=409314119&amp;comment_type=review_comment" href="https://github.com/elastic/elasticsearch/pull/53352#discussion_r409314119">#53352 (comment)</a> )</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Settle on the specification for encrypted and searchable snapshots (ping David about it)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Investigate if versioning if individual encrypted blobs is necessary <a href="https://github.com/elastic/elasticsearch/pull/53352/files#r444383568">https://github.com/elastic/elasticsearch/pull/53352/files#r444383568</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Investigate if we can guarantee that DEKs do not change inside a given shard</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> Report encryption stats (from <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="578762432" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/53352" data-hovercard-type="pull_request" data-hovercard-url="/elastic/elasticsearch/pull/53352/hovercard?comment_id=432261031&amp;comment_type=review_comment" href="https://github.com/elastic/elasticsearch/pull/53352#discussion_r432261031">#53352 (comment)</a>)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> test FIPS negative behaviour (that a short password doesn't crash the node or something).</li>
</ul>