I'm writing a serverless function on AWS Lambda.
On certain instances I need to use kms:GenerateDataKey* permissions.
What exactly is the purpose of this. I checked the AWS documentation but it is too cryptic. Can someone give a practical example of where this is used?
A Lambda function that requires kms:GenerateDataKey permission is most likely encrypting large amounts of data using a symmetric data key.
kms:GenerateDataKey is used to implement envelope encryption, which is the process of encrypting a key with another key. Symmetric key algorithms are faster and produce smaller ciphertexts than public key algorithms, whereas public key algorithms provide inherent separation of roles and easier key management. Envelope encryption combines the strengths of each strategy.
Envelope Encryption in AWS
Create a Customer Master Key in KMS. Even though a CMK can be used to encrypt data up to 4K in size, it is primarily used to encrypt/decrypt data encryption keys.
Generate a Data Encryption Key - Used to encrypt data by using symmetric encryption algorithms.
Encrypt the data key by using the CMK.
Store encrypted data and encrypted data key together.
When a user calls kms:GenerateDataKey, KMS generates a data key, encrypts it with the CMK and finally returns plaintext and encrypted data key pair back (steps 2 & 3 above).
The user is responsible for managing these keys. Plaintext data key is usually discarded immediately after encrypting data, whereas encrypted data key is stored together with encrypted data. Data encryption key must be decrypted by using kms:decrypt before decrypting data.
I'm not familiar with permission itself, but I found this in the documentation:
From Using Key Policies in AWS KMS - AWS Key Management Service:
kms:GenerateDataKey* – Allows key users to successfully request data encryption keys (data keys) to use for client-side encryption. Key users can choose to receive two copies of the data key—one in plaintext form and one that is encrypted with this CMK—or to receive only the encrypted form of the data key.
Related
I have recently been reading about Amazon KMS, including envelope encryption. The source I used for envelope encryption (Free Code Camp) seems to consider 4 levels of envelope encryption (data encryption key, key encryption key, KMS master key and root KMS master key), as shown in the figure below:
On the other hand, what I have read in aws only seems to consider two levels (KMS key or key encryption key and data encryption key), as shown in the figure below:
Am I missing something here? Is this disparity only apparent?
Thanks in advance for your answers!
What you see here are just different levels of abstraction.
The upper diagram shows the whole chain of keys involved in creating data keys for encryption/decryption. It peeks under the hood of the KeyId abstraction.
The docs are quite extensive as cryptography is famously littered with pitfalls and edge cases, but here's my not entirely accurate but useful enough mental model of the process:
Through heavily-audited and secure random magic hardware appliances (HSMs) can come up with keys that (almost, except for replication) never leave the appliance. Besides creating these keys, HSMs can also encrypt and decrypt data using these keys as well as create more keys.
KMS exposes these HSMs in some ways to you. It allows you to manage these keys through the KMS API and create an abstraction called a master Key (there are different versions...) identified by a KeyId. This master key is not necessarily a single key (key material is what AWS uses in the docs).
Rather there are multiple keys under the hood, but they are not directly exposed to the user. The underlying keys can be rotated according to a schedule. Effectively this key is versioned, but only the most recent version is used to encrypt new data; older versions are only used to decrypt data.
The other keys in the chain in the diagram are pretty much just implementation details that you don't need to know unless you plan to audit KMS in which case my mental model won't be enough. In practice, the lower diagram is much more important.
The lower diagram shows what KMS looks like from a client that uses KMS to manage encryption keys to encrypt data in another system. To use KMS in a system, you only need to know the identifier of a KMS Key (KeyId) and have the appropriate permissions.
The flow looks like this for encryption:
You make a GenerateDataKey API Call to KMS and pass in the KeyId of the key you wish to use. KMS will then return (among other things) a plain text data key and an encrypted data key.
You use the plain text data key to encrypt your data locally and store the encrypted data alongside the encrypted data key. Next, you delete the plain text data key from memory.
For decryption the flow is as follows:
You read the encrypted data key from the encrypted object and make a call to the Decrypt API passing in the encrypted data key. Assuming you have permission, KMS will return the plain text data key [1][2].
You use the plain text data key to decrypt your encrypted data.
From the client's perspective, all you need to know about the keys that KMS uses is the KeyId (by default, there are more advanced features like encryption contexts).
[1] for symmetric encryption, it can infer the correct KeyId from the metadata, but as a best practice, you should send the KeyId as well.
[2] AWS infers the correct key-material for decryption from metadata in the encrypted blob.
I am expecting very high traffic on one of my services, and I would like to add encryption for a new feature. I know KMS makes an API call each encrypt/decrypt call, but is it possible to use KMS for key management and cache the keys in memory to encrypt/decrypt locally without additional API calls?
The KMS key never leaves its hardware. That's it.
By default the KMS is intended for the envelope encryption. There is a data encryption key and KMS is used to encrypt the data key.
You can call the KMS to generate a random data key along the its encrypted value and then use the data-key to encrypt the data itself.
If you are encrypting for the same system (data are encrypted for the same target ), you may reuse the same data key and use a unique IV to encrypt multiple messages.
Edit: I'd suggest using the AWS Encryption SDK a bit helping the developers to do it properly
How exactly does encryption key rotation work? I understand it's a very good practice to continuously rotate your encryption keys for security purposes, but rotating a key would require too much work.
Case:
Let's just say I have a database storing 30GB of data, and we're using an internal key to encrypt data at rest, and I plan to rotate my keys every month.
Questions:
Does that mean all my data will be decrypted by the old key and re-encrypted by the new one every month?
The whole encryption-decryption would take a lot of time and compute resources.
If my DB ( or any encrypted dataset ) scales tomorrow, does that mean the same process would duplicate when my key rotates? This does not look like a scalable solution.
Other Details:
I've also seen AWS KMS rotates it's keys if we've selected the rotation option. How does AWS manage to rotate it's keys and all encrypted data for all the underlying services?
You need to familiarize yourself with Envelope Encryption. Each time you want to encrypt data in AWS, you first generate a unique data-key. You then encrypt your data with this key. This key is not the key that is rotated!
Then you take this key, and you encrypt it with a key from KMS. Now if you want to decrypt this data, you must first get the decrypted data key, and to decrypt this data key, you will need the KMS key.
Now if you want to rotate the key, you don't need to re-encrypt all the data, instead you need to decrypt the data key using your key to be rotated from KMS, and then get a new key, and re-encrypt the unencrypted data key. That way you don't need to re-encrypt all the data.
Here are two important links that can help you understand envelop encryption and key rotation in AWS.
https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html
https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html
I have quoted some important concepts from the above links:
Envelop encryption:
When you encrypt your data, your data is protected, but you have to protect your encryption key. One strategy is to encrypt it. Envelope encryption is the practice of encrypting plaintext data with a data key, and then encrypting the data key under another key.
Customer Master Keys
CMKs are created in AWS KMS. Symmetric CMKs and the private keys of asymmetric CMKs never leave AWS KMS unencrypted. This strategy differs from data keys. AWS KMS does not store, manage, or track your data keys. You must use them outside of AWS KMS.
Data Keys
Data keys are encryption keys that you can use to encrypt data, including large amounts of data and other data encryption keys. You can use AWS KMS customer master keys (CMKs) to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys. You must use and manage data keys outside of AWS KMS.
Key rotation
When you enable automatic key rotation for a customer managed CMK, AWS KMS generates new cryptographic material for the CMK every year. AWS KMS also saves the CMK's older cryptographic material in perpetuity so it can be used to decrypt data that it encrypted. AWS KMS does not delete any rotated key material until you delete the CMK.
An important concept in Key rotation is the HSM backing key(HBK):
(https://docs.aws.amazon.com/kms/latest/cryptographic-details/key-hierarchy.html)
Within the hierarchy of a specific CMK, the HBK can be thought of as a version of the CMK. When you want to rotate the CMK through AWS KMS, a new HBK is created and associated with the CMK as the active HBK for the CMK. The older HBKs are preserved and can be used to decrypt and verify previously protected data. But only the active cryptographic key can be used to protect new information.
Does that mean all my data will be decrypted by the old key and re-encrypted by the new one every month?
As already answered, the simple answer is no. But the previous answers miss the purpose of the key rotation
The reason behind the key rotation is limit amout of data encrypted by a single key.
How does AWS manage to rotate it's keys and all encrypted data for all the underlying services?
The basic idea (at least in KMS) is, that the KMS key is not a single key, but it is a set of keys, which the last one is the current one. You can imagine that as "key versioning". After each key rotation the current key is saved so you can still decrypt the previously encrypted ciphertext (data key - as mentioned in other answers). I believe in the KMS this whole set is hidden, but Azure KeyValt shows the whole set as key version.
The whole encryption-decryption would take a lot of time and compute resources
No, you don't need to do anything. AWS manages the "key versions" for you. Even if you'd re-encrypt the envelope keys as suggested in another answer, then you would actually fail to fulfill the purpose of the key rotation.
The ciphertext generated by the KSM must contain some identification of the key version too, so the KMS is able to decrypt the ciphertext even after the key is rotated.
Here is the encryption flow of Server-side encryption with S3 managed keys (SSE-S3)
(this picture is from Oreilly)
The client selects their object(s) to upload to S3 and indicates the encryption mechanism of SSE-S3 during this process.
S3 then takes control of the object and encrypts it with a plaintext data key that's generated by S3. The result is an encrypted version of the object, which is then stored within your chosen S3 bucket.
The plaintext data key that used to encrypt the object is then encrypted with an S3 master key, resulting in an encrypted version of the key. This now encrypted key is also stored on S3 and has an association to the encrypted data object. Finally, the plaintext data key is removed from memory in S3.
As you can see, SSE-S3 use a unique data key to encrypt the object, rather than the master key.
I heard a lot that SSE-KMS is envelope encryption, my question is:
Is SSE-S3 envelop encryption too?
Reference
Stack Overflow: Does AWS KMS use envelope encryption?
AWS KMS Envelope Encryption
AWS Encryption-At-Rest(server-side encryption) uses Envelope Encryption irrespective of what key is used. Not just for S3, for every service at AWS. Here is a nice blog about it.
Only difference is who manages data key and encryption key.
SSE-S3 Both keys are managed by AWS
SSE-KMS We manage CMK, where as AWS manages data key. Gives us more control on rotation, access, etc.
SSE-C We manage both encryption key and data key(we can choose to not use data key but use same key), we provide data key and algorithm, AWS encrypts for us and we need to keep track of which key is used for which object.
I am reading AWS encrypt cli document from https://docs.aws.amazon.com/cli/latest/reference/kms/encrypt.html and https://docs.aws.amazon.com/cli/latest/reference/kms/decrypt.html. I found that I am able to encrypt/decrypt without creating a data key. When I read https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html, it says that I need to use KMS CMK to generate a data key which is used to encrypt my data.
So I am confused about whether I need a data key at all?
CMK is designed to encrypt/decrypt the data keys. Therefore, there is a limit of 4 KB on the amount of plaintext that can be encrypted in a direct call to the encrypt function. You can easily test this by passing in message larger than 4 KB.
These operations are designed to encrypt and decrypt data keys. They use an AWS KMS customer master key (CMK) in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. Although you might use them to encrypt small amounts of data, such as a password or RSA key, they are not designed to encrypt application data.
You are likely using a default CMK that was created by another AWS service that uses KMS encryption.
Of course all encryption and decryption operations require a key. If you did not explicitly create one for your application, then you are using the current default key.
Ensure that KMS Customer Master Keys (CMKs) are used by your AWS services and resources instead of default KMS keys, in order to have full control over data encryption/decryption process and meet compliance requirements. A KMS default master key is used by an AWS service such as RDS, EBS, Lambda, Elastic Transcoder, Redshift, SES, SQS, CloudWatch, EFS, S3 or Workspaces when no other key is defined to encrypt a resource for that service. The default key cannot be modified to ensure its availability, durability and security. On the other side, a KMS Customer Master Key (CMK) provides the ability to create, rotate, disable, enable and audit the encryption key used to protect the data.
See https://www.cloudconformity.com/knowledge-base/aws/KMS/default-key-usage.html