I have a AWS Lambda function that needs to send messages to an encrypted SQS queue. At one point I got it working that my AWS Lambda function could send messages to this encrypted SQS queue. After I got it working I made some adjustments to the permissions so they are more specific. Then it suddenly stopped working even after I reverted back to the really broad permissions.
Now I always get this following error:
KMS.AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access.
My CMK has the following policy and ONLY allows in it:
{
"Sid": "Allow specific lambda to use this key",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey*"
],
"Resource": "arn:aws:lambda:XXXXXX:function:XXXXXX"
}
I redacted the resource but I triple checked the ARN and it is correct.
My Lambda function has the following policy attached to it:
{
"Action": "sqs:*",
"Resource": "*",
"Effect": "Allow"
},
{
"Action": "kms:*",
"Resource": "*",
"Effect": "Allow"
},
Shouldn't this be enough for my Lambda to send a message to this SQS queue? Like I mentioned above at one point this worked than I tried to restrict the policies on my Lambda and then it stopped working even after I changed it back.
What am I missing here? I'm not sure what else I could change and why it worked and then it stopped working...
Edit
Permissions that seemed to work and now doesn't:
CMK policy
{
"Sid": "Allow AWS Lambda to use this key",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey*"
],
"Resource": "*"
}
Lambda Policy
{
"Action": "sqs:*",
"Resource": "*",
"Effect": "Allow"
},
{
"Action": "kms:*",
"Resource": "*",
"Effect": "Allow"
}
Edit 2
There has to be something in play that I don't understand because this doesn't behave predictable...
I made the following changes with CDK:
First change:
IAM Statement Changes
┌───┬──────────┬────────┬─────────────────────┬──────────────────────────────────────────────────────────────────┬───────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────┼────────┼─────────────────────┼──────────────────────────────────────────────────────────────────┼───────────┤
│ + │ * │ Allow │ kms:GenerateDataKey │ AWS:${lambda---------------------------------------/ServiceRole} │ │
└───┴──────────┴────────┴─────────────────────┴──────────────────────────────────────────────────────────────────┴───────────┘
Result: I can send messages to the SQS queue again
Second change:
IAM Statement Changes
┌───┬──────────┬────────┬─────────────────────┬──────────────────────────────────────────────────────────────────┬───────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────┼────────┼─────────────────────┼──────────────────────────────────────────────────────────────────┼───────────┤
│ - │ * │ Allow │ kms:GenerateDataKey │ AWS:${lambda---------------------------------------/ServiceRole} │ │
├───┼──────────┼────────┼─────────────────────┼──────────────────────────────────────────────────────────────────┼───────────┤
│ + │ * │ Allow │ kms:* │ AWS:${lambda---------------------------------------/ServiceRole} │ │
└───┴──────────┴────────┴─────────────────────┴──────────────────────────────────────────────────────────────────┴───────────┘
Result: I can't send messages to the SQS queue anymore ?!? Why?
Third change:
IAM Statement Changes
┌───┬──────────┬────────┬─────────────────────┬──────────────────────────────────────────────────────────────────┬───────────┐
│ │ Resource │ Effect │ Action │ Principal │ Condition │
├───┼──────────┼────────┼─────────────────────┼──────────────────────────────────────────────────────────────────┼───────────┤
│ - │ * │ Allow │ kms:* │ AWS:${lambda---------------------------------------/ServiceRole} │ │
├───┼──────────┼────────┼─────────────────────┼──────────────────────────────────────────────────────────────────┼───────────┤
│ + │ * │ Allow │ kms:GenerateDataKey │ AWS:${lambda---------------------------------------/ServiceRole} │ │
└───┴──────────┴────────┴─────────────────────┴──────────────────────────────────────────────────────────────────┴───────────┘
Result: I still can't send messages to the SQS queue anymore ?!? What is happening here? It is the same policy state as with the first change....
Related
My first attempt at doing this resulted in a circular reference:
resource "aws_kms_key" "cloudtrails-key" {
description = "KMS Master Key for trails logs"
key_usage = "ENCRYPT_DECRYPT"
customer_master_key_spec = "SYMMETRIC_DEFAULT"
deletion_window_in_days = 30
is_enabled = true
enable_key_rotation = false
policy = data.aws_iam_policy_document.cloudtrails-key-policy.json
multi_region = true
tags = {
ManagedBy = "terraform"
Name = "${var.project_name}-${var.environment}-${var.aws_account_id}-cloudtrails-key"
Environment = var.environment
Usage = "monitoring"
}
}
data "aws_iam_policy_document" "cloudtrails-key-policy" {
statement {
principals {
type = "Service"
identifiers = [ "cloudtrail.amazonaws.com" ]
}
actions = [
"kms:CallerAccount",
"kms:EncryptionAlgorithm",
"kms:EncryptionContext:*",
"kms:EncryptionContextKeys",
"kms:RequestAlias",
"kms:ViaService"
]
resources = [ aws_kms_key.cloudtrails-key.arn ]
condition {
test = "ForAnyValue:StringLike"
variable = "aws:SourceArn"
values = toset([for k, v in var.all_account_ids : "arn:aws:cloudtrail:*:${v}:trail/full-cloudtrails"])
}
}
}
As I couldn't determine how to remove the circular reference, I decided to try using grants instead:
resource "aws_kms_grant" "cloudtrails-key-grant" {
depends_on = [ aws_kms_key.cloudtrails-key ]
for_each = var.all_account_ids
name = "${each.key}-cloudtrails-key-grant"
key_id = aws_kms_key.cloudtrails-key.key_id
grantee_principal = "arn:aws:cloudtrail::${each.value}:trail/full-cloudtrails"
operations = [ "Encrypt" ]
}
var.all_account_ids refers to a map of account names and account IDs.
I then updated the aws_kms_key block to remove the policy argument. However, this resulted instead in the following output on apply:
Terraform v1.3.8
on linux_amd64
aws_kms_grant.cloudtrails-key-grant["users"]: Creating...
aws_kms_grant.cloudtrails-key-grant["root"]: Creating...
aws_kms_grant.cloudtrails-key-grant["dev"]: Creating...
aws_kms_grant.cloudtrails-key-grant["shared"]: Creating...
aws_kms_grant.cloudtrails-key-grant["monitoring"]: Creating...
aws_cloudtrail.full-cloudtrails: Modifying... [id=full-cloudtrails]
...
aws_kms_grant.cloudtrails-key-grant["users"]: Still creating... [2m50s elapsed]
aws_kms_grant.cloudtrails-key-grant["monitoring"]: Still creating... [2m50s elapsed]
╷
│ Error: updating CloudTrail Trail (full-cloudtrails): InsufficientEncryptionPolicyException: Insufficient permissions to access S3 bucket full-cloudtrails or KMS key arn:aws:kms:us-east-1:703617067304:key/mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b.
│
│ with aws_cloudtrail.full-cloudtrails,
│ on monitoring.tf line 1, in resource "aws_cloudtrail" "full-cloudtrails":
│ 1: resource "aws_cloudtrail" "full-cloudtrails" {
│
╵
╷
│ Error: creating KMS Grant for Key (mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b): InvalidArnException: ARN is not valid: arn:aws:cloudtrail:us-east-1:323924548037:trail/full-cloudtrails
│
│ with aws_kms_grant.cloudtrails-key-grant["root"],
│ on security.tf line 19, in resource "aws_kms_grant" "cloudtrails-key-grant":
│ 19: resource "aws_kms_grant" "cloudtrails-key-grant" {
│
╵
╷
│ Error: creating KMS Grant for Key (mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b): InvalidArnException: ARN is not valid: arn:aws:cloudtrail:us-east-1:739800733271:trail/full-cloudtrails
│
│ with aws_kms_grant.cloudtrails-key-grant["shared"],
│ on security.tf line 19, in resource "aws_kms_grant" "cloudtrails-key-grant":
│ 19: resource "aws_kms_grant" "cloudtrails-key-grant" {
│
╵
╷
│ Error: creating KMS Grant for Key (mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b): InvalidArnException: ARN is not valid: arn:aws:cloudtrail:us-east-1:703617067304:trail/full-cloudtrails
│
│ with aws_kms_grant.cloudtrails-key-grant["monitoring"],
│ on security.tf line 19, in resource "aws_kms_grant" "cloudtrails-key-grant":
│ 19: resource "aws_kms_grant" "cloudtrails-key-grant" {
│
╵
╷
│ Error: creating KMS Grant for Key (mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b): InvalidArnException: ARN is not valid: arn:aws:cloudtrail:us-east-1:457407591303:trail/full-cloudtrails
│
│ with aws_kms_grant.cloudtrails-key-grant["dev"],
│ on security.tf line 19, in resource "aws_kms_grant" "cloudtrails-key-grant":
│ 19: resource "aws_kms_grant" "cloudtrails-key-grant" {
│
╵
╷
│ Error: creating KMS Grant for Key (mrk-c0a4616ba4f74ae9b5ac4ecd1f96e95b): InvalidArnException: ARN is not valid: arn:aws:cloudtrail:us-east-1:158889104371:trail/full-cloudtrails
│
│ with aws_kms_grant.cloudtrails-key-grant["users"],
│ on security.tf line 19, in resource "aws_kms_grant" "cloudtrails-key-grant":
│ 19: resource "aws_kms_grant" "cloudtrails-key-grant" {
│
╵
I tried adding the region (though I would prefer multi-region), taking off the trail name after the slash, using wildcards for region, but everything resulted in the above invalid ARN exception.
I was curious to know if anyone has any working recipes for granting cloudtrails encryption access to a KMS key, as I'm unsure what I'm doing wrong with either method (policy or grant)
The solution for the cyclical reference was to broaden the resource scope to all keys, and not just the one key. This was done on all examples in AWS documentation. I also had to add the default key policy offered by AWS otherwise I got a warning that I would be locking myself out.
The Default Key Policy can be found documented here. The specific key policy required for CloudTrail is found here. Using these two resources, the final working configuration is below:
data "aws_iam_policy_document" "cloudtrail-key-policy" {
statement {
principals {
type = "Service"
identifiers = [ "cloudtrail.amazonaws.com" ]
}
actions = [
"kms:DescribeKey",
"kms:GenerateDataKey*"
]
resources = [ "*" ]
condition {
test = "ForAnyValue:StringLike"
variable = "kms:EncryptionContext:aws:cloudtrail:arn"
values = toset([for k, v in var.all_account_ids : "arn:aws:cloudtrail:*:${v}:trail/${k}-full-cloudtrail"])
}
condition {
test = "ForAnyValue:StringLike"
variable = "aws:SourceArn"
values = toset([for k, v in var.all_account_ids : "arn:aws:cloudtrail:*:${v}:trail/${k}-full-cloudtrail"])
}
}
statement {
principals {
type = "AWS"
identifiers = [ "arn:aws:iam::${var.aws_account_id}:role/terraform-${var.environment}" ]
}
actions = [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
]
resources = [ "*" ]
}
statement {
principals {
type = "AWS"
identifiers = [ "arn:aws:iam::${var.aws_account_id}:root" ]
}
actions = [ "kms:*" ]
resources = [ "*" ]
}
}
Additionally, no grants were needed as we're doing this for a service. According to the documentation, service ARNs are excluded:
The grantee principal can be any AWS principal, including an AWS
account (root), an IAM user, an IAM role, a federated role or user, or
an assumed role user. The grantee principal can be in the same account
as the KMS key or a different account. However, the grantee principal
cannot be a service principal, an IAM group, or an AWS organization
The above text can be found here
Hopefully this helps future readers with the same issue.
EDIT: Updated with final working config
I am creating a lambda function that will send emails (nothing new). I am having issues with the policy that I am creating. TFSec busted me with a the Resource meta-attribute and indicated that "*" isn't allowed. So all I am trying to do is limit my policy to my lambda function. I figured I could dynamically create the ARN as i have done in the past. Well, the only way I found that it works is leveraging the "condition" meta-attribute BUT..., when I used the condition, I get this error:
│ Error: error creating Lambda Function (1):
InvalidParameterValueException: The provided execution role does not
have permissions to call CreateNetworkInterface on EC2
│ { │ RespMetadata: { │ StatusCode: 400, │ RequestID:
"3b8fc1dd-21d8-xxxx-xxx-xxxxxxx" │ }, │ Message_: "The provided
execution role does not have permissions to call
CreateNetworkInterface on EC2", │ Type: "User" │ }
Whats killing me is that the permission is set in my policy, see below:
resource "aws_iam_role_policy" "lambda_sendmail_policy" {
name = "lambda-sendmail"
role = aws_iam_role.lambda_sendmail_role.id
policy = jsonencode({
"Version" : "2012-10-17",
"Statement" : [
{
"Sid" : "Stmt163546987325",
"Effect" : "Allow",
"Action" : [
"ec2:DescribeInstances",
"ec2:CreateNetworkInterface",
"ec2:AttachNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface",
"autoscaling:CompleteLifecycleAction"
],
"Resource" : "*",
"Condition" : {
"StringEquals" : {
"lambda:FunctionArn" : "arn:aws:lambda:us-east-1:${local.accountid}:function:${var.appenv}-${var.lambda_appname}"
}
}
},
{
"Effect" : "Allow",
"Action" : [
"logs:PutLogEvents",
"logs:CreateLogStream"
],
"Resource" : [
"arn:aws:logs:us-east-1:${local.accountid}:log-group:/aws/lambda/*"
]
},
{
"Effect" : "Allow",
"Action" : [
"xray:PutTraceSegments",
"xray:PutTelemetryRecords"
],
"Resource" : "*"
}
]
})
}
Clearly the permission is there but I am passing the Lambda ARN incorrectly need some help here.
If I remove the condition it works but TFSEC catches me telling me that I need to specify the resource.
I am not sure if Lambda is supported by the condition attribute but any help here woudl be great.
You can't put this condition for the ec2 actions mentioned:
"Condition" : {
"StringEquals" : {
"lambda:FunctionArn" : "arn:aws:lambda:us-east-1:${local.accountid}:function:${var.appenv}-${var.lambda_appname}"
}
}
The condition to use for most of those ec2:* actions is...
"Condition": {
"ArnLikeIfExists": {
"ec2:Vpc": "your_vpc_arn"
}
}
This block limits the permission to the vpc shared by the function and fails to true for certain lambda actions like creation/update where the api calls don't include the vpc arn. This is about as tight as you can get this policy.
I'd move autoscaling to it's own block also, because you can definitely tighten that up.
I am trying to create a role in Cloud9 with Terraform.
resource "aws_iam_role" "firehose_role" {
name = "firehose_test_role"
assume_role_policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "sts:AssumeRole",
"Principal": {
"Service": "firehose.amazonaws.com"
},
"Effect": "Allow",
"Sid": ""
}
]
}
EOF
}
Error:
failed creating IAM Role (firehose_test_role): InvalidClientTokenId: The security token included in the request is invalid │ status code: 403, request id: 6ea2ca59-70ac-47b8-b83c-6547ad98aec1
And when I disable the "AWS managed temporary credentials", this is what I get:
Error: Invalid provider configuration
│
│ Provider "registry.terraform.io/hashicorp/aws" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described
│ in the provider documentation.
│
╵
╷
│ Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found.
│
│ Please see https://registry.terraform.io/providers/hashicorp/aws
│ for more information about providing credentials.
│
│ Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, http response error StatusCode: 404, request to EC2 IMDS failed
│
│
│ with provider["registry.terraform.io/hashicorp/aws"],
│ on <empty> line 0:
│ (source code not available)
So, what can I do here to make it work on Cloud9?
I am getting this error messages when i run terragrunt apply.I need to be able to specify each element when creating SQS and DeadLetterQueue with the policy.I am unsure how to fix this issue.I want to find a way to reference individual DLQ directly with corresponding SQS to create the required resources
resource "aws_sqs_queue_policy" "Cloudtrail_SQS_Policy" {
for_each = toset(var.sqs_queue_names)
queue_url = aws_sqs_queue.CloudTrail_SQS[each.key].id
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "AllowSQSInvocation",
"Effect": "Allow",
"Principal": {"AWS":"*"},
"Action": "sqs:*",
"Resource": "${aws_sqs_queue.CloudTrail_SQS[each.key].arn}"
resource "aws_sqs_queue_policy" "CloudTrail_SQS_DLQ"{
for_each = toset(var.dead_queue_names)
queue_url = aws_sqs_queue.CloudTrail_SQS_DLQ[each.key].id
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "DLQ Policy",
"Effect": "Allow",
"Principal": {"AWS":"*"},
"Action": "sqs:*",
"Resource": "${aws_sqs_queue.CloudTrail_SQS_DLQ[each.key].arn}
Error Messages:
Error: Invalid index
│
│ on iam.tf line 3, in resource "aws_sqs_queue_policy" "Cloudtrail_SQS_Policy":
│ 3: queue_url = aws_sqs_queue.CloudTrail_SQS[each.key].id
│ ├────────────────
│ │ aws_sqs_queue.CloudTrail_SQS is object with 2 attributes
│ │ each.key is "CloudTrail_SQS_Management_Event"
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│ on iam.tf line 15, in resource "aws_sqs_queue_policy" "Cloudtrail_SQS_Policy":
│ 15: "Resource": "${aws_sqs_queue.CloudTrail_SQS[each.key].arn}",
│ ├────────────────
│ │ aws_sqs_queue.CloudTrail_SQS is object with 2 attributes
│ │ each.key is "CloudTrail_SQS_Data_Event"
│
│ The given key does not identify an element in this collection value.
╵
╷
│ Error: Invalid index
│
│ on iam.tf line 15, in resource "aws_sqs_queue_policy" "Cloudtrail_SQS_Policy":
│ 15: "Resource": "${aws_sqs_queue.CloudTrail_SQS[each.key].arn}",
│ ├────────────────
│ │ aws_sqs_queue.CloudTrail_SQS is object with 2 attributes
│ │ each.key is "CloudTrail_SQS_Management_Event"
│
│ The given key does not identify an element in this collection value.
I refactored your entire code, as any changes will cascade to more errors. In your case, the best way to deal with your requirement is to use map, not list or set. You don't have to make your input variables a map if you don't want, though it would be easier. But keeping your input variables as they are, you can create local.sqs_dlq_map and use that:
locals {
sqs_dlq_map = zipmap(var.sqs_queue_names, var.dead_queue_names)
}
resource "aws_sqs_queue" "CloudTrail_SQS"{
for_each = local.sqs_dlq_map
name = each.key
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.CloudTrail_SQS_DLQ[each.key].arn
maxReceiveCount = 2 #var.max_receive_count
})
#tags = var.default_tags
}
resource "aws_sqs_queue" "CloudTrail_SQS_DLQ"{
for_each = local.sqs_dlq_map
name = each.value
#tags = var.default_tags
}
resource "aws_sqs_queue_policy" "Cloudtrail_SQS_Policy" {
for_each = local.sqs_dlq_map
queue_url = aws_sqs_queue.CloudTrail_SQS[each.key].id
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "AllowSQSInvocation",
"Effect": "Allow",
"Principal": {"AWS":"*"},
"Action": "sqs:*",
"Resource": "${aws_sqs_queue.CloudTrail_SQS[each.key].arn}"
}]
}
POLICY
}
resource "aws_sqs_queue_policy" "CloudTrail_SQS_DLQ"{
for_each = local.sqs_dlq_map
queue_url = aws_sqs_queue.CloudTrail_SQS_DLQ[each.key].id
policy = <<POLICY
{
"Version": "2012-10-17",
"Id": "sqspolicy",
"Statement": [
{
"Sid": "DLQ Policy",
"Effect": "Allow",
"Principal": {"AWS":"*"},
"Action": "sqs:*",
"Resource": "${aws_sqs_queue.CloudTrail_SQS_DLQ[each.key].arn}"
}]
}
POLICY
}
Ive been able to test this role by assuming it using the CLI with the given the profile. However, I still get this error on apply:
│ Error: error configuring Terraform AWS Provider: IAM Role (arn:aws:iam::<omitted>:role/<omitted>) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│ * The credentials used in order to assume the role are invalid
│ * The credentials do not have appropriate permission to assume the role
│ * The role ARN is not valid
│
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│ For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│
│
│ with provider["registry.terraform.io/hashicorp/aws"],
│ on providers.tf line 3, in provider "aws":
│ 3: provider "aws" {
│
The provider configuration looks like so:
provider "aws" {
region = var.aws_region
profile = var.aws_profile
assume_role {
role_arn = var.assume_role_arn
session_name = "assume-role-${timestamp()}"
}
}
Assuming all variables are correct - what am I doing wrong?
I was able to re-create this in a simple repositiory:
https://github.com/SparkPost/tf-recreate-assume-role-bug
You'll obviously have to create a correct assume role setup with permissions both in the assume-role policy of the role and the policy of the user/role doing the assuming.
Edit:
It's been requested, so here's the trust policy of the assumed role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::<omitted>:user/<my username>"
]
},
"Action": "sts:AssumeRole"
}
]
}
Additionally, it should be noted that I am an admin user with permission to assume all roles.
Here's the command I used to test the assume role from the CLI:
aws sts --profile <profile used in module> assume-role --role-arn <role arn from error message> --role-session-name test
This returned successfully.
Your timestamp() will return illegal characters for session name. You have to format it to have only good characters, e.g.:
session_name = "assume-role-${formatdate("MMM-DD-YYYY", timestamp())}"