I have created a aws_iam_policy_document with ssm and ec2 actions :
data "aws_iam_policy_document" "AdminEc2Actions" {
statement {
effect = "Allow"
sid = "sid1"
actions = [
"ssm:SendCommand",
"ec2:Start*",
"ssm:StartSession"
]
resources = [
"*"
]
}
}
# Create IAM policy for ec2
resource "aws_iam_policy" "AdminEc2Actions" {
name = "test-AdminEc2Actions-policies"
description = "ec2 policies"
path = "/"
policy = data.aws_iam_policy_document.AdminEc2Actions.json
}
# Attaches customer managed IAM poloicy to an IAM admins group
resource "aws_iam_group_policy_attachment" "AdminEc2Actions" {
group = aws_iam_group.groups[0].name
policy_arn = aws_iam_policy.AdminEc2Actions.arn
}
But, for some reason, any changes in actions sometimes are taken into account sometimes are ignored. eg. when I run this command, I got :
[ssm-user#ip-xxx.yy.zz.1]$ aws s3 mv /tmp/file* s3://test-dent-backup1/daily/
move failed: ../../tmp/file1.txt to s3://test-dent-backup1/daily/file1.txt [Errno 1] Operation not permitted: '/tmp/file1.txt'
Any suggestions?
Related
We have deployed a DMS replication task to replicate our entire Postgres database to Redshift. The tables are getting created with the correct schemas, but the data isn't coming through to Redshift and getting held up in the S3 bucket DMS uses as an intermediary step. This is all deployed via Terraform.
We've configured the IAM roles as described in the replication instance Terraform docs with all three of dms-access-for-endpoint, dms-cloudwatch-logs-role, and dms-vpc-role IAM roles created. The IAM roles are deployed via a different stack to where DMS is deployed from as the roles are used by another, successfully deployed, DMS instance running a different task.
data "aws_iam_policy_document" "dms_assume_role_document" {
statement {
actions = ["sts:AssumeRole"]
principals {
identifiers = [
"s3.amazonaws.com",
"iam.amazonaws.com",
"redshift.amazonaws.com",
"dms.amazonaws.com",
"redshift-serverless.amazonaws.com"
]
type = "Service"
}
}
}
# Database Migration Service requires the below IAM Roles to be created before
# replication instances can be created. See the DMS Documentation for
# additional information: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.html#CHAP_Security.APIRole
# * dms-vpc-role
# * dms-cloudwatch-logs-role
# * dms-access-for-endpoint
resource "aws_iam_role" "dms_access_for_endpoint" {
name = "dms-access-for-endpoint"
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"]
force_detach_policies = true
}
resource "aws_iam_role" "dms_cloudwatch_logs_role" {
name = "dms-cloudwatch-logs-role"
description = "Allow DMS to manage CloudWatch logs."
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"]
force_detach_policies = true
}
resource "aws_iam_role" "dms_vpc_role" {
name = "dms-vpc-role"
description = "DMS IAM role for VPC permissions"
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_document.json
managed_policy_arns = ["arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"]
force_detach_policies = true
}
However, on runtime, we're seeing the following logs in CloudWatch:
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Not retriable error: <AccessDenied> Access Denied [1001705] (anw_retry_strategy.cpp:118)
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Failed to list bucket 'dms-sandbox-redshift-intermediate-storage': error code <AccessDenied>: Access Denied [1001713] (s3_dir_actions.cpp:105)
2022-09-01T16:51:38 [SOURCE_UNLOAD ]E: Failed to list bucket 'dms-sandbox-redshift-intermediate-storage' [1001713] (s3_dir_actions.cpp:209)
We also enabled S3 server access logs on the bucket itself to see whether this would give us more information. This is what we're seeing (anonymised):
<id> dms-sandbox-redshift-intermediate-storage [01/Sep/2022:15:43:32 +0000] 10.128.69.80 arn:aws:sts::<account>:assumed-role/dms-access-for-endpoint/dms-session-for-replication-engine <code> REST.GET.BUCKET - "GET /dms-sandbox-redshift-intermediate-storage?delimiter=%2F&max-keys=1000 HTTP/1.1" 403 AccessDenied 243 - 30 - "-" "aws-sdk-cpp/1.8.80/S3/Linux/4.14.276-211.499.amzn2.x86_64 x86_64 GCC/4.9.3" - <code> SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader s3.eu-west-2.amazonaws.com TLSv1.2 -
The above suggests that a service dms-session-for-replication is the service in question that is receiving the AccessDenied responses, but we're unable to pinpoint what this is and how we can fix it.
We attempted to add a bucket policy to the S3 bucket itself but this did not work (this also includes the S3 server access logs bucket):
resource "aws_s3_bucket" "dms_redshift_intermediate" {
# Prefixed with `dms-` as that's what the AmazonDMSRedshiftS3Role policy filters on
bucket = "dms-sandbox-redshift-intermediate-storage"
}
resource "aws_s3_bucket_logging" "log_bucket" {
bucket = aws_s3_bucket.dms_redshift_intermediate.id
target_bucket = aws_s3_bucket.log_bucket.id
target_prefix = "log/"
}
resource "aws_s3_bucket" "log_bucket" {
bucket = "${aws_s3_bucket.dms_redshift_intermediate.id}-logs"
}
resource "aws_s3_bucket_acl" "log_bucket" {
bucket = aws_s3_bucket.log_bucket.id
acl = "log-delivery-write"
}
resource "aws_s3_bucket_policy" "dms_redshift_intermediate_policy" {
bucket = aws_s3_bucket.dms_redshift_intermediate.id
policy = data.aws_iam_policy_document.dms_redshift_intermediate_policy_document.json
}
data "aws_iam_policy_document" "dms_redshift_intermediate_policy_document" {
statement {
actions = [
"s3:*"
]
principals {
identifiers = [
"dms.amazonaws.com",
"redshift.amazonaws.com"
]
type = "Service"
}
resources = [
aws_s3_bucket.dms_redshift_intermediate.arn,
"${aws_s3_bucket.dms_redshift_intermediate.arn}/*"
]
}
}
How do we fix the <AccessDenied> issues that we're seeing on CloudWatch and enable data loading to Redshift? DMS is able to PUT items in the S3 bucket as we're seeing encrypted CSVs appearing in there (the server access logs also confirm this), but DMS is unable to then GET the files back out of it for Redshift. The AccessDenied responses also suggest that it's an IAM role issue not a security group issue but our IAM roles are configured as per the docs so we're confused as to what could be causing this issue.
What we thought was an IAM issue, was actually a security group issue. The COPY command for Redshift was struggling to access S3. By adding a 443 egress rule for HTTPS to the Redshift security group, we were able to pull data through again
resource "aws_security_group_rule" "https_443_egress" {
type = "egress"
description = "Allow HTTP egress from DMS SG"
protocol = "tcp"
to_port = 443
from_port = 443
security_group_id = aws_security_group.redshift.id
cidr_blocks = ["0.0.0.0/0"]
}
So if you're experiencing the same issue as the question, check whether Redshift has access to S3 via HTTPS.
You are right this is an IAM role issue, make sure the role in questions has the following statements added to the policy document,
{
"Effect": "Allow",
"Action": [
"s3:*"
],
"Resource":"arn:aws:s3:::<yourbucketnamehere>/*"
},
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource":"arn:aws:s3:::<yourbucketnamehere>"
},
{
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::*"
}
My setup is as follows:
iam user --> iam user group --> assume role group policy --> eks cluster role --> eks role bindings
So, I have an assume role policy:
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Sid = ""
Principal = {
AWS = "arn:aws:iam::xxxxxxxxxxxxx:root"
}
},
]
Then I have a user group, with the following group policy:
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"sts:AssumeRole",
]
Sid = "AllowAssumeOrganizationAccountRole"
Effect = "Allow"
Resource = "arn:aws:iam::xxxxxxxxxxxx:role/ro-k8s-role"
},
]
I then have the following cluster role for EKS:
rule {
api_groups = ["apps", "batch", "extensions"]
resources = ["pods", "deployments", "statefulsets", "events", "services", "pods/log", "pods/portforward"]
verbs = ["get", "list", "watch", "describe"]
}
And then a bunch of role bindings which are used for a list of namespaces:
metadata {
name = "k8s-ro-${each.value}"
namespace = each.value
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "Role"
name = "k8s-ro"
}
subject {
kind = "User"
name = "k8s-user"
api_group = "rbac.authorization.k8s.io"
}
If I then use the profile of a user in the group, and check the caller-identity...
{
"UserId": "XXXXXXXXXXXXXXXXXX",
"Account": "xxxxxxxxxxxxxxx",
"Arn": "arn:aws:iam::xxxxxxxxxxxxx:user/test-user"
}
I would expect to see the assume role here, right? Which explains why I can't view anything running on the cluster.
I'll close this as the initial problem is resolved. I hadn't properly configured the AWS Profiles to automatically handle the assume role. Which having done has resolved this issue.
~/.aws/config
[profile test]
cli_pager =
role_arn = arn:aws:iam::xxxxxxxxxxxx:role/k8s-role
source_profile = test
region = eu-west-1
~/.aws/credentials
[test]
aws_access_key_id = XXXXXXXXXXXXXXXXXX
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
I'm trying to create elasticsearch cluster using terraform.
Using terraform 0.11.13
Please can someone point out why I'm not able to create log groups? What is the Resource Access Policy? is it the same as the data "aws_iam_policy_document" I'm creating?
Note: I'm using elasticsearch_version = "7.9"
code:
resource "aws_cloudwatch_log_group" "search_test_log_group" {
name = "/aws/aes/domains/test-es7/index-logs"
}
resource "aws_elasticsearch_domain" "amp_search_test_es7" {
domain_name = "es7"
elasticsearch_version = "7.9"
.....
log_publishing_options {
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.search_test_log_group.arn}"
log_type = "INDEX_SLOW_LOGS"
enabled = true
}
access_policies = "${data.aws_iam_policy_document.elasticsearch_policy.json}"
}
data "aws_iam_policy_document" "elasticsearch_policy" {
version = "2012-10-17"
statement {
effect = "Allow"
principals {
identifiers = ["*"]
type = "AWS"
}
actions = ["es:*"]
resources = ["arn:aws:es:us-east-1:xxx:domain/test_es7/*"]
}
statement {
effect = "Allow"
principals {
identifiers = ["es.amazonaws.com"]
type = "Service"
}
actions = [
"logs:PutLogEvents",
"logs:PutLogEventsBatch",
"logs:CreateLogStream",
]
resources = ["arn:aws:logs:*"]
}
}
I'm getting this error
aws_elasticsearch_domain.test_es7: Error creating ElasticSearch domain: ValidationException: The Resource Access Policy specified for the CloudWatch Logs log group /aws/aes/domains/test-es7/index-logs does not grant sufficient permissions for Amazon Elasticsearch Service to create a log stream. Please check the Resource Access Policy.
For ElasticSearch (ES) to be able to write to CloudWatch (CW) Logs, you have to provide a resource-based policy on your CW logs.
This is achieved using aws_cloudwatch_log_resource_policy which is missing from your code.
In fact, TF docs have a ready to use example of how to do it for ES, thus you should be able to just copy and paste it.
ES access policies are different from CW log policies, as they determine who can do what on your ES domain. Thus, you would have to adjust that part of your code to meet your requirements.
as i'm new with terraform, i'd like to ask your help once i got stuck for almost a day.
When trying to apply a IAC to deploy a Nginx service into a ECS(EC2 launch type) on aws i'm facing the following problem:
Error: Error creating IAM Role nginx-iam_role: MalformedPolicyDocument: Has prohibited field Resource status code: 400, request id: 0f1696f4-d86b-4ad1-ba3b-9453f3beff2b
I have already checked the documentation and the syntax is fine. What else could be wrong?
Following the snippet code creating the IAM infra:
provider "aws" {
region = "us-east-2"
}
data "aws_iam_policy_document" "nginx-doc-policy" {
statement {
sid = "1"
actions = [
"ec2:*"
]
resources = ["*"]
}
}
resource "aws_iam_role" "nginx-iam_role" {
name = "nginx-iam_role"
path = "/"
assume_role_policy = "${data.aws_iam_policy_document.nginx-doc-policy.json}"
}
resource "aws_iam_group_policy" "nginx-group-policy" {
name = "my_developer_policy"
group = "${aws_iam_group.nginx-iam-group.name}"
policy = "${data.aws_iam_policy_document.nginx-doc-policy.json}"
}
resource "aws_iam_group" "nginx-iam-group" {
name = "nginx-iam-group"
path = "/"
}
resource "aws_iam_user" "nginx-user" {
name = "nginx-user"
path = "/"
}
resource "aws_iam_user_group_membership" "nginx-membership" {
user = "${aws_iam_user.nginx-user.name}"
groups = ["${aws_iam_group.nginx-iam-group.name}"]
}
If you guys need the remaining code: https://github.com/atilasantos/iac-terraform-nginx.git
You are trying to use the aws_iam_policy_document.nginx-doc-policy policy as an assume_role_policy which does not work as an assume role policy needs to define a principal that you trust and want to grant access to assume the role you are creating.
An assume role policy could look like this is you want to grant access to the role to EC2 instances via instance profiles. At the end you can attach your initial role via a new resource as an inline policy to the role:
data "aws_iam_policy_document" "instance-assume-role-policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "nginx-iam_role" {
name = "nginx-iam_role"
path = "/"
assume_role_policy = data.aws_iam_policy_document.instance-assume-role-policy.json
}
resource "aws_iam_role_policy" "role_policy" {
name = "role policy"
role = aws_iam_role.nginx-iam_role.id
policy = data.aws_iam_policy_document.nginx-doc-policy.json
}
Instead of attaching the policy as an inline policies you can also create an IAM Policy and attach it to the various iam resources. (e.g.: aws_iam_policy and aws_iam_role_policy_attachment for roles.)
We created a bunch of open-source IAM modules (and others) to make IAM handling easier: Find them here on github. But there are more modules out there that you can try.
I'm using terraform to provision an ELB & want to Enable Access logs for ELB in a S3 bucket. When I try to apply the resources, I get the below error - InvalidConfiguration: Access Denied for bucket:
Below are my TF resources with the S3 bucket policy created using the IAM Policy Document.
resource "aws_lb" "this" {
name = var.name
load_balancer_type = "application"
access_logs {
bucket = aws_s3_bucket.this.bucket
prefix = var.name
enabled = true
}
}
resource "aws_s3_bucket" "this" {
bucket = "${var.bucket_name}"
acl = "log-delivery-write"
force_destroy = true
}
resource "aws_s3_bucket_policy" "this" {
bucket = "aws_s3_bucket.this.id"
policy = "${data.aws_iam_policy_document.s3_bucket_lb_write.json}"
}
data "aws_iam_policy_document" "s3_bucket_lb_write" {
policy_id = "s3_bucket_lb_logs"
statement {
actions = [
"s3:PutObject",
]
effect = "Allow"
resources = [
"${aws_s3_bucket.this.arn}/*",
]
principals {
identifiers = ["${data.aws_elb_service_account.main.arn}"]
type = "AWS"
}
}
statement {
actions = [
"s3:PutObject"
]
effect = "Allow"
resources = ["${aws_s3_bucket.this.arn}/*"]
principals {
identifiers = ["delivery.logs.amazonaws.com"]
type = "Service"
}
}
statement {
actions = [
"s3:GetBucketAcl"
]
effect = "Allow"
resources = ["${aws_s3_bucket.this.arn}"]
principals {
identifiers = ["delivery.logs.amazonaws.com"]
type = "Service"
}
}
}
output "bucket_name" {
value = "${aws_s3_bucket.this.bucket}"
}
I get the following error
Error: Error putting S3 policy: NoSuchBucket: The specified bucket does not exist
status code: 404, request id: 5932CFE816059A8D, host id: j5ZBQ2ptHXivx+fu7ai5jbM8PSQR2tCFo4IAvcLkuocxk8rn/r0TG/6YbfRloBFR2WSy8UE7K8Q=
Error: Failure configuring LB attributes: InvalidConfigurationRequest: Access Denied for bucket: test-logs-bucket-xyz. Please check S3bucket permission
status code: 400, request id: ee101cc2-5518-42c8-9542-90dd7bb05e3c
terraform version
Terraform v0.12.23
provider.aws v3.6.0
There is mistake in:
resource "aws_s3_bucket_policy" "this" {
bucket = "aws_s3_bucket.this.id"
policy = "${data.aws_iam_policy_document.s3_bucket_lb_write.json}"
}
it should be:
resource "aws_s3_bucket_policy" "this" {
bucket = aws_s3_bucket.this.id
policy = data.aws_iam_policy_document.s3_bucket_lb_write.json
}
The orginal version (bucket = "aws_s3_bucket.this.id") will just try to look for bucket literally called "aws_s3_bucket.this.id".