I have a sagemaker model that will be deployed in different aws regions. This model will download models from an s3 bucket that is in region x. As long as the model is deployed in region x, the endpoint works.
However, when I deploy the sagemaker model from region y, it fails with the message.
Error: error creating SageMaker model: ValidationException: Could not
access model data at s3://mmmm/. Please ensure that the role
"arn:aws:iam::xxxx:role/dev-xxx-iam-role" exists and that its trust
relationship policy allows the action "sts:AssumeRole" for the service
principal "sagemaker.amazonaws.com". Also ensure that the role has
"s3:GetObject" permissions and that the object is located in region x.
My iam role permissions are as follows:
resource "aws_iam_policy_attachment" "sm_full_access_attach" {
name = "sm-full-access-attachment"
roles = [aws_iam_role.sagemaker_inferencer_iam_role.name]
policy_arn = "arn:aws:iam::aws:policy/AmazonSageMakerFullAccess"
}
resource "aws_iam_policy_attachment" "s3_full_access_attach" {
name = "s3-full-access-attachment"
roles = [aws_iam_role.sagemaker_inferencer_iam_role.name]
policy_arn = "arn:aws:iam::aws:policy/AmazonS3FullAccess"
}
with the assume role policy of:
data "aws_iam_policy_document" "sm_assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["sagemaker.amazonaws.com"]
}
}
}
The iam role is as follows:
resource "aws_iam_role" "sagemaker_inferencer_iam_role" {
name = "${var.app_environment}-inferencer-sm-${var.aws_region}-iam-role"
assume_role_policy = data.aws_iam_policy_document.sm_assume_role_policy.json
}
And the above works for same region buckets.
I can of course create have buckets in each region with replication rules from the original bucket. However, this is costly as the model files are huge. So I created a multi-region accesspoint for my original bucket with an alias zzz.mrap
However, when I specify the accesspoint alias in the aws_sagemaker_model resource as follows:
resource "aws_sagemaker_model" "sagemaker_multimodel" {
name = "${var.app_environment}-inferencer-sm-${var.aws_region}-model"
execution_role_arn = aws_iam_role.sagemaker_inferencer_iam_role.arn
primary_container {
image = local.multi_model_inferencer_container_name
mode = "MultiModel"
model_data_url = "s3://zzz.mrap/"
}
}
I get the following error:
Error: error creating SageMaker model: ValidationException: Could not
access model data at s3://zzzz.mrap/. Please ensure that the role
"arn:aws:iam::878435376106:role/dev-xxx-iam-role" exists and that its
trust relationship policy allows the action "sts:AssumeRole" for the
service principal "sagemaker.amazonaws.com". Also ensure that the role
has "s3:GetObject" permissions and that the object is located in
region x.
AWS says that I only have to replace the bucket name by the alias and that they support sagemaker, however this does not seem to be the case.
What am I doing wrong?
Related
I'm trying to create elasticsearch cluster using terraform.
Using terraform 0.11.13
Please can someone point out why I'm not able to create log groups? What is the Resource Access Policy? is it the same as the data "aws_iam_policy_document" I'm creating?
Note: I'm using elasticsearch_version = "7.9"
code:
resource "aws_cloudwatch_log_group" "search_test_log_group" {
name = "/aws/aes/domains/test-es7/index-logs"
}
resource "aws_elasticsearch_domain" "amp_search_test_es7" {
domain_name = "es7"
elasticsearch_version = "7.9"
.....
log_publishing_options {
cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.search_test_log_group.arn}"
log_type = "INDEX_SLOW_LOGS"
enabled = true
}
access_policies = "${data.aws_iam_policy_document.elasticsearch_policy.json}"
}
data "aws_iam_policy_document" "elasticsearch_policy" {
version = "2012-10-17"
statement {
effect = "Allow"
principals {
identifiers = ["*"]
type = "AWS"
}
actions = ["es:*"]
resources = ["arn:aws:es:us-east-1:xxx:domain/test_es7/*"]
}
statement {
effect = "Allow"
principals {
identifiers = ["es.amazonaws.com"]
type = "Service"
}
actions = [
"logs:PutLogEvents",
"logs:PutLogEventsBatch",
"logs:CreateLogStream",
]
resources = ["arn:aws:logs:*"]
}
}
I'm getting this error
aws_elasticsearch_domain.test_es7: Error creating ElasticSearch domain: ValidationException: The Resource Access Policy specified for the CloudWatch Logs log group /aws/aes/domains/test-es7/index-logs does not grant sufficient permissions for Amazon Elasticsearch Service to create a log stream. Please check the Resource Access Policy.
For ElasticSearch (ES) to be able to write to CloudWatch (CW) Logs, you have to provide a resource-based policy on your CW logs.
This is achieved using aws_cloudwatch_log_resource_policy which is missing from your code.
In fact, TF docs have a ready to use example of how to do it for ES, thus you should be able to just copy and paste it.
ES access policies are different from CW log policies, as they determine who can do what on your ES domain. Thus, you would have to adjust that part of your code to meet your requirements.
as i'm new with terraform, i'd like to ask your help once i got stuck for almost a day.
When trying to apply a IAC to deploy a Nginx service into a ECS(EC2 launch type) on aws i'm facing the following problem:
Error: Error creating IAM Role nginx-iam_role: MalformedPolicyDocument: Has prohibited field Resource status code: 400, request id: 0f1696f4-d86b-4ad1-ba3b-9453f3beff2b
I have already checked the documentation and the syntax is fine. What else could be wrong?
Following the snippet code creating the IAM infra:
provider "aws" {
region = "us-east-2"
}
data "aws_iam_policy_document" "nginx-doc-policy" {
statement {
sid = "1"
actions = [
"ec2:*"
]
resources = ["*"]
}
}
resource "aws_iam_role" "nginx-iam_role" {
name = "nginx-iam_role"
path = "/"
assume_role_policy = "${data.aws_iam_policy_document.nginx-doc-policy.json}"
}
resource "aws_iam_group_policy" "nginx-group-policy" {
name = "my_developer_policy"
group = "${aws_iam_group.nginx-iam-group.name}"
policy = "${data.aws_iam_policy_document.nginx-doc-policy.json}"
}
resource "aws_iam_group" "nginx-iam-group" {
name = "nginx-iam-group"
path = "/"
}
resource "aws_iam_user" "nginx-user" {
name = "nginx-user"
path = "/"
}
resource "aws_iam_user_group_membership" "nginx-membership" {
user = "${aws_iam_user.nginx-user.name}"
groups = ["${aws_iam_group.nginx-iam-group.name}"]
}
If you guys need the remaining code: https://github.com/atilasantos/iac-terraform-nginx.git
You are trying to use the aws_iam_policy_document.nginx-doc-policy policy as an assume_role_policy which does not work as an assume role policy needs to define a principal that you trust and want to grant access to assume the role you are creating.
An assume role policy could look like this is you want to grant access to the role to EC2 instances via instance profiles. At the end you can attach your initial role via a new resource as an inline policy to the role:
data "aws_iam_policy_document" "instance-assume-role-policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_role" "nginx-iam_role" {
name = "nginx-iam_role"
path = "/"
assume_role_policy = data.aws_iam_policy_document.instance-assume-role-policy.json
}
resource "aws_iam_role_policy" "role_policy" {
name = "role policy"
role = aws_iam_role.nginx-iam_role.id
policy = data.aws_iam_policy_document.nginx-doc-policy.json
}
Instead of attaching the policy as an inline policies you can also create an IAM Policy and attach it to the various iam resources. (e.g.: aws_iam_policy and aws_iam_role_policy_attachment for roles.)
We created a bunch of open-source IAM modules (and others) to make IAM handling easier: Find them here on github. But there are more modules out there that you can try.
I am new to terraform. I was working with terraform v0.12 previously and since I wanted to bring in a dependency between modules, I started using terraform v0.13 recently. I am trying to create an IAM role and attach a few policies to the created role. But the issue arises in policy attachment to the role. Few policies are getting attached to the role but some policies throw an error saying no such role exists while the other policies are attached to the role properly. Is there anything wrong in my implementation?
module.tf
provider "aws" {
region = "ap-southeast-1"
}
#Control Plane role and policies
module "ControlPlane_Role" {
source = "../../templates/IAM/roles"
role_name = var.EKS-master-role
}
module "ControlPlane_Policy1" {
source = "../../templates/IAM/aws_policy"
role_name = var.EKS-master-role
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy"
depends_on = [module.ControlPlane_Role.role_create]
}
module "ControlPlane_Policy2" {
source = "../../templates/IAM/aws_policy"
role_name = var.EKS-master-role
policy_arn = "arn:aws:iam::aws:policy/AmazonEKSServicePolicy"
depends_on = [module.ControlPlane_Role.role_create]
}
templates/IAM/roles/role.tf
resource "aws_iam_role" "role_create" {
assume_role_policy = data.aws_iam_policy_document.trusted_entity.json
name = var.role_name
}
aws_policy.tf
resource "aws_iam_role_policy_attachment" "aws_policy" {
role = var.role_name
policy_arn = var.policy_arn
}
I'll be passing the variable files separately and there are no issues with that.
Error:
Error: Error attaching policy arn:aws:iam::aws:policy/AmazonEKSClusterPolicy to IAM Role EKS-master: NoSuchEntity: The role with name EKS-master cannot be found.
Error attaching policy arn:aws:iam::aws:policy/AmazonEKSServicePolicy to IAM Role EKS-master: NoSuchEntity: The role with name EKS-master cannot be found.
If I re-run the command terraform apply again on the same resources without any change, the policies are getting attached.
You shouldn't use depends_on except for some exceptional cases. From the templates/IAM/roles, define an output that is the name of the role and in the other modules pass this output (role_name = module.ControlPlane_Role.output_role_name). With this setup, the dependency graph is clear (create the role, create the things that depend on the role) instead of having to manually define dependencies with depends_on.
We use a combination of cloud formation and terraform where some common resources like DynamoDB, S3 are created using terraform and others like APIGateway are created using serverless and cloudformation. All resources are in the same AWS account
I have an S3 bucket in terraform
resource "aws_s3_bucket" "payment_bucket" {
bucket = "payment-bucket-${var.env_name}"
acl = "private"
tags = merge(
module.tags.base_tags,
{
"Name" = "payment-bucket-${var.env_name}"
}
)
lifecycle {
ignore_changes = [tags]
}
}
This creates a private bucket payment-bucket-dev in my AWS account when I run the tf-apply
We have an APIGateway in the same AWS account which is created using serverless and one of the lambda needs accesses to this bucket so I have created an IAM role for the lambda function to grant permission to access the bucket.
makePayment:
name: makePayment-${self:provider.stage}
handler: src/handler/makePayment.default
events:
- http:
path: /payment
method: post
private: true
cors: true
iamRoleStatementsName: ${self:service}-${self:provider.stage}-makePayment-role
iamRoleStatements:
- Effect: Allow
Action:
- s3:PutObject
Resource:
- arn:aws:s3:::#{AWS::Region}:#{AWS::AccountId}:payment-bucket-${self:provider.stage}/capture/batch/*
But when I run this lambda make-payment-dev , it throws an AccessDenied error unless I add bucket policy granting access to the lambda role
resource "aws_s3_bucket_policy" "payment_service_s3_bucket_policy" {
..
..
}
Why do I need to add S3 bucket policy when both s3 bucket and the lambda function and role are in the same account? Am I missing something?
Also, If I created the bucket using AWS::S3::Bucket as part of the cloud formation stack the Apigateway is in (we are using serverless), I don't need add bucket policy and it all works fine.
I think the problem is simply that the S3 bucket ARN is incorrect.
S3 bucket ARNs do not have account IDs or regions in them. Use arn:aws:s3:::mybucket/myprefix/*.
The answer depends on what AWS IAM role is applying the terraform plan because the AWS s3 bucket canned ACL rule: "private" restricts bucket access as: Owner gets FULL_CONTROL. No one else has access rights (default). per documentation: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html
You have to be relatively explicit at this point as to who can access the bucket. Often if I'm going with private ACL but want every other role in my AWS account to have access to the bucket I attach a bucket policy to the terraform aws_s3_bucket resource to first allow access to the bucket. Then I explicitly grant the lambda's role access to said bucket via another inline policy.
In your case it would look something like this:
// Allow access to the bucket
data "aws_iam_policy_document" "bucket_policy" {
statement {
sid = "S3 bucket policy for account access"
actions = [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
]
principals {
type = "AWS"
identifiers = [
"arn:aws:iam::{your_account_id_here}:root",
]
}
resources = [
"arn:aws:s3:::test_bucket_name",
"arn:aws:s3:::test_bucket_name/*",
]
condition {
test = "StringEquals"
variable = "aws:PrincipalArn"
values = ["arn:aws:iam::{your_account_id_here}:role/*"]
}
}
}
resource "aws_s3_bucket" "this" {
bucket = "test_bucket_name"
acl = "private"
policy = data.aws_iam_policy_document.bucket_policy.json
}
// Grant the lambda IAM role permissions to the bucket
data "aws_iam_policy_document" "grant_bucket_access" {
statement {
sid = "AccessToTheAppAuxFilesBucket"
actions = [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
]
resources = [
"arn:aws:s3:::test_bucket_name/*",
"arn:aws:s3:::test_bucket_name"
]
}
}
// Data call to pull the arn of the lambda's IAM Role
data "aws_iam_role" "cloudformation_provisioned_role" {
name = "the_name_of_the_lambdas_iam_role"
}
resource "aws_iam_role_policy" "iam_role_inline_policy" {
name = "s3_bucket_access"
role = data.aws_iam_role.cloudformation_provisioned_role.arn
policy = data.aws_iam_policy_document.grant_bucket_access.json
}
It's an open bug. acl and force_destroy aren't well imported with terraform import : https://github.com/hashicorp/terraform-provider-aws/issues/6193
I'm using this terraform module to create eks cluster: https://github.com/terraform-aws-modules/terraform-aws-eks
Then I create an additional role and added to map_roles input similar to the example in the repo
(my role is to use CloudWatch)
{
rolearn = "arn:aws:iam::66666666666:role/role1"
username = "role1"
groups = ["system:masters"]
}
I can verify that the role is added to the aws-auth config map together with a role created by the module.
I got this error when the app trying to use CloudWatch:
User: arn:aws:sts::xxx:assumed-role/yyy/zzz is not authorized to perform: logs:DescribeLogGroups on resource: arn:aws:logs:xxx:yyy:log-group::log-stream
the User arn in the error message has the yyy part match the role arn created by the module. So I thought I'm using the wrong role? if so how can I choose the correct credential? (I'm using .NETcore, create AmazonCloudWatchLogsClient without specify any credential)
When I manually edit that role and add the log's permission, the app works. Not sure if it's the right way, if so how can I add the permission in terraforming?
I ended up pulling the eks module to local and add more policies to the existing role:
resource "aws_iam_policy" "my_new_policy" {
name_prefix = "eks-worker-my_new_policy-${aws_eks_cluster.this.name}"
description = "EKS worker node my_new_policy policy for cluster ${aws_eks_cluster.this.name}"
policy = data.aws_iam_policy_document.my_new_policy.json
path = var.iam_path
}
data "aws_iam_policy_document" "my_new_policy" {
statement {
sid = "my_new_policy"
effect = "Allow"
actions = [
"logs:DescribeLogGroups"
]
resources = ["*"]
}
}