I want to create files(key) in s3. By below code, I can successfully create the files in s3 -
locals {
rules = [
{
user = "user-1",
roles = "test"
}
}
resource "aws_s3_bucket_object" "this" {
for_each = zipmap(local.rules.*.user, local.rules.*)
bucket = "test-09129"
acl = "private"
source = "/dev/null"
key = "${each.value.roles}/${each.key}"
}
file in s3 - test/user-1 created
But roles can be multiple so I create list for roles.
locals {
rules = [
{
user = "user-1",
roles = ["test"]
},
{
user = "user-2",
roles = ["test", "dev", "prd"]
}]
}
Now I want to create files based on above list. How to iterate list(roles) to create files based on user and role?
desire output -
file in s3 -
test/user-1
test/user-2
dev/user-2
prd/user-2
You can flatten your rules into helper_rules (assuming I understand correctly your desired outcome):
locals {
rules = [
{
user = "user-1",
roles = ["test"]
},
{
user = "user-2",
roles = ["test", "dev", "prd"]
}]
helper_rules = merge([
for user, rules in zipmap(local.rules.*.user, local.rules.*.roles):
{ for rule in rules:
"${user}-${rule}" => {user = user, rule = rule}
}
]...)
}
This will result in helper_rules:
{
"user-1-test" = {
"rule" = "test"
"user" = "user-1"
}
"user-2-dev" = {
"rule" = "dev"
"user" = "user-2"
}
"user-2-prd" = {
"rule" = "prd"
"user" = "user-2"
}
"user-2-test" = {
"rule" = "test"
"user" = "user-2"
}
}
With that, your objects could be:
resource "aws_s3_bucket_object" "this" {
for_each = local.helper_rules
bucket = "test-09129"
acl = "private"
source = "/dev/null"
key = "${each.value.role}/${each.value.user}"
}
Related
So I have multiple IAM groups which I am looping through as follows :
resource "aws_iam_group" "all_iam_groups" {
for_each = var.iam_user_groups
name = "${local.csi}-${each.key}"
path = "/"
}
This will create multiple IAM groups. Now for each IAM groups I will have an IAM Policy which I will attatch. Rather than having to do it manually and create multiple resources what is the best approach to take here. So far I have been doing it as follows :
resource "aws_iam_policy" "finance_read_only" {
name = "${local.csi}-finance-read-only"
path = "/"
policy = data.aws_iam_policy_document.finance_read_only.json
}
resource "aws_iam_policy" "security_read_only" {
name = "${local.csi}-security-read-only"
path = "/"
policy = data.aws_iam_policy_document.security_read_only.json
}
resource "aws_iam_group_policy_attachment" "security_read_only" {
group = aws_iam_group.security_team.name
policy_arn = aws_iam_policy.security_read_only.arn
}
resource "aws_iam_group_policy_attachment" "finance_read_only" {
group = aws_iam_group.finance_team.name
policy_arn = aws_iam_policy.finance_read_only.arn
}
For example purposes I have added in the group names but I know i can use a for-each loop to go through the names but am not sure how I would be able to look through the IAM Policies and policy attatchments.
I do understand that I will have multiple aws_iam_policy_document which is fine.
There's a direct relationship between the following resources: aws_iam_policy , the data source for aws_iam_policy_document, the aws_iam_group_policy_attachment and the aws_iam_group resource.
My suggestion is to create a variable of a list of objects type, where you define different attributes that are needed for the resources required.
Example:
variable "policies" {
type = list(object({
name = string
statement = object
}))
default = [
{
name = "finance-read-only"
statement = {
sid = "XXX"
actions = ["ec2:XXX"]
resources = ["XXX"]
}
},
{
name = "security-read-only"
statement = {
sid = "XXX"
actions = ["ec2:XXX"]
resources = ["XXX"]
}
}
]
}
data "aws_iam_policy_document" "this" {
for_each = { for policy in var.policies : policy.name => policy }
statement = each.value.statement
}
resource "aws_iam_policy" "this" {
for_each = { for policy in var.policies : policy.name => policy }
name = format("${local.csi}-%s", each.key)
path = "/"
policy = data.aws_iam_policy_document.this[each.key].json
}
resource "aws_iam_group" "this" {
for_each = { for policy in var.policies : policy.name => policy }
name = each.key
}
resource "aws_iam_group_policy_attachment" "security_read_only" {
for_each = { for policy in var.policies : policy.name => policy }
group = aws_iam_group.this[each.key].name
policy_arn = aws_iam_policy.this[each.key].arn
}
I want to create two Amazon SNS topics with the same aws_iam_policy_document, aws_sns_topic_policy & time_sleep configs.
This is my terraform, my_sns_topic.tf:
resource "aws_sns_topic" "topic_a" {
name = "topic-a"
}
resource "aws_sns_topic" "topic_b" {
name = "topic-b"
}
data "aws_iam_policy_document" "topic_notification" {
version = "2008-10-17"
statement {
sid = "__default_statement_ID"
actions = [
"SNS:Publish"
]
# Cut off some lines for simplification.
## NEW LINE ADDED
statement {
sid = "allow_snowflake_subscription"
principals {
type = "AWS"
identifiers = [var.storage_aws_iam_user_arn]
}
actions = ["SNS:Subscribe"]
resources = [aws_sns_topic.topic_a.arn] # Troubles with this line
}
}
resource "aws_sns_topic_policy" "topic_policy_notification" {
arn = aws_sns_topic.topic_a.arn
policy = data.aws_iam_policy_document.topic_policy_notification.json
}
resource "time_sleep" "topic_wait_10s" {
depends_on = [aws_sns_topic.topic_a]
create_duration = "10s"
}
As you can see here, I set up the configuration only for topic-a. I want to loop this over to apply for topic-b as well.
It would be better to use map and for_each, instead of separately creating "a" and "b" topics:
variable "topics" {
default = ["a", "b"]
}
resource "aws_sns_topic" "topic" {
for_each = toset(var.topics)
name = "topic-${each.key}"
}
data "aws_iam_policy_document" "topic_notification" {
version = "2008-10-17"
statement {
sid = "__default_statement_ID"
actions = [
"SNS:Publish"
]
# Cut off some lines for simplification.
}
resource "aws_sns_topic_policy" "topic_policy_notification" {
for_each = toset(var.topics)
arn = aws_sns_topic.topic[each.key].arn
policy = data.aws_iam_policy_document.topic_policy_notification.json
}
resource "time_sleep" "topic_wait_10s" {
for_each = toset(var.topics)
depends_on = [aws_sns_topic.topic[each.key]]
create_duration = "10s"
}
I am creating my very own EKS cluster and had some issues adding in roles via my namespace declared in locals dynamically
I am using the terraform-aws-eks-auth Tag v1.0.0 to populate my IAM roles into the groups that i have created in terraform.
I expect that each.key variable would populate/append in the list of maps entries of map_roles but so far it doesn't do so. The end result in aws auth config map shows that it only contains AdminAccessRole, my_eks_admin_role, dev-kiwi-ingress-nginx_admin_role and dev-kiwi-ingress-nginx_readonly_role. The previous namespaces that was supposed to be dynamically created were overwritten.
Should I even use for_each in such case? How can I populate aws-auth dynamically by namespaces used, combined with 2 static entries.
I appreciate any advice or help!
locals {
namespaces = [
"${var.env}-devops",
"${var.env}-devops-ingress-nginx",
"${var.env}-shared",
"${var.env}-shared-ingress-nginx",
"${var.env}-pear",
"${var.env}-pear-ingress-nginx",
"${var.env}-apple",
"${var.env}-apple-ingress-nginx",
"${var.env}-banana",
"${var.env}-banana-ingress-nginx",
"${var.env}-kiwi",
"${var.env}-kiwi-ingress-nginx"
]
}
module "eks_auth" {
source = "./terraform-modules/terraform-aws-eks-auth-1.0.0"
eks = module.eks
# Additional IAM roles to add to the aws-auth configmap.
for_each = toset(local.namespaces)
map_roles = [
{
rolearn = "arn:aws:iam::1234567890:role/AdminAccessRole"
username = "AdminAccessRole"
groups = ["system:masters"]
},
{
rolearn = "arn:aws:iam::1234567890:role/my-eks-admin-role"
username = "my_eks_admin_role"
groups = ["system:masters"]
},
{
rolearn = aws_iam_role.eks_namespace_admin_roles[each.key].arn
username = "${each.key}_admin_role"
groups = ["${each.key}:${each.key}_group"]
},
{
rolearn = aws_iam_role.eks_namespace_readonly_roles[each.key].arn
username = "${each.key}_readonly_role"
groups = ["${each.key}:${each.key}_group"]
}
]
map_users = [
{
userarn = "arn:aws:iam::1234567890:user/terraform-service-account"
username = "terraform-service-account"
groups = ["system:masters"]
}
]
}
Terraform v1.0.11
aws provider version = "~> 4.9.0"
By adding the for_each in the level shown in the question, terraform will attempt to create multiple instances of the eks_auth module
In your case you need to apply the looping in the variable creation.
Also in order to not loop through the static definitions we use concat to merge the lists E.g.
map_roles = concat( [ for index,value in toset(local.namespaces) :
{
rolearn = aws_iam_role.eks_namespace_admin_roles[value].arn
username = "${value}_admin_role"
groups = ["${value}:${value}_group"]
},
{
rolearn = aws_iam_role.eks_namespace_readonly_roles[value].arn
username = "${value}_readonly_role"
groups = ["${value}:${value}_group"]
}
],
[
{
rolearn = "arn:aws:iam::1234567890:role/AdminAccessRole"
username = "AdminAccessRole"
groups = ["system:masters"]
},
{
rolearn = "arn:aws:iam::1234567890:role/my-eks-admin-role"
username = "my_eks_admin_role"
groups = ["system:masters"]
}
])
With the help from Tolis, this is the answer to the issue.
It was still throwing out errors when tried to FOR loop a couple of map entries, so i split them up and concat multiple list instead.
module "eks_auth" {
source = "./terraform-modules/terraform-aws-eks-auth-1.0.0"
eks = module.eks
# Additional IAM roles to add to the aws-auth configmap.
for_each = toset(local.namespaces)
map_roles = concat(
[
for key in toset(local.namespaces) :
{
rolearn = aws_iam_role.eks_namespace_admin_roles[key].arn
username = "${key}_admin_role"
groups = ["${key}:${key}_group"]
}
],
[
for key in toset(local.namespaces) :
{
rolearn = aws_iam_role.eks_namespace_readonly_roles[key].arn
username = "${key}_readonly_role"
groups = ["${key}:${key}_group"]
}
],
[
{
rolearn = "arn:aws:iam::1234567890:role/AdminAccessRole"
username = "AdminAccessRole"
groups = ["system:masters"]
},
{
rolearn = "arn:aws:iam::1234567890:role/my-eks-admin-role"
username = "my_eks_admin_role"
groups = ["system:masters"]
}
])
# Additional IAM users to add to the aws-auth configmap.
map_users = [
]
}
I am creating an AWS cloudsearch domainfrom terraform, after successful creation of this domain, I need to trigger one REST API. can anyone have idea how we can do this in terraform ?
sample Terraform code :
resource "aws_cloudsearch_domain" "example" {
name = "example-domain"
scaling_parameters {
desired_instance_type = "search.medium"
}
index_field {
name = "headline"
type = "text"
search = true
return = true
sort = true
highlight = false
analysis_scheme = "_en_default_"
}
index_field {
name = "price"
type = "double"
search = true
facet = true
return = true
sort = true
}
}
resource "aws_cloudsearch_domain_service_access_policy" "example" {
domain_name = aws_cloudsearch_domain.example.id
access_policy = <<POLICY
{
"Version":"2012-10-17",
"Statement":[{
"Sid":"search_only",
"Effect":"Allow",
"Principal":"*",
"Action":[
"cloudsearch:search",
"cloudsearch:document"
]
}]
}
POLICY
}
i am trying to build the terraform for sagemaker private work force with private cognito
Following : https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sagemaker_workforce
it working fine
main.tf
resource "aws_sagemaker_workforce" "workforce" {
workforce_name = "workforce"
cognito_config {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
}
}
resource "aws_cognito_user_pool" "user_pool" {
name = "sagemaker-cognito-userpool"
}
resource "aws_cognito_user_pool_client" "congnito_client" {
name = "congnito-client"
generate_secret = true
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_group" "user_group" {
name = "user-group"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_pool_domain" "domain" {
domain = "sagemaker-user-pool-ocr-domain"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_sagemaker_workteam" "workteam" {
workteam_name = "worker-team"
workforce_name = aws_sagemaker_workforce.workforce.id
description = "worker-team"
member_definition {
cognito_member_definition {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
user_group = aws_cognito_user_group.user_group.id
}
}
}
resource "aws_sagemaker_human_task_ui" "template" {
human_task_ui_name = "human-task-ui-template"
ui_template {
content = file("${path.module}/sagemaker-human-task-ui-template.html")
}
}
resource "aws_sagemaker_flow_definition" "definition" {
flow_definition_name = "flow-definition"
role_arn = var.aws_iam_role
human_loop_config {
human_task_ui_arn = aws_sagemaker_human_task_ui.template.arn
task_availability_lifetime_in_seconds = 1
task_count = 1
task_description = "Task description"
task_title = "Please review the Key Value Pairs in this document"
workteam_arn = aws_sagemaker_workteam.workteam.arn
}
output_config {
s3_output_path = "s3://${var.s3_output_path}"
}
}
it's creating the cognito user pool with callback urls. These callback urls is coming from aws_sagemaker_workforce.workforce.subdomain and getting set in cognito automatically which is what i want.
But i also want to set config in cognito userpool like
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
now when i add above two line we need to add callbackurl also which i dont want.
i tried
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
callback_urls = [aws_sagemaker_workforce.workforce.subdomain]
which is giving error :
Cycle: module.sagemaker.aws_cognito_user_pool_client.congnito_client, module.sagemaker.aws_sagemaker_workforce.workforce
as both resource are dependent on each other, i want to pass those two line but it forces me to add callback url also.
here is the final main.tf which is failing with that three line
resource "aws_sagemaker_workforce" "workforce" {
workforce_name = "workforce"
cognito_config {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
}
}
resource "aws_cognito_user_pool" "user_pool" {
name = "sagemaker-cognito-userpool"
}
resource "aws_cognito_user_pool_client" "congnito_client" {
name = "congnito-client"
generate_secret = true
user_pool_id = aws_cognito_user_pool.user_pool.id
explicit_auth_flows = ["ALLOW_REFRESH_TOKEN_AUTH", "ALLOW_USER_PASSWORD_AUTH", "ALLOW_CUSTOM_AUTH", "ALLOW_USER_SRP_AUTH"]
allowed_oauth_flows_user_pool_client = true
supported_identity_providers = ["COGNITO"]
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
callback_urls = [aws_sagemaker_workforce.workforce.subdomain]
}
resource "aws_cognito_user_group" "user_group" {
name = "user-group"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_pool_domain" "domain" {
domain = "sagemaker-user-pool-ocr-domain"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_sagemaker_workteam" "workteam" {
workteam_name = "worker-team"
workforce_name = aws_sagemaker_workforce.workforce.id
description = "worker-team"
member_definition {
cognito_member_definition {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
user_group = aws_cognito_user_group.user_group.id
}
}
}
resource "aws_sagemaker_human_task_ui" "template" {
human_task_ui_name = "human-task-ui-template"
ui_template {
content = file("${path.module}/sagemaker-human-task-ui-template.html")
}
}
resource "aws_sagemaker_flow_definition" "definition" {
flow_definition_name = "flow-definition"
role_arn = var.aws_iam_role
human_loop_config {
human_task_ui_arn = aws_sagemaker_human_task_ui.template.arn
task_availability_lifetime_in_seconds = 1
task_count = 1
task_description = "Task description"
task_title = "Please review the Key Value Pairs in this document"
workteam_arn = aws_sagemaker_workteam.workteam.arn
}
output_config {
s3_output_path = "s3://${var.s3_output_path}"
}
}
You do not need to specify the callback URL for the workforce. It is sufficient to specify the following in order to create the aws_cognito_user_pool_client resource:
callback_urls = [
"https://${aws_cognito_user_pool_domain.domain>.cloudfront_distribution_arn}",
]
Then you reference the user pool client in your workforce definition:
resource "aws_sagemaker_workforce" "..." {
workforce_name = "..."
cognito_config {
client_id = aws_cognito_user_pool_client.<client_name>.id
user_pool = aws_cognito_user_pool_domain.<domain_name>.user_pool_id
}
}
Existence of the callback URLs can be proven after applying the terraform configuration by running aws cognito-idp describe-user-pool-client --user-pool-id <pool_id> --client-id <client_id>:
"UserPoolClient": {
...
"CallbackURLs": [
"https://____.cloudfront.net",
"https://____.labeling.eu-central-1.sagemaker.aws/oauth2/idpresponse"
],
"LogoutURLs": [
"https://____.labeling.eu-central-1.sagemaker.aws/logout"
],
It seems as terraform itself does not do anything special on workforce creation (see https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/sagemaker/workforce.go). So the callback urls seem to be added by AWS SageMaker itself.
This means that you have to instruct terraform to ignore changes on those attributes in the aws_cognito_user_pool_client configuration:
lifecycle {
ignore_changes = [
callback_urls, logout_urls
]
}