How can i add resource aws_cloudwatch_event_api_destination and aws_cloudwatch_event_connection to resource aws_cloudwatch_event_rule in terraform?
This Code in below
resource "aws_cloudwatch_event_rule" "test123_schedule_everyday" {
name = "${var.test123_bucket_name}-schedule-everyday-${var.env}"
description = "Meter reader get api data every 11:00 PM at lotus"
schedule_expression = "cron(0 16 * * ? *)"
is_enabled = "${var.test123_cloudwatch_is_enable}"
lifecycle {
ignore_changes = [schedule_expression, description, is_enabled]
}
}
resource "aws_cloudwatch_event_api_destination" "test123_event_api" {
name = "test-dev-api"
description = "test-dev-api destination"
invocation_endpoint = "https://test.com"
invocation_rate_limit_per_second = "1"
http_method = "GET"
connection_arn = aws_cloudwatch_event_connection.test123_event_connection.arn
}
resource "aws_cloudwatch_event_connection" "test123_event_connection" {
name = "test-dev-connection"
description = "A connection description"
authorization_type = "API_KEY"
auth_parameters {
api_key {
key = "test-key"
value = "TUVURVJSRUFESU5HOkNCWktXWFU3NzZDS0dGTk5LNjdGWUFVNFFRNE1HV0o3"
}
}
}
I believe what you seek is an aws_cloudwatch_event_target. This allows you to specify your destination as a target for a given rule, as documented in the official API destinations documentation.
resource "aws_cloudwatch_event_target" "this" {
rule = aws_cloudwatch_event_rule.test123_schedule_everyday.name
arn = aws_cloudwatch_event_api_destination.test123_event_api.arn
}
Related
I have created two lambda functions. now i wanted to pass all the cloudwatch logs from first lambda to second lambda. i have created new log group name and subscription filter to pass the cloudwatch logs to second lambda.
I am not sure if this configuration needs to be added any resource.
resource "aws_lambda_function" "audit-logs" {
filename = var.audit_filename
function_name = var.audit_function
source_code_hash = filebase64sha256(var.audit_filename)
role = module.lambda_role.arn
handler = "cloudwatch.lambda_handler"
runtime = "python3.9"
timeout = 200
description = "audit logs"
depends_on = [module.lambda_role, module.security_group]
}
resource "aws_cloudwatch_log_group" "splunk_cloudwatch_loggroup" {
name = "/aws/lambda/audit_logs"
}
resource "aws_lambda_permission" "allow_cloudwatch_for_splunk" {
statement_id = "AllowExecutionFromCloudWatch"
action = "lambda:InvokeFunction"
function_name = aws_lambda_function.splunk-logs.arn
principal = "logs.amazonaws.com" #"logs.region.amazonaws.com"
source_arn = "${aws_cloudwatch_log_group.splunk_cloudwatch_loggroup.arn}:*"
}
resource "aws_cloudwatch_log_subscription_filter" "splunk_cloudwatch_trigger" {
depends_on = [aws_lambda_permission.allow_cloudwatch_for_splunk]
destination_arn = aws_lambda_function.splunk-logs.arn
filter_pattern = ""
log_group_name = aws_cloudwatch_log_group.splunk_cloudwatch_loggroup.name
name = "splunk_filter"
}
# splunk logs lambda function
resource "aws_lambda_function" "splunk-logs" {
filename = var.splunk_filename
function_name = var.splunk_function
source_code_hash = filebase64sha256(var.splunk_filename)
role = module.lambda_role.arn
handler = "${var.splunk_handler}.handler"
runtime = "python3.9"
timeout = 200
description = "audit logs"
depends_on = [module.lambda_role, module.security_group]
}
how can i pass all the logs from first lambda to newly created log group ? any help ?
I have a lambda that I trigger with an EventBridge.
I have allowed_triggers in my lambda_function:
allowed_triggers = {
"RunDaily" = {
principal = "events.amazonaws.com"
source_arn = module.eventbridge.eventbridge_rule_arns["crons"]
}
}
And I have an eventbridge module:
module "eventbridge" {
source = "terraform-aws-modules/eventbridge/aws"
version = "1.14.0"
create_bus = false
create_role = false
create_rules = true
rules = {
crons = {
description = "deafault"
schedule_expression = "rate(1 day)"
}
}
targets = {
crons = [
{
arn = module.lambda_function.lambda_function_arn
input = jsonencode({ "job" : "crons" })
}
]
}
}
Now, this works great, as the rule is created and attached properly.
But when I want to change the name of the rule along with its description, terraform pickups only the description change:
module "eventbridge" {
...
rules = {
crons = {
description = "My custom cron rule"
schedule_expression = "rate(1 day)"
}
}
targets = {
crons = [
{
name = "my-custom-cron-rule-name"
arn = module.lambda_function.lambda_function_arn
input = jsonencode({ "job" : "crons" })
}
]
}
}
Plan:
Terraform will perform the following actions:
# module.eventbridge.aws_cloudwatch_event_rule.this["crons"] will be updated in-place
~ resource "aws_cloudwatch_event_rule" "this" {
~ description = "deafault" -> "My custom cron rule"
id = "crons-rule"
name = "crons-rule"
tags = {
"Name" = "crons-rule"
}
# (5 unchanged attributes hidden)
}
Plan: 0 to add, 1 to change, 0 to destroy.
Question: How do I change the name attribute for an eventbridge rule?
As per the module definition [1], the aws_cloudwatch_event_rule name is derived from value of the key of the rules block, i.e.:
rules = {
crons = {
description = "My custom cron rule"
schedule_expression = "rate(1 day)"
}
}
Based on the lines from the GitHub repo, the name is formed with:
locals {
eventbridge_rules = flatten([
for index, rule in var.rules :
merge(rule, {
"name" = index
"Name" = "${replace(index, "_", "-")}-rule"
})
])
... # rest of locals goes here
}
If you take a look at your definition and this part of code, you can see that the name will be crons-rule, which is visible in both the name and tags.Name arguments:
name = "crons-rule"
tags = {
"Name" = "crons-rule"
}
So in order to change the name of the rule, you would have to change the key of the rules block, i.e.:
rules = {
very-nice-new-crons = { # <----- here is where the change should be made
description = "My custom cron rule"
schedule_expression = "rate(1 day)"
}
}
You can verify this by looking at [2]:
resource "aws_cloudwatch_event_rule" "this" {
for_each = { for k, v in local.eventbridge_rules : v.name => v if var.create && var.create_rules }
name = each.value.Name
...
tags = merge(var.tags, {
Name = each.value.Name
})
}
EDIT: As pointed out, there are two more changes that need to be made after the name is changed:
The allowed_triggers of the Lambda function should now use the new key to reference the event rule that is allowed to trigger it. It has to be changed from
source_arn = module.eventbridge.eventbridge_rule_arns["crons"]
to
source_arn = module.eventbridge.eventbridge_rule_arns["very-nice-new-crons"]
The same name change has to be used in the targets block as well, i.e., the crons key in the targets has to be replaced with the same key name as in the rules block:
targets = {
very-nice-new-crons = [
{
name = "my-custom-cron-rule-name"
arn = module.lambda_function.lambda_function_arn
input = jsonencode({ "job" : "crons" })
}
]
}
[1] https://github.com/terraform-aws-modules/terraform-aws-eventbridge/blob/master/main.tf#L2-L6
[2] https://github.com/terraform-aws-modules/terraform-aws-eventbridge/blob/master/main.tf#L44
i am trying to build the terraform for sagemaker private work force with private cognito
Following : https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sagemaker_workforce
it working fine
main.tf
resource "aws_sagemaker_workforce" "workforce" {
workforce_name = "workforce"
cognito_config {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
}
}
resource "aws_cognito_user_pool" "user_pool" {
name = "sagemaker-cognito-userpool"
}
resource "aws_cognito_user_pool_client" "congnito_client" {
name = "congnito-client"
generate_secret = true
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_group" "user_group" {
name = "user-group"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_pool_domain" "domain" {
domain = "sagemaker-user-pool-ocr-domain"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_sagemaker_workteam" "workteam" {
workteam_name = "worker-team"
workforce_name = aws_sagemaker_workforce.workforce.id
description = "worker-team"
member_definition {
cognito_member_definition {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
user_group = aws_cognito_user_group.user_group.id
}
}
}
resource "aws_sagemaker_human_task_ui" "template" {
human_task_ui_name = "human-task-ui-template"
ui_template {
content = file("${path.module}/sagemaker-human-task-ui-template.html")
}
}
resource "aws_sagemaker_flow_definition" "definition" {
flow_definition_name = "flow-definition"
role_arn = var.aws_iam_role
human_loop_config {
human_task_ui_arn = aws_sagemaker_human_task_ui.template.arn
task_availability_lifetime_in_seconds = 1
task_count = 1
task_description = "Task description"
task_title = "Please review the Key Value Pairs in this document"
workteam_arn = aws_sagemaker_workteam.workteam.arn
}
output_config {
s3_output_path = "s3://${var.s3_output_path}"
}
}
it's creating the cognito user pool with callback urls. These callback urls is coming from aws_sagemaker_workforce.workforce.subdomain and getting set in cognito automatically which is what i want.
But i also want to set config in cognito userpool like
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
now when i add above two line we need to add callbackurl also which i dont want.
i tried
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
callback_urls = [aws_sagemaker_workforce.workforce.subdomain]
which is giving error :
Cycle: module.sagemaker.aws_cognito_user_pool_client.congnito_client, module.sagemaker.aws_sagemaker_workforce.workforce
as both resource are dependent on each other, i want to pass those two line but it forces me to add callback url also.
here is the final main.tf which is failing with that three line
resource "aws_sagemaker_workforce" "workforce" {
workforce_name = "workforce"
cognito_config {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
}
}
resource "aws_cognito_user_pool" "user_pool" {
name = "sagemaker-cognito-userpool"
}
resource "aws_cognito_user_pool_client" "congnito_client" {
name = "congnito-client"
generate_secret = true
user_pool_id = aws_cognito_user_pool.user_pool.id
explicit_auth_flows = ["ALLOW_REFRESH_TOKEN_AUTH", "ALLOW_USER_PASSWORD_AUTH", "ALLOW_CUSTOM_AUTH", "ALLOW_USER_SRP_AUTH"]
allowed_oauth_flows_user_pool_client = true
supported_identity_providers = ["COGNITO"]
allowed_oauth_flows = ["code", "implicit"]
allowed_oauth_scopes = ["email", "openid", "profile"]
callback_urls = [aws_sagemaker_workforce.workforce.subdomain]
}
resource "aws_cognito_user_group" "user_group" {
name = "user-group"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_cognito_user_pool_domain" "domain" {
domain = "sagemaker-user-pool-ocr-domain"
user_pool_id = aws_cognito_user_pool.user_pool.id
}
resource "aws_sagemaker_workteam" "workteam" {
workteam_name = "worker-team"
workforce_name = aws_sagemaker_workforce.workforce.id
description = "worker-team"
member_definition {
cognito_member_definition {
client_id = aws_cognito_user_pool_client.congnito_client.id
user_pool = aws_cognito_user_pool_domain.domain.user_pool_id
user_group = aws_cognito_user_group.user_group.id
}
}
}
resource "aws_sagemaker_human_task_ui" "template" {
human_task_ui_name = "human-task-ui-template"
ui_template {
content = file("${path.module}/sagemaker-human-task-ui-template.html")
}
}
resource "aws_sagemaker_flow_definition" "definition" {
flow_definition_name = "flow-definition"
role_arn = var.aws_iam_role
human_loop_config {
human_task_ui_arn = aws_sagemaker_human_task_ui.template.arn
task_availability_lifetime_in_seconds = 1
task_count = 1
task_description = "Task description"
task_title = "Please review the Key Value Pairs in this document"
workteam_arn = aws_sagemaker_workteam.workteam.arn
}
output_config {
s3_output_path = "s3://${var.s3_output_path}"
}
}
You do not need to specify the callback URL for the workforce. It is sufficient to specify the following in order to create the aws_cognito_user_pool_client resource:
callback_urls = [
"https://${aws_cognito_user_pool_domain.domain>.cloudfront_distribution_arn}",
]
Then you reference the user pool client in your workforce definition:
resource "aws_sagemaker_workforce" "..." {
workforce_name = "..."
cognito_config {
client_id = aws_cognito_user_pool_client.<client_name>.id
user_pool = aws_cognito_user_pool_domain.<domain_name>.user_pool_id
}
}
Existence of the callback URLs can be proven after applying the terraform configuration by running aws cognito-idp describe-user-pool-client --user-pool-id <pool_id> --client-id <client_id>:
"UserPoolClient": {
...
"CallbackURLs": [
"https://____.cloudfront.net",
"https://____.labeling.eu-central-1.sagemaker.aws/oauth2/idpresponse"
],
"LogoutURLs": [
"https://____.labeling.eu-central-1.sagemaker.aws/logout"
],
It seems as terraform itself does not do anything special on workforce creation (see https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/sagemaker/workforce.go). So the callback urls seem to be added by AWS SageMaker itself.
This means that you have to instruct terraform to ignore changes on those attributes in the aws_cognito_user_pool_client configuration:
lifecycle {
ignore_changes = [
callback_urls, logout_urls
]
}
all our AWS infra managed by Terraform, including the Sagemaker resources. We want to implement Autoscaling in our SM resources. We can't find Terraform solution to build our infra as a code.
In generally, ASG should be located in aws_sagemaker_endpoint_configuration >> production_variants blocks
references:
AWS documentation: https://aws.amazon.com/blogs/aws/auto-scaling-is-now-available-for-amazon-sagemaker/
TF documentation: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sagemaker_endpoint_configuration
Thanks in advance for your response
so, from my researches it should be something as:
resource "aws_appautoscaling_target" "sagemaker_target" {
max_capacity = var.max_instance_count
min_capacity = var.min_instance_count
resource_id = "endpoint/${aws_sagemaker_endpoint.endpoint.name}/variant/${var.service_name}-${var.site}-${var.environment}"
role_arn = aws_iam_role.sm_execution.arn
scalable_dimension = "sagemaker:variant:DesiredInstanceCount"
service_namespace = "sagemaker"
}
resource "aws_appautoscaling_policy" "sagemaker_policy" {
name = "${var.service_name}-${var.site}-${var.environment}-target-tracking"
policy_type = "TargetTrackingScaling"
resource_id = aws_appautoscaling_target.sagemaker_target.resource_id
scalable_dimension = aws_appautoscaling_target.sagemaker_target.scalable_dimension
service_namespace = aws_appautoscaling_target.sagemaker_target.service_namespace
target_tracking_scaling_policy_configuration {
predefined_metric_specification {
predefined_metric_type = "SageMakerVariantInvocationsPerInstance"
}
target_value = var.target_invocations
scale_in_cooldown = var.target_scale_in_cooldown
scale_out_cooldown = var.target_scale_out_cooldown
}
}
Taken a reference from the other answer, below is what my code looks like. Pasting here for some one's reference.
Note : I had to remove role_arn key from aws_appautoscaling_target resource so that it uses default service IAM role & also had to use SageMakerEndpointInvocationScalingPolicy string for policy name in aws_appautoscaling_policy resource to get same behavior as we get when we create auto scalling policy for sagemaker endpoint from aws console.
PS : Without above adjustments, TargetValue was not getting rendered wehn viewing from aws console & I was not manually able to change target value of terraform created auto scalling policy of sagemaker endpoint (as while updating TargetValue from console , was getting Validation Exception like Only one Target Tracking Scaling policy for a given metric specification is allowed. error etc
resource "aws_appautoscaling_target" "register_myendpoint_target" {
max_capacity = 2
min_capacity = 1
resource_id = "endpoint/${aws_sagemaker_endpoint.my_model-endpoint.name}/variant/${variant_name}"
scalable_dimension = "sagemaker:variant:DesiredInstanceCount"
service_namespace = "sagemaker"
}
resource "aws_appautoscaling_policy" "autoscale_policy_my_endpoint" {
name = "SageMakerEndpointInvocationScalingPolicy" # Had to use this name ditto so that on console it doesn't show some custom policy is configured etc etc
policy_type = "TargetTrackingScaling"
resource_id = aws_appautoscaling_target.register_myendpoint_target.resource_id
scalable_dimension = aws_appautoscaling_target.register_myendpoint_target.scalable_dimension
service_namespace = aws_appautoscaling_target.register_myendpoint_target.service_namespace
target_tracking_scaling_policy_configuration {
target_value = 100.0
predefined_metric_specification {
predefined_metric_type = "SageMakerVariantInvocationsPerInstance"
}
scale_in_cooldown = 300
scale_out_cooldown = 300
}
}
I have an issue with my terragrunt/terraform code as below.
I don't know the right way to retrieve my both crawlers created by my for_each loop.
Normally I create it with for and count.
I can't retrieve the correct values in my action triggers (main.tf).
terragrunt file (input):
inputs = {
glue_crawler = {
crawler = {
crawler_name = "test",
description = "test crawler"
},
crawler1 = {
crawler_name = "test2",
description = "test2 crawler"
}
}
}
main.tf
#crawler declaration
resource "aws_glue_crawler" "default" {
for_each = var.glue_crawler
database_name = aws_glue_catalog_database.database.name
name = "Crawler_${each.value.crawler_name}"
description = each.value.description
role = aws_iam_role.svc-glue-crawler.id
table_prefix = "raw_"
tags = var.tags
s3_target {
path = "${var.s3_glue_name}/${each.value.crawler_name}"
}
configuration = jsonencode(var.crawler_configuration)
}
...
#trigger
resource "aws_glue_trigger" "my_trigger" {
name = var.trigger_name
schedule = "cron(00 01 * * ? *)"
type = "SCHEDULED"
enabled = "false"
tags = var.tags
actions {
job_name = aws_glue_crawler.default[0].name
}
actions {
job_name = aws_glue_crawler.default[1].name
}
variable.tf
variable "glue_crawler" {
type = map(object({
crawler_name = string
description = string
}))
default = {}
description = "glue crawler definitions."
}
When i run this code i have the following errors:
Error: Invalid index
on main.tf line 294, in resource "aws_glue_trigger" "my_trigger": 294: job_name = aws_glue_crawler.default[0].name
|----------------
| aws_glue_crawler.default is object with 2 attributes
The given key does not identify an element in this collection value.
Error: Invalid index
on main.tf line 298, in resource "aws_glue_trigger" "my_trigger": 298: job_name = aws_glue_crawler.default[1].name
|----------------
| aws_glue_crawler.default is object with 2 attributes
The given key does not identify an element in this collection value.
When you use for_each instead of count you need to access the specific element with the key and not the index. So this will be crawler and crawler1 instead of 0 and 1 in your example:
resource "aws_glue_crawler" "default" {
for_each = var.glue_crawler
database_name = aws_glue_catalog_database.database.name
name = "Crawler_${each.value.crawler_name}"
description = each.value.description
role = aws_iam_role.svc-glue-crawler.id
table_prefix = "raw_"
tags = var.tags
s3_target {
path = "${var.s3_glue_name}/${each.value.crawler_name}"
}
configuration = jsonencode(var.crawler_configuration)
}
...
#trigger
resource "aws_glue_trigger" "my_trigger" {
name = var.trigger_name
schedule = "cron(00 01 * * ? *)"
type = "SCHEDULED"
enabled = "false"
tags = var.tags
actions {
job_name = aws_glue_crawler.default["crawler"].name
}
actions {
job_name = aws_glue_crawler.default["crawler1"].name
}
}
But of course that only works that specific input. Instead you should consider making the actions parameter dynamic and using for_each over the crawlers here too:
resource "aws_glue_trigger" "my_trigger" {
name = var.trigger_name
schedule = "cron(00 01 * * ? *)"
type = "SCHEDULED"
enabled = "false"
tags = var.tags
dynamic "actions" {
for_each = aws_glue_crawler.default
content {
job_name = actions.name
}
}
}