We are creating confluent kafka sink connector (https://registry.terraform.io/providers/confluentinc/confluent/latest/docs/resources/confluent_connector) using terraform.
resource "confluent_connector" "gcs-sink" {
for_each = { for topic in var.topics : "${topic.name} ${topic.tasks}" => topic }
environment {
id = var.env_id
}
kafka_cluster {
id = var.cluster_id
}
config_nonsensitive = {
"name" = "${each.value.name}-gcs-connector"
"connector.class" = "GcsSink"
"topics" = "${each.value.name}"
"kafka.auth.mode" = "SERVICE_ACCOUNT"
"kafka.service.account.id" = "${var.connector_sa}"
"gcs.bucket.name" = "${var.gcs_bucket_name}"
"input.data.format" = "AVRO"
"output.data.format" = "AVRO"
"time.interval" = "HOURLY"
"flush.size" = "1000"
"tasks.max" = "${each.value.tasks}"
"topics.dir" = "avro-hourly"
"path.format" = "'process_date'=YYYY-MM-dd/'hour'=HH"
"rotate.schedule.interval.ms" = "60000"
"gcs.credentials.config" = var.gcs_sa_json
}
}
We need to pass service json key file to gcs.credentials.config, so i'm placing the json file in gcp secret manager and reading it from secret manger on runtime and storing it in variable gcs_sa_json, but I'm running into the below issue
Error: error waiting for Connector "g-gg-prod-gcs-connector" to provision: connector "display_name"="g-gg-prod-gcs-connector" provisioning status is "FAILED": Unable to validate configuration. If an update was made to the configuration, this means that the configuration was invalid, and the connector continues to operate on a previous configuration that passed validation. Errors:
│ gcs.credentials.config: Unable to retrieve credentials
│ gcs.bucket.name: Unable to retrieve credentials
│ . You might need to remove Connector manually before retrying.
│
│ with module.sink_connector.confluent_connector.gcs-sink["g-gg-prod-topic 2"],
│ on ../../terraform-modules/confluent-kafka/sink-connector/main.tf line 1, in resource "confluent_connector" "gcs-sink":
│ 1: resource "confluent_connector" "gcs-sink" {
I used the jsondecode function as well, but in vain.
Related
I am trying to update a test AWS Transfer Server because I was unable to connect to it via SFTP
Now trying to use the FTP / FTPS protocols, I have used the same layout as the example here
This is the example in the docs
resource "aws_transfer_server" "example" {
endpoint_type = "VPC"
endpoint_details {
subnet_ids = [aws_subnet.example.id]
vpc_id = aws_vpc.example.id
}
protocols = ["FTP", "FTPS"]
certificate = aws_acm_certificate.example.arn
identity_provider_type = "API_GATEWAY"
url = "${aws_api_gateway_deployment.example.invoke_url}${aws_api_gateway_resource.example.path}"
}
And here is my code
resource "aws_transfer_server" "transfer_x3" {
tags = {
Name = "${var.app}-${var.env}-transfer-x3-server"
}
endpoint_type = "VPC"
endpoint_details {
vpc_id = data.aws_vpc.vpc_global.id
subnet_ids = [data.aws_subnet.vpc_subnet_pri_commande_a.id, data.aws_subnet.vpc_subnet_pri_commande_b.id]
}
protocols = ["FTP", "FTPS"]
certificate = var.certificate_arn
identity_provider_type = "API_GATEWAY"
url = "https://${aws_api_gateway_rest_api.Api.id}.execute-api.${var.region}.amazonaws.com/latest/servers/{serverId}/users/{username}/config"
invocation_role = data.aws_iam_role.terraform-commande.arn
}
And here is the error message
╷
│ Error: error creating Transfer Server: InvalidRequestException: Bad value in IdentityProviderDetails
│
│ with aws_transfer_server.transfer_x3,
│ on transfer-x3.tf line 1, in resource "aws_transfer_server" "transfer_x3":
│ 1: resource "aws_transfer_server" "transfer_x3" {
│
╵
My guess is, it doesn't like the value in the url parameter
I have tried using the same form as one provided in the example: url = "${aws_api_gateway_deployment.ApiDeployment.invoke_url}${aws_api_gateway_resource.ApiResourceServerIdUserUsernameConfig.path}", but encountered the same error message
I have tried ordering the parameters around if it was that, but I had the same error over and over when I use the command terraform apply
The commands terraform validate and terraform plan didn't show the error message at all
What value could the url parameter need? Or is there a parameter missing in my resource declaration?
As per the documentation (CloudFormation in this case) [1], the examples say the only thing needed is the invoke URL of the API Gateway:
.
.
.
"IdentityProviderDetails": {
"InvocationRole": "Invocation-Role-ARN",
"Url": "API_GATEWAY-Invocation-URL"
},
"IdentityProviderType": "API_GATEWAY",
.
.
.
Comparing that to the attributes provided by the API Gateway stage resource in terraform, the only thing that is needed is the invoke_url attribute [2].
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-transfer-server.html#aws-resource-transfer-server--examples
[2] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_stage#invoke_url
I prepared the following terraform scripts to assign an LF-tag to a database in lake formation.
resource "aws_lakeformation_resource_lf_tags" "gm_access" {
count = length(var.db_config)
database {
name = "gm_${var.db_config[count.index].name}_${terraform.workspace}"
}
lf_tag {
key = "access"
value = var.db_config[count.index].access
}
}
The LF Tag access has already been created in AWS manually (historically) with values defined.
I received errors:
│ Error: creating AWS Lake Formation Resource LF Tags (): attempted to add 1 tags, 1 failures
│
│ with aws_lakeformation_resource_lf_tags.gm_access[0],
│ on self_serve.tf line 72, in resource "aws_lakeformation_resource_lf_tags" "gm_access":
│ 72: resource "aws_lakeformation_resource_lf_tags" "gm_access" {
│
Any advice, please?
I have created two shared VPCs for my organization, one for prod and one for non-prod usage.
For the these shared VPCs I want to create a description and define subnets, but I cannot find the right entries in terraform for these elements.
I.e. Here is how I defined the resource block to specify the host project (and create a shared VPC) :
resource "google_compute_shared_vpc_host_project" "dev-shared-shared-vpc-host" {
provider = google.as_network_admin
project = google_project.dev-shared-vpc-host.project_id
}
Now when I try to create the subnet :
resource "google_compute_subnetwork" "dev-subnetwork" {
provider = google.as_network_admin
name = var.vpc_and_subnet_info.for_dev_env.subnetwork.name
ip_cidr_range = var.vpc_and_subnet_info.for_dev_env.subnetwork.ip_cidr_range
region = var.region
secondary_ip_range {
range_name = var.vpc_and_subnet_info.for_dev_env.subnetwork.secondary_ip_range.name
ip_cidr_range = var.vpc_and_subnet_info.for_dev_env.subnetwork.secondary_ip_range.ip_cidr_range
}
network = google_compute_shared_vpc_host_project.dev-shared-shared-vpc-host.id
project = google_project.dev-shared-vpc-host.id
}
I get an error like
╷
│ Error: Error creating Subnetwork: googleapi: Error 400: Invalid value for field 'resource.network': 'projects/projects/<redacted_project_id>/global/networks/<redacted_project_id>'. The URL is malformed., invalid
│
│ with google_compute_subnetwork.dev-subnetwork,
│ on networking.tf line 5, in resource "google_compute_subnetwork" "dev-subnetwork":
│ 5: resource "google_compute_subnetwork" "dev-subnetwork" {
│
Obviously the projects/projects/.. is messing up the network parameter, but in the documentation for google_compute_shared_vpc_host_project there isn't any other output other than id. And for the input arguments there is no description. However, when I try to manually create the shared VPC, I can enter a description, and create a subnet.
Mind you, the google_compute_network that creates regular VPCs is quite well documented and the subnet that I defined above works well with it.
EDIT :
Fixing the project argument into project = google_project.dev-shared-vpc-host.project_id instead of id removes the projects/projects/... network error, but gives this error instead :
╷
│ Error: Error creating Subnetwork: googleapi: Error 404: The resource 'projects/<redacted_project_id>/global/networks/<redacted_project_id>' was not found, notFound
│
│ with google_compute_subnetwork.dev-subnetwork,
│ on networking.tf line 5, in resource "google_compute_subnetwork" "dev-subnetwork":
│ 5: resource "google_compute_subnetwork" "dev-subnetwork" {
│
╵
I seem to have misunderstood the creation of a shared VPC via google_compute_shared_vpc_host_project, this does not create a vpc perse, but only designates a project as the host project, thus sharing a vpc that must exist beforehand.
Therefore I should have created a google_compute_network beforehand, here is the HCL necessary to achieve what I wanted in the question :
resource "google_compute_network" "dev-vpc-network" {
provider = google.as_network_admin
name = var.vpc_and_subnet_info.for_dev_env.vpc.name
auto_create_subnetworks = var.vpc_and_subnet_info.for_dev_env.vpc.auto_create_subnetworks
project = google_project.dev-shared-vpc-host.project_id
description = var.vpc_and_subnet_info.for_dev_env.vpc.description
}
resource "google_compute_shared_vpc_host_project" "dev-shared-shared-vpc-host" {
provider = google.as_network_admin
project = google_project.dev-shared-vpc-host.project_id
}
resource "google_compute_subnetwork" "dev-subnetwork" {
provider = google.as_network_admin
name = var.vpc_and_subnet_info.for_dev_env.subnetwork.name
ip_cidr_range = var.vpc_and_subnet_info.for_dev_env.subnetwork.ip_cidr_range
region = var.region
secondary_ip_range {
range_name = var.vpc_and_subnet_info.for_dev_env.subnetwork.secondary_ip_range.name
ip_cidr_range = var.vpc_and_subnet_info.for_dev_env.subnetwork.secondary_ip_range.ip_cidr_range
}
network = google_compute_network.dev-vpc-network.id
project = google_project.dev-shared-vpc-host.project_id
}
Of course, in these examples I used variables declared in .tfvars to fill in the name, and other arguments needed in the resource blocks.
I am building a lambda in terraform using it's AWS module and my code is as below:
module "lambda_function" {
# * Lambda module configs
source = "terraform-aws-modules/lambda/aws"
version = "3.0.0"
# * Lambda Configs
function_name = "${var.function_name}-${var.env}"
description = "My Project"
handler = local.constants.lambda.HANDLER
runtime = local.constants.lambda.VERSION
memory_size = 128
cloudwatch_logs_retention_in_days = 14
source_path = "./function/"
timeout = local.constants.lambda.TIMEOUT
create_async_event_config = true
maximum_retry_attempts = local.constants.lambda.RETRIES_ATTEMPT
layers = [
data.aws_lambda_layer_version.layer_requests.arn
]
environment_variables = {
AWS_ACCOUNT = var.env
SLACK_HOOK_CHANNEL = var.SLACK_HOOK_CHANNEL
}
tags = {
Name = "${var.function_name}-${var.env}"
}
trusted_entities = local.constants.lambda.TRUSTED_ENTITIES
}
This code works fine and the lambda get's deployed. Now i need to put the lambda in the VPC. When i add the code below in the resource block, i get the error error modifying Lambda Function (lambda_name) configuration : ValidationException: │ status code: 400, request id: de2641f6-1125-4c83-87fa-3fe32dee7b06 │ │ with module.lambda_function.aws_lambda_function.this[0], │ on .terraform/modules/lambda_function/main.tf line 22, in resource "aws_lambda_function" "this": │ 22: resource "aws_lambda_function" "this" {
The code for the vpc is:
# * VPC configurations
vpc_subnet_ids = ["10.21.0.0/26", "10.21.0.64/26", "10.21.0.128/26"]
vpc_security_group_ids = ["sg-ffffffffff"] # Using a dummy value here
attach_network_policy = true
If i use the same values in the AWS console and deploy the lambda in the VPC, it works fine.
Can someone please help ?
You have to provide valid subnet ids, not CIDR ranges. So instead of
vpc_subnet_ids = ["10.21.0.0/26", "10.21.0.64/26", "10.21.0.128/26"]
it should be
vpc_subnet_ids = ["subnet-asfid1", "subnet-asfid2", "subnet-as4id1"]
I'm trying deploy Service catalog via Terraform. When I try deploy Service catalog product by code:
#Service catalog product
resource "aws_servicecatalog_product" "linuxDesktop" {
name = "Linux Desktop"
description= "Cloud development environment configured for engineering staff. Runs AWS Linux."
owner = "IT"
type = "CLOUD_FORMATION_TEMPLATE"
provisioning_artifact_parameters {
template_url = "https://fdfdasfadfdf.s3.us-west-2.amazonaws.com/development-environment.yaml"
}
}
I got error from terraform:
aws_servicecatalog_portfolio.portfolio: Creation complete after 2s [id=port-xe2ql6s2myy3s]
╷
│ Error: error creating Service Catalog Product: InvalidParametersException: The CLOUD_FORMATION_TEMPLATE Product Type only supports the following ProvisioningArtifact Types: CLOUD_FORMATION_TEMPLATE, ACCOUNT_FACTORY
│
│ with aws_servicecatalog_product.linuxDesktop,
│ on main.tf line 31, in resource "aws_servicecatalog_product" "linuxDesktop":
│ 31: resource "aws_servicecatalog_product" "linuxDesktop" {
You have to add type to provisioning_artifact_parameters as well:
resource "aws_servicecatalog_product" "linuxDesktop" {
name = "Linux Desktop"
description= "Cloud development environment configured for engineering staff. Runs AWS Linux."
owner = "IT"
type = "CLOUD_FORMATION_TEMPLATE"
provisioning_artifact_parameters {
template_url = "https://fdfdasfadfdf.s3.us-west-2.amazonaws.com/development-environment.yaml"
type = "CLOUD_FORMATION_TEMPLATE"
}
}