I'm trying to deploy a Dataflow template with Terraform in GCloud.
There are several tutorial which include some terraform code. There are 2 options:Use module like the following link or use resource like the following link
With both options I have the following error:
Error: googleapi: got HTTP response code 502 with body: <!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 502 (Server Error)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>502.</b> <ins>That’s an error.</ins>
<p>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds. <ins>That’s all we know.</ins>
on .terraform\modules\dataflow-job\terraform-google-modules-terraform-google-dataflow-722fc1d\main.tf line 17, in resource "google_dataflow_job" "dataflow_job":
17: resource "google_dataflow_job" "dataflow_job" {
I have tried running from my local computer and also from the cloud shell inside GCP.
The problem should be in the dataflow module, because I also tried to create other resource like a bucket and compute engine and it works without any problem.
The dataflow template is storaged in a bucket before I run the terraform script.
Terraform version: 0.12.19
The code:
main.tf
variable "project_id" {}
<...>
provider "google" {
version = "~> 2.8.0"
region = var.region
}
resource "google_dataflow_job" "dataflow_job" {
project = var.project_id
region = var.region
zone = "${var.region}-a"
name = var.project_name
on_delete = "cancel"
max_workers = var.max_workers
template_gcs_path = var.template_location
temp_gcs_location = "gs://${var.gcs_location}/tmp_dir"
service_account_email = var.controller_service_account_email
parameters = {
inputPubSub = var.input_PubSub_subscription
outputPubSub = var.output_PubSub_subscription
}
machine_type = var.machine_type
}
terraform.tfvars
<...>
template_location = "gs://www/zzz/template"
gcs_location= "gs://www/yyy"
<...>
To test if my code is wrong, I also tried directly from the code of link and also the same error.
Am I missing any dependence to add to the code?
Note that you have declared temp_gcs_location as "gs://${var.gcs_location}/tmp_dir" but then, in terraform.tvars you set gcs_location as "gs://www/yyy" (so the gs:// prefix appears twice). In any case, the job should be launched but then fail to create.
I made a minimal example with the following versions:
$ terraform --version
Terraform v0.12.20
+ provider.google v3.5.0
and using the Google-provided word count template. My main.tf file is:
variable "project_id" {
type = string
description = "GCP Project ID."
}
variable "gcs_location" {
type = string
description = "GCS bucket name (no gs:// prefix)."
}
provider "google" {
project = var.project_id
region = "us-central1"
zone = "us-central1-c"
}
resource "google_dataflow_job" "wordcount" {
name = "wordcount"
template_gcs_path = "gs://dataflow-templates/latest/Word_Count"
temp_gcs_location = "gs://${var.gcs_location}/temp"
parameters = {
inputFile = "gs://dataflow-samples/shakespeare/kinglear.txt"
output = "gs://${var.gcs_location}/wordcount/output"
}
}
and df.tfvars (change with the appropriate values):
project_id = "PROJECT_ID"
gcs_location = "BUCKET_NAME"
I run it with:
terraform apply -var-file="df.tvars"
and the job is successfully created:
google_dataflow_job.wordcount: Creating...
google_dataflow_job.wordcount: Creation complete after 3s [id=2020-01-27_...]
Let me know if this helps.
Related
I am trying to update a test AWS Transfer Server because I was unable to connect to it via SFTP
Now trying to use the FTP / FTPS protocols, I have used the same layout as the example here
This is the example in the docs
resource "aws_transfer_server" "example" {
endpoint_type = "VPC"
endpoint_details {
subnet_ids = [aws_subnet.example.id]
vpc_id = aws_vpc.example.id
}
protocols = ["FTP", "FTPS"]
certificate = aws_acm_certificate.example.arn
identity_provider_type = "API_GATEWAY"
url = "${aws_api_gateway_deployment.example.invoke_url}${aws_api_gateway_resource.example.path}"
}
And here is my code
resource "aws_transfer_server" "transfer_x3" {
tags = {
Name = "${var.app}-${var.env}-transfer-x3-server"
}
endpoint_type = "VPC"
endpoint_details {
vpc_id = data.aws_vpc.vpc_global.id
subnet_ids = [data.aws_subnet.vpc_subnet_pri_commande_a.id, data.aws_subnet.vpc_subnet_pri_commande_b.id]
}
protocols = ["FTP", "FTPS"]
certificate = var.certificate_arn
identity_provider_type = "API_GATEWAY"
url = "https://${aws_api_gateway_rest_api.Api.id}.execute-api.${var.region}.amazonaws.com/latest/servers/{serverId}/users/{username}/config"
invocation_role = data.aws_iam_role.terraform-commande.arn
}
And here is the error message
╷
│ Error: error creating Transfer Server: InvalidRequestException: Bad value in IdentityProviderDetails
│
│ with aws_transfer_server.transfer_x3,
│ on transfer-x3.tf line 1, in resource "aws_transfer_server" "transfer_x3":
│ 1: resource "aws_transfer_server" "transfer_x3" {
│
╵
My guess is, it doesn't like the value in the url parameter
I have tried using the same form as one provided in the example: url = "${aws_api_gateway_deployment.ApiDeployment.invoke_url}${aws_api_gateway_resource.ApiResourceServerIdUserUsernameConfig.path}", but encountered the same error message
I have tried ordering the parameters around if it was that, but I had the same error over and over when I use the command terraform apply
The commands terraform validate and terraform plan didn't show the error message at all
What value could the url parameter need? Or is there a parameter missing in my resource declaration?
As per the documentation (CloudFormation in this case) [1], the examples say the only thing needed is the invoke URL of the API Gateway:
.
.
.
"IdentityProviderDetails": {
"InvocationRole": "Invocation-Role-ARN",
"Url": "API_GATEWAY-Invocation-URL"
},
"IdentityProviderType": "API_GATEWAY",
.
.
.
Comparing that to the attributes provided by the API Gateway stage resource in terraform, the only thing that is needed is the invoke_url attribute [2].
[1] https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-transfer-server.html#aws-resource-transfer-server--examples
[2] https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/api_gateway_stage#invoke_url
I'm having problems creating a new EKS version 1.22 in a dev environment.
I'm using the module in Terraform registry, trimming some parts since it's only for testing purposes (we just want to test the version 1.22).
I'm using a VPC that was created for testing EKS's, and 2 public subnets and 2 private subnets.
This is my main.tf:
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "18.21.0"
cluster_name = "EKSv2-update-test"
cluster_version = "1.22"
cluster_endpoint_private_access = true
cluster_endpoint_public_access = true
cluster_addons = {
coredns = {
resolve_conflicts = "OVERWRITE"
}
kube-proxy = {}
vpc-cni = {
resolve_conflicts = "OVERWRITE"
}
}
vpc_id = "vpc-xxx" # eks-vpc
subnet_ids = ["subnet-priv-1-xxx", "subnet-priv-2-xxx", "subnet-pub-1-xxx", "subnet-pub-2-xxx"]
}
Terraform apply times out after 20 min (it just hangs on module.eks.aws_eks_addon.this["coredns"]: Still creating... [20m0s elapsed])
and this is the error
│ Error: unexpected EKS Add-On (EKSv2-update-test:coredns) state returned during creation: timeout while waiting for state to become 'ACTIVE' (last state: 'DEGRADED', timeout: 20m0s)
│ [WARNING] Running terraform apply again will remove the kubernetes add-on and attempt to create it again effectively purging previous add-on configuration
│
│ with module.eks.aws_eks_addon.this["coredns"],
│ on .terraform/modules/eks/main.tf line 305, in resource "aws_eks_addon" "this":
│ 305: resource "aws_eks_addon" "this" {
The EKS gets created, but this is clearly not the way to go.
Regarding coredns, what am I missing?
Thanks
a minimum of 2 cluster nodes are required for addon coredns to meet its requirements for its replica set
I am working to build the packer pipeline which would use the market place ami to install certain softwares and create an ami. I had created the json template which are working fine but as per the packer recommendation, I am working to upgrade it to the hcl2 template.
When i run the hcl2_upgrade command. I see a the json template is converted to the .pkr.hcl template but while running it. I have done some customization to template as per the recommded in packer documentation.It gives me below error.
data "amazon-ami" "autogenerated_1"{
access_key = "${var.aws_access_key}"
filters = {
root-device-type = "ebs"
virtualization-type = "hvm"
name = "**** Linux *"
}
most_recent = true
region = "${var.aws_region}"
owners = ["${var.owner_id}"]
secret_key = "${var.aws_secret_key}"
}
when I am trying to consume this ami id in the source block It gives me error.
ami_name = "${var.ami_name}"
associate_public_ip_address = false
force_deregister = true
iam_instance_profile = "abc"
instance_type = "****"
region = "${var.aws_region}"
source_ami = "{data.amazon-ami.autogenerated_1.id}"
ssh_interface = "private_ip"
ssh_username = "user"
subnet_id = "subnet-********"
vpc_id = "vpc-***********"
}
Error details are below:
amazon-ebs.pqr_ami: status code: 400, request id: *********
Build 'amazon-ebs.pqr_ami' errored after 1 second 49 milliseconds: Error querying AMI: InvalidAMIID.Malformed: Invalid id: "{data.amazon-ami.autogenerated_1.id}" (expecting "ami-...")
status code: 400, request id: ************
Your AMI is literally a string source_ami = "{data.amazon-ami.autogenerated_1.id}". It should be:
source_ami = "${data.amazon-ami.autogenerated_1.id}"
or for HCL2:
source_ami = data.amazon-ami.autogenerated_1.id
Ensure that the AMI ID is in the same region as what was specified in the script
Despite using depends_on directive, it looks like zip is not created before trying to put it in the bucket. Considering pipeline output, somehow it just omits archiving the file before firing upload to bucket. Both files (index.js and package.json) exists.
resource "google_storage_bucket" "cloud-functions" {
project = var.project-1-id
name = "${var.project-1-id}-cloud-functions"
location = var.project-1-region
}
resource "google_storage_bucket_object" "start_instance" {
name = "start_instance.zip"
bucket = google_storage_bucket.cloud-functions.name
source = "${path.module}/start_instance.zip"
depends_on = [
data.archive_file.start_instance,
]
}
data "archive_file" "start_instance" {
type = "zip"
output_path = "${path.module}/start_instance.zip"
source {
content = file("${path.module}/scripts/start_instance/index.js")
filename = "index.js"
}
source {
content = file("${path.module}/scripts/start_instance/package.json")
filename = "package.json"
}
}
Terraform has been successfully initialized!
$ terraform apply -input=false "planfile"
google_storage_bucket_object.stop_instance: Creating...
google_storage_bucket_object.start_instance: Creating...
Error: open ./start_instance.zip: no such file or directory
on cloud_functions.tf line 41, in resource "google_storage_bucket_object" "start_instance":
41: resource "google_storage_bucket_object" "start_instance" {
LOGS:
2020-11-18T13:02:56.796Z [DEBUG] plugin.terraform-provider-google_v3.40.0_x5: 2020/11/18 13:02:56 [WARN] Failed to read source file "./start_instance.zip". Cannot compute md5 hash for it.
2020/11/18 13:02:56 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_storage_bucket_object.stop_instance, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .detect_md5hash: planned value cty.StringVal("different hash") does not match config value cty.NullVal(cty.String)
2020/11/18 13:02:56 [WARN] Provider "registry.terraform.io/hashicorp/google" produced an invalid plan for google_storage_bucket_object.start_instance, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .detect_md5hash: planned value cty.StringVal("different hash") does not match config value cty.NullVal(cty.String)
I have exactly the same issue with GitLab CI/CD pipeline. After some digging, according to the discussion I found out that with this setup, the plan and apply stages are run in separate containers, and the archiving step is executed in the plan stage.
A workaround is to create a dummy trigger with null_resource and force the archive_file to depend on it, and, hence, to be executed in the apply stage.
resource null_resource dummy_trigger {
triggers = {
timestamp = timestamp()
}
}
resource "google_storage_bucket" "cloud-functions" {
project = var.project-1-id
name = "${var.project-1-id}-cloud-functions"
location = var.project-1-region
}
resource "google_storage_bucket_object" "start_instance" {
name = "start_instance.zip"
bucket = google_storage_bucket.cloud-functions.name
source = "${path.module}/start_instance.zip"
depends_on = [
data.archive_file.start_instance,
]
}
data "archive_file" "start_instance" {
type = "zip"
output_path = "${path.module}/start_instance.zip"
source {
content = file("${path.module}/scripts/start_instance/index.js")
filename = "index.js"
}
source {
content = file("${path.module}/scripts/start_instance/package.json")
filename = "package.json"
}
depends_on = [
resource.null_resource.dummy_trigger,
]
}
google_cloud_run_service.skills-online: Creating...
Error: Error creating Service: googleapi: got HTTP response code 404 with body:
<title>Error 404 (Not Found)!!1</title> <style> *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px} </style>
404. That’s an error.
The requested URL /apis/serving.knative.dev/v1/namespaces/skills-online/services was not found on this server. That’s all we know.
on main.tf line 11, in resource "google_cloud_run_service" "skills-online":
11: resource "google_cloud_run_service" "skills-online" {
creating google cloud run using infrastructure as code terraform
after $ terraform apply return HTTP response 404 with body, during the creation
Terraform Version
terraform version: v0.12.24
Google provider: v3.19.0
Terraform Configuration Files
provider "google" {
credentials = file("skills-online.json")
project = "skills-online"
region = "asia-south1"
}
resource "google_project_service" "run" {
service = "run.googleapis.com"
}
resource "google_cloud_run_service" "skills-online" {
name = "skills-online"
location = "asia-south1"
template {
spec {
containers {
image = "gcr.io/skills-online/imagename"
}
}
}
traffic {
percent = 100
latest_revision = true
}
}
terraform.tf file Affected Resource(s) google_cloud_run_service
Debug Output file
https://gist.github.com/Ameen-Alam/7765d293028a9fd7b29630edfb8e45a3
Expected Behavior
Cloud run service created
Important Factoids
image = "gcr.io/skills-online/imagename"
image = "gcr.io/skills-online/---------------"
due to security reason, I wrote the name of the image as a dummy
opened this issue by Ameen Alam
https://github.com/terraform-providers/terraform-provider-google/issues/6258
The google cloud run service is not available in asia-south1 region
which is why
"Error creating Service: googleapi: got HTTP response code 404 with body:"
is occurring, during creation google_cloud_run_service
I changed the region in my code, according to this list
https://cloud.google.com/run/docs/locations.
I have selected asia-east1 (Taiwan) region.
The error has been resolved.
closed this issue
https://github.com/terraform-providers/terraform-provider-google/issues/6258