I m trying to create terraform script to launch the fastai instance from the marketplace.
I m adding image name as,
boot_disk {
initialize_params {
image = "<image name>"
}
}
When I add
click-to-deploy-images/deeplearning
from url
https://console.cloud.google.com/marketplace/details/click-to-deploy-images/deeplearning
is giving error,
Error: Error resolving image name 'click-to-deploy-images/deeplearning': Could not find image or family click-to-deploy-images/deeplearning
on fastai.tf line 13, in resource "google_compute_instance" "default":
13: resource "google_compute_instance" "default" {
If I use
debian-cloud/debian-9
from url
https://console.cloud.google.com/marketplace/details/debian-cloud/debian-stretch?project=<>
is working.
Can we deploy fastai image through terraform?
I made a deployment from the deep learning marketplace VM instance you share and review the source image[1], you should be able to use that url I provided to deploy with Terraform. I also notice a warning image stating that image is deprecated and there is this new version[2].
Hope this helps!
[1]sourceImage: https://www.googleapis.com/compute/v1/projects/click-to-deploy-images/global/images/tf2-2-1-cu101-20200109
[2]https://www.googleapis.com/compute/v1/projects/click-to-deploy-images/global/images/tf2-2-1-cu101-20200124
In this particular case, the name was "deeplearning-platform-release/pytorch-latest-gpu",
boot_disk {
initialize_params {
image = "deeplearning-platform-release/pytorch-latest-gpu"
...
}
}
Now I m able to create the instance.
To other newbies like me:
Apparently GCP Marketplace is using Deployment Manager which is google's own declarative tool to manage infrastructure. (I think modules are the closest abstraction in terraform to it.)
Hence, there is no simple/single answer to the question in the title.
In my opinion - if you start from scratch and/or can afford the effort the time - the best is to use terraform modules instead of GCP marketplace solutions - if such exists.
However, changes are good that you are importing an existing infra and you cannot just replace it immediately (or there is no such module).
In this case, I think the best that you can do is go to Deployment Manager in google console and open the particular deployment you need to import.
At this point you can see what resources make up the deployment. Probably there will be vm template(s), vm(s), firewall rule(s), etc...
Clicking on vm instance and the template will show you a lot of useful details.
Most importantly you can deduce what image was used.
E.g.:
In my case it showed:
sourceImage https://www.googleapis.com/compute/v1/projects/openvpn-access-server-200800/global/images/aspub275
From this I could define (based on an answer on issue #7319)
data "google_compute_image" "openvpn_server" {
name = "aspub275"
project = "openvpn-access-server-200800"
}
Which I could in turn use in google_compute_instance resource.
This will force a recreation of the VM though.
Related
Is it possible to create a terraform module that updates a specific resource which is created by another module?
Currently, I have two modules...
linux-system: which creates a linux vm with boot disks
disk-updater: which I'm planning to use to update the disks I created from the first module
The reason behind is I want to create a pipeline that will do disk operations tasks via terraform like disk resizing.
data "google_compute_disk" "boot_disk" {
name = "linux-boot-disk"
zone = "europe-west2-b"
}
resource "google_compute_disk" "boot_disk" {
name = data.google_compute_disk.boot_disk.name
zone = data.google_compute_disk.boot_disk.zone
size = 25
}
I tried to use data block to retrieve the existing disk details and pass it to resource block hoping to update the same disk but it seems like it will just try to create a new disk with the same name thats why im getting this error.
Error creating Disk: googleapi: Error 409: The resource ... already exists, alreadyExists
I think I'm doing it wrong, can someone give me advice how to proceed without using the first module I built. btw I'm a newbie when it comes to terraform
updates a specific resource which is created by another module?
No. You have to update the resource using its original definition.
The only way to update it from other module, is to import to the other module, which is bad design, as now you will have to definitions for the same resource, resulting in out-sync state files.
I am developing series of lambdas that are using docker images. The first step is to create them and registering in AWS ECR (not sure if everything I am doing is ok, so any advice is welcomed :-) ):
terraform {
...
required_providers {
docker = {
source = "kreuzwerker/docker"
version = ">= 2.12"
}
}
}
resource aws_ecr_repository lambda_repo {
name = "lambda"
}
resource docker_registry_image lambda_image {
name = "<account_id>.dkr.ecr.<region>.amazonaws.com/lambda:latest"
build {
context = "./code/lambda"
}
depends_on = [
aws_ecr_repository.lambda_repo
]
keep_remotely = true
}
resource aws_lambda_function lambda {
...
image_uri = "<account_id>.dkr.ecr.<region>.amazonaws.com/lambda:latest"
source_code_hash = docker_registry_image.lambda_image.sha256_digest
...
}
So with this code:
docker_registry_image > lambda_image : build the image and uploaded it in AWS
aws_lambda_function > lambda : if the image "lambda:latest" the lambda is updated with the new code
The problem I have is how to "force" docker_registry_image > lambda_image to rebuild the image and update the "lambda:latest" when the Dockerfile or app.py (the main code that is added in the file) has changed. Also I am not sure if this is the way to build the images.
Thanks!!
I was stuck with the exact same problem, and was disappointed to find your question hadn't been answered. I struggled a good bit, but I just clicked late tonight and got mine working.
The problem is incorrect thinking based on bad Docker habits (guilty of the same here!):
latest is a bad habit: it's based on tag mutability, which isn't how docker was designed, and pulling latest is non-deterministic, anyway - you never know what you're going to get. Usually, latest will pull the most recent version on a docker pull.
More on tag immutability: as developers, when encountering a small bug, we often quickly rebuild and overwrite the existing image with the same tag because "nobody will know, and it's just a small fix".
Thinking of the Lambda code files as something with state that should trigger a Terraform replace - the code files are not a resource, they're an asset for creating the Lambda.
Here is the better way to think about this:
docker_registry_image and the kreusewerker/docker provider are based on tag immutability.
docker_registry_image gets "replaced" in Terraform state (you'll see that in the Terraform plan when you try it), but the
effect in your ECR repository is to add a new image with a the next
sequential tag number, not to replace the actual image as one
usually thinks with Terraform.
Each change to your Lambda source code files requires a new image build with a new tag number.
If you find your images piling up on you, then you need a lifecycle policy to automate managing that.
Looking at your code, here is the solution:
Create a new variable called image_tag:
variable "image_tag" {
default = 1
}
Modify your docker_registry_image so that it uses the image_tag variable (also touching up the docker_registry_image name so you're not doing to much error-prone string building):
resource docker_registry_image lambda_image {
name = "${aws_ecr_repository.lambda_repo.repository_url}:${var.image_tag}"
build {
context = "./code/lambda"
}
...
}
Modify your aws_lambda_function. Change the image_uri to the name of the docker_registry_image so that those two are never out of sync:
resource aws_lambda_function lambda {
image_uri = docker_registry_image.lambda_image.name
source_code_hash = docker_registry_image.lambda_image.sha256_digest
}
Each time you change the code in the Lambda's source file(s), increment the image_tag variable by 1. Then try a terraform plan and you'll see that the docker_registry_image and aws_lambda_function will be replaced. A good exercise, would be to look at your ECR repo and Lambda function in the console while you do this. You'll see the images appearing in your ECR repo, and the Lambda function's image uri being updated with the new image_tag.
If you don't like how your image versions are piling up in your ECR repo, look into implementing a ecr_lifecycle_policy
Hope this helps. I sure feel a whole lot better tonight!
I'm trying to deploy a container in Cloud Run using Terraform. I've already deployed the container manually and run correctly. Now, I need to deploy it with Terraform to make a replicable environment.
I have already managed to configure practically all the parameters, except for this one.
CPU is always allocated
Change this parameter is very easy where I'm deploying manually.
Manual deploy
But I don't find where configure this parameter in the terraform files, main.tf or variables.tf. I'm using this module: https://github.com/GoogleCloudPlatform/terraform-google-cloud-run
Has anyone done it or could help me by indicating if it is possible and where it is configured.
Thank you so much.
You have the solution in the documentation here.You have to add an annotation, as you can find in the terraform documentation (but not for CPU Throttling), and add the correct entry, like that
resource "google_cloud_run_service" "default" {
name = "cloudrun-srv"
location = "us-central1"
template {
spec {
containers {
image = "us-docker.pkg.dev/cloudrun/container/hello"
}
}
metadata {
annotations = {
"run.googleapis.com/cpu-throttling" = false
}
}
}
autogenerate_revision_name = true
}
I have custom resource defined in my terraform module:
resource "aws_alb_target_group" "whatever"
{
....
}
Turns out whatever is not good name, and I need to update it.
Classic way of doing it would be to login onto each environment and execute terraform state mv, however I have lots of environments, and no automation for such action.
How I can change name of resource without manually moving state (only through editing terraform modules and applying plans)?
Based on the explanation in the question, I guess your best bet would be to use the moved block [1]. So for example, in your case that would be:
resource "aws_alb_target_group" "a_much_better_whatever"
{
....
}
moved {
from = aws_alb_target_group.whatever
to = aws_alb_target_group.a_much_better_whatever
}
EDIT: As #Matt Schuchard noted, the moved block is available only for Terraform versions >=1.1.0.
EDIT 2: As per #Martin Atkins' comments, changed the resource name to be the name of the resource moving to instead of moving from.
[1] https://www.terraform.io/language/modules/develop/refactoring#moved-block-syntax
I'm in the same situation.
My plan is to create the new resource group in Terraform, apply it, move the resources in the Azure portal to the new resource group, and then do terraform state mv to move the resources in terraform.
Yes, if you have a lot of resources its boring.. but I guess I won't brake anything this way.
I have a secret stored in AWS secret manager and trying to integrate that within terraform during runtime. We are using terraform 0.11.13 version, and updating to latest terraform is in the roadmap.
We all want to use the jsondecode() available as part of latest terraform, but need to get few things integrated before we upgrade our terraform.
We tried to use the below helper external data program suggested as part of https://github.com/terraform-providers/terraform-provider-aws/issues/4789.
data "external" "helper" {
program = ["echo", "${replace(data.aws_secretsmanager_secret_version.map_example.secret_string, "\\\"", "\"")}"]
}
But we ended up getting this error now.
data.external.helper: can't find external program "echo"
Google search didn't help much.
Any help will be much appreciated.
OS: Windows 10
It sounds like you want to use a data source for the aws_secretsmanager_secret.
Resources in terraform create new resources. Data sources in terraform reference the value of existing resources in terraform.
data "aws_secretsmanager_secret" "example" {
arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-123456"
}
data "aws_secretsmanager_secret_version" "example" {
secret_id = data.aws_secretsmanager_secret.example.id
version_stage = "example"
}
Note: you can also use the secret name
Docs: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/secretsmanager_secret
Then you can use the value from this like so:
output MySecretJsonAsString {
value = data.aws_secretsmanager_secret_version.example.secret_string
}
Per the docs, the secret_string property of this resource is:
The decrypted part of the protected secret information that was originally provided as a string.
You should also be able to pass that value into jsondecode and then access the properties of the json body individually.
but you asked for a terraform 0.11.13 solution. If the secret value is defined by terraform you can use the terraform state datasource to get the value. This does trust that nothing else is updating the secret other than terraform. But the best answer is to upgrade your terraform. This could be a useful stopgap until then.
As a recommendation, you can make the version of terraform specific to a module and not your whole organization. I do this through the use of docker containers that run specific versions of the terraform bin. There is a script in the root of every module that will wrap the terraform commands to come up in the version of terraform meant for that project. Just a tip.