I am try to creat a compute instance using Googles deployment manager. It has shall get two disk, one based on a boot image and a second that shall be blank. The blank disk will later on be formated and mounted correctly by salt stack. Deployment manager complains with "Source image must be specified." How do I create a second blank disk for a compute instance using deployment manager?
My compute-instance.jinja:
resources:
- type: compute.v1.instance
name: {{ env["deployment"] }}-{{ env["name"] }}
properties:
zone: europe-west1-c
machineType: zones/europe-west1-c/machineTypes/n1-standard-1
disks:
- deviceName: {{ env["deployment"] }}-{{ env["name"] }}
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: global/images/XXXXXXX
- deviceName: {{ env["deployment"] }}-{{ env["name"] }}-data
type: PERSISTENT
boot: false
autoDelete: true
initializeParams:
diskSizeGb: {{ properties["size"] }}
networkInterfaces:
- network: global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
Solved by creating a separate resource for the disk using:
- type: compute.v1.disk
name: {{ env["deployment"] }}-{{ env["name"] }}-1-data
properties:
sizeGb: {{ properties["size"] }}
zone: europe-west1-c
Then refer to it from the compute-instance:
- deviceName: {{ env["deployment"] }}-{{ env["name"] }}-1-data
boot: false
autoDelete: true
source: $(ref.{{ env["deployment"] }}-{{ env["name"] }}-1-data.selfLink)
If you must supply the source image, you can create an image of an empty disk.
The downside of this, is that you will start paying (not much tough) for 10gb of image storage when your actually storing nothing.
gcloud compute disks create emptydisk --size 10GB
gcloud compute images create empty-disk-image --source-disk emptydisk
And then use it as the source image
- deviceName: {{ env["deployment"] }}-{{ env["name"] }}-data
type: PERSISTENT
boot: false
autoDelete: true
initializeParams:
sourceImage: projects/your-project-id/global/images/empty-disk-image
diskSizeGb: {{ properties["size"] }}
Related
I am using the below jijna instance creation template in GCP. When I create this I get the below error.
I have already tried giving even Owner and iam.serviceAccountUser permission to xxxxxxxxxx#cloudservices.gserviceaccount.com account.
I have also added xxxxxxxxxx#cloudservices.gserviceaccount.com user with iam.serviceAccountUser role under Access control section as member in my new service account I have created which is test-vm-gke-init-vm#.iam.gserviceaccount.com.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1663746707027-5e92b3777f488-7748cc3b-a3d3cc8f]: errors:
- code: RESOURCE_ERROR
location: /deployments/demo-vm/resources/test-vm
message: "{\"ResourceType\":\"compute.v1.instance\",\"ResourceErrorCode\":\"SERVICE_ACCOUNT_ACCESS_DENIED\"\
,\"ResourceErrorMessage\":\"The user does not have access to service account 'serviceAccount:test-vm-gke-init-vm#<myproject>.iam.gserviceaccount.com'.\
\ User: 'xxxxxxxxxx#cloudservices.gserviceaccount.com'. Ask a project owner\
\ to grant you the iam.serviceAccountUser role on the service account\"}"
resources:
- type: compute.v1.instance
name: {{ properties.name }}
properties:
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/{{ properties["machineType"] }}
zone: {{ properties["zone"] }}
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/debian-11-bullseye-v20220822
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
serviceAccounts:
- email: serviceAccount:$(ref.{{ properties.name }}-sa.email)
scopes:
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring.write
- https://www.googleapis.com/auth/servicecontrol
- https://www.googleapis.com/auth/service.management.readonly
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/cloud-platform
- name: {{ properties.name }}-sa
type: gcp-types/iam-v1:projects.serviceAccounts
properties:
accountId: {{ properties.name }}-gke-init-vm
displayName: {{ properties.name }}-gke-init-vm
accessControl:
gcpIamPolicy:
bindings:
- role: roles/iam.serviceAccountUser
members:
- "serviceAccount:<myproject>#cloudservices.gserviceaccount.com"
- name: {{ env["project"] }}-{{ properties.name }}-initnode-sa-binding
type: gcp-types/cloudresourcemanager-v1:virtual.projects.iamMemberBinding
properties:
resource: {{ env["project"] }}
member: serviceAccount:$(ref.{{ properties.name }}-sa.email)
role: roles/container.clusterAdmin
- name: {{ env["project"] }}-{{ properties.name }}-initnode-sa-binding-v2
type: gcp-types/cloudresourcemanager-v1:virtual.projects.iamMemberBinding
properties:
resource: {{ env["project"] }}
member: serviceAccount:$(ref.{{ properties.name }}-sa.email)
role: roles/container.admin
For reference for other communities for this issue
specify the reference below to resolve the issue.
$(ref.{{ properties.name }}-sa.email)
I'm carrying out the lab of the GCP platform "Configure a Firewall and a Startup Script with Deployment Manager", i changed the qwicklabs.jinja for this code:
resources:
- name: default-allow-http
type: compute.v1.firewall
properties:
targetTags: ["http"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/f1-micro
# For examples on how to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
tags:
items: ["http"]
metadata:
items:
- key: startup-script
value: "apt-get update \n apt-get install -y apache2"
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
The VM and Disk are made succesfully but i can't complete the last task "Check that Deployment manager includes startup script and firewall resource" because i have problems making the firewall rule an this appear:
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1598852175371-5a
e25c7f61bda-1c55c951-22ca1242]: errors:
- code: RESOURCE_ERROR
location: /deployments/deployment-templates/resources/http-firewall-rule
message: '{"ResourceType":"compute.v1.firewall","ResourceErrorCode":"400","ResourceErrorMessage":{
"code":400,"message":"Request
contains an invalid argument.","status":"INVALID_ARGUMENT","statusMessage":"Bad
Request","requestPath":"https://compute.googleapis.com/compute/v1/projects/qwiklabs-gcp-01-888e7
df2843f/global/firewalls","httpMethod":"POST"}}'
Could someone help me pls? I have to finish this lab!
Your file was giving me for some reason "invalid format" error so I created a new Deployment Manager config file; took VM template from here, added your external IP configuration and also firewall rule part (without any changes).
My yaml file looks like this (I didn't use any variables though).
resources:
- name: vm-created-by-deployment-manager
type: compute.v1.instance
properties:
zone: us-central1-a
machineType: zones/us-central1-a/machineTypes/n1-standard-1
tags:
items: ["http"]
metadata:
items:
- key: startup-script
value: "apt-get update \n apt-get install -y apache2"
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
- name: default-allow-http3
type: compute.v1.firewall
properties:
targetTags: ["http"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
When I ran the file everything worked as intended:
wbogacz#cloudshell:~/fire (wojtek)$ gcloud deployment-manager deployments create test1 --config dm1.yaml
The fingerprint of the deployment is b'n63E-AtErTCKtWOvktfUsA=='
Waiting for create [operation-1599036146720-5ae5-----99-2a45880e-addbce89]...done.
Create operation operation-1599036146720-5ae-----99-2a45880e-addbce89 completed successfully.
NAME TYPE STATE ERRORS INTENT
default-allow-http3 compute.v1.firewall COMPLETED []
vm-created-by-deployment-manager compute.v1.instance COMPLETED []
At the end I logged in via SSH to the VM and verified that the startup script was executed - and again success.
I'm trying to create a CloudSQL instance and two databases using Google Deployment Manager. I can't get a reliable first-time deployment where both databases create successfully. Instead, each time I run it, one (or both!) fail with the status "FAILED_PRECONDITION", error message "Bad Request" and no further explanation as to which precondition has failed or how to fix it. Has anyone else come across this before or have any clues on how I can find the issue?
The properties {{ SQL_NAME }} etc. are all defined at the top of my jinja template, but I've ommitted them for clarity.
resources:
- name: {{ SQL_NAME }}
type: sqladmin.v1beta4.instance
properties:
backendType: SECOND_GEN
instanceType: CLOUD_SQL_INSTANCE
region: {{ properties["region"] }}
databaseVersion: {{ properties["dbType"] }}
settings:
tier: db-n1-standard-1
dataDiskSizeGb: 10
dataDiskType: PD_SSD
storageAutoResize: true
replicationType: SYNCHRONOUS
locationPreference:
zone: {{ properties['zone']}}
ipConfiguration:
privateNetwork: {{ properties["network"] }}
- name: {{ DB_NAME }}
type: sqladmin.v1beta4.database
properties:
name: db1
instance: $(ref.{{ SQL_NAME }}.name)
charset: utf8
collation: utf8_general_ci
metadata:
dependsOn:
- {{ SQL_NAME }}
- name: {{ DB2_NAME }}
type: sqladmin.v1beta4.database
properties:
name: db2
instance: $(ref.{{ SQL_NAME }}.name)
charset: utf8
metadata:
dependsOn:
- {{ SQL_NAME }}
- name: {{ USER_NAME }}
type: sqladmin.v1beta4.user
properties:
name: dbroot
host: "%"
instance: $(ref.{{ SQL_NAME }}.name)
password: {{ properties['password'] }}
metadata:
dependsOn:
- {{ SQL_NAME }}
So, I found the answer. It turns out Google is even less helpful with their error message than I thought when I came across the issue. What it seems to be (I still have no concrete evidence that this was the precondition, but the below seems to solve it) is that you can't create two databases at the same time on the same CloudSQL instance, and Deployment Manager tries to because they're both only dependant on the CloudSQL instance itself. I solved the issue by adding dependencies on each successive resource:
- name: {{ DB_NAME }}
type: sqladmin.v1beta4.database
properties:
name: db1
instance: $(ref.{{ SQL_NAME }}.name)
charset: utf8
collation: utf8_general_ci
metadata:
dependsOn:
- {{ SQL_NAME }}
- name: {{ DB2_NAME }}
type: sqladmin.v1beta4.database
properties:
name: db2
instance: $(ref.{{ SQL_NAME }}.name)
charset: utf8
metadata:
dependsOn:
- {{ SQL_NAME }}
- {{ DB_NAME }}
- name: {{ USER_NAME }}
type: sqladmin.v1beta4.user
properties:
name: dbroot
host: "%"
instance: $(ref.{{ SQL_NAME }}.name)
password: {{ properties['password'] }}
metadata:
dependsOn:
- {{ SQL_NAME }}
- {{ DB2_NAME }}
BTW, my coworker just discovered today that as of right meow, explicitly setting backendType to SECOND_GEN and databaseVersion to MYSQL_5_6 also yielded a 400 error. You can use this combination in the console. This is a very recent API break. Just a heads up.
I have tried creating a firewall rule with sourceTags and have my VM tagged with http. But it still does not allow HTTP traffic. Why is that?
resources:
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/f1-micro
# For examples on how to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
metadata:
items:
- key: startup-script
value: |
#!/bin/bash
apt-get update
apt-get install -y apache2
tags:
items:
- http
Tag has to be the same as the network tag attached. The default ones are "http-server" or "https-server", so the script should have that present like this:
tags
items
- http-server
- https-server
Bear in mind also to properly configure firewall rule and check that http server is running and listening on that port.
You need to add the firewall section there as well. Here is the one which is working for me:
resources:
- type: compute.v1.firewall
name: tcp-firewall-rule
properties:
network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
sourceRanges: ["0.0.0.0/0"]
targetTags: ["http","http-server"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties['zone'] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env['project'] }}/zones/{{ properties['zone'] }}/machineTypes/f1-micro
tags:
items: ["http","http-server"]
metadata:
items:
# For more ways to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
- key: startup-script
value: |
#!/bin/bash
apt-get update
apt-get install -y apache2
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
How do I create an instance template that would create empty disks in the instances? It seems the source attribute is required for the disks while creating instance template.
- name: {{ TEMPLATE_NAME }}
type: compute.v1.instanceTemplate
properties:
properties:
machineType: {{ properties["machine_type"] }}
canIpForward: False
networkInterfaces:
- network: "projects/mapreng-1/global/networks/{{ VPC }}"
#No internet access if this is not specified
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: https://www.googleapis.com/compute/v1/projects/{{ PROJECT }}/global/images/{{ IMAGE }}
- deviceName: dev1
type: SCRATCH
autoDelete: true
initializeParams:
diskType: pd-ssd
diskSizeGb: 20
This fails at runtime with " Source image must be specified"
I think the point of instance templates is that they allow you to easily create many machines that all look the same when they boot by using disk images. How are you thinking of using instances that don't actually boot because they don't have an OS on disk?