Configure a Firewall and a Startup Script with Deployment Manager - google-cloud-platform

I'm carrying out the lab of the GCP platform "Configure a Firewall and a Startup Script with Deployment Manager", i changed the qwicklabs.jinja for this code:
resources:
- name: default-allow-http
type: compute.v1.firewall
properties:
targetTags: ["http"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/f1-micro
# For examples on how to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
tags:
items: ["http"]
metadata:
items:
- key: startup-script
value: "apt-get update \n apt-get install -y apache2"
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
The VM and Disk are made succesfully but i can't complete the last task "Check that Deployment manager includes startup script and firewall resource" because i have problems making the firewall rule an this appear:
ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1598852175371-5a
e25c7f61bda-1c55c951-22ca1242]: errors:
- code: RESOURCE_ERROR
location: /deployments/deployment-templates/resources/http-firewall-rule
message: '{"ResourceType":"compute.v1.firewall","ResourceErrorCode":"400","ResourceErrorMessage":{
"code":400,"message":"Request
contains an invalid argument.","status":"INVALID_ARGUMENT","statusMessage":"Bad
Request","requestPath":"https://compute.googleapis.com/compute/v1/projects/qwiklabs-gcp-01-888e7
df2843f/global/firewalls","httpMethod":"POST"}}'
Could someone help me pls? I have to finish this lab!

Your file was giving me for some reason "invalid format" error so I created a new Deployment Manager config file; took VM template from here, added your external IP configuration and also firewall rule part (without any changes).
My yaml file looks like this (I didn't use any variables though).
resources:
- name: vm-created-by-deployment-manager
type: compute.v1.instance
properties:
zone: us-central1-a
machineType: zones/us-central1-a/machineTypes/n1-standard-1
tags:
items: ["http"]
metadata:
items:
- key: startup-script
value: "apt-get update \n apt-get install -y apache2"
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
sourceImage: projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: global/networks/default
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
- name: default-allow-http3
type: compute.v1.firewall
properties:
targetTags: ["http"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
When I ran the file everything worked as intended:
wbogacz#cloudshell:~/fire (wojtek)$ gcloud deployment-manager deployments create test1 --config dm1.yaml
The fingerprint of the deployment is b'n63E-AtErTCKtWOvktfUsA=='
Waiting for create [operation-1599036146720-5ae5-----99-2a45880e-addbce89]...done.
Create operation operation-1599036146720-5ae-----99-2a45880e-addbce89 completed successfully.
NAME TYPE STATE ERRORS INTENT
default-allow-http3 compute.v1.firewall COMPLETED []
vm-created-by-deployment-manager compute.v1.instance COMPLETED []
At the end I logged in via SSH to the VM and verified that the startup script was executed - and again success.

Related

Google deployment manager fails when the MIG (managed instance group) and the load balancer are in the same config file

I'm working on IaC (Goole Deployment Manager - Python and YAML) and tinkering with Google's external load balancer. As part of the PoC I created:
An instance template based on ubuntu with startup script installing apache2 and an index.html file.
Create an instance group based on the above template.
Place an external HTTP load balancer in front of the instance group.
A network and a firewall rule to make debugging easier.
I heavily borrowed from Cloud Foundation Toolkit, and I'm showing only the config part of the code below. I used Python for templating.
imports:
- path: ../network/vpc/vpc_network.py
name: vpc_network.py
- path: ../network/firewall/firewall_rule.py
name: firewall_rule.py
- path: ../compute/instance_template/instance_template.py
name: instance_template.py
- path: ../compute/health_checks/health_check.py
name: health_check.py
- path: ../compute/instance_group/mananged/instance_group.py
name: instance_group.py
- path: ../network_services/load_balancing/external_loadBalancers/external_load_balancer.py
name: external_load_balancer.py
resources:
- name: demo-firewall-rules-1
type: firewall_rule.py
properties:
rules:
- name: "allow-ssh-for-all"
description: "tcp firewall enable from all"
network: $(ref.net-10-69-16.network)
priority: 1000
action: "allow"
direction: "INGRESS"
sourceRanges: ['0.0.0.0/0']
ipProtocol: "tcp"
ipPorts: ["22"]
- name: "test-health-check-for-ig"
description: "enable health check to work on the project"
network: $(ref.net-10-69-16.network)
priority: 1000
action: "allow"
direction: "INGRESS"
sourceRanges: ['130.211.0.0/22', '35.191.0.0/16']
ipProtocol: "tcp"
ipPorts: ["80", "443"]
- name: "allow-http-https-from-anywhere"
description: "allow http and https from anywhere"
network: $(ref.net-10-69-16.network)
priority: 1000
action: "allow"
direction: "INGRESS"
sourceRanges: ['0.0.0.0/0']
ipProtocol: "tcp"
ipPorts: ["80", "443"]
- name: net-10-69-16
type: vpc_network.py
properties:
subnetworks:
- region: australia-southeast1
cidr: 10.69.10.0/24
- region: australia-southeast1
cidr: 10.69.20.0/24
- region: australia-southeast1
cidr: 10.69.30.0/24
- name: mig-regional-1
type: instance_group.py
properties:
region: australia-southeast1
instanceTemplate: $(ref.it-demo-1.selfLink)
targetSize: 2
autoHealingPolicies:
- healthCheck: $(ref.demo-http-healthcheck-MIG-1.selfLink)
initialDelaySec: 400
- name: it-demo-1
type: instance_template.py
properties:
machineType: n1-standard-1
tags:
items:
- http
disks:
- deviceName: boot-disk-v1
initializeParams:
sourceImage: projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20201211
diskType: pd-ssd
networkInterfaces:
- network: $(ref.net-10-69-16.network)
subnetwork: $(ref.net-10-69-16.subnetworks[0])
accessConfigs:
- type: ONE_TO_ONE_NAT
metadata:
items:
- key: startup-script
value: |
sudo apt-get install -y apache2
sudo apt-get install -y php7.0
sudo service apache2 restart
sudo echo "Ho Ho Ho from $HOSTNAME" > /var/www/html/index.html
- name: demo-http-healthcheck-MIG-1
type: health_check.py
properties:
type: HTTP
checkIntervalSec: 5
timeoutSec: 5
unhealthyThreshold: 2
healthyThreshold: 2
httpHealthCheck:
port: 80
requestPath: /
- name: http-elb-1
type: external_load_balancer.py
properties:
portRange: 80
backendServices:
- resourceName: backend-service-for-http-1
sessionAffinity: NONE
affinityCookieTtlSec: 1000
portName: http
healthCheck: $(ref.demo-http-healthcheck-MIG-1.selfLink)
backends:
- group: $(ref.mig-regional-1.selfLink)
balancingMode: UTILIZATION
maxUtilization: 0.8
urlMap:
defaultService: backend-service-for-http-1
The problem is with the load balancer. When it is present in the same file, the deployment fails saying the managed instance group mig-regional-1 does not exist. But when I move the load balancer part out of this file and deploy the load balancer separately, (after a bit of delay) it all goes through well.
The most probable explanation is, the instance group is not ready when the load balancer is trying to reference it which is the exact situation ref.*.selfLink is supposed to handle.
This is not exactly a blocker, but it is nice to have the entire config in 1 file.
So, my questions:
Have any of you faced this before or am I missing something here?
Do you have any solution for this?
Your template need to explicit its dependencies.
You can have dependencies between your resources, such as when you
need certain parts of your environment to exist before you can deploy
other parts of the environment.
You can specify dependencies using the dependsOn option in your templates.
Seems that your External LB needs dependsOn attribute that has MIG value.
refer here to get more information about expliciting dependencies.

Google Deployment Manager: Whats the equivalent of Allow HTTP traffic when creating a compute engine instance?

I have tried creating a firewall rule with sourceTags and have my VM tagged with http. But it still does not allow HTTP traffic. Why is that?
resources:
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/f1-micro
# For examples on how to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
metadata:
items:
- key: startup-script
value: |
#!/bin/bash
apt-get update
apt-get install -y apache2
tags:
items:
- http
Tag has to be the same as the network tag attached. The default ones are "http-server" or "https-server", so the script should have that present like this:
tags
items
- http-server
- https-server
Bear in mind also to properly configure firewall rule and check that http server is running and listening on that port.
You need to add the firewall section there as well. Here is the one which is working for me:
resources:
- type: compute.v1.firewall
name: tcp-firewall-rule
properties:
network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
sourceRanges: ["0.0.0.0/0"]
targetTags: ["http","http-server"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- type: compute.v1.instance
name: vm-test
properties:
zone: {{ properties['zone'] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env['project'] }}/zones/{{ properties['zone'] }}/machineTypes/f1-micro
tags:
items: ["http","http-server"]
metadata:
items:
# For more ways to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
- key: startup-script
value: |
#!/bin/bash
apt-get update
apt-get install -y apache2
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT

GCP deployment maanger template unable to create tagged firewall

I am trying to deploy a Linux VM with tag "http" and creating firewall for allowing HTTP Port 80 access as tagged firewall "http". The VM is getting deployed but no external access is working for the VM. Also gave startup script for VM but its not working
resources:
- type: compute.v1.instance
name: vm-test
properties:
metadata:
items:
- key: startup-script-url
value: https://storage.googleapis.com/cf405bucket/install-web.sh
zone: {{ properties["zone"] }}
machineType: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/n1-standard-2
# For examples on how to use startup scripts on an instance, see:
# https://cloud.google.com/compute/docs/startupscript
tags:
items: ["http"]
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: disk-{{ env["deployment"] }}
sourceImage: https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/family/debian-9
networkInterfaces:
- network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
# Access Config required to give the instance a public IP address
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
- type: compute.v1.firewall
name: default-allow-http
properties:
sourceRanges: ["0.0.0.0/0"]
network: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/global/networks/default
targetTags: ["http"]
allowed:
- IPProtocol: TCP
ports: ["80"]
Try SSH'ing into the deployment's created VM instance and run the command apache2 --version.
What happens? I assume you'll be told it isn't a recognised command or something...given it looks like the web server hasn't been installed for some reason. If so, perhaps try updating install-web.sh to include sudo before commands, i.e.,
#!/bin/bash
sudo apt-get update
sudo apt-get install -y apache2
Failing that, why not ditch the install-web.sh file altogether and just include the script in the config file directly (as there isn't much to it), e.g., something like:
metadata:
items:
- key: startup-script
value: |
#!/bin/bash
apt-get update
apt-get install -y apache2

How to make a regional cluster in GKE w/ deployment-manager?

"zone" is a required field when I try to create but it says in the documentation that it is "deprecated". This is kinda misleading. Then everytime I include "zone". It is the one followed; Let us say I put "asia-east2-a" then it will be a zonal where the master node is in asia-east2-a.
Below is my jinja template
resources:
- name: practice-gke-clusters
type: container.v1.cluster
properties:
zone: asia-east2-a
cluster:
name: practice-gke-clusters
location: asia-east2
network: $(ref.practice-gke-network.selfLink)
subnetwork: $(ref.practice-gke-network-subnet-1.selfLink)
nodePools:
- name: default-pool
config:
machineType: n1-standard-1
diskSizeGb: 10
diskType: pd-ssd
preemptible: True
oauthScopes:
- https://www.googleapis.com/auth/compute
- https://www.googleapis.com/auth/devstorage.read_only
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/monitoring
initialNodeCount: 1
autoscaling:
enabled: True
minNodeCount: 1
maxNodeCount: 100
management:
autoUpgrade: False
autoRepair: True
loggingService: logging.googleapis.com
monitoringService: monitoring.googleapis.com
Currently v1 API does not support the creation of regional clusters. However you can use v1beta1 API which supports this feature and use the following resource type:
type: gcp-types/container-v1beta1:projects.locations.clusters
Rather than using the 'zone' or 'region' key in the YAML, you would instead use a parent property that includes locations.
So your YAML would look something like this (replace PROJECT_ID and REGION with your own).
resources:
- type: gcp-types/container-v1beta1:projects.locations.clusters # previously container.v1.clusters
name: source-cluster
properties:
parent: projects/PROJECT_ID/locations/REGION
cluster:
name: source
initialNodeCount: 3

How to include firewall in VM instance creation by DeploymentManager in GCP

my yaml template is as follows, I want to add firewall property to allow http traffic:
resources:
- name: deployed-vm2222
type: compute.v1.instance
properties:
zone: us-central1-f
machineType: https://www.googleapis.com/compute/v1/projects/myproject/zones/us-central1-f/machineTypes/f1-micro
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
In the firewall, we use:
targetTags: ["http"]
Then, in the instance, we use:
tags:
items: ["http"]
The complete file can be as shown:
resources:
- name: default-allow-http
type: compute.v1.firewall
properties:
targetTags: ["http"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- name: vm-test
type: compute.v1.instance
properties:
zone: xxxx
machineType: xxxx
tags:
items: ["http"]
disks:
- deviceName: boot
type: PERSISTENT
boot: true
autoDelete: true
initializeParams:
diskName: xxxx
sourceImage: xxxx
networkInterfaces:
- network: xxxx
accessConfigs:
- name: External NAT
type: ONE_TO_ONE_NAT
A couple things to note when performing this action, make sure the instance is correctly tagged to enable the labelling to be applied. For example, tagging the instance, http-server or https-server ensure the firewall is aware it is processing public traffic.
Adding a firewall entry can be achieved in the following way.
resources:
- name: instance
type: xxxxxxxx
properties:
zone: us-east1-b
tags:
items: ["http-server", "tensorboard"]
- name: default-allow-http
type: compute.v1.firewall
properties:
network: https://www.googleapis.com/compute/v1/projects/myproject/global/networks/default
targetTags: ["http-server"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["80"]
- name: default-allow-tensorboard
type: compute.v1.firewall
properties:
network: https://www.googleapis.com/compute/v1/projects/myproject/global/networks/default
targetTags: ["tensorboard"]
sourceRanges: ["0.0.0.0/0"]
allowed:
- IPProtocol: TCP
ports: ["6006"]
You can add a firewall rule in your template as follow:
- name: allow-http-fw
type: compute.v1.firewall
properties:
allowed:
- IPProtocol: TCP
ports: 80
sourceRanges: [ 0.0.0.0/0 ]
You can define the properties listed for the firewall resource.
#LundinCast is almost totally correct
network: under properties is missing.
It would be the same value as under networkInterfaces: