Connecting function to empty cloud storage bucket - google-cloud-platform

I am trying to use the following info from Google deployment manager examples on GitHub.
empty_bucket_in_function.yaml
empty_bucket_cf.yaml
imports:
- path: empty_bucket_cf.jinja
resources:
- name: my-function
type: empty_bucket_cf.jinja
properties:
project: <PROJECT_NAME>
region: europe-west1
entryPoint: handler
runtime: nodejs8
bucket: lskflsjfsj
empty_bucket_cf.jinja
{% set BUCKET = properties['bucket'] + '-bucket' %} resources:
#- type: cloudfunctions.v1.function
- type: gcp-types/cloudfunctions-v1:projects.locations.functions name: my-function properties:
parent: projects/{{ properties['project'] }}/locations/{{ properties['region'] }}
location: {{ properties['region'] }}
function: my-{{ properties['bucket'] }}
sourceArchiveUrl: gs://$(ref.{{ BUCKET }}.name)/my-function
entryPoint: {{ properties['entryPoint'] }}
runtime: {{ properties['runtime'] }}
eventTrigger:
resource: $(ref.my-topic.name)
eventType: providers/cloud.pubsub/eventTypes/topic.publish
#- type: pubsub.v1.topic
- type: gcp-types/pubsub-v1:projects.topics name: my-topic properties:
topic: {{ properties['bucket'] }}-topic
#- type: storage.v1.bucket
- type: gcp-types/storage-v1:buckets name: {{ BUCKET }} properties:
predefinedAcl: projectPrivate
projection: full
location: US
storageClass: STANDARD
While deploying using deployment manager I am getting error as
testsetup has resource warnings
my-function: {"ResourceType":"gcp-types/cloudfunctions-v1:projects.locations.functions","ResourceErrorCode":"400","ResourceErrorMessage":"Failed to retrieve function source code"}
Deployment properties
Any idea why this is not a bug in Google Cloud Platform GitHub repository. Isn't it the purpose of empty_bucket config to create CFs with empty bucket.
Note: Sometimes it executes successfully as well.

I don't what Google had in mind when they publish this example, but it can't work. If your bucket is empty, the function has no code. However, when you deploy a function, the code is compiled/parsed, the entry-point checked (exists, correct signature,...), and deployed on the environment.
Here, no entry point, no code to compile/parse thus no deployment -> It's normal, but the example is disturbing. You can open an issue on the repos.

Related

Unable to delete and upload object containg bucket in GCP using deployment manager

I am using this Deployment Manager jinja template to create a bucket and upload files into the bucket.
resources:
- type: storage.v1.bucket
name: {{ properties['bucket_name'] }}
properties:
location: {{ properties['region'] }}
- name: {{ properties['build_name'] }}
action: gcp-types/cloudbuild-v1:cloudbuild.projects.builds.create
metadata:
runtimePolicy:
- CREATE
properties:
steps:
- name: gcr.io/cloud-builders/git
args: ['clone', 'https://<token>#github.com/{{ properties['username'] }}/{{ properties['repo_name'] }}.git']
- name: gcr.io/cloud-builders/gsutil
args: ['-m', 'cp', '-r', '{{ properties['repo_name'] }}/{{ properties['file_path_name_in_repo'] }}*', 'gs://{{ properties['bucket_name'] }}/']
timeout: 120s
I have taken idea of above code from How to write a file to Google Cloud Storage using Deployment Manager?
While running
gcloud deployment-manager deployments create name --config=file.yaml
then it is working fine, but while executing
gcloud deployment-manager deployments delete name
it is showing that cannot delete a bucket which contains files. Also
gcloud deployment-manager deployments update name --config=file.yaml
command is not working and showing that update option is not supported for Cloud Build.
My goal is to create a bucket and uploads files into that from github. Also if required then I can delete that and update (to put updated file) that using deployment manager template.
I have been told to do that using jinja or yaml template.
It will be good if you give a look into this and clear my problem on that.

Cloud SQL creation with Deployment Manager - "Precondition check failed." error

I'm using the gcp-types/sqladmin-v1beta4:instances Resource Type to create a Cloud SQL instance using the Deployment Manager and I'm getting the error below:
{
"ResourceType":"gcp-types/sqladmin-v1beta4:instances",
"ResourceErrorCode":"400",
"ResourceErrorMessage":{
"code":400,
"message":"Precondition check failed.",
"status":"FAILED_PRECONDITION",
"statusMessage":"Bad Request",
"requestPath":"https://www.googleapis.com/sql/v1beta4/projects/[PROJECT_NAME]/instances",
"httpMethod":"POST"
}
}
Here's the configuration inside the JINJA file:
{% set deployment_name = env['deployment'] %}
{% set INSTANCE_NAME = deployment_name + '-instance' %}
resources:
- name: {{ INSTANCE_NAME }}
type: gcp-types/sqladmin-v1beta4:instances
properties:
region: us-central1
rootPassword: root
settings:
tier: db-n1-standard-1
backupConfiguration:
binaryLogEnabled: true
enabled: true
- name: demand_ml_db
type: gcp-types/sqladmin-v1beta4:databases
properties:
name: demand_ml_db
instance: $(ref.{{ INSTANCE_NAME }}.name)
charset: utf8
The FAILED_PRECONDITION error - while not very descriptive, tends to be thrown when you're attempting to deploy over a previous Cloud SQL instance that was recently deleted; as a matter of fact, the instance you selected for deletion is not cleaned up instantly. There's an Issue Tracker thread regarding this here.
I was able to verify this on my end as well. The deployment using the JINJA file you've specified worked fine at first, but when I deleted it, and re-deployed - I received the same error.
The most simple approach is to try using a different deployment (or instance) name.

How to add cloudsql role to a service account via cloud-deployment-manager

I'm trying to add the cloudsq.client role to a service account via the cloud deployment manager. How do i do this?
I figured out that i can add primitive roles like role/owner with the pubsub.v1.topic resourceType. See the official google example:
resources:
- name: {{ env['name'] }}
type: pubsub.v1.topic
properties:
topic: {{ env['name'] }}
accessControl:
gcpIamPolicy:
bindings:
- role: roles/pubsub.subscriber
members:
- "serviceAccount:$(ref.{{ properties['serviceAccountId'] }}.email)"
But it seems like that this doesn't work with role/cloudsql.client:
It says:
"message":"Role roles/cloudsql.client is not supported for this resource."
I figured out through Can't create cloudsql role for Service Account via api that i most likely have to use the cloudresourcemanager.v1.project resource. But just replacing the resources doesn't work either:
resources:
- name: {{ env['name'] }}
type: cloudresourcemanager.v1.project
properties:
projectId: {{ env['project'] }}
accessControl:
gcpIamPolicy:
bindings:
- role: roles/cloudsql.client
members:
- "serviceAccount:$(ref.{{ properties['serviceAccountId'] }}.email)"
Error:
- code: RESOURCE_ERROR
location: /deployments/test-cluster/resources/cloudsql-client-role
message: '{"ResourceType":"cloudresourcemanager.v1.project","ResourceErrorCode":"403","ResourceErrorMessage":{"code":403,"message":"Service
accounts cannot create projects without a parent.","status":"PERMISSION_DENIED","statusMessage":"Forbidden","requestPath":"https://cloudresourcemanager.googleapis.com/v1/projects","httpMethod":"POST"}}'
Im kinda stuck so i appreciate every help i can get!
To fix this you need to use the following type in your "resources": https://github.com/GoogleCloudPlatform/deploymentmanager-samples/blob/ae293a455f90746fb2e25142dbc11250cc51aad3/community/cloud-foundation/templates/iam_member/iam_member.py#L30
The correct way to add a role to a service account is: getIamPolicy > setIamPolicy
First you need to get the policies then you will be able to set the policies, this process is called "binding".
Please use the following template to add roles to service accounts:
https://github.com/GoogleCloudPlatform/deploymentmanager-samples/tree/master/community/cloud-foundation/templates/iam_member
If you have further questions I would be glad to help you.

Deployment Manager cannot update instance templates - NO_METHOD_TO_UPDATE_FIELD

I have a deployment comprising a managed instance group and two instance templates (A and B). The deployment was initially created with the instance group referencing instance template A.
I tried updating the sourceImage in instance template B using deployment manager (gcloud beta deployment-manager deployments update my-deployment --template ...), but got the following error:
ERROR: (gcloud.beta.deployment-manager.deployments.update) Error in
Operation [operation-1538798895713-57787898f4ae9-8b478716-0bb72a09]:
errors:
- code: NO_METHOD_TO_UPDATE_FIELD
message: No method found to update field 'properties' on
resource 'fwp-app-preprod-instance-template-a' of type
'compute.v1.instanceTemplate'. The resource may need to be
recreated with the new field.
I should make it clear that the only change I made from the original deployment is the instance template's sourceImage.
Is it possible to perform an update of an instance template via deployment manager so that it references an updated sourceImage?
The error states clearly that the resource (instance template) may need to be recreated, and I'm happy for deployment manager to do that. But I have no idea how to instruct/force deployment manager to take that action.
I don't doubt it can be done outside of deployment manager, but I want to avoid configuration drift.
My app.jinja.schema:
imports:
- path: instance-group.jinja
- path: instance-template.jinja
My app.jinja:
resources:
- name: instance-template-a
type: instance-template.jinja
properties:
name: {{ env["deployment"] }}-instance-template-a
machineType: g1-small
sourceImage: "projects/my-project/global/images/my-image"
diskSizeGb: '30'
- name: instance-template-b
type: instance-template.jinja
properties:
name: {{ env["deployment"] }}-instance-template-b
machineType: g1-small
sourceImage: "projects/my-project/global/images/my-image"
diskSizeGb: '30'
- name: fwp-instance-group
type: instance-group.jinja
My instance-group.jinja:
resources:
- name: 'instance-group-{{ env["deployment"] }}'
type: compute.v1.regionInstanceGroupManager
properties:
baseInstanceName: ig-instance-{{ env["deployment"] }}
instanceTemplate: '$(ref.{{ env["deployment"] }}-instance-template-a.selfLink)'
targetSize: 1
region: australia-southeast1
- name: 'autoscaler-{{ env["deployment"] }}'
type: compute.v1.regionAutoscalers
properties:
autoscalingPolicy:
coolDownPeriodSec: 60
loadBalancingUtilization:
utilizationTarget: 0.9
maxNumReplicas: 10
minNumReplicas: 2
target: $(ref.instance-group-{{ env["deployment"] }}.selfLink)
region: australia-southeast1
And my instance-template.jinja
resources:
- name: {{ properties["name"] }}
type: compute.v1.instanceTemplate
properties:
name: {{ properties["name"] }}
description: ''
properties:
machineType: {{ properties["machineType"] }}
tags:
items:
- no-ip
- web-server
- http-server
- https-server
disks:
- type: 'PERSISTENT'
boot: true
mode: 'READ_WRITE'
autoDelete: true
deviceName: instance-device
initializeParams:
sourceImage: {{ properties["sourceImage"] }}
diskType: 'pd-standard'
diskSizeGb: {{ properties["diskSizeGb"] }}
canIpForward: false
networkInterfaces:
- network: projects/my-project/global/networks/vpc-fwp-nonprod
subnetwork: projects/my-project/regions/australia-southeast1/subnetworks/subnet-private-fwp-nonprod
aliasIpRanges: []
labels: { environment: {{ env["deployment"] }}, tenancy: "fwp-nonprod" }
scheduling:
preemptible: false
onHostMaintenance: MIGRATE
automaticRestart: true
nodeAffinities: []
serviceAccounts:
- email: some-service-account#developer.gserviceaccount.com
scopes:
- https://www.googleapis.com/auth/cloud-platform
To recap the comments:
The DM config includes an instance template for the managed instance group. The change of source image is attempting to change the image used in the template.
Unfortunately, instance templates are immutable once created
"So it is not possible to update an existing instance template or change an instance template after it has been created."
This explains the error message returned. The proper way to change the image you want to use for a Managed Instance Group is to create a new template and perform a rolling update on the group and using the new instance template.

CLOUD DEPLOYMENT MANAGER: Internal Load Balancer create issue

I am using the following to try and create an internal load balancer via Deployment Manager using the following code
- name: {{ env["name"] }}-port389-healthcheck
type: compute.v1.healthChecks
properties:
type: tcp
tcpHealthCheck: {
port: 389
}
- name: {{ env["name"] }}-port389-backend-service
type: compute.beta.backendService
properties:
healthChecks:
- $(ref.{{ env["name"] }}-port389-healthcheck.selfLink)
backends:
- group: $(ref.{{ env['name'] }}-master-instance-groups-managed.instanceGroup)
- group: $(ref.{{ env['name'] }}-slave-instance-groups-managed.instanceGroup)
protocol: TCP
region: {{ properties['region'] }}
loadBalancingScheme: INTERNAL
- name: {{ env["name"] }}-port389-forwarding-rule
type: compute.beta.forwardingRule
properties:
loadBalancingScheme: INTERNAL
ports:
- 389
network: default
region: {{ properties["region"] }}
backendService: $(ref.{{ env["name"] }}-port389-backend-service.selfLink)
It errrors when run with the following
Waiting for create operation-1478651694403-540d36cfdcdb9-cba25532-08697daf...failed.
ERROR: (gcloud.beta.deployment-manager.deployments.create) Error in Operation operation-1478651694403-540d36cfdcdb9-cba25532-08697daf:
errors:
- code: RESOURCE_ERROR
location: /deployments/forgerock/resources/forgerock-frontend-port389-backend-service-us-central1
message: 'Unexpected response from resource of type compute.beta.backendService:
400 {"code":400,"errors":[{"domain":"global","message":"Invalid value for field
''resource.loadBalancingScheme'': ''INTERNAL''. Load balancing scheme must be
external for a global backend service.","reason":"invalid"}],"message":"Invalid
value for field ''resource.loadBalancingScheme'': ''INTERNAL''. Load balancing
scheme must be external for a global backend service.","statusMessage":"Bad Request","requestPath":"https://www.googleapis.com/compute/beta/projects/carbide-tenure-557/global/backendServices"}'
It would appear to be creating using the https://www.googleapis.com/compute/beta/projects/carbide-tenure-557/global/backendServices instead of https://www.googleapis.com/compute/beta/projects/carbide-tenure-557/backendServices
I know this is beta functionality, but trying to develop this solution using GDM instead of a mixture of gcloud commands and GDM
For type instead of type: compute.beta.backendService use:
type: compute.v1.regionBackendService