How to update machine type property through GCP deployment manager - google-cloud-platform

I have a python template as shown below for a compute instance type along with other needed config.yaml file.
...
CONTROLLER_MACHINE_TYPE='n1-standard-8'
controller_template = {
'name': 'controller-it',
'type': 'it_template.py',
'properties': {
'machineType': CONTROLLER_MACHINE_TYPE,
'dockerImage': CONTROLLER_IMAGE,
'dockerEnv': {
'ADC_LISTEN_QUEUE': 'controller-subscriber'
},
'zone': ZONE,
'network': NETWORK_NAME,
'saEmail': SA_EMAIL
}
}
it_template.py contents
def GenerateConfig(context):
resources = [{
'name': context.env['name'],
'type': 'compute.v1.instanceTemplate',
'properties': {
'zone': context.properties['zone'],
'properties': {
"machineType": context.properties['machineType'],
"metadata": {
"items": [{
"key": 'gce-container-declaration',
"value": GenerateManifest(context)
}]
}, ...
I have deployed it with the environment named qa. Now after sometime I realized that I need to change the machine type of this instance. For eg instead of n1-standard-8, I want my qa environment to update the type of machine for this resource.
However I don't see any example which mentions about updating the property of any resource.
Can we update the property of a resource in an environment using gcp deployment manager?
Or I need to add a new resource with another name and desired machine type property?
Update
As suggested by #jordi Miralles, I modified my template to have machineType as n1-standard-16 and tried to update the deployment.
But I got below error
cloud deployment-manager deployments update qa --config dm_config.yaml
The fingerprint of the deployment is KkD38j9KYiBTiaIW8SltbA==
Waiting for update [operation-1525444590548-56b623ef1b421-b4733efd-53174d1b]...failed.
ERROR: (gcloud.deployment-manager.deployments.update) Error in Operation [operation-1525444590548-56b623ef1b421-b4733efd-53174d1b]: errors:
- code: NO_METHOD_TO_UPDATE_FIELD
message: No method found to update field 'properties' on resource 'controller-it'
of type 'compute.v1.instanceTemplate'. The resource may need to be recreated with
the new field.
Please help.

You can update resources in your deployment manager, there is no need to create a new one with a different name if you want to change a resource. See updating a deployment
You can use your existing deployment yaml file with the changes you want to apply and then update your existing deployment:
gcloud deployment-manager deployments update [EXISTING DEPLOYMENT] --config [UPDATED YAML]
Take into account that your instance must be stopped. All other implications on changing machine type apply. Also, no data on any of your persistent disks will be lost.
Remember to turn on the instances when the update finishes!

Related

'403 Permission denied while getting Drive credentials' when using Deployment Manager to create an 'external table' in BigQuery

Steps to reproduce:
Create sheet in Google Sheets
Enable Deployment Manager & Google Drive API in Google Cloud Platform
add deployment manager service-account with view permissions on sheet
Create dataset with deployment manager
Create table with deployment manager, reference external sheet in sourceUris
partial python template:
def GenerateConfig(context):
name: str = context.env['name']
dataset: str = context.properties['dataset']
tables: [] = context.properties['tables']
location: str = context.properties.get('location', 'EU')
resources = [{
'name': name,
'type': 'gcp-types/bigquery-v2:datasets',
'properties': {
'datasetReference': {
'datasetId': dataset,
},
'location': location
},
}]
for t in tables:
resources.append({
'name': '{}-tbl'.format(t["name"]),
'type': 'gcp-types/bigquery-v2:tables',
'properties': {
'datasetId': dataset,
'tableReference': {
'tableId': t["name"]
},
'externalDataConfiguration': {
'sourceUris': ['https://docs.google.com/spreadsheets/d/123123123123123-123123123123/edit?usp=sharing'],
'sourceFormat': 'GOOGLE_SHEETS',
'autodetect': True,
'googleSheetsOptions':
{
"skipLeadingRows": '1',
}
}
},
})
return {'resources': resources}
I've found a few leads such as this, but they all reference using 'scopes' to add https://www.googleapis.com/auth/drive.
I'm not sure of how to add scopes to a deployment manager request, or really how scopes work.
Any help would be appreciated.
Yes, using scopes solves the problem. However, even after adding the scopes, I was facing the same error. Sharing the google sheets document with the GCP service account helped me get rid of this error.
To summarize - use scopes and share the document with the GCP service account that you will use for querying the table.
Also, this document is helpful for querying external tables
I was having the same issue when running Airflow DAGs on Cloud Composer, which is the managed Airflow service on Google Cloud Platform.
Essentially you need to:
Share the file with the email of the service account (give Viewer or Editor permissions based on what the DAG is supposed to execute)
Enable Google Drive OAuth Scopes
Depending on the Cloud Composer version you are using, the second step should be executed in a slightly different way:
For Cloud Composer 1
You should add the Google Drive OAuth Scope through the User Interface:
"https://www.googleapis.com/auth/drive"
Alternatively, if you are using Infrastructure as a Code (e.g. Terraform), you can specify oauth_scopes as shown below:
config {
...
node_config {
...
oauth_scopes = [
"https://www.googleapis.com/auth/drive",
]
}
}
For Cloud Composer 2
Since Cloud Composer v2 uses GKE Autopilot, it does not support OAuth on the environment level. You can however specify the scope at the connection level, that is being used by your Airflow Operator in order to initiate the connection.
If you are using the default GCP connection (i.e. google_cloud_default which is automatically created upon deployment of the Cloud Composer instance), then all you need to do is specify Google Drive ("https://www.googleapis.com/auth/drive") in the scopes of the connection (through Airflow Connections UI).
Alternatively, you can even create your new connection and once again specify the Google Drive in the scopes field and then pass the name of this connection in the gcp_conn_id argument of your Operator.

Set 'maxActiveInstances' error

I am using AWS data-pipeline to export a DDB table, but when I activate I get an error:
Web service limit exceeded: Exceeded number of concurrent executions. Please set the field 'maxActiveInstances' to a higher value in your pipeline or wait for the currenly running executions to complete before trying again (Service: DataPipeline; Status Code: 400; Error Code: InvalidRequestException; Request ID: efbf9847-49fb-11e8-abef-1da37c3550b5)
How do I set this maxActiveInstances property using the AWS UI?
You can set it as a property on your Ec2Resource[1] (or EmrActivity[2]) object. Using the UI, click Edit Pipeline, click on Resources on the right hand side of the screen (it's a collapsable menu). There should be an Ec2Resource object. There should be a drop down on this object called "Add an additional field" and you should see max active instances in the drop down.
[1]https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-ec2resource.html
[2] https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-emractivity.html
We ran into this too. For an on-demand pipeline, it looks like after a certain number of retries, you have to give it time to finish terminating the provisioned resources before you will be allowed to try again.
Solution: Patience.
With an on-demand pipline you can specify it in the 'Default object', like this
{
"objects": [
{
"failureAndRerunMode": "CASCADE",
"scheduleType": "ONDEMAND",
"name": "Default",
"id": "Default",
"maxActiveInstances": "5"
},
...
I couldn't add it in Architect, I had to create another pipeline from the json. But once that was done I could edit it in Architect (under 'Others' section).

Editing cloud formation templates terminates existing instances and creates new instances

We are using a cloud formation template for a set of VMs and each time after the code deployment , we need to edit the package version on the template parameters for auto scaling to take the latest package from the s3 bucket.
The issue is, editing cloud formation template triggers a cloudformation-based upgrade of the instances(which involves destroying the existing machines and creating new ones from scratch, which is time consuming).
Is there anyway we can prevent this.
Basically, we dont need the cloud formation template to destroy and recreate the instances whenever we edit it.?
EDIT : This my autoscaling group setting
"*********":{
"Type":"AWS::AutoScaling::AutoScalingGroup",
"Properties":{
"AvailabilityZones":[
{
"Ref":"PrimaryAvailabilityZone"
}
],
"Cooldown":"300",
"DesiredCapacity":"2",
"HealthCheckGracePeriod":"300",
"HealthCheckType":"EC2",
"LoadBalancerNames":[
{
"Ref":"elbxxbalancer"
}
],
"MaxSize":"8",
"MinSize":"1",
"VPCZoneIdentifier":[
{
"Ref":"PrivateSubnetId"
}
],
"Tags":[
{
"Key":"Name",
"Value":"my-Server",
"PropagateAtLaunch":"true"
},
{
"Key":"VPCRole",
"Value":{
"Ref":"VpcRole"
},
"PropagateAtLaunch":"true"
}
],
"TerminationPolicies":[
"Default"
],
"LaunchConfigurationName":{
"Ref":"xxlaunch"
}
},
"CreationPolicy":{
"ResourceSignal":{
"Timeout":"PT10M",
"Count":"1"
}
},
"UpdatePolicy":{
"AutoScalingRollingUpdate":{
"MinInstancesInService":"1",
"MaxBatchSize":"1",
"PauseTime":"PT10M",
"WaitOnResourceSignals":"true"
}
}
},
You can look at the documentation and view the Update requires: field on the property you modifying on your CF template.
If it says Replacement it will recreate the instance, with a new logical id
If it says Some Interruption it will make the instance unavailable, in the ec2 case, restarting it, but will not recreate the instance, keeping the same logical id
If it says No interruption it will not impact the instance at all
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html

How to import manual changes into Terraform remote state

I am new to terraform - I have created remote tfstate in s3, and now there are some manual changes too that are done in my AWS infrastructure. I need to import those manual changes into tfstate.
I used the import command for some resources, but for some resources such as IAM policy etc, there is no such import command.
Also some resources such as DB are changed with new parameters added, and I need to import them as well. When I try to import those changes it says:
Error importing: 1 error(s) occurred:
* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide
with an existing resource.
Please remove or rename this resource before continuing.
Any help would be appreciated. Thanks.
Before directly answering this question I think some context would help:
Behind the scenes, Terraform maintains a state file that contains a mapping from the resources in your configuration to the objects in the underlying provider API. When you create a new object with Terraform, the id of the object that was created is automatically saved in the state so that future commands can locate the referenced object for read, update, and delete operations.
terraform import, then, is a different way to create an entry in the state file. Rather than creating a new object and recording its id, instead the user provides an id on the command line. Terraform reads the object with that id and adds the result to the state file, after which it is indistinguishable in the state from a resource that Terraform created itself.
So with all of that said, let's address your questions one-by-one.
Importing Resources That Don't Support terraform import
Since each resource requires a small amount of validation and data-fetching code to do an import, not all resources are supported for import at this time.
Given what we know about what terraform import does from the above, in theory it's possible to skip Terraform's validation of the provided id and instead manually add the resource to the state. This is an advanced operation and must be done with care to avoid corrupting the state.
First, retrieve the state into a local file that you'll use for your local work:
terraform state pull >manual-import.tfstate
This will create a file manual-import.tfstate that you can open in a text editor. It uses JSON syntax, so though its internal structure is not documented as a stable format we can carefully edit it as long as we remain consistent with the expected structure.
It's simplest to locate an existing resource that is in the same module as where you want to import and duplicate and edit it. Let's assume we have a resources object like this:
"resources": {
"null_resource.foo": {
"type": "null_resource",
"depends_on": [],
"primary": {
"id": "5897853859325638329",
"attributes": {
"id": "5897853859325638329"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
}
},
Each attribute within this resources object corresponds to a resource in your configuration. The attribute name is the type and name of the resource. In this case, the resource type is null_resource and the attribute name is foo. In your case you might see something like aws_instance.server here.
The id attributes are, for many resources (but not all!), the main thing that needs to be populated. So we can duplicate this structure for a hypothetical IAM policy:
"resources": {
"null_resource.foo": {
"type": "null_resource",
"depends_on": [],
"primary": {
"id": "5897853859325638329",
"attributes": {
"id": "5897853859325638329"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
},
"aws_iam_policy.example": {
"type": "aws_iam_policy",
"depends_on": [],
"primary": {
"id": "?????",
"attributes": {
"id": "?????"
},
"meta": {},
"tainted": false
},
"deposed": [],
"provider": ""
}
},
The challenge at this step is to figure out what sort of id this resource requires. The only sure-fire way to know this is to read the code, which tells me that this resource expects the id to be the full ARN of the policy.
With that knowledge, we replace the two ????? sequences in the above example with the ARN of the policy we want to import.
After making manual changes to the state it's necessary to update the serial number at the top-level of the file. Terraform expects that any new change will have a higher serial number, so we can increment this number.
After completing the updates, we must upload the updated state file back into Terraform:
terraform state push manual-import.tfstate
Finally we can ask Terraform to refresh the state to make sure it worked:
terraform refresh
Again, this is a pretty risky process since the state file is Terraform's record of its relationship with the underlying system and it can be hard to recover if the content of this file is lost. It's often easier to simply replace a resource than to go to all of this effort, unless it's already serving a critical role in your infrastructure and there is no graceful migration strategy available.
Imports Colliding With Existing Resources
The error message given in your question is talking about an import "colliding" with an existing resource:
Error importing: 1 error(s) occurred:
* Can't import aws_security_group.Q8SgProdAdminSshInt, would collide with an existing resource.
Please remove or rename this resource before continuing.
The meaning of this message is that when Terraform tried to write the new resource to the state file it found a resource entry already present for the name aws_security_group.Q8SgProdAdminSshInt. This suggests that either it was already imported or that a new security group was already created by Terraform itself.
You can inspect the attributes of the existing resource in state:
terraform state show aws_security_group.Q8SgProdAdminSshInt
Compare the data returned with the security group you were trying to import. If the ids match then there's nothing left to do, since the resource was already imported.
If the ids don't match then you need to figure out which of the two objects is the one you want to keep. If you'd like to keep the one that Terraform already has, you can manually delete the one you were trying to import.
If you'd like to keep the one you were trying to import instead, you can drop the unwanted one from the Terraform state to make way for the import to succeed:
terraform state rm aws_security_group.Q8SgProdAdminSshInt
Note that this just makes Terraform "forget" the resource; it will still exist in EC2, and will need to be deleted manually via the console, command line tools, or API. Be sure to note down its id before deleting it to ensure that you can find it in order to to clean it up.
For resources you have to use import function or manually add terraform state like block..
or if there is any change in config like you mentioned dB configs.. if the dB resource is managed by terraform remote state.. terraform refresh will help you..

Cloudformation - Redeploy environment that uses a recordset (with Jenkins)

TL;DR - What's the recommended way, using a CI server, to keep an AWS environment up to date, and always pointed to from the same CNAME?
We're just starting to use AWS with a new project, and as part of the project I've been tasked with creating a simple demo environment, and updating this environment each night to show the previous days progress.
I'm using Jenkins and the Cloudformation plugin to do this, and it works great in creating a simple EC2 instance in an existing security group, pointed to by a Route53 CNAME so it can be browsed at subdomain.example.com.
The problem I have is that I can't redeploy the same stack, because the recordset already exists, and CF won't overwrite it.
There are lots of guides on how to deploy an environment, but I'm struggling to find one on how to keep an environment up to date.
So I guess my question is: What's the recommended way, using a CI server, to keep an AWS environment up to date, and always pointed to from the same CNAME?
I agree with the comments in your question i.e. probably better to create a clean server and upload / update to it via continuous integration (Jenkins). Docker is super useful in this scenario which you mentioned in a later comment.
However, if you are leaning towards "immutable infrastructure" and want everything encapsulated in your CloudFormation template (Including creating a record in Route53) you could do something like the following code snippet in your AWS::CloudFormation::Init section - (See 'http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource- init.html' for more info)
"Resources": {
"MyServer": {
"Type": "AWS::EC2::Instance",
"Metadata": {
"AWS::CloudFormation::Init": {
"configSets" : { "Install" : [ "UpdateRoute53", "ConfigSet2, .... ] },
"UpdateRoute53" : {
"files" : {
"/usr/local/bin/cli53" : {
"source" : "https://github.com/barnybug/cli53/releases/download/0.6.3/cli53-linux-amd64",
"mode" : "000755", "owner" : "root", "group" : "root"
},
"/tmp/update_route53.sh" : {
"content" : { "Fn::Join" : ["", [
"#!/bin/bash\n\n",
"PRIVATE_IP=`curl http://169.254.169.254/latest/meta-data/local-ipv4/`\n",
"/usr/local/bin/cli53 rrcreate ",
{"Ref": "Route53HostedZone" },
" \"", { "Ref" : "ServerName" },
" 300 A $PRIVATE_IP\" --replace\n"
]]},
"mode" : "000755", "owner" : "root", "group" : "root"
}
},
"commands" : {
"01_UpdateRoute53" : {
"command" : "/tmp/update_route53.sh > /tmp/update-route53.log 2>&1"
}
}
}
}
},
"Properties": { ... }
}
}
....
I've ommitted large chunks of the template to focus on the important info. The "UpdateRoute53" section creates 2 files:
/usr/local/bin/cli53 - CLI53 is a great little wrapper program around AWS Route53 (as AWS CLI version of route53 is pretty horrible to use i.e. requires creating large chunks of JSON) - see https://github.com/barnybug/cli53 for more info on CLI53
/tmp/update_route53.sh - creates a script to upload to Route53 via the CLI53 script we installed in (1). This script determines the PRIVATE_IP via a curl comand to the special AWS meta data endpoint (see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html for more details). The "zone id" to the correct hosted zone is injected via a CloudFormation parameter (i.e. {"Ref": "Route53HostedZone" }). Finally the name of the record comes from the "ServerName" parameter but how this is set could vary from template to template.
In the "commands" section we run the script we created in "files" section (2) and output the results a log file in the /tmp folder.
NOTE (1) - The parameter Route53HostedZone can be declared as follows: -
"Route53HostedZone": {
"Description": "Route 53 hosted zone for updating internal DNS",
"Type": "AWS::Route53::HostedZone::Id",
"Default": "VIWIWK4PYAC23B"
}
The cool thing about the "AWS::Route53::HostedZone::Id") parameter type is that it displays a combo box (when running a CloudFormation template via the AWS web console) showing the zone name with the value being the Zone ID.
NOTE (2) - The --replace attribute in the CLI53 script overrides existing records which is probably what you want.
NOTE (3) - Another option would be to SSH via Jenkins (e.g. using the the "Publish Over SSH Plugin" - https://wiki.jenkins-ci.org/display/JENKINS/Publish+Over+SSH+Plugin), determine the private IP and using the CLI53 script update Route53 either from the server you've logged into or even the build server (when Jenkins is running).
Lots of options - hope you get it sorted! :-)