I need to move more than 50 compute instances from a Google Cloud project to another one, and I was wondering if there's some tool that can take care of this.
Ideally, the needed steps could be the following (I'm omitting regions and zones for the sake of simplicity):
Get all instances in source project
For each instance get machine sizing and the list of attached disks
For each disk create a disk-image
Create a new instance, of type machine sizing, in target project using the first disk-image as source
Attach remaining disk-images to new instance (in the same order they were created)
I've been checking on both Terraform and Ansible, but I have the feeling that none of them supports creating disk images, meaning that I could only use them for the last 2 steps.
I'd like to avoid writing a shell script because it doesn't seem a robust option, but I can't find tools that can help me doing the whole process either.
Just as a side note, I'm doing this because I need to change the subnet for all my machines, and it seems like you can't do it on already created machines but you need to clone them to change the network.
There is no tool by GCP to migrate the instances from one project to another one.
I was able to find, however, an Ansible module to create Images.
In Ansible:
You can specify the “source_disk” when creating a “gcp_compute_image” as mentioned here
Frederic
Related
I am building a lab utility to deploy my teams development environments (testing / stress etc).
At present, the pipeline is as follows:
Trigger pipeline via HTTP request, args contain the distribution, web server and web server version using ARGs that are passed too multi stage dockerfiles.
Dockerx builds the container (if it doesn't exist in ECR)
Pipeline job pushes that container to ECR (if it doesn't already exist).
Terraform deploys the container using Fargate, sets up VPCs and a ALB to handle ingress externally.
FQDN / TLS is then provisioned on ...com
Previously when I've made tools like this that create environments, environments were managed and deleted solely at project level, given each environment had it's own project, given this is best practice for isolation and billing tracking purposes, however given the organisation security constraints of my company, I am limited to only 1 project wherein I can create all the resources.
This means I have to find a way to manage/deploy 30 (the max) environments in one project without it being a bit of a clustered duck.
More or less, I am looking for a way that allows me to keep track, and tear down environments (autonomously) and their associated resources relevant to a certain identifier, most likely these environments can be separated by resource tags/groups.
It appears the CDKTF/Pulumi looks like a neat way of achieving some form of "high-level" structure, but I am struggling to find ways to use them to do what I want. If anyone can recommend an approach, it'd be appreciated.
I have not tried anything yet, mainly because this is something that requires planning before I start work on it (don't like reaching deadends ha).
I have two GCP projects communicating with each over over a Classic VPN, I'd like to duplicate this entire configuration to another GCP account with two projects. So in addition to the tunnels and gateways, I have one network in each project to duplicate, some firewall rules, and a custom routing rule on one project.
I've found how I can largely dump these using various
gcloud compute [networks | vpn-tunnels | target-vpn-gateways] describe
commands, but looking at the create commands they don't seem setup to be piped to, nor use this output data as a file, not to mention there are some items that won't be applicable in the new projects.
I'm not just trying to save time, I'm trying to make sure I don't miss anything and I also want a hard copy of sorts, of my current configuration.
Is there any way to do this? thank you!
As clarified in other similar cases - like here and here - it's not possible to clone or duplicate entire projects in Google Cloud Platform. As explained in these other cases, you can use Terraformer as to generate Terraform files from existing infrastructure (reverse Terraform) and then, recreate the files in your new instance as explained here.
To summarize, you can try this CLI as a possible alternative to copy part of your structure, but as emphasized in this answer here, there is no automatic way or magic tool that will copy everything, so even your VMs configuration, your app contents, your data content, won't be duplicated.
I'm trying to create a GCP instance template which has the most recent version of my repo on it. My repository is private and I cant figure out how to clone it in the instance groups. I don't think I can use SSH because the machines will be randomly destroyed and created and therefore the generated keys will be inconsistent. Whats the best way to do this?
An Instance Template is based on an Image. This image can be a clean Ubuntu/Windows/Debian copy or a custom image created by you.
Saying that, I can think of 2 ways for you to get your repository inside there.
Using a custom image.
In essence, A snapshot of an instance with your latest code and dependencies installed on it.
There are two paths you can go with here.
a. Create a custom image when you clone the repository to the instance. You might need to that for every update in the code.
b. An alternative is to use some sort of Network File System (NFS/SMB). This will usually require more resources like another server that is always available.
If you want to avoid creating images, or as a solution to the issue mentioned in 1a you can set up a Startup Script to run on the server at boot(creation) time to clone/pull the latest copy.
There are Pros and Cons for both. I guess only you can tell what is best for you. I hope it gets you in the right direction.
Read more about creating an image here.
Read more about startup scripts here.
We are using queue based managed instance scaling. We need to setup environment variables on the VMs by instance-groups (so that same VM image can be used to subscribe to different queues in different instance-groups). I don’t see the option to define an environment variables when I create an instance-group.
Is there a way to use the same image across multiple instance groups and still achieve different VM behavior based on either different environment variable at instance group level or some other way?
Example: Create 2 managed-instance-groups with the same VM image. One has environment variable 'queue-name' set to 'queue-1' and the other has 'queue-name' set to 'queue-2'. The application deployed to the VMs in first instance-group pulls tasks from pub/sub queue 'queue-1' and on other group pulls tasks from 'queue-1'.
Using two templates same VM image
In order to create two instances groups with the same VM image having a different behaviour you can definitely make use of two different instance templates.
In this way you will be able to change networking configuration, startup and shutdown scripts or metadata.
For example you could make use of a startup script to set up the different environment variables and in this way connect to one. Fort example like here.
Using same Template same VM image
On the other hand if you cannot make use of two different templates I would propose a small hack but I guess there are several ways to do it.
As you noticed there is no a direct way to do it (since there is the possibility to customise already in the template creation).
I would add in the startup script a small portion of code that making use of the gcloud command understand which is the name of the instance group it belongs to and basing itself on this info set up in different ways an environment variable.
In this way you will need merely to follow some kind of pattern naming your instances, but I am sure you can find more elegant solution.
Or you could even base your decision on the hostname of the machine (but I like this solution even less).
Let's say I've created an AMI from one of my EC2 instances. Now, I can add this manually to then LB or let the AutoScaling group to do it for me (based on the conditions I've provided). Up to this point everything is fine.
Now, let's say my developers have added a new functionality and I pull the new code on the existing instances. Note that the AMI is not updated at this point and still has the old code. My question is about how I should handle this situation so that when the autoscaling group creates a new instance from my AMI it'll be with the latest code.
Two ways come into my mind, please let me know if you have any other solutions:
a) keep AMIs updated all the time; meaning that whenever there's a pull-request, the old AMI should be removed (deleted) and replaced with the new one.
b) have a start-up script (cloud-init) on AMIs that will pull the latest code from repository on initial launch. (by storing the repository credentials on the instance and pulling the code directly from git)
Which one of these methods are better? and if both are not good, then what's the best practice to achieve this goal?
Given that anything (almost) can be automated using the AWS using the API; it would again fall down to the specific use case at hand.
At the outset, people would recommend having a base AMI with necessary packages installed and configured and have init script which would download the the source code is always the latest. The very important factor which needs to be counted here is the time taken to checkout or pull the code and configure the instance and make it ready to put to work. If that time period is very big - then it would be a bad idea to use that strategy for auto-scaling. As the warm up time combined with auto-scaling & cloud watch's statistics would result in a different reality [may be / may be not - but the probability is not zero]. This is when you might consider baking a new AMI frequently. This would enable you to minimize the time taken for the instance to prepare themselves for the war against the traffic.
I would recommend measuring and seeing which every is convenient and cost effective. It costs real money to pull down the down the instance and relaunch using the AMI; however thats the tradeoff you need to make.
While, I have answered little open ended; coz. the question is also little.
People have started using Chef, Ansible, Puppet which performs configuration management. These tools add a different level of automation altogether; you want to explore that option as well. A similar approach is using the Docker or other containers.
a) keep AMIs updated all the time; meaning that whenever there's a
pull-request, the old AMI should be removed (deleted) and replaced
with the new one.
You shouldn't store your source code in the AMI. That introduces a maintenance nightmare and issues with autoscaling as you have identified.
b) have a start-up script (cloud-init) on AMIs that will pull the
latest code from repository on initial launch. (by storing the
repository credentials on the instance and pulling the code directly
from git)
Which one of these methods are better? and if both are not good, then
what's the best practice to achieve this goal?
Your second item, downloading the source on server startup, is the correct way to go about this.
Other options would be the use of Amazon CodeDeploy or some other deployment service to deploy updates. A deployment service could also be used to deploy updates to existing instances while allowing new instances to download the latest code automatically at startup.