How would you use an existing Compute Engine VM instance for a Google Cloud Build pipeline?
I know there's been a similar question in the past, however, the suggested answer is not really what I want - creating and then destroying a Compute Engine with every build.
In settings, Cloud Build allows you to enable "service account permissions" for Compute Engine (Compute Instance Admin (v1)), but I've found no information how to use that permission and service for running the build process with one of your predefined VM instances.
Or maybe I misunderstand the answer in the linked thread above and
COMMAND=sudo supervisorctl restart
actually restarts the existing VM supervisorctl? Any help would be appreciated.
You can't run a Cloud Build build on a GCE instance. The most customizable option you would have is to run the build on a private pool. But even in those cases it's always managed, you never have access to the underlying VM.
Another option would be to start a powerful GCE instance with Cloud Build via the GCE API, run your operations there and then stop the GCE instance.
Related
I need a Compute Engine instance to import the exact configuration (IP, services, files, etc...) of the original machine, without impacting the frontend if it concerns a web server for example. While running this machine, I would be able to shut down the original machine to increase its RAM or vCPUs before starting it again and deleting the cloned instance.
The problem is that I want to automate this process, and that's why I need the gcloud command. So is there a way to clone an entire gcp instance using the gcloud command or another tool?
This is not possible with the gcloud. This is possible with the cloud console, but as you can see in this documentation:
Restrictions
You can only use this feature in the Cloud Console; this feature is not supported in the gcloud tool or the API.
What you could do is create similar (not completely equal) instances from a custom image, using that all you have to do is use the following command:
gcloud compute instances create --image=IMAGE
More details on that command can be found here
In a GCP project, I have 10 virtual machines in GCE (runs sshd).
I have a need to run a script on each of the 10 virtual machines (in GCE) once an hour. I would like this to be centralized because number of VMs will grow over time and I do not want to have to do this on every single VM. In addition, I would want to analyze the data I get back in a central place.
However, I do not want to use a bastion VM, because I would like a cloud-native solution that does not require maintaining yet another virtual machine.
Which GCP service can do this?
I have looked into Cloud Run and Cloud Composer. I was not able to do this with Cloud Run, although that may be my own lack of familiarity with the product. Cloud Composer seems like a overkill.
As #JohnHanley mentioned, you will need to write code or scripts to launch commands on VMs dynamically because GCP doesn't have the type of service you require.
You may want to consider Cloud Identity-Aware Proxy (IAP) as it can be used for building your solution:
IAP helps to protect SSH access to your VMs without needing to provide
your VMs with public IP addresses, and without having to set up
bastion hosts.
For instance, you can check the enable IAP on Compute Engine guide.
You can also create a feature request for Google to consider implementing this solution.
Your best solution, with no additional charges, would be to :
Use a start-up script on each GCE
In order to set a CRON instruction to execute your script
crontab.guru can help you fin the CRON instruction, hourly is 0 * * * *
When running vm instance cluster+ nodes even if I am using and running things on the cluster/ dataproc, the vm instance shuts off automatically after about 30 minutes or so. I cannot find this setting and would appreciate any help re: how to disable this to prevent it from shutting off or even how to configure a new cluster in a way that will prevent this from happening.
Thank you
Default Dataproc clusters do not have any kind of automatic shutdown.
If you are using the older Datalab initialization action, you are probably seeing Datalab's own non-Dataproc-aware shutdown functionality, which you can disable one of the ways suggested here: How to keep Google Dataproc master running?
Otherwise, if you're using some kind of template or copy/paste arguments for creating your Dataproc cluster, perhaps you're accidentally setting "scheduled deletion": https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/scheduled-deletion
If neither of those settings explain your situation, you should visit your "activity logs" from the "Cloud Logging" interface, selecting Cloud Dataproc Cluster, and opening up the activity_log type of logs to see an audit log of who was deleting your cluster. Alternatively, if the cluster still existed in Dataproc, but the underlying VM was being shut down, visit the "Compute Engine VM" log category and also look at "activity logs" to see who was stopping your VMs. Sometimes, in a shared project, a project admin might be running some kind of script to automatically shut down VMs to save cost.
It probably is a silly question. In my project we'll be using Dataflow and Cloud composer. For that I had asked permission to create a VM instance in the GCP project to keep the both the Dataflow and Cloud composer python program. But the client asked me the reason of creation of a VM instance and told me that you can execute the Dataflow without the VM instance.
Is that possible? If yes how to achieve it? Can anyone please explain it? It'll be really helpful to me.
You can run Dataflow pipelines or manage Composer environments in you own computer once your credentials are authenticated and you have both the Google SDK and Dataflow Python library installed. However, this depends on how you want to manage your resources. I prefer to use a VM instance to have all the resources I use in the cloud where it is easier to set up VPC networks including different services. Also, saving data from a VM instance into GCS buckets is usually faster than from an on-premise computer/server.
I want to start/stop a set of Compute engine instances in Google Cloud Platform using Google Cloud Scheduler. How can I do it?
In order to start and stop a Compute Engine using the Cloud Scheduler you can follow Google this tutorial, or this other
I won’t be copy-pasting the required code here because the tutorial it's very complete but I will resume here the steps to follow.
Set up your Compute Engine instances
Deploy the starter Cloud Function. You can see an example in here
Deploy the stop Cloud Function. You can see an example in here
Set up the Cloud Scheduler jobs
If you need any help with the tutorial please just let me know!
I still wonder why gcp has still not have this feature in the first place.
Anyways These simple steps did the job for me
Create a new JobScheduler.
Fill in the required details
Choose frequency which suits your requirement.
Choose the target to Pub/Sub.
Choose the topic name (Create a new topic if not created ).
In the payload section use this stop script
gcloud compute instances stop instance-name.
To verify the change you can run the job manually and check
I use vm instance API directly. No need for a cloud function.
Here is the link to the api description:
https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop
The API Call: POST https://compute.googleapis.com/compute/v1/projects/{project}/zones/{zone}/instances/{resourceId}/stop
You can start the engine in a similiar way.
Example how to configure the scheduler:
You can look at Google Article to achieve your goal https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule.
Also, If these VM instances are stateless then I would suggest to look at Google Cloud Run service which can help you to save cost and operation overhead to configure auto-shutdown/auto-startup.
Hope this helps.
The new Google Compute Engine feature of Instance Schedules can now be used to start and stop instances through the Cloud Console UI, using gcloud or via the API:
https://cloud.google.com/compute/docs/instances/schedule-instance-start-stop