We noticed that multiple compute instances got deleted at the same time after hours of 100% CPU usage. Because of this deletion, the hours of computation was lost.
Can anyone tell us why they got deleted?
I have created a gist with the only log we could find in Stackdriver logging around the time of deletion.
The log files show the following pieces of information:
The deleter's source IP address 34.89.101.139. Check if this matches the public IP address of the instance that was deleted. This IP address is within Google Cloud.
The User-Agent specifies that the Google Cloud SDK CLI gcloud is the program that deleted the instance.
The Compute Engine Default Service Account provided the permissions to delete the instance.
In summary, a person or script ran the CLI and deleted the instance using your project's Compute Engine Default Service Account key from a Google Cloud Compute service.
Future Suggestions:
Remove the permission to delete instances from the Compute Engine Default Service Account or (better) create a new service account that only has the required permissions for this instance.
Do not share service accounts in different Compute Engine instances.
Create separate SSH keys for each user that can SSH into the instance.
Enable Stackdriver logging of the SSH Server auth.log file. You will then know who logged into the instance.
Related
This might be a stupid question.
I'm just curious. I'm new to Redis and would like to experiment with it.
However, I would like to turn the instance on and off whenever I am experimenting as I want to save on costs rather than have the instance running all the time.
But I don't see a stop button like other products such as compute.
Is there a reason for this?
Thank you
You won't be able to manage a Cloud Memorystore for Redis instance as a Compute Engine instance as they are different products with different billing requirements and therefore you can't stop a Cloud Memorystore for Redis instance.
If you are only interested in learning more about Redis you can always install Redis on a Compute Engine instance (see the following tutorial for a clear path as to how to accomplish this or this other tutorial as to how to accomplish this task using docker) and afterwards delete the Compute Engine instance in order for charges to stop accruing.
To avoid incurring charges to your Google Cloud account for the resources used in this quickstart:
Go to the Memorystore for Redis page in the Cloud Console.
Memorystore for Redis
Click the instance ID of the instance you want to delete.
Click the Delete button.
In the prompt that appears, enter the instance ID.
Click Delete.
https://cloud.google.com/memorystore/docs/redis/quickstart-console#clean_up
I'm trying to execute a Script on a Google VM through Terraform.
First I tried it via Google Startup Scripts. But since the metadata is visible in the Google Console (startup scripts count as metadata) and that would mean that anybody with read access can see that script which is not acceptable.
So i tried to get the script from a Storage Account. But for that i need to attach a service account to the VM so the VM has the rights to access the Storage Account. Now people that have access to the VM also have access to my script as long as the service account is attached to the VM. In order to "detach" the service account i would have to stop the VM. Also if i don't want to permanently keep the attachment of the service account i would have to attach the service account via a script which requires another stop and start of the VM. This is probably not possible and also really ugly.
I don't understand how the remote-exec ressource works on GCP VMs. Because i have to specify a user and a userpassword to connect to the VM and then execute the script. But the windows password needs to be set manually via the google console, so i can't specify those things at this point in time.
So does anybody know how I can execute a Script where not anybody has access to my script via Terraform?
Greetings :) and Thanks in advance
I ended up just running a gcloud script in which i removed the Metadata from the VM after the Terraform apply was finished. In my Gitlab pipeline i just called the script in the "after_script"-section. Unfortunately the credentials are visible for approximately 3min.
While trying for the first time to use Google Kubernetes Cloud solution, and according to the tutorial... I am trying to create new cluster.
But after pressing Create i receive
The request contains invalid arguments: "EXTERNAL: service account
"****#developer.gserviceaccount.com" does not exist.". Error code: "7"
in a red circle near the Kubernetes cluster name.
After some investigations it's looks like the default service account which google generated for my account.
I've looked over the create cluster options, but there isn't any option to change the service account.
Do I need to change Google Compute Engine default service account? how i can do it?
How I can overcome this issue?
Thank you
Default Compute Engine Service Account is essential for functions related to Compute Engine and is being generated automatically. Kubernetes Engine utilizes Compute Engine VM Instances as Nodes used for the cluster. GKE uses the Compute Engine Service Account to authorize the creation of these nodes.
In order to regenerate default service there are two options:
Regenerate by Disabling and Re-enabling the Google Compute Engine API. In the "API's & Services" dashboard. If for some reason performing this option encountering errors when disabling the API, then try option 2.
run command gcloud services enable compute.googleapis.com in Cloud SDK or Cloud Shell which is in the header of the page.
Looks like you either do not have any default service account or have more than one.
Simply go to the "Service Accounts" section "IAM & Admin" and select the app engine default service account, and provide this as an argument while creating cluster from gcloud or gshell as below:
gcloud container clusters create my-cluster --zone=us-west1-b --machine-type=n1-standard-1 --disk-size=100 --service-account=abc#appspot.gserviceaccount.com
To initialize GKE, go to the GCP Console. Wait for the "Kubernetes Engine is getting ready. This may take a minute or more" message to disappear.
Please open the page and wait for a while
I have a Google Cloud VM instance, but I overlooked setting the Storage permission to read-write when creating it.
Now further down the line, I'm looking to experiment with cloud storage, but my instance is read-only.
How can this be changed? I understand it may be possible by relaxing the storage buckets policy, but I'd prefer my instance had write access to all future project buckets.
I presume there is an option in gcloud to change the devstorage parameter?
So, you can't change the option to grant the VM permission, but I did find that you can just run
gcloud auth login
then you can sign in with your management account from the VM and you'll be able to run commands that way.
Unfortunately you can't change the scopes of the VM. You will have to create a new one to change them.
When you create a new one you can reuse the disk of the old VM, if that helps avoid the pain.
You can update the permissions of your VM instance now but only when it is shut down. Check the following documentation:
Changing the service account and access scopes for an instance
If you want to update the API permissions for the kuberntes clusters VM instance then you cannot do that unless you create a new cluster and give the API access to the nodes associated with the clusters.
I believe they have added the option to change it now, without creating another VM.
Once you have stopped the instances, click on the instances you want to change. On the top there is a Edit button, you can click on it and change any the permissions.
Hope the image helps
edit button
Once you have changed the permission to read_write and it still says Access Denied. Go in your instance SSH browser window and enter 'gcloud auth login', follow the steps and hopefully it works!
You need to stop your vm first, then click on edit and change the cloud api access scopes for storage.
You may find more information here: https://ismailyenigul.medium.com/setting-access-scope-of-google-cloud-vm-instances-c8637718f453
I created a Amazon AWS EC2 instance under my account and made an website/ftp on it, now a new partner wants to move the instance under his company account so his company can pay the bills.
We can't change the instance IP because banks in the region are communicating with the server.
How can I move the instance to a different account without having to change anything on the configuration?
The short answer is: No, you cannot move an running instance from one account to another unless and ofcourse AWS Technical support has some magic available behind the curtains.
You can However, Create an AMI from this instance and share this AMI with other users/account. refer: http://aws.amazon.com/articles/530
To share or migrate EC2 instances from a source account to a target
account follow these steps:
Create a custom Amazon Machine Image (AMI)
from the instance you want to share or migrate. Be sure to include all
required EBS data volumes in the AMI.
Note: Data stored on instance store volumes isn't preserved in AMIs, and won't be on the instance store volumes of the instances
that you launch from the AMI.
Share the AMI with the target account
using either the EC2 console or the AWS Command Line Interface (CLI).
From the target account, find the AMI
using the EC2 console or the AWS CLI.
Launch a new instance from the shared AMI
on the target account.
Note: The private IP address of VPC instances will be different in the new account, unless you specifically set them during
launch.
Related information
Changing the Encryption State of Your Data
AWS CLI Command Reference (EC2)
Source: Transfer Amazon EC2 Instance
This is not possible.
AWS Support does not have access to copy Amazon EC2 resources or
manipulate any configuration options in AWS accounts. You can't
separate an AWS account from an Amazon.com account or transfer
resources between AWS accounts. It is possible to manually migrate
Amazon EC2 resources from one account to another by completing the
steps described here.
Source : https://aws.amazon.com/premiumsupport/knowledge-center/account-transfer-ec2-instance/
I'm working with several hundreds on EC2 instances in several AWS regions and accounts. You can move an EC2 instance to another AWS account, however, you can't move the Elastic IP and it will take up 16 steps with AWS CLI, if you want to migrate Tags and clone the Security Groups. I wrote a detailed post with the whole process at https://medium.com/#gmusumeci/how-to-move-an-ec2-instance-to-another-aws-account-e5a8f04cef21.
there are more than 10 steps involved in doing the cloud move. I would suggest you use Infrastructure as a Configuration (terraform and CloudFormation) or Infrastructure as a real code (pulumi and CDK)
however if you want to give a go at a nice tool I found called KopiCloud. Please feel welcome to try it and leave your comments below. Is good if you need to move instances on a quick lift and shift scenario.
You can re-think the design of having the banks in the region communicating to your servers via IP.
If the banks communicate using DNS names, you have much more flexibility to move your servers around.
You can also achieve improvements in high availability and resiliency by moving to DNS connections.
So a plan might be
Setup a DNS record for your existing server
Get the banks who connect to your server to connect via the DNS name
Setup your new server in the other account (other answers describe this)
Cut the banks over to your new server in the new account simply by updating the DNS record
I haven't tried load balancing across accounts, but that may be another option, which would give you HA as a bonus. By registering your current instance, and new instance in another account as targets with a load balancer and getting your clients to connect to the load balancer, you could cut over to the other account. The only part I haven't tried is registering targets in different accounts, but looks like this should be possible with an AWS Network Load Balancer