We run only a small handful of instances on Google Cloud Platform and we don't run them all the time. Generally we just fire one up, do what we need to do then shut it down... which is great, except when "we" forget to shut them down.
I've been able to track down the relevant REST APIs and the gcloud sdk but I don't see anything that says who started the instance. Actually it also doesn't have a timestamp on when it was started.
I did find this python app engine script that I might be able to rewrite to stop the instances after X amount of time, but I'd rather find a way to notify the user who started it and let them know the instance is still running.
Has anyone tried to do something similar or seen a way to get the "starter" of the instance in GCP?
You can look into the Audit Logs to determine who did what, where, and when. Further, you can use the Stackdriver Logging API method entries.list to retrieve audit log entries for your use case.
Also you can choose use the Activity Logs to know the details such as the authorized user who made the API request.
With the new API you have to filter on the following:
resource.type="gce_instance"
resource.labels.instance_id="ID"
protoPayload.methodName="v1.compute.instances.start"
Related
I have a deep learning web application deployed in GCE. I created a template to build a VM instance group. Then added loading balancing to it.
I plan that when each user accesses the URL, the requests from the same user will always be assigned to a VM instance. I use gunicorn -b 0.0.0.0:5000 wsgi:app -t 600 as part of the startup script. (I also tried with workers, gevent. But requests from different users can be handled in the same instance, as a result of which, the results were affected by each other. So I want requests from different users to be handled in different instances.)
To do so, I tried different CPU utilization for autoscaling. It can autoscale with new instances. But from the results, sometimes the requests are still handled in the same instance.
I also tried Kurbenetes, app engine, and cloud run. Mistakes are similar. I feel I am working in the wrong direction.
Thanks in advance.
---UPDATE---
As mentioned by #John Hanley, assigning requests from a user always to the same instance is not the targeted feature of these products. If you are looking for the answer to this question, you may try the Cloud Tasks + App Engine.
Actually, I want requests from different users to be handled in different instances so that the back-end deep learning algorithm's results cannot affect each other.
So, instead of spinning up an instance, another way to solve this is to store necessary data from each user in a common database with a unique session ID.
A simple demo can be found in https://cloud.google.com/python/docs/getting-started/session-handling-with-firestore
Hope this can be helpful for anyone struggling with similar problems.
As mentioned in the comment section by #John Hanley, the best approach would be to use App Engine + Cloud Tasks, I suggest you to check the next tutorial, although it uses ngrok instead of unicorn the idea of the of the workflow should be similar for what you want to achieve.
I am a total beginner in cloud service management, so this is a very basic question.
I have inherited a kubernetes based project running in Google Cloud. I have discovered recently that there are millions of errors I am unaware of in APIs & Services > Compute Engine API > Metrics menu:
I have tried searching for these values both on google in the docs to no avail. With no link to the list of logs and hundreds of sub menu items I feel completely lost on where to start.
How can I get more information about these errors?
How can I navigate to the relevant logs?
Your question is rather general so I will make some assumptions and educated guesses about your project and try to explain.
This level of error with API calls is of course unusually high and suggesting that some things don't work (for example someone deleted a backend service but left the load balancer without any health checks and it's accepting requests from the outside but there's nothing in the backend to process them).
That is just an exmaple - without more details I'm not even speculate further.
If you want to read more about the messages take the second one from the top - documentation for compute.v1.BackendServicesService.delete.
You can also explore other Compute Engine API methods to see what they do to give you more insight what is happening with your project.
This should give you a good starting point to explore the API.
Now - regarding logs. Just navigate to Logs Viewer and select as a resource whatever you want to analyse (all or a single VM, Load Balancer, firewall rule, etc). You can also include (or exclude) certain level of logs (warning, error etc). Pissibilities are endless.
Your query may look something like this:
Here's more documentation on GCP Logs Viewer to help you out.
I have a large number of instances across multiple cloud providers. Each of them is running a single Java program. I want to check that all of these Java programs are running and haven't crashed, and if/when one of them crashes, I want to be alerted about it.
At the moment I have a hacked-together solution that I run from my local computer, which will loop through an array of all the IP addresses, and send a command through SSH to each of them to check ps -ef and count the number of Java processes are running. If that number is zero then I will popup something on my screen to alert me.
Is there a better solution? Ideally I could use a Zabbix-style tool to handle it for me but I don't know if anything exists that serves this need.
If you have large number of applications running in the cloud then you may like to consider Cloud monitoring tools instead of re-inventing the wheel. I am sure you would like to monitor more than just the process up/down status. There are plenty of cloud monitoring tools, which allow you to monitor both the platform(machines) and the processes. Also, the different types of notifications can be configured depending on the need.
I would suggest that you look into Cloud monitoring solutions such as New Relic/Datadog/Pager Duty/etc. If it is commercially viable then I would highly recommend you to use them.
You can have all of your services write a status metric to CloudWatch Metrics and create an alarm when any do not report a status. This example shows using CloudWatch metrics to report on linux performance counters.
I have just recently started to work with Google Cloud and I am trying to wrap my head around some of its inner workings, mainly the audit logging part.
What I want do is get the log activity from when my keys are used for anything and also when someone actually logged into the Google Console Cloud (it could be the Key Vault or the Key Ring, too).
I have been using power shell to extract these logs using gcloud read logging and this is where I start to doubt whether I have the right place. I will explain:
I have created new keys and I see in the Activity Panel this action, and I can already extract this through gcloud read logging resource.type=cloudkms_cryptokey (there could be a typo on the command line, since I am writing it from the top of my head, sorry for that!).
Albeit I have this information, I am rather curious if this is the correct course of action here. I saw the CreateCryptoKey and SetIamPolicy methods on my logs, alright, but am I going to see all actions related to these keys? By reading the GCloud docs, I feel as though I am only getting some of the actions?
As I have said, I am trying to work my way around the GCloud Documentation, but it is such an overwhelming amount of information that I am not really getting the proper answer I am looking for, this is why I thought about resorting to this community.
So, to summarize, am I getting all the information related to my keys the way I am doing right now? And what about the people that have access to the Google Cloud Console page, is there a way to find who accessed it and which part (Crypto Keys page, Crypto Vault page for example)? That's something I have not understood from the docs as well, sadly. Perhaps someone could show me the proper page where I can make references to what I am looking for? Because the Cloud Audit Logging page doesn't feel totally clear to me on this front (and I assume I could be at fault here, these past weeks have been harsh!)
Thanks for anyone that takes some time to answer my question!
Admin activities such as creating a key or setting IAM policy are logged by default.
Data access activities such as listing Cloud KMS resources (key rings, keys, etc.), or performing cryptographic operations (encryption, decryption, etc.) are not logged by default. You can enable data access logging, via the steps at https://cloud.google.com/kms/docs/logging. I'm not sure if that is the topic you are referring to, or https://cloud.google.com/logging/docs/audit/.
Is it possible to open a website,like facebook.com for example, on an amazon web service?
My objective is to automate a certain task in a game and to do so without having to be online on my computer. The point is to spend less time on that game, but to not be left behind on the progress. (I'm building a bot to automate the daily tasks there, just need to know if i can now leave everything running on amazon)
Another project i want to do is to automate access to my email account and perform certain tasks depending on the emails i receive.
You get the point, i tried searching on google but i only find results about creating or hosting your own website in there and not about accessing existing websites and using automation in them.
It sounds like what you want is a virtual private server - basically a computer in the cloud that you control and is always on.
AWS have a service called LightSail for this kind of purpose. Under the hood lightsail just uses EC2, but lightsail takes away a lot of the options and configuration to provide a simpler 'click and go' kind of service.
Once you have a server you can schedule regular tasks. Depending on the complexity of your needs, you could look at using Cron as a scheduler and curl for you http requests.
For the specifics of any project you have I would suggest opening a new question with details of what you are trying to do, the reading you have done, and examples of any code you have tried.