I have not used my GCP account in months and shut everything down yet I still see this
I have never use the compute engine api. I went in and disabled it. I am just curious for self-knowledge here as to why it would be recording anything BUT 'my personal requests'. I wonder if it is recording hackers hitting the api? or port scanners? not really sure her. Anyone know?
You can have a look on the metrics page and add the view Traffic by credential. All that traffic is Anonymous or Unspecified - meaning that is not performed by any of the service accounts that you created in the project.
Hence the requests have a stable pattern and never stop, I think those aren't external port scanners :).
I guess the traffic is generated by google's system services (billing/monitoring etc), especially because the traffic is present only on the compute engine API.
Confirmed with a Google Cloud Platform support agent, in short:
There is nothing to be worry on the traffic that can be seen under Compute Engine API as it is designed to run for different purposes on your project.
More precisely:
These methods are part of the Compute Engine API for your vm instance and the logs of traffic does not mean that these are the traffic that goes inside and outside of your VM instance from any users. Instead these traffics means the response of the VM instances for the services under the API even though it is stopped which is normal.
For example, there is a method on the table logged named instances.getScreenshot, this method returns the screenshot from the specified instance. Screenshot is used as part of the project service for troubleshooting your VM instances whenever you goes to console under VM instances information > Screenshot tab, the traffic was recorded by the API and response to your request.
Another example is the method named backendServices.list, this retrieves the list of BackendService resources available to the specified project. This is part of the API for the project to return that information. All of these traffic are used within the project. For more information regarding GCP compute engine API, you can refer to this guide to understand more of the methods used. Meaning, there is nothing to be worry on the traffic that can be seen under Compute Engine API as it is designed to run for different purposes on your project.
Moreover, regarding potential charges of that traffic:
That traffic will not contribute to your charges as charges in GCP pertains to those resources only consumed.
[...] please note that the charges on your VM instances that are currently stopped are the resources attached to it like the disks and the external IP address if there's any.
You can check more details on pricing in this VM Instance Pricing guide.
Additionally, here's a guide on how to access this traffic metric:
Go to GCP Console
Click on hamburger menu
Click on APIs & Services
Scroll down to the bottom of the Dashboard and click on the Compute Engine API from the list of filters
Click on 'View metrics' button on the bottom of 'Traffic by response code' card
This is what it looks like:
Related
I had a VM in my account, and out of nowhere, the VM just disappeared. Is there any way to review what was done and why?
Seems to be if you are using free trial You need to explicitly enable billing while during the trial, otherwise your instances will be shut down when the trial runs out. It is not possible to retrieve the instances that have been deleted once. If it has been stopped, it can be retrieved back by simply starting it again.
But During the creation of the Instance you could configure deletion rules to keep the boot disk when the instance is deleted. This can be configured in the submenu “Management, security, disks, networking, sole tenancy” in the Disks section.
Refer to this SO for more information.
You can review what has been done by Audit Logs on GCP. Audit logs help you answer "who did what, where, and when?" within your Google Cloud resources with the same level of transparency as in on-premises environments. This could help you determine what happened to your VM.
To view Audit Logs for Compute Engine, please refer to this doc. To read more about the Compute Engine Audit Logs, you can review this doc.
I need to setup a shared processing service that uses a load balancer and several EC2 instances to process incoming requests using a custom .NET application. My issue is that I need to be able to bill based on usage. Only white-listed IPs will be able to call the application, but each IP only gets a set number of calls before each call is a billable event.
Since the AWS documentation for the ELB states "We recommend that you use access logs to understand the nature of the requests, not as a complete accounting of all requests", I do not feel the Access Logs on the ELB is what I'm looking for.
The question I have is how to best manage this so that the accounting team has an easy report each month that says how many calls each client made.
Actually you can use Access logs and since access logs will be written to S3, you can query each IP with Athena by using standard SQL. You can analyze your logs and extract reports.
References:
https://docs.aws.amazon.com/athena/latest/ug/what-is.html
https://aws.amazon.com/premiumsupport/knowledge-center/athena-analyze-access-logs/
So I am helping out on a project in Google cloud and recently it has come to my attention that in the "activity" tab, right next to the dashboard on the home page there is a set of logs. My problem is that I am getting a series of logs on repeat with the following order:
1.Create VM
2.Add instances to instance group
3.Remove instances from instance group
4.Delete VM
Also, the actions are being done by a service account.
This project had people working before in it, so I am not aware of everything that has been done in the past. That being said, I am tasked with finding out where this is coming from.
In my search I came up with things like there might have been an managed instance group with load balancer and autoscaler enabled which could be causing the automatic recreation of instances. However, when searching on this section there is nothing on instance groups, load balancers or anything of the like. There aren't even VMs on compute engine.
Any idea on what could be causing it or how I can begin to search for this?
I am running django on google VM instance using apache and mod wsgi... i however am unsure of the concurrent requests that my app shall receive from the users and would like to know if i can transfer the surplus load of the VM to the App engine automatically to prevent the server from crashing.
I am unable to find any solution expect running kubernetes cluster or docket containers to effectively manage the load. but in need to be free of this hassle and send off the excess load to GAE.
If you want to analyze the traffic, latency and load of your resources and applications, I would recommend you to start with Stackdriver Trace.
As per documentation, Stackdriver Trace is a distributed tracing system that collects latency data from your applications and displays it in the Google Cloud Platform Console. You can track how requests propagate through your application and receive detailed near real-time performance insights. Stackdriver Trace automatically analyzes all of your application's traces to generate in-depth latency reports to surface performance degradations, and can capture traces from all of your VMs, containers, or Google App Engine projects.
Once you have determine the user traffic or you have a better idea about this, then you can try using "Instance Groups".
GCE offers two kind of VM instance groups:
Managed instance groups (MIGs) allow you to operate applications on multiple identical VMs. You can make your workloads scalable and highly available by taking advantage of automated MIG services, including: autoscaling, autohealing, regional (multi-zone) deployment, and auto-updating.
Unmanaged instance groups allow you to load balance across a fleet of VMs that you manage yourself.
I have successful secured my backend services using cloud armor and applying white listing or black listing.
Supposing that there is traffic coming from some specific addresses, is there any way to detect them automatically based on the frequency without iterating over the StackDriver logs?
If so is there any way to blacklist them in an automated way?
Cloud Armor does not offer "intelligent" features at the moment. As stated in the Google public docs, it has straight policies for white-listing or blacklisting CIDR ranges. What can be done (not simple, considerable effort required) is to create Stackdriver sink and export the logs for blacklisted IPs. based on the logs captured, there could be cloud function jobs to monitor the logs and then kick off the creation of cloud armor policies to block the offending IPs.But as mentioned, this is not simple. considerable effort required.