I have successful secured my backend services using cloud armor and applying white listing or black listing.
Supposing that there is traffic coming from some specific addresses, is there any way to detect them automatically based on the frequency without iterating over the StackDriver logs?
If so is there any way to blacklist them in an automated way?
Cloud Armor does not offer "intelligent" features at the moment. As stated in the Google public docs, it has straight policies for white-listing or blacklisting CIDR ranges. What can be done (not simple, considerable effort required) is to create Stackdriver sink and export the logs for blacklisted IPs. based on the logs captured, there could be cloud function jobs to monitor the logs and then kick off the creation of cloud armor policies to block the offending IPs.But as mentioned, this is not simple. considerable effort required.
Related
I'm new to AWS wolrd. My purpose is to find as soon as possible in case of problems using Elastic Load Balancer logs top ips from requests, if possible who they are or some inspection on it. I only found paid services. Does anyone know a free application or maybe a website that analyzes AWS ELB logs?
Completely free solution isn't available as I know. Btw, there are cheap solutions.
You can monitor your load balancer by "Access logs", "CloudWatch metrics", "Request tracing" and "CloudTrail logs".
I don't understand exactly what you want, but there are some possible solutions.
If you're afraid of being attacked and you need immediate protection (against security scans, DDoS etc), you can use AWS's own services. "AWS Shield Standard" is automatically included at no extra cost. Btw, "For added protection against DDoS attacks, AWS offers AWS Shield Advanced". https://docs.aws.amazon.com/shield/
WAF is also good against attacks. You can create rules, rule-actions etc. Sadly it's not completely free. It runs "pay-as-you-use" style. https://aws.amazon.com/waf/pricing/
you can store the access log in S3 and analyse it later, but this can be costly in the end (and it's not real time)
you can analyse your log records with Lambda function. In this case, you need to use some NoSQL or something to store states or logics. (Lambda and DynamoDB is "pay-as-you-use" style and cheap, but not for free)
Keep in mind that:
The load balancer and lambda also increments the corresponding CloudWatch metric (it's cheap, but not for free)
You will pay for the outgoing data transfer. I mean from AWS to internet 1TB/month/account is always free (through CloudFront): https://aws.amazon.com/free/
you should use AWS's own services if you want a cheap and good solution
Elastic Load Balancing provides access logs that capture detailed information about requests sent to your load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.
But keep in mind that access logging is an optional feature of Elastic Load Balancing that is disabled by default. After you enable access logging for your load balancer, Elastic Load Balancing captures the logs and stores them in the Amazon S3 bucket that you specify as compressed files. You can disable access logging at any time.
There are many complex and paid application that returns information regard access log but i advise you a simple, easy to use website that i use when i want to see top requester on our load balancer.
Website is https://vegalog.net
You shoud only upload your log file taken from S3 bucket and it returns to you a report with top requester, who they are (using whois function), response time and other useful informations.
I have not used my GCP account in months and shut everything down yet I still see this
I have never use the compute engine api. I went in and disabled it. I am just curious for self-knowledge here as to why it would be recording anything BUT 'my personal requests'. I wonder if it is recording hackers hitting the api? or port scanners? not really sure her. Anyone know?
You can have a look on the metrics page and add the view Traffic by credential. All that traffic is Anonymous or Unspecified - meaning that is not performed by any of the service accounts that you created in the project.
Hence the requests have a stable pattern and never stop, I think those aren't external port scanners :).
I guess the traffic is generated by google's system services (billing/monitoring etc), especially because the traffic is present only on the compute engine API.
Confirmed with a Google Cloud Platform support agent, in short:
There is nothing to be worry on the traffic that can be seen under Compute Engine API as it is designed to run for different purposes on your project.
More precisely:
These methods are part of the Compute Engine API for your vm instance and the logs of traffic does not mean that these are the traffic that goes inside and outside of your VM instance from any users. Instead these traffics means the response of the VM instances for the services under the API even though it is stopped which is normal.
For example, there is a method on the table logged named instances.getScreenshot, this method returns the screenshot from the specified instance. Screenshot is used as part of the project service for troubleshooting your VM instances whenever you goes to console under VM instances information > Screenshot tab, the traffic was recorded by the API and response to your request.
Another example is the method named backendServices.list, this retrieves the list of BackendService resources available to the specified project. This is part of the API for the project to return that information. All of these traffic are used within the project. For more information regarding GCP compute engine API, you can refer to this guide to understand more of the methods used. Meaning, there is nothing to be worry on the traffic that can be seen under Compute Engine API as it is designed to run for different purposes on your project.
Moreover, regarding potential charges of that traffic:
That traffic will not contribute to your charges as charges in GCP pertains to those resources only consumed.
[...] please note that the charges on your VM instances that are currently stopped are the resources attached to it like the disks and the external IP address if there's any.
You can check more details on pricing in this VM Instance Pricing guide.
Additionally, here's a guide on how to access this traffic metric:
Go to GCP Console
Click on hamburger menu
Click on APIs & Services
Scroll down to the bottom of the Dashboard and click on the Compute Engine API from the list of filters
Click on 'View metrics' button on the bottom of 'Traffic by response code' card
This is what it looks like:
I have a small number of HTTP servers on GCP VMs. I have a mixture of different server languages and Linux based OS's.
Questions
A. It it possible to use the Stackdriver monitoring service to set alerts at specific percentiles for HTTP response latencies?
B. Can I do this without editing the code of each server process?
C. Will installing the agent into the VM report HTTP latencies?
For example, if the 95th percentile goes over 100ms for a certain time period I want to know.
I know I can do this for CPU utilisation and other hypervisor provided stats using:
https://console.cloud.google.com/monitoring/alerting
Thanks.
Request latencies are extracted by cloud load balancers. As long as you are using cloud load balancer you don't need to install monitoring agent to create alerts based 95th Percentile Metrics.
Monitoring agent captures latencies for some preconfigured systems such as riak, cassandra and some others. Here's a full list of systems and metrics monitoring agent supports by default: https://cloud.google.com/monitoring/api/metrics_agent
But if you want anything custom, i.e. you want to measure request latencies from VM you would need to capture response times yourself and configure logging agent to create a custom metric which you can use to create alerts. And as long as you are capturing them as distribution metrics you should be able to visualise different percentiles (i.e. 25, 50, 75, 80, 90, 95 and 99th etc.) and create alert based on that.
see: https://cloud.google.com/logging/docs/logs-based-metrics/distribution-metrics
A. It it possible to use the Stackdriver monitoring service to set
alerts at specific percentiles for HTTP response latencies?
If you want to simply consider network traffic, yes it is possible. Also if you are using a load balancer it's also possible to set alerts on that.
What you want to do should be pretty straight forward from the interface, however you can also find more info in the documentation.
If you want to use some advanced metric on top of tomcat/apache2 etc, you should check the list of metrics provided by the stackdriver monitoring agent here.
B. Can I do this without editing the code of each server process?
Yes, no need to update any program, stackdriver monitoring works transparently and will be able to fetch basic metrics from a GCP VMs without the need of the monitoring agent, including network traffic and cpu utilization.
C. Will installing the agent into the VM report HTTP latencies?
No, the agent shouldn't cause any http latencies.
when I make an unauthenticated (public) Cloud Run endpoint to host an API, what are my options to protect this endpoint from malicious users making billions of HTTP requests?
For $10 you can launch a Layer 7 HTTP flood attack that can send 250k requests per second. Let's assume your Cloud Run endpoints scale up and all requests are handled. For invocations alone, you will pay $360,-/hour (at $0.40 per million requests).
Note that there is a concurrency limit and a max instance limit that you might hit if the attack is not distributed over multiple Cloud Run endpoints. What other controls do I have?
As I understand, the usual defenses with Cloud Armor and Cloud CDN are bound to the Global Load Balancer, which is unavailable for Cloud Run, but is available for Cloud Run on GKE.
For unauthenticated invocations to a Cloud Run service with an IAM Cloud Run Invoker role set to the allUsers member type, I would expect the answer to be the same as those provided here - https://stackoverflow.com/a/49953862/7911479
specifically:
Cloud Functions sits behind the Google Front End which mitigates and absorbs many Layer 4 and below attacks, such as SYN floods, IP fragment floods, port exhaustion, etc.
It would certainly be great to get a clear Y/N answer on Cloud Armor support.
[Edit]: I have been thinking on this quite a lot and have come to the following conclusion:
if you expect you are likely to become a victim of an attack of this type then I would monitor your regular load/peak and set your account's ability to scale just above that load. Monitoring will allow you to increase this as your regular traffic grows over time. It appears to be the only good way. Yes, your service will be down once you reach your account limits, but that seems preferable in the scenario where you are the target.
An idea which I am yet to try is a protected route with Firebase Authentication and anonymous authentication.
I want help on understanding the AWS cost explorer graph to track the huge data transfer usage.
I have noticed the AWS account bills for jan, Feb and March (till current date) where it is showing a huge data transfer charge as a bill line item (image attached AWS Bill line Item)
regional data transfer - in/out/between EC2 AZs or using elastic IPs
or ELB
. Further i checked it in AWS Cost Explorer reports by applying Group by filter Region wise and can see that it has data transfer for each region but also for
No Region
, i am not able able to understand this bar graph (please see the image attached and yellow graph AWS Cost Explorer Reports Region Wise) with level "No Region".
A good starting point would be to enable VPC Flow Logs. VPC Flow Logs will show you where the source and destination of all the traffic within your VPC. After you've analysed the logs, you should have a good indication of where to begin investigating.
Out of context but adding it here as it might help you: for some services such as S3, you can enable object-level logging to help you understand what is accessing your objects, which could help you further understand why you're paying for data transfers.
You can avoid paying for data transfer charges between AWS services by using VPC Endpoints. VPC endpoints allow you to connect directly to the service rather than over the internet, which will avoid incurring extra data charges. More on VPC Endpoints here.