I'm trying to estimate how much GuardDuty is going to cost me per month and according to https://aws.amazon.com/guardduty/pricing/ I should look at how many cloudtrail logs I produce a month as well as how much VPC logs in GB I produce a month.
Using boto3 S3 I can count how many logs are in my bucket, which tells me how much I am going to spend having GuardDuty read my logs. Now I wish to find how many GB's of data my VPC logs are producing, but I can't seem to figure out where I can pull that kind of information from. I want to programmatically see how many GB's of VPC flow logs I produce a month to best estimate how much I would spend on GD.
This code snippet is to show how to get the size of VPC flow flogs associated with each network interface in the VPC. You have to modify the script to get the logs for the entire month and sum it.
import boto3
logs = boto3.client('logs')
# List the log groups and identify the VPC flow log group
for log in logs.describe_log_groups()['logGroups']:
print log['logGroupName']
# Get the logstreams in 'vpc-flow-logs'
for log in logs.describe_log_streams(logGroupName='vpc-flow-logs')['logStreams']:
print log['logStreamName'], log['storedBytes']
describe_log_streams
Lists the log streams for the specified log group. You can list all
the log streams or filter the results by prefix. You can also control
how the results are ordered.
This operation has a limit of five transactions per second, after which transactions are throttled.
Request Syntax
response = client.describe_log_streams(
logGroupName='string',
logStreamNamePrefix='string',
orderBy='LogStreamName'|'LastEventTime',
descending=True|False,
nextToken='string',
limit=123
)
Related
Is there a quick way to check how many data (volume wise, GBs, TBs etc) did my specific DMS task transfered for example within last month?
I can't find any note in the documentation on that, I could probably try with boto3 but want to double check first. Thanks for help!
Even with Boto3, you can check the API - DescribeReplicationTasks but likely, there is no information about your data transfer.
Reference: https://docs.aws.amazon.com/dms/latest/APIReference/API_DescribeReplicationTasks.html
If you have only 1 data replication task that is associated with only 1 replication instance, you can check that replication instance's network metric via CloudWatch metric. From CloudWatch metrics, AWS DMS namespace, there will be several network metrics such as NetworkTransitThroughput or NetworkReceiveThroughput. You can choose one and try as below:
Statistic: Sum
Period: 30 days (or up to you)
And you have a 30DAYS_THROUGHPUT.
I want to query AWS load balancer log to automatically and on schedule send report for me.
I am using Amazon Athena and AWS Lambda to trigger Athena. I created data table based on guide here: https://docs.aws.amazon.com/athena/latest/ug/application-load-balancer-logs.html
However, I encounter following issues:
Logs bucket increases in size day by day. And I notice if Athena query need more than 5 minutes to return result, sometimes, it produce "unknown error"
Because the maximum timeout for AWS Lambda function is 15 minutes only. Therefore, I can not continue to increase Lambda function timeout to wait for Athena to return result (if in the case that Athena needs >15 minutes to return result, for example)
Can you guys suggest for me some better solution to solve my problem? I am thinking of using ELK stack but I have no experience in working with ELK, can you show me the advantages and disadvantages of ELK compared to the combo: AWS Lambda + AWS Athena? Thank you!
First off, you don't need to keep your Lambda running while the Athena query executes. StartQueryExecution returns a query identifier that you can then poll with GetQueryExecution to determine when the query finishes.
Of course, that doesn't work so well if you're invoking the query as part of a web request, but I recommend not doing that. And, unfortunately, I don't see that Athena is tied into CloudWatch Events, so you'll have to poll for query completion.
With that out of the way, the problem with reading access logs from Athena is that it isn't easy to partition them. The example that AWS provides defines the table inside Athena, and the default partitioning scheme uses S3 paths that have segments /column=value/. However, ALB access logs use a simpler yyyy/mm/dd partitioning Scheme.
If you use AWS Glue, you can define a table format that uses this simpler scheme. I haven't done that so can't give you information other than what's in the docs.
Another alternative is to limit the amount of data in your bucket. This can save on storage costs as well as reduce query times. I would do something like the following:
Bucket_A is the destination for access logs, and the source for your Athena queries. It has a life-cycle policy that deletes logs after 30 (or 45, or whatever) days.
Bucket_B is set up to replicate logs from Bucket_A (so that you retain everything, forever). It immediately transitions all replicated files to "infrequent access" storage, which cuts the cost in half.
Elasticsearch is certainly a popular option. You'll need to convert the files in order to upload it. I haven't looked, but I'm sure there's a Logstash plugin that will do so. Depending on what you're looking to do for reporting, Elasticsearch may be better or worse than Athena.
I'd like to know if possible to discover which resource is behind this cost in my Cost Explorer, grouping by usage type I can see it is Data Processing bytes, but I don't know which resource would be consuming this amount of data.
Have some any idea how to discover it on CloudWatch?
This is almost certainly because something is writing more data to CloudWatch than previous months.
As stated this AWS Support page about unexpected CloudWatch logs bill increases:
Sudden increases in CloudWatch Logs bills are often caused by an
increase in ingested or storage data in a particular log group. Check
data usage using CloudWatch Logs Metrics and review your Amazon Web
Services (AWS) bill to identify the log group responsible for bill
increases.
Your screenshot identifies the large usage type as APS2-DataProcessing-Bytes. I believe that the APS2 part is telling you it's about the ap-southeast-2 region, so start by looking in that region when following the instructions below.
Here's a brief summary of the steps you need to take to find out which log groups are ingesting the most data:
How to check how much data you're ingesting
The IncomingBytes metric shows you how much data is being ingested in your CloudWatch log groups in near-real time. This metric can help you to determine:
Which log group is the highest contributor towards your bill
Whether there's been a spike in the incoming data to your log groups or a gradual increase due to new applications
How much data was pushed in a particular period
To query a small set of log groups:
Open the Amazon CloudWatch console.
In the navigation pane, choose Metrics.
For each of your log groups, select the IncomingBytes metric, and then choose the Graphed metrics tab.
For Statistic, choose Sum.
For Period, choose 30 Days.
Choose the Graph options tab and choose Number.
At the top right of the graph, choose custom, and then choose Absolute. Select a start and end date that corresponds with the last 30 days.
For more details, and for instructions on how to query hundreds of log groups, read the full AWS support article linked above.
Apart from the steps which Gabe mentioned what helped me identify the resource which was creating large number of logs was by:
heading over to Cloudwatch
selecting the region which showed in Cost explorer
Selecting Log Groups
From settings under Log Groups, Enabling column Stored bytes to be visible
This showed me which service was causing a lot of logs to be written to Cloudwatch.
We have recently huge cost increasing (x8 times) on CloudWatch GetMetricData operation. We have a lot of log groups and different teams on the same Aws Account.
Do you know how could we find out the GetMetricData is for which log group ?
Thanks.
Unfortunately, there's no easy answer your question. We had the same issue where a line on the bill call "GetMetricsData API" was getting completely out of control. It's a shame AWS CloudTrail does not log such request. To discover the root cause, we had to disable all external monitoring tool we had plugged on this account one by one and monitor for a dent in the bill. See this article.
AWS does not tie the charges of GetMetricData to specific CloudWatch Log Groups so sadly this is not possible to see. The only things that you can see on a per log group basis are "processing bytes" and storage. If you believe that those could be close proxies, then you can query them directly via Cost and Usage Reports...but it may be that ingestion costs are not at all tied to querying of metric data.
An alternative hosted solution for seeing all of this data aggregated together would be https://www.vantage.sh/ which will query for all CloudWatch log groups and show you all the costs that it can on a per Log Group basis but you'll need to enabled "Advanced Analytics" from them.
Is it possible to have some metrics about how many search requests were processed over a certain time on ElasticSearch at AWS? Something like the cloudwatch monitoring for Cloudsearch that you can check the number of successful requests per minute (RPM):
Just find out the endpoint _stats that allow user to retrieve interesting metrics, so basically you will have to divide indices.search.query_total for indices.search.query_time_in_millis to have an average time for each query.
Still don't know a good way to have a real time data to plot a monitoring graph
Source #1
Source #2