I'm working on Centralized Logging of my environnement AWS presented in this guide:https://docs.aws.amazon.com/solutions/latest/centralized-logging/overview.html
I'm using the same architecture as presented in AWS Doc.
I'm searching the possibility to send metrics from CloudWatch of each account to OpenSearch in centralized account. I was thinking of a Lambda function that retrieves each X minutes the data with get_metric_data ? I'm collecting the metrics with CloudWatch Agent
Is there an another method to do it ?
Thank you
Related
Suppose multiple aws services are running (like EC2,S3) and If someone changes the configuration of these services , I want to know immediately and want to pull the updated configuration. So how can i listen to these services to pull configuration immediately . I went through couple of options like aws SNS and all. But i don't want to implement this using any aws services for this.
I am using spring boot application. Any help is appreciated , Thanks in advance.
The two services you should be interested in are:
AWS CloudTrail:
AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs.
AWS Config:
AWS Config provides a detailed view of the configuration of AWS resources in your AWS account. This includes how the resources are related to one another and how they were configured in the past so that you can see how the configurations and relationships change over time.
You can also configure Amazon CloudWatch Events rules that trigger when a particular API call is made (eg when a user changes some configuration). This could trigger a notification, a Lambda function, etc.
See:
Creating a CloudWatch Events Rule That Triggers on an AWS API Call Using AWS CloudTrail - Amazon CloudWatch Events
How to monitor AWS account activity with Cloudtrail, Cloudwatch Events and Serverless
I'm trying to retrieve all open Alarms (alerts) from Amazon Monitoring tool: Cloudwatch, using REST API GET.
And while Azure Monitor has a very clear documentation that contains the Alerts Management endpoint:
https://management.azure.com/subscriptions/{SubscriptionID}/providers/Microsoft.AlertsManagement/alerts?includeContext=True&api-version=2018-05-05
On Amazon side, on the other hand. i got confused.
After reading Amazon Cloudwatch documentation.
I understood that i need to pull the data from Cloudwatch Alarms, and not from Cloudwatch Log or Cloudwatch Events.
But i didn't find a corresponding endpoint to use in order to retrieve all the open Alarms (alerts) in Cloudwatch using REST API Get, from the available AWS Service Points:
https://docs.aws.amazon.com/general/latest/gr/rande.html
Can someone shed some light on this please?
https://github.com/awsdocs/aws-doc-sdk-examples/blob/master/ruby/example_code/cloudwatch/cw-ruby-example-show-alarms.rb
here is a ruby example of describe alarms end point. You should be able to do the same using other programming languages as well.
I am planning to export logging from MuleSoft CloudHub to AWS CloudWatch.
I saw there is one AWS CloudWatch Connector in GitHub:
https://github.com/mulesoft-labs/mule-amazon-cloudwatch-connector/tree/master/mule-cloudwatch-connector
Is there any examples on how to implement this?
And which AWS CloudWatch features is supported in this connector?
I found other export logs method with examples, but not for AWS CloudWatch:
https://help.mulesoft.com/s/question/0D52T00004mXUALSA4/export-log-to-external-system
Thanks.
You could try to create a Mule application that reads logs from CloudHub using CloudHub's API and pushes them to CloudWatch. The readme for the connector doesn't seem to mention an operation to put logs into CloudWatch, but if CloudWatch has a REST API for it -every AWS product has it- you could use it in your application.
This KB article shows how to get the logs from CloudHub: https://help.mulesoft.com/s/article/How-to-get-whole-Application-logs-from-Cloudhub-through-API
Is there a way where we can check how many S3 List requests are happening on a bucket
The general solution on AWS to finding out what is going on at the API level is to use CloudTrail.
See Logging Amazon S3 API Calls by Using AWS CloudTrail.
This can be done by enabling the request metrics on S3 bucket by using CloudWatch service. The metrics are available at 1-minute intervals after some latency to process. These CloudWatch metrics are billed at the same rate as the Amazon CloudWatch Custom Metrics.
You can refer this Document to Configure Request Metrics for an S3 Bucket.
I've setup some aws cloudwatch agents to send logs to a cloudwatch log group and what I want to do now is get those into an elasticsearch cluster (on ec2). Ideally, what I want to do is write a lambda function that parses it and then sends it to elasticsearch, but also have the lambda function be triggered anytime a message is written to the log group. Is it possible to trigger lambda from a cloudwatch log group?
Subscribing AWS Lambda functions to CloudWatch Logs is documented here. Also, streaming CloudWatch Logs to the AWS ElasticSearch service is documented here.
Here you can find the documentation about adding a subscription to CloudWatch logs for Lambda.
For streaming to AWS ElasticSearch see this (as mentioned before).