I have a serverless architecture with a few AWS lambdas up and running sending logs to Cloudwatch right now.
Question: Is there any option to avoid send logs to Cloudwatch and redirect them to another tool?
Example: Catch all logs from stdout, avoid send it to Cloudwatch (of course, I don't need to pay for Cloudwatch storage) and send all these logs to another external tool such as New Relic, Splunk, etc?
Thank you very much for your help!
You can do the tricky via removing Cloudwatch logs permissions from the role of your lambda.
Related
I can see that in AWS alerting works with SNS where protocols are available as Email, https and SMS...Basically push mechanisms..
Is it possible to make these alerts pull-based? any ideas around how this could be tried?
You would need to create a solution for this, natively when an alert triggers it can be configured to go to either an SNS topic which has 1 or more subscriptions to notify.
Instead you would need to use the describe-alarms function to retrieve a list of alarms, this can be narrowed down to alarming only by using the --state-value parameter such as the example below
aws cloudwatch describe-alarms --state-value ALARM
Does this fit your needs https://awscli.amazonaws.com/v2/documentation/api/latest/reference/cloudwatch/describe-alarm-history.html ?
Please check aws cloudwatch there are several options for alarms.
I want to send logs data from my EC2 instance (Ubuntu) to an AWS eventbridge where I can then send them to multiple endpoints. e.g. if someone performs a root user operation on the server this is written to/var/log/auth.log, I would then like this change in the log to be sent to eventbridge where it can then be routed to other locations, how can achieve this?
cheers
N.B.
I have tried using the cloudwatch agent but I can't figure how to get the logs to eventbridge once they're in a log group, so if there is a way I can this that would also work.
I can't figure how to get the logs to eventbridge once they're in a log group, so if there is a way I can this that would also work.
Once your CloudWatch Agent writes relevant logs to CloudWatch logs, you can setup a subscription filter on your log group.
The filter would stream logs of interest (e.g. those that contain ssh) into a lambda function. How to set it up is shown in:
Example 2: Subscription Filters with AWS Lambda
The lambda, using events api, e.g. in boto3, could process the log stream, filter out messages, construct events and publish them to the event bridge.
Once logs are sent to CloudWatch using the CloudWatch client, we want to cleanup the disk. We have following 2 use cases.
We don't login into some of the servers. We don't need to keep the logs on disk. So cleanup can happen immediately after sending logs to Server.
On some servers, we want to keep logs for the last N days, after which these need to be deleted.
The Cloudwatch Logs Agent is compatible with logrotate, just make sure that you use one of the supported patterns.
See: CloudWatch Logs Agent Reference - Amazon CloudWatch Logs
With CloudWatch you can monitor applications running on AWS. Is it also possible to monitor an external service?
For example, I have a REST API and I want to get notified once that API is not accessible anymore. Does AWS offer you a monitoring tool for that purpose?
Not Cloudwatch just by itself, but you can use a combination of Cloudwatch and Lambdas to do what you're asking. You can use cloudwatch events to run lambdas on a schedule, something like once every 5 mins.
CloudwatchEvents -> HealthCheck Lambda -> Cloudwatch Custom Metrics
Your lambda can then ping the API you're monitoring the health of, and either send its status to cloudwatch as a custom metric; or potentially if your lambda throws an error when the API fails, the lambda error metric which is already in cloudwatch becomes your API failure metric
Once the metric exists in cloudwatch, either as a custom metric or the lambda metric by proxy, you're able to do usual cloudwatch things like alarms and notifications.
Now there is a simple way to monitor external resources - CloudWatch Synthetics. Just create a canary to regularly monitor a website, API or even validate a multi-step UI flow.
Read more in the docs: CloudWatch > Using Synthetic Monitoring
Amazon CloudWatch supports custom metrics generated by your applications and services that you do not run on AWS. In this way, CloudWatch can be an integrated storage and aggregation point, allowing you to monitor all of the metrics that you collect, and track on a single platform.
There might be more than one way to reach your goal by using the AWS CLI, an API/SDK, or the CloudWatch collectd plugin etc. I'd recommend you take a look at these links for more details: link-1, link-2, link-3, link-4
I would like to create a cloudwatch alarm for the sns metric NumberOfMessagesPublished on a new sns topic. If I attempt to do this before I trigger the topic (and there is no data for it) then the option to create an alarm does not exist in the web console.
How can I do this with the web console? Is using a cli tool the only option?
I noticed that too -- you cannot select a metric until the data has been sent to CloudWatch.
I suspect that CloudWatch actually doesn't have any visibility into metrics until they are sent from the originating service into CloudWatch. Thus, it can't populate the console because it doesn't know what data will be sent. That would be a nice, loosely-coupled design.
My experiments show that it is possible create an alarm via the AWS Command-Line Interface (CLI) on metrics that do not exist -- even within the EC2 namespace. That's the way to go!
I suggest you to use Cloudformation. It is the best way to create resources stack in AWS and overall you can easily experiment with it and delete everything when you are done. Here there is a sample application that uses Cloudformation and Cloudwatch https://github.com/awslabs/cloudwatch-dashboards-cloudformation-sample
The manual solution I went with was to create the sns topic and then use the Publish to topic option in the sns menu to send out a test message to my subscribers. This creates a set of notifications without the need to set off the cloudwatch rule. After this the option appears in the cloudwatch alarm creation menu.
The other CLI options would be better for automation.
Yes, it is possible. The AWS Official Documentation answers your question. Just to quote the relevant lines:
You can create an alarm for a custom metric before you've created that custom metric. For the alarm to be valid, you must include all of the dimensions for the custom metric in addition to the metric namespace and metric name in the alarm definition.