I have an on-premise app deployed in an Application Server (e.g. Tomcat) and it generates its own log file. If I decide to migrate this to an AWS EC2, including the Application Server, is it possible to port my application logs in Cloudwatch instead? or is Cloudwatch only capable of logging the runtime logs in my application server? is it a lot of work to do this or is this even possible?
Kind of confuse on Cloudwatch. Seems it can do multiple things but is it really right to make it do that? Its only supposed to log metrics right, so it can alert whatever or whoever needs to be alerted.
If you have already developed application that produces its own log files, you can use CloudWatch Logs Agent to ingest the logs into CloudWatch Logs:
After installation is complete, logs automatically flow from the instance to the log stream you create while installing the agent. The agent confirms that it has started and it stays running until you disable it.
The metrics, such as RAM usage, disk space, can also be monitored and pushed to CloudWatch through the agent.
In both cases, logs and metrics, you can setup CloudWatch Alarms to automatically detect anomalies and notify you, or perform other actions, when they are detected. For logs, this is done through metric filters:
You can search and filter the log data coming into CloudWatch Logs by creating one or more metric filters. Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs. CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics that you can graph or set an alarm on.
update
You can also have your application to inject logs directly to CloudWatch logs using AWS SDK. For example, in python, you can use put_log_events.
Related
I am trying to send custom metric data of my application to CloudWatch using Cloudwatch Agent.
I am able to successfully send the data to CloudWatch using aws cloudwatch put-metric-data command with AWS CLI, but instead i would like to use the cloudwatch agent to receive my application metrics (connection count , queue count etc) and send that data to cloudwatch via Cloudwatch agent. Please help.
At first, you need to check what metrics CWAgent supports and collects.
Reference: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html
From my point of view, CWAgent mostly collects OS metrics that it can read directly from your machine instead of metrics such as connection-count from your application.
The way you did using the put-metric-data is a good direction, you can customize it with your own code.
But recently, AWS did mention about OpenTelemetry which supports application metrics and integrates with CloudWatch Agent. I believe this way is the one you are looking for.
Reference:
https://aws-otel.github.io/docs/introduction
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-open-telemetry.html
Was studying about Amazon web services and fundamentals when came across these 2 concepts:
Amazon CloudWatch
Amazon CloudWatch Events
Even while going through the official documents on AWS, I couldn't find a difference between the two even when Amazon mentions that they are different. Excerpt is:
CloudWatch provides you with data and actionable insights to monitor
your applications, respond to system-wide performance changes,
optimize resource utilization, and get a unified view of operational
health. CloudWatch collects monitoring and operational data in the
form of logs, metrics, and events, providing you with a unified view
of AWS resources, applications, and services that run on AWS and
on-premises servers. You can use CloudWatch to detect anomalous behavior in your environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep your applications
running smoothly.
Documentation of AWS CloudWatch
Amazon CloudWatch Events delivers a near real-time stream of system
events that describe changes in Amazon Web Services (AWS) resources.
Using simple rules that you can quickly set up, you can match events
and route them to one or more target functions or streams. CloudWatch
Events becomes aware of operational changes as they occur. CloudWatch
Events responds to these operational changes and takes corrective
action as necessary, by sending messages to respond to the
environment, activating functions, making changes, and capturing
state information.
Documentation of AWS CloudWatch Events
CloudWatch
CloudWatch is a monitoring service for your AWS resources. You can log your log files. By default the resources created within AWS logs in CloudWatch(CW). You can monitor the performance of resources too for example you can monitor how is the CPU utilisation of your EC2 instances. You can set Alarms for your resources
threshold and get an SNS alert on that. For example you can create an Alarm for your DynamoDB if Write capacity is exceeding. You can set an alarm for your billing too. So basically CW is used as a Monitoring solution.
CloudWatch Events
CW Events is also the part of CloudWatch. CloudWatch Events is helpful when you want to schedule something. Say you to want run your lambda every other day, you can create a Rule for that or you want to trigger your lambda by Event Pattern. There are bunch of services supported by CloudWatch Events, you can use anyone of them as your target not just Lambda. Event Buses is used to send your events to multiple accounts also. For example if you have a CICD account and every month you bake new AMI there, to notify all accounts you can use Event Buses, after getting the event from Event Buses other accounts can trigger some important tasks.
Once logs are sent to CloudWatch using the CloudWatch client, we want to cleanup the disk. We have following 2 use cases.
We don't login into some of the servers. We don't need to keep the logs on disk. So cleanup can happen immediately after sending logs to Server.
On some servers, we want to keep logs for the last N days, after which these need to be deleted.
The Cloudwatch Logs Agent is compatible with logrotate, just make sure that you use one of the supported patterns.
See: CloudWatch Logs Agent Reference - Amazon CloudWatch Logs
I'm looking for a way to filter and transform logs before they are sent to AWS by CloudWatch Logs Agent.
Let's assume I'm using a third party software which generates logs on an EC2 instance. The problems that I'm trying to solve are:
Not sending all the logs generated by the third party software, there are too many logs generated and only a few of them are valuable
The logs are not in JSON and I'm interested in transforming them into one before sending them to CloudWatch (for search purposes)
Is this possible and how?
With CloudWatch you can monitor applications running on AWS. Is it also possible to monitor an external service?
For example, I have a REST API and I want to get notified once that API is not accessible anymore. Does AWS offer you a monitoring tool for that purpose?
Not Cloudwatch just by itself, but you can use a combination of Cloudwatch and Lambdas to do what you're asking. You can use cloudwatch events to run lambdas on a schedule, something like once every 5 mins.
CloudwatchEvents -> HealthCheck Lambda -> Cloudwatch Custom Metrics
Your lambda can then ping the API you're monitoring the health of, and either send its status to cloudwatch as a custom metric; or potentially if your lambda throws an error when the API fails, the lambda error metric which is already in cloudwatch becomes your API failure metric
Once the metric exists in cloudwatch, either as a custom metric or the lambda metric by proxy, you're able to do usual cloudwatch things like alarms and notifications.
Now there is a simple way to monitor external resources - CloudWatch Synthetics. Just create a canary to regularly monitor a website, API or even validate a multi-step UI flow.
Read more in the docs: CloudWatch > Using Synthetic Monitoring
Amazon CloudWatch supports custom metrics generated by your applications and services that you do not run on AWS. In this way, CloudWatch can be an integrated storage and aggregation point, allowing you to monitor all of the metrics that you collect, and track on a single platform.
There might be more than one way to reach your goal by using the AWS CLI, an API/SDK, or the CloudWatch collectd plugin etc. I'd recommend you take a look at these links for more details: link-1, link-2, link-3, link-4