We are using log-entries as driver on AWS ECS service for sending logs to our logentries account. We have configured AWS ECS service with required parameters like logentries-token but it's observed that after certain amount of time certain containers are not able to send logs to logentries.
Appreciate your help in advance, I am unable to find proper documentation for this on both logenries as well as AWS.
Thanks,
We had the same issue, so I started digging deeper than usual.
Actual driver implementation is quite simple.
The dragon is a dependency that does the socket, tls handling
There is a open issue and a PR to solve a very similar issue.
The PR is stale and I don't see chance for it to land, so I move away from logentries and recommend doing the same. Probably cloudwatch will be better.
Related
first time asker.
So I've been trying to implement AWS Cloud Watch to monitor Disk Usage on an EC2 instance running EC2 Linux. I'm interesting in doing this just using the CW Agent and I've installed it according to the how-to found here. The install runs fine and I've made sure I've created an IAM Role for the instance as is described here. Unfortunately whenever I run the amazon-cloudwatch-agent.service it only sends log files and not the custom used_percent measurement specified. I receive this error when I tail the logs.
2021-06-18T15:41:37Z E! WriteToCloudWatch failure, err: RequestError: send request failed
caused by: Post "https://monitoring.us-west-2.amazonaws.com/": dial tcp 172.17.1.25:443: i/o timeout
I've done my best googlefu but gotten nowhere thus far. If you've got any advice it would be appreciated.
Thank you
Belated answer to my own question. I had to create a security group that would accept traffic from that same security group!
Having the same issue, it definitely wasn't a network restriction as I was still able to telnet to the monitoring endpoint.
From AWS docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/create-iam-roles-for-cloudwatch-agent.html
One role or user enables CloudWatch agent to be installed on a server
and send metrics to CloudWatch. The other role or user is needed to
store your CloudWatch agent configuration in Systems Manager Parameter
Store. Parameter Store enables multiple servers to use one CloudWatch
agent configuration.
If you're using the default cloudwatchagent configuration wizard, you may require extra policy CloudWatchAgentAdminRole in your role for the agent to connect to the monitoring service.
Yesterday our infrastructure started throwing lots of connection errors. We started debugging and the more we looked, the more perplexing the issue appeared to be; until someone noticed the bell icon (Alerts) on the AWS page had an orange dot on it.
Behold! there were lots of AWS operational issues in our availability region that AWS were fixing.
To avoid this situation in the future I wanted to subscribe to these 'Alert' so we get an email notification.
Does anyone know how to set up an email alert for AWS operational issues in the specified region?
Much to my astonishment, there was no obvious way to set this up.
Easiest way is to subscribe RSS feed on AWS Service Health Dashboard.
If you want customized stuffs, you can checkout AWS Personal Health Dashbaord. It shows your AWS services and whether they are experiencing issues.
This AWS documentation provides a really comprehensive guide on how to setup alerts. Checkout this aws-health-tools github repository for fully functional examples.
I run a spring boot application in AWS with Docker. Sometimes Amazon have to restart a hardware. Then Environment Health of instance in Beanstalk goes Degraded, Warning, and restarts.
I do want my app logs from the last 7 days but it was restarted due to unforeseen AWS hardware issues so I lost my information. How can I avoid it and make AWS to save all my logs even after restart?
It is true that archiving logs to S3 would work for the most part but you may want to consider installing and configuring the CloudWatch Logs agent - http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
This will stream logs directly to CloudWatch and save them upon termination. You also could consider numerous other solutions for this such as Sumo Logic, ELK, Splunk, etc.
You should always build solutions so as to be ready even when hardware crashes. One possible solution could be that while rotating log files send them to S3 bucket. You can create a cron-job to do this.
I tried searching for this topic on Google and after many failed attempts I decided to post this as questions here.
What I want to achieve: Monitoring my aws limits using Nagios.
As I have understood aws cli can be used to get the limits of only few aws services, for more in depth cost management and service limit management one has to opt for trusted advisor. Unfortunately it's quite expensive.
So I was wondering if there's a much simpler way with Nagios in which I could get notified if any of the aws services for my account is hitting a limit?
What kind of service limit notification strategy is used by organizations(That can't afford to buy a subscription of trusted advisor) that use Aws?
You're right: only few services can show their limit (and current usage) through CLI or API. I don't like it either :) We have three options here:
Create a parser that grabs information from AWS Console (there is an example code here: https://forrestbrazeal.com/2015/07/20/adventures-in-aws-automating-service-limit-checks/).
Buy Trusted Advisor (btw, you can get a Trusted Advisor report with API call).
Try using awslimitchecker. Cause someone already tried to solve this problem.
https://awslimitchecker.readthedocs.io/en/latest/
I am programming a Jersey service on Tomcat via EBS with LoadBalancer. I am finding getting the EC2's S3 catalina files very cumbersome. Currently I need to determine the EC2 instance(s) then work my way to each of the S3 locations, download the files, then I can diagnose.
The snapshot doesn't help due to the amount of requests that come in, it doesn't hold enough info and by the time I get the snapshot, it has "rolled" off the snapshot.
Two questions:
1) Is there an easier approach to logs files via AWS? (Increase time before rotation which I don't believe is supported as of now, scripts, etc)
2) Is there any software or scripts to access all the logs under load balancer? I am basically wanting to say "give me all logs for this EBS" and have it get all logs for that day under all servers for that load balancer (up or down)". The clincher is down. Problem becomes more complex when the load balancer takes down an instance right when the issue occurs.
Thanks!
As an immediate solution to your problem you can follow the approach suggested in this answer. Essentially you can modify the logrotate configuration to rotate for a bigger log size using ebextensions.
Then snapshot logs should work for you.
Let me know if you need more clarifications on this approach.
AWS has released CloudWatch Logs just last week, which enables you to to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files:
You can send your existing system, application, and custom log files to CloudWatch Logs and monitor these logs in near real-time. [...] you can store your logs using highly durable, low-cost storage for later access.
See the introductory blog post Store and Monitor OS & Application Log Files with Amazon CloudWatch for an illustrated walk through, which touches on using Elastic Beanstalk and CloudWatch Logs already - this is further detailed in Using AWS Elastic Beanstalk with Amazon CloudWatch Logs.