I am new to AWS and was experimenting with some of its services. I managed to create a python application running inside an EC2 instance. The application creates a log file with the analysis data.
I want to connect this log file with AWS's Elasticsearch and Kibana service to begin running analytics on it.
Can someone point me to the best way of streaming my EC2 app's logs to AWS elasticsearch service.
You have multiple options to deal with this problem. in case of AWS
Install aws cloud watch log agent
Start log-agent with log file
Stream cloud watch log to lambda
Lamda will push logs to ELK
But I will go with the below approach as it will not need Lambda and log-group and the logs will send to ELK directly.
Filebeat
Logagent (node based pacakge)
Filebeat is part of the Elastic Stack, meaning it works seamlessly with Logstash, Elasticsearch, and Kibana. Whether you want to
transform or enrich your logs and files with Logstash, fiddle with
some analytics in Elasticsearch, or build and share dashboards in
Kibana, Filebeat makes it easy to ship your data to where it matters
most.
All you need to sepcify application log files.
paths:
- /app/log/*.log
- /app/log/*/*.log
Logagent is a modern, open-source, light-weight log shipper. It is like Filebeat and Logstash in one, without the JVM memory > footprint. It comes with out of the box and extensible log parsing, > on-disk buffering, secure transport, and bulk indexing to > Elasticsearch, Sematext Logs, and other destinations. Its low memory > footprint and low CPU overhead make it suitable for deploying on edge > nodes and devices, while its ability to parse and structure logs makes > it a great Logstash alternative.
sudo npm i -g #sematext/logagent
shipping-data-to-aws-elasticsearch-with-logagent
Related
We have couple of application running on AWS. Currently we are redirecting all our logs to single bucket. However for ease of access to users, I am thinking to install ELK Stack on EC2 instance.
Would want to check if there is alternate way available where I don't have to maintain this stack.
Scaling won't be an issue, as this is only for logs generated through application running on AWS, so not ingestion or processing is required. mostly log4j logs.
You can go for either the managed Elasticsearch available in AWS or setup your own in an EC2 instance
It usually comes down to the price involved and the amount of time you have in hand in setting up and maintaining your own setup
With your own setup, you can do a lot more configurations than that provided by the managed service and also helps in reducing the cost
You can find more info on this blog
We have a setup with multiple containers running NodeJS services(node:11-alpine docker image) deployed in AWS ECS Fargate.
We already have a running ElasticSearch instance collecting logs from non-Fargate application. I would like to pass the logs from the Fargate containers into this ElasticSearch instance, but I have a hard time to figure out what is the best approach.
1)
It seem one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?
2)
I was hoping i could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?
3)
Should I install something like FileBeat on each container and let that send the logs?
Any help is appreciated.
It seems one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?
If you're looking for an AWS-based managed solution, that is one of the ways. You don't really need to write a Lambda function, AWS does it for you. Although, you have to bear the cost of Lambda and CloudWatch.
There is another solution that is recommended by AWS and that is the use of fluent-bit as a sidecar container to export logs directly to Elasticsearch/OpenSearch from other containers running within a service. Using this solution, you save money by not using AWS CloudWatch. This solution also provides better results with regard to the loss of logs upon failure.
I was hoping I could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?
Yes, that is possible if you run that container along with the other container.
Should I install something like FileBeat on each container and let that send the logs?
You can use Fluent Bit, Filebeat, Fluentd, Functionbeat, or Logstash as you like.
Note: If you're thinking of running your own logs exporter container like Logstash, Fluent Bit, etc, don't enable CloudWatch logging to save money as you're not going to use that.
Is there any way to stream/push docker app logs to S3 bucket?
I know following 2 ways
Configure cloud watch logs/stream - All logs (both info & Error logs) are getting merged in this approach
Configure graylogs2 to push every log message and collect and then push to S3 bucket - Need to maintain graylogs2 app.
I am looking for any easy way to push docker app/error logs S3 Bucket
Thanks
A possible solution, though it's hard to tell for your case, is to run logstash in a separate container, and have your app direct logs to logstash. Since Logstash’s logging framework is based on Log4j 2 framework, it will likely be familiar to you. A plugin already exists for logstash to push to S3 on your behalf.
You can configure your existing log4j2 to emit to a port that logstash is running on.
If even this is considered too much maintenance for you, your best solution is probably just setting up a cron to run rsync.
I have a classic scala app, it produces three different logs in the location
/var/log/myapp/log1/mylog.log
/var/log/myapp/log2/another.log
/var/log/myapp/log3/anotherone.log
I containerized the app and working fine, I can get those logs by docker volume mount.
Now the app/container will be deployed in AWS ECS with auto scaling group. in this case multiple container may run on one single ecs host.
I would like to use cloud watch to monitor my application logs.
One solution could be put aws log agent inside my application container.
Is there any better way to get those application logs from container to cloudwatch log.
help is very much appreciated.
When using docker, the recommended approach is to not log to files, but to send logs to stdout and stderr. Doing so prevents the logs from being written to the container's filesystem, and (depending on the logging driver in use), allows you to view the logs using the docker logs / docker container logs subcommand.
Many applications have a configuration option to log to stdout/stderr, but if that's not an option, you can create a symlink to redirect output; for example, the official NGINX image on Docker Hub uses this approach.
Docker supports logging drivers, which allow you to send logging to (among others) AWS cloud watch. After you modified your image to make it log to stdout/stderr, your can configure the AWS logging driver.
More information about logging in Docker can be found in the "logging" section in the documentation
You don't need log agent if you can change the code.
You can directly publish Custom Metric Data into ColudWatch like this page said: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-cloudwatch-publish-custom-metrics.html
I am programming a Jersey service on Tomcat via EBS with LoadBalancer. I am finding getting the EC2's S3 catalina files very cumbersome. Currently I need to determine the EC2 instance(s) then work my way to each of the S3 locations, download the files, then I can diagnose.
The snapshot doesn't help due to the amount of requests that come in, it doesn't hold enough info and by the time I get the snapshot, it has "rolled" off the snapshot.
Two questions:
1) Is there an easier approach to logs files via AWS? (Increase time before rotation which I don't believe is supported as of now, scripts, etc)
2) Is there any software or scripts to access all the logs under load balancer? I am basically wanting to say "give me all logs for this EBS" and have it get all logs for that day under all servers for that load balancer (up or down)". The clincher is down. Problem becomes more complex when the load balancer takes down an instance right when the issue occurs.
Thanks!
As an immediate solution to your problem you can follow the approach suggested in this answer. Essentially you can modify the logrotate configuration to rotate for a bigger log size using ebextensions.
Then snapshot logs should work for you.
Let me know if you need more clarifications on this approach.
AWS has released CloudWatch Logs just last week, which enables you to to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files:
You can send your existing system, application, and custom log files to CloudWatch Logs and monitor these logs in near real-time. [...] you can store your logs using highly durable, low-cost storage for later access.
See the introductory blog post Store and Monitor OS & Application Log Files with Amazon CloudWatch for an illustrated walk through, which touches on using Elastic Beanstalk and CloudWatch Logs already - this is further detailed in Using AWS Elastic Beanstalk with Amazon CloudWatch Logs.