I'm trying to use AWS Firelens for our FARGATE container logs. I'm are doing so by using the side-car method. Everything is working as expected. However I have one concern, i.e. if the logs in the source container is rotational. In other words, should I be worried about the disk space in our containers to run out because of the logs?
The reason why we ask this is because after enabling Cloudwatch Agent in the source container, I see that there is no change in the disk_used_percent at all. This is why I couldn't come to a conclusion. Can someone assist me with some information on the aforementioned question.
Related
I need to deploy an application in AWS using ECS Fargate. This application has multiple services and a docker-compose file. I see there are two main ways to do this:
Using Docker's Context ECS cli, the official docs I found: Docker doc and AWS doc.
Using Amazon's ECS cli as described here.
I am trying to understand the following but didn't find any comparison on the web:
Which are the advantages/disadvantages of each way?
Can the same result be achieved with both options, or is there something one can do that the other can't?
What should I take in consideration when I choose one?
Thanks,
So I've been trying this this past week, both approaches, and here's what I have found.
ecs-cli and docker support different sets of tags for nontrivial features, even things such as how much CPU and memory your container needs.
For example docker wants the deploy config in a deploy tag under the service. See https://docs.docker.com/compose/compose-file/deploy/#memory
However ECS-CLI wants things in a acs-params.yml. There are almost no examples out there other than trivial ones though, and the ones published don't actually work with the current tooling. What they publish doesn't work.
Docker ECS integration works and handles tons of details for you, including VPC, subnet creation, LBs, security groups, everything. This is an amazing part.
ecs-cli offers tons more options than Docker CLI but you need to do a lot more work yourself, manual security group config, etc.
I was never able to get ecs-cli to really work. It kept choking on cpu config, and what was written in AWS docs did not actually work\
docker compose logs doesn't work
Overall neither CLI seems to be in production shape, but docker one seems to be far ahead of where the ecs-cli is IMO.
We have a setup with multiple containers running NodeJS services(node:11-alpine docker image) deployed in AWS ECS Fargate.
We already have a running ElasticSearch instance collecting logs from non-Fargate application. I would like to pass the logs from the Fargate containers into this ElasticSearch instance, but I have a hard time to figure out what is the best approach.
1)
It seem one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?
2)
I was hoping i could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?
3)
Should I install something like FileBeat on each container and let that send the logs?
Any help is appreciated.
It seems one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?
If you're looking for an AWS-based managed solution, that is one of the ways. You don't really need to write a Lambda function, AWS does it for you. Although, you have to bear the cost of Lambda and CloudWatch.
There is another solution that is recommended by AWS and that is the use of fluent-bit as a sidecar container to export logs directly to Elasticsearch/OpenSearch from other containers running within a service. Using this solution, you save money by not using AWS CloudWatch. This solution also provides better results with regard to the loss of logs upon failure.
I was hoping I could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?
Yes, that is possible if you run that container along with the other container.
Should I install something like FileBeat on each container and let that send the logs?
You can use Fluent Bit, Filebeat, Fluentd, Functionbeat, or Logstash as you like.
Note: If you're thinking of running your own logs exporter container like Logstash, Fluent Bit, etc, don't enable CloudWatch logging to save money as you're not going to use that.
I need to set up a monitoring into an aws account to ping certain servers from outside the account, create a custom cloudwatch metric with the package loss and i need to deploy the solution without any EC2 instance.
My first choice was lambda, but it seems that lambda does not allow pinging from it.
Second choice was a container, as FARGATE has the ability to execute containers without any EC2 instance. The thing is im able to run the task definition and i see the task in RUNNNING state in the cluster, but the cloudwatch metric is never received.
If I use the normal EC2 cluster, the container works perfectly, so i assume I have some error within the configuration, but I'm lost why. I have added admin rights to the ECS Task Execution Role and opened all ports in the sec group.
I have tried public/private subnets with no success.
Anyone could please help me?
Here you can find that the task is certainly RUNNING, however the app dont generate any further action
So i solved the problem. There was problem inside the container. It seems Fargate doesn't like cron, so i removed my cron schedule from the container and used a cloudwatch event rule instead and it works perfectly
We can add tags to EC2 instances to help us better track billing usages and to manage instances.
Is there a way to achieve when deploying containers in ECS? I would like the running container to have the ability to know what tag it currently have attached.
It really depends on what you're ultimately trying to visualize after the fact. I'll share a few off-the-cuff thoughts below, and maybe you can extrapolate on these to build something that satisfies your needs.
As you probably are aware, ECS Tasks themselves don't support the notion of tags, however there are some workarounds that you could consider. For example, depending on how you're logging your application's behavior (eg. batching logs to CloudWatch Logs), you could create a Log Stream name, for each ECS Task, that contains a delimited array of tags.
As part of a POC I was building recently, I used the auto-generated computer name to dynamically create CloudWatch Log Stream names. You could easily append or prepend the tag data that you embed in your container images, and then query the tag information from the CloudWatch Log Streams later on.
Another option would be to simply log a metric to CloudWatch Metrics, based on the number of ECS Tasks running off of each unique Task Definition in ECR.
You could build a very simple Lambda function that queries your ECS Tasks, on each cluster, and writes the Task count, for each unique Task Definition, to CloudWatch Metrics on a per-minute basis. CloudWatch Event Rules allow you to trigger Lambda functions on a cron schedule, so you can customize the period to your liking.
You can use this metric data to help drive scaling decisions about the ECS Cluster, the Services and Tasks running on it, and the underlying EC2 compute instances that support the ECS Cluster.
Hope this helps.
Just found this while trying to work out the current situation. For future searchers: I believe tagging was added some time after this question, in late 2018.
I've not yet worked out if you can set this up in the Console or if it's a feature of the API only, but e.g. the Terraform AWS provider now lets you set service or task definition tags to 'bubble through' to tasks – including Fargate ones – via propagate_tags.
I've just enabled this and it works but forces a new ECS service – I guess this is related to it not being obviously editable in the Console UI.
I am programming a Jersey service on Tomcat via EBS with LoadBalancer. I am finding getting the EC2's S3 catalina files very cumbersome. Currently I need to determine the EC2 instance(s) then work my way to each of the S3 locations, download the files, then I can diagnose.
The snapshot doesn't help due to the amount of requests that come in, it doesn't hold enough info and by the time I get the snapshot, it has "rolled" off the snapshot.
Two questions:
1) Is there an easier approach to logs files via AWS? (Increase time before rotation which I don't believe is supported as of now, scripts, etc)
2) Is there any software or scripts to access all the logs under load balancer? I am basically wanting to say "give me all logs for this EBS" and have it get all logs for that day under all servers for that load balancer (up or down)". The clincher is down. Problem becomes more complex when the load balancer takes down an instance right when the issue occurs.
Thanks!
As an immediate solution to your problem you can follow the approach suggested in this answer. Essentially you can modify the logrotate configuration to rotate for a bigger log size using ebextensions.
Then snapshot logs should work for you.
Let me know if you need more clarifications on this approach.
AWS has released CloudWatch Logs just last week, which enables you to to monitor and troubleshoot your systems and applications using your existing system, application, and custom log files:
You can send your existing system, application, and custom log files to CloudWatch Logs and monitor these logs in near real-time. [...] you can store your logs using highly durable, low-cost storage for later access.
See the introductory blog post Store and Monitor OS & Application Log Files with Amazon CloudWatch for an illustrated walk through, which touches on using Elastic Beanstalk and CloudWatch Logs already - this is further detailed in Using AWS Elastic Beanstalk with Amazon CloudWatch Logs.