I'm looking for a way to establish custom metrics over StatsD protocol for Amazon Elastic Container Service. I've found a documentation on how to establish Amazon CloudWatch Agent on EC2. It works well. However I'm failing to find a correct configuration for Dockerfile. Quite probably some set of custom IAM permissions will also be required there.
Is it possible to have Docker containers working from AWS ECS with custom metrics using StatsD reporting to AWS CloudWatch?
Rather than building your own container, you can use the one provided by Amazon. This article explains how, including a link to an example daemon service task configuration.
Related
The question:
Does anyone now how to configure cwagent to make it only create service discovery files for prometheus?
Why:
I am trying to find an approach to monitor my amazon ECS services with prometheus. So I'd prefer to do this via some sort of "service discovery". Unfortunatelly, prometheus cannot do this from scratch. However, I found this article: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup-autodiscovery-ecs.html
Well, cloudwatch agent has the worst documentation I ever seen.
Does anyone now how exactly should I configure cwagent to make it only create service discovery files for prometheus?
Ideally, I want to run it as a side container in a kubernetes pod with prometheus. The only thing cwagent should do is to hit API (in different accounts) via roles and then create sd file for prometheus to scrape.
So my understanding is that when I deploy a new service to ECS using AWS Copilot, logs are forwarded to CloudWatch automatically by default.
Copilot creates log groups for each service, I can see that in CloudWatch Logs.
However, according to AWS docs, logging can be also implemented using Copilot sidecars and AWS FireLens, which uses FluentD or FluentBit to collect logs, and then it forwards stuff CloudWatch.
I don't understand why is this necessary. I mean, why to create a sidecar for logging to CloudWatch, when logging seems to work automatically, without any sidecar.
https://aws.github.io/copilot-cli/docs/developing/sidecars/
There is an example here for logging via FireLens. What's the benefit of doing this over the logging mechanism that just works by default?
Thanks in advance!
AWS Copilot builds an image for you application that already has an agent configured to forward logs to CloudWatch, however you might want to deploy other images to ECS that don't have this agent installed. For example, suppose you wanted to deploy an nginx container to ECS, you might choose to use a sidecar to forward logs instead of customizing the nginx image.
How can I see the used resources per ECS container? Now im able to see the resources from the whole cluster but I want to see it per container.
I know I can see it via docker stats when I SSH to my cluster, but I prefer Cloudwatch for it.
ECS doesn't provide per container metrics on CloudWatch.
You could install the CloudWatch Agent in your containers via the Docker file. You can configure CloudWatch agent to publish a bunch of host level metrics.
However, note that all of these metrics would be Custom Metrics and would cost a lot.
In spring boot logs by default go to stdout. that's nice standard - less config, no directory configuration etc. but I want to build a docker image and run it on aws.
how can i get all the logs from dockerized spring-boot stdout? does cloudwatch support it? is there a simple solution or do i have to switch to logging to a file, doing docker volumes mount etc?
It depends how your architecture looks like and what do you want to do with logs.
Nowadays you can use a myriad of tools in order to read logs. You can use AWS Cloudwatch Logs and through this you can configure alertings through CloudWatch itself.
In order to use it, you can configure your slf4j backend.
<appender name="cloud-watch" class="io.github.dibog.AwsLogAppender">
<awsConfig>
<credentials>
<accessKeyId></accessKeyId>
<secretAccessKey></secretAccessKey>
</credentials>
<region></region>
<clientConfig class="com.amazonaws.ClientConfiguration">
<proxyHost></proxyHost>
<proxyPort></proxyPort>
</clientConfig>
</awsConfig>
<createLogGroup>false</createLogGroup>
<queueLength>100</queueLength>
<groupName>group-name</groupName>
<streamName>stream-name</streamName>
<dateFormat>yyyyMMdd_HHmm</dateFormat>
<layout>
<pattern>[%thread] %-5level %logger{35} - %msg %n</pattern>
</layout>
Obviously it depends from your architecture: if you have for example filebeat, you can configure filebeat to use cloudwatch.
If you use ecs-optimized AMI for the ec2 instances (it should be at least 1.9.0), you can also use the aws logdriver for your containers:
1. Before launch the ecs agent, you must change /etc/ecs/ecs.config and adjust ECS_AVAILABLE_LOGGING_DRIVERS with: ["json-file","awslogs"]
2. Activate the auto-configuration feature to create log group for ecs tasks (you can also create the groups manually, but I think you want here more automation)
For more informations about aws logdriver, you can look on aws documentation:
AWS Logs Driver
Install ECS Agent
I have an AWS account, and have several EC2 servers and an ElasticSearch domain set up to take the syslogs from these servers. However, in Cloudwatch and when investigating a specific server instance in the EC2 control panel, I see specific metrics and graphs for things like CPU, memory load, storage use, etc. Is there some way I can pipe this information into my ElasticSearch as well?
Set up Logstash and use this plugin https://github.com/EagerELK/logstash-input-cloudwatch
Or go the other way and use AWS Logs agent to put your syslogs into Cloudwatch and stop using ElasticSearch