I have a monolith that is currently being transferred to aws (using ecs/fargate), and will later be broken up into microservices. It uses an amazon linux 1 image provisioned with apache, php, and all of my production website data. Currently, it sends logs to several files in /etc/httpd/logs and /var/www/vhosts/logs.
supposedly there's some stuff I can do in ecs task Definitions with log configurations and volumes, but I haven't been able to find anything that explains the details on how to do so.
In case of a container, I will never suggest writing logs to file, better to write logs container stdout and stderr.
Another interesting thing, how you will deal with logfile if you moved to fargate? so do not write logs to file and do not treat the container like instance machine.
The beauty of AWS log driver is, it pushes to logs to Cloud watch logs and from cloud watch it also super easy to push these to ELK.
Go for AWS log driver, design your entrypoint in way that it writes logs to stdout and stderr in the container. normally this is super easy when you run the process in the foreground it automatically writes log to container stdout.
Just add this line in your task definition and add cloud watch role.
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-wordpress",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "awslogs-example"
}
}
or
once it configured you will see the logs
Using the awslogs Log Driver
You can configure the containers in your tasks to send log information
to CloudWatch Logs. If you are using the Fargate launch type for your
tasks, this allows you to view the logs from your containers. If you
are using the EC2 launch type, this enables you to view different logs
from your containers in one convenient location, and it prevents your
container logs from taking up disk space on your container instances.
This topic helps you get started using the awslogs log driver in your
task definitions.
Note
The type of information that is logged by the containers in your task
depends mostly on their ENTRYPOINT command. By default, the logs that
are captured show the command output that you would normally see in an
interactive terminal if you ran the container locally, which are the
STDOUT and STDERR I/O streams. The awslogs log driver simply passes
these logs from Docker to CloudWatch Logs. For more information on how
Docker logs are processed, including alternative ways to capture
different file data or streams, see View logs for a container or
service in the Docker documentation.
aws-ecs-log-driver
Related
I need to collect the custom metrics from my ECS instances and from the documentation
These are steps I need to follow
Install aws cloudwatch agent
Install collectd daemon
Configure cloudwatch agent to get the metrics from collectd daemon
It seems I can
Dockerize the cloudwatch agent (which seems to be already done, but with lack of documentation)
Dockerize collectd daemon
Why to dockerize or not use awslogs driver for collecting metrics ?
Currently we already have some services running as docker instances managed by amazon ecs and configured to use awslog driver to send logs to amazon cloudwatch logs.
But in order to collect more custom metrics from the services e.g number of requests per particular user from service a the only solution which aws suggested is to use colletd with curl plugin along with cloud watch agent.
As due to some scaling issues, instead of running cloudwatch agent and collectd in a instance. I want to run those as containers.
Question:
Is there any way to run cloudwatch agent in docker container which can read the metrics from collectd daemon which runs in different container but on the same machine ?
you do not need to run cloudwatch agent in your container, do not forget the rule of thumb, one process per container.
All you need to push application logs to stdout or stderr of the container and docker daemon will take care of it.
Important Configuration:
All you need to set log driver to AWS log driver in the task definition.
Amazon CloudWatch Logs logging driver
The awslogs logging driver sends container logs to Amazon CloudWatch
Logs. Log entries can be retrieved through the AWS Management Console
or the AWS SDKs and Command Line Tools.
Specifying a Log Configuration in your Task Definition
Before your containers can send logs to CloudWatch, you must
specify the awslogs log driver for containers in your task definition.
This section describes the log configuration for a container to use
the awslogs log driver. For more information, see Creating a Task
Definition.
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-mysql",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "awslogs-example"
}
}
AmazonECS-using_awslogs
How can I configure ECS Fargate + Cloudwatch to include specific file location.
I have app-access.log where my framework puts all of my access logs.
Cloudwatch currently consumes logs from my server command IO only. How can I tell ECS Fargate to use app-access.log also.
PS. I am using Cloudformation.
ECS or Cloudwatch don't watch files in the container. ECS has integration with the docker logs, if the docker container emits logs from the access.log then these will be available for Cloudwatch. That's why you're only seeing the IO commands.
So it's not about ECS but rather about how docker logging works. See here for more details on docker logging.
You have to make sure any logline is written to STDOUT or STDERR.
One method is to symlink /path/to/app-access.log -> /dev/stdout.
But usually, it's easier to make sure there's a console appender for your service.
I'm an application developer with very limited knowledge of infrastructure. At my last job we frequently deployed Java web services (built as WAR files) to Elastic Beanstalk, and much of the infrastructure had already been set up before I ever started there, so I got to focus primarily on the code and not how things were tied together. One feature of Elastic Beanstalk that often came in handy was the button to "Request Logs," where you can select either the "Last 100 Lines" or the "Full Logs." What I'm used to seeing when clicking this button is to directly view the logs generated by my web service.
Now, at the new job, the infrastructure requirements are a little different, as we have to Dockerize everything before deploying it. I've been trying to stand up a Spring Boot web app inside a Docker container in Elastic Beanstalk, and have been running into trouble with that. And I also noticed a bizarre difference in behavior when I went to "Request Logs." Now when I choose one of those options, instead of dropping me into the relevant log file directly, it downloads a ZIP file containing the entire /var/log directory, with quite a number of disparate and irrelevant log files in there. I understand that there's no way for Amazon to know, necessarily, that I don't care about X log file but do care about Y log file, but was surprised that the behavior is different from what I was used to. I'm assuming this means the EB configuration at the last job was set up in a specific way to filter the one relevant log file, or something like that.
Is there some way to configure an Elastic Beanstalk application to only return one specific log file when you "Request Logs," rather than a ZIP file of the /var/log directory? Is this done with ebextensions or something like that? How can I do this?
Not too sure about the Beanstalk console, but using the EBCLI, if you enable CloudWatch log streaming (note that this would cost you to store logs in CloudWatch) for your Beanstalk instances, you can perform:
eb logs --stream --log-group <CloudWatch logGroup name>
The above command basically gives you the logs for your instance specific to the file/log group you specified. In order for the above command to work, you need to enable CloudWatch log streaming:
eb logs --stream enable
As an aside, to determine which log groups your environment presently has, perform:
aws logs describe-log-groups --region <region> | grep <beanstalk environment name>
I have a classic scala app, it produces three different logs in the location
/var/log/myapp/log1/mylog.log
/var/log/myapp/log2/another.log
/var/log/myapp/log3/anotherone.log
I containerized the app and working fine, I can get those logs by docker volume mount.
Now the app/container will be deployed in AWS ECS with auto scaling group. in this case multiple container may run on one single ecs host.
I would like to use cloud watch to monitor my application logs.
One solution could be put aws log agent inside my application container.
Is there any better way to get those application logs from container to cloudwatch log.
help is very much appreciated.
When using docker, the recommended approach is to not log to files, but to send logs to stdout and stderr. Doing so prevents the logs from being written to the container's filesystem, and (depending on the logging driver in use), allows you to view the logs using the docker logs / docker container logs subcommand.
Many applications have a configuration option to log to stdout/stderr, but if that's not an option, you can create a symlink to redirect output; for example, the official NGINX image on Docker Hub uses this approach.
Docker supports logging drivers, which allow you to send logging to (among others) AWS cloud watch. After you modified your image to make it log to stdout/stderr, your can configure the AWS logging driver.
More information about logging in Docker can be found in the "logging" section in the documentation
You don't need log agent if you can change the code.
You can directly publish Custom Metric Data into ColudWatch like this page said: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/examples-cloudwatch-publish-custom-metrics.html
Is it possible to dynamically set values in AWS ECS task definitions? For example. I have the following volume definition.
"volumes": [
{
"host": {
"sourcePath": "/tmp/logs/registrations"
},
"name": "logs"
}
],
I would like do do something like /tmp/logs/<container_id>.
I am trying to do this because I have multiple containers running the same app. I need to mount each containers log directory for a sumologic collector. The problem is if the directories aren't namespaced by container then the mounts will conflict.
Cloudwatch logs is another option if saving to the file system is not a requirement.
If you are using the ECS optimized AMI it's already configured and all you need to do is turn it on in the container task definition:
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html
You would also need to configure log groups or log streams for each container.
Is saving logs to the file system a hard requirement? If not, Sumo Logic has a pretty good article about another option: Update On Logging With Docker.
You run the Sumo Logic Collector for Docker in one container and your application in another. You configure Docker to send logs from your application container to the Sumo Logic collector. This is built into Docker, so your application shouldn't need to change. The collector container will then send those logs to Sumo Logic.
I've seen this pattern called the sidecar, in case you're looking for other examples of it.