I have create an eks cluster on fargate. I want to send container logs to cloudwatch. for that i have use this guideline(eks-fargate-logging. I want to create cloudwatch stream as namespace-pod-name. how can we do this ??
Need to use older cloudwatch plugins. dont use cloudwatch_logs plugin.cloudwatch plugin
Related
My understanding is that the Cloudwatch agent is available both as a Linux binary and as a Kubernetes deamonset.
I am aware that the EKS container logs could be forwarded to Cloudwatch using the cloudwatch agent that runs as EKS daemonset.
I have a query on how to send OS logs from EKS nodes to cloudwatch? Would the cloudwatch agent daemonset service be able to send the OS logs to Cloudwatch? or is the Linux binary required to be run on the EKS nodes to send OS logs?
I believe that either can be used independently.
You can set up Fluentd or Fluent Bit as a DaemonSet to send logs to CloudWatch Logs.
Alternatively, you can install CloudWatch agent on Amazon EKS nodes using Distributor and State Manager. You may also consider including it in the launch template to automate installation because the EKS-optimized AMI does not include the agent by default.
So my understanding is that when I deploy a new service to ECS using AWS Copilot, logs are forwarded to CloudWatch automatically by default.
Copilot creates log groups for each service, I can see that in CloudWatch Logs.
However, according to AWS docs, logging can be also implemented using Copilot sidecars and AWS FireLens, which uses FluentD or FluentBit to collect logs, and then it forwards stuff CloudWatch.
I don't understand why is this necessary. I mean, why to create a sidecar for logging to CloudWatch, when logging seems to work automatically, without any sidecar.
https://aws.github.io/copilot-cli/docs/developing/sidecars/
There is an example here for logging via FireLens. What's the benefit of doing this over the logging mechanism that just works by default?
Thanks in advance!
AWS Copilot builds an image for you application that already has an agent configured to forward logs to CloudWatch, however you might want to deploy other images to ECS that don't have this agent installed. For example, suppose you wanted to deploy an nginx container to ECS, you might choose to use a sidecar to forward logs instead of customizing the nginx image.
I'm trying to automate the turning on and off process of Redis Cluster in aws. I saw the following link for reference (https://forums.aws.amazon.com/thread.jspa?threadID=149772). Is there a way to do it via cloudwatch ?
I am very new to aws platform.
Check the documentation regarding scale in/out
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/redis-cluster-resharding-online.html It also has commands to reshard a cluster manually.
Check CloudWatch metrics from the Redis cluster. https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.HostLevel.html and https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html Choose the metrics that will trigger autoscaling
You can trigger an AWS Lambda on some event for a metric https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
From the Lambda you cal call aws cli to reshard the cluster as described in 1. Example: https://alestic.com/2016/11/aws-lambda-awscli/
If you need to turn off the cluster completely, instead of the resharding commands just use https://docs.aws.amazon.com/cli/latest/reference/elasticache/delete-cache-cluster.html
I'm looking for a way to establish custom metrics over StatsD protocol for Amazon Elastic Container Service. I've found a documentation on how to establish Amazon CloudWatch Agent on EC2. It works well. However I'm failing to find a correct configuration for Dockerfile. Quite probably some set of custom IAM permissions will also be required there.
Is it possible to have Docker containers working from AWS ECS with custom metrics using StatsD reporting to AWS CloudWatch?
Rather than building your own container, you can use the one provided by Amazon. This article explains how, including a link to an example daemon service task configuration.
Had couple of questions on AWS:
Is there a way by which I can recreate/write AWS CloudWatch metrics to DynamoDB?
If an Amazon EC2 instance is deleted or if I change a VPC, I need to recreate all CloudWatch metrics manually every time. Is there a way by which I can automate CloudWatch metrics creation for every new VPC instance? Through Terraform, I can only create CloudWatch metric alarms, events and logs but not CloudWatch metrics (eg, EC2, RDS metrics etc).
#1 I could achieve it via AWS CLI and via Python script thereby writing it to dynamodb as well. #2 is still open.