For an EKS cluster, cloudtrail logs cluster events such as create, update and delete. However we are using kubeadm to provision clusters. How do we log an audit trail of these cluster events? Thanks.
CloudTrail logs API events in AWS, so I don't think you can use it for K8S events. However, you can use log shippers to send custom metrics to CloudWatch. From there you can emit events and create dashboards.
For this you have a couple of options, you can use the CloudWatch agent, An Elastic Beat, Logstash, or maybe use something like Splunk if you don't want to use CloudWatch.
From the K8S documentation, there's an Audit log (possibly at /var/log/kube-audit for your cluster) which...
Kubernetes auditing provides a security-relevant chronological set of records documenting the sequence of activities that have affected system by individual users, administrators or other components of the system. It allows cluster administrator to answer the following questions:
You can ship/parse this log with another service.
If you need more control over the outcome, you can write a custom Beat, based on the libbeat specification. https://github.com/elastic/beats/tree/master/libbeat
Otherwise, I think a lot of people use Filebeat: https://github.com/elastic/beats/tree/master/deploy/kubernetes
K8S also supports custom Audit Policies for further control
Related
I am kind of confused with the difference between aws system manager and aws cloudwatch?
Could someone help me to get clear with the difference?
Thank you very much.
They have different purposes.
aws system manager in the core of its functionality allows you to manage a fleet of instances as well as on-premise servers. Using the manger you can updated hundreds of instances with just a single command, execute custom scripts on all of them, monitor their patch compliance (i.e. do all your instances of interest have latest updates) and so on.
aws cloudwatch is primary used as a central location for storing variety of logs, from your applications (e.g. lambda execution logs), aws services and so on. It also allows you to monitor performance metrics of your instances (e.g. CPU utilization) as well as other resources. Other functionality is to respond to live events from resources (e.g. execute lambda whenever an instance is terminated)
In short, AWS System Manger is a centralized tool to automate management of AWS resources.
Whereas AWS Cloudwatch is centralized tool for monitoring AWS resource logs.
These short video resources might help -
AWS System Manager -
https://www.youtube.com/watch?v=MK4ZoCs-muo&ab_channel=AmazonWebServices
AWS Cloudwatch -
https://www.youtube.com/watch?v=a4dhoTQCyRA&ab_channel=AmazonWebServices
I'm trying to automate the turning on and off process of Redis Cluster in aws. I saw the following link for reference (https://forums.aws.amazon.com/thread.jspa?threadID=149772). Is there a way to do it via cloudwatch ?
I am very new to aws platform.
Check the documentation regarding scale in/out
https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/redis-cluster-resharding-online.html It also has commands to reshard a cluster manually.
Check CloudWatch metrics from the Redis cluster. https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.HostLevel.html and https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/CacheMetrics.Redis.html Choose the metrics that will trigger autoscaling
You can trigger an AWS Lambda on some event for a metric https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
From the Lambda you cal call aws cli to reshard the cluster as described in 1. Example: https://alestic.com/2016/11/aws-lambda-awscli/
If you need to turn off the cluster completely, instead of the resharding commands just use https://docs.aws.amazon.com/cli/latest/reference/elasticache/delete-cache-cluster.html
I'm looking for a way to establish custom metrics over StatsD protocol for Amazon Elastic Container Service. I've found a documentation on how to establish Amazon CloudWatch Agent on EC2. It works well. However I'm failing to find a correct configuration for Dockerfile. Quite probably some set of custom IAM permissions will also be required there.
Is it possible to have Docker containers working from AWS ECS with custom metrics using StatsD reporting to AWS CloudWatch?
Rather than building your own container, you can use the one provided by Amazon. This article explains how, including a link to an example daemon service task configuration.
As far as I understand there are special logs (like for EC2 Instances, RDS) which can be enabled to collect metrics and are not enabled automatically. Is there an OVERVIEW somewhere stating for which resources do special AWS Logs exist and how they can be enabled (via CloudFormation)?
If not I would suggest to create such an Overview since AWS DOCUMENTATION doesn’t have it.
Merci A
Amazon CloudWatch collects metrics from AWS services that you use. (Or, more accurately, the services you use send metrics to Amazon CloudWatch.)
The documentation for each service shows the metrics that are collected.
The only logs that go beyond this are special ones sent from within an Amazon EC2 instance. By default, the Amazon EC2 service can only examine your virtual machine ('instance') from outside of the instance. This provides basic metrics for CPU, disk and network. It is not possible to see inside your instance.
However, if you run additional software inside the instance that communicates with the operating system, then the software can send additional metrics to Amazon CloudWatch. This is typically done with the CloudWatch Agent.
See: Collect Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent - Amazon CloudWatch
There are additional Metrics Collected by the CloudWatch Agent that can provide details such as free memory and free disk space because the agent communicates with the Operating System running inside the instance. These metrics are then sent to Amazon CloudWatch as custom metrics.
The CloudWatch Agent can also send application logs from an Amazon EC2 instance to Amazon CloudWatch Logs, which retains the individual log entries and can trigger alarms based on their content.
The CloudWatch Agent must be installed into the Amazon EC2 instance, either via a startup script ('User Data' script) or via AWS Systems Manager.
Similarly, Amazon RDS can send database logs to Amazon CloudWatch Logs. However, I couldn't see an option in Amazon CloudFormation to activate these log exports.
I have an AWS account, and have several EC2 servers and an ElasticSearch domain set up to take the syslogs from these servers. However, in Cloudwatch and when investigating a specific server instance in the EC2 control panel, I see specific metrics and graphs for things like CPU, memory load, storage use, etc. Is there some way I can pipe this information into my ElasticSearch as well?
Set up Logstash and use this plugin https://github.com/EagerELK/logstash-input-cloudwatch
Or go the other way and use AWS Logs agent to put your syslogs into Cloudwatch and stop using ElasticSearch