So, I created the cloudwatch index and streaming the cloud watch logs to elastic search and I am seeing data, however, I am only seeing current date data. I dont see old logs in elastic search which are in same log group in cloudwatch. I changed the date filter in elastic search, but dont see any change. Any idea why?
The index name created is, cwl-2018.03.20
That's the expected behavior. The streaming of logs from CloudWatch to Elasticsearch relies of a feature called subscription filters which only forwards new data to the destination.
Related
Our setup is this, AWS Services produce and publish logs to the CloudWatch Service. From there we use the standard Lambda function to publish the logs to the AWS ElasticSearch
The lambda function pushes the logs to ES using the file format cloudwatch-logs-<date> This creates a new index every day
We have an issue with mapping of the data. So for example when a service (eg. aurora db) publish its first set of logs and the field CPU value is 0 the ES set that as a long. When that same service publishes a second set of logs and the CPU is set to 10.5 the ES rejects that set of data with the error mapper cannot change type [long] to [float]
We have allot of services publishing logs with allot of data sets. Is the best way to resolve this for lambda to push the logs with format of cloudwatch-logs so only one index is created and then manual fix the mapping issue for that index ? or is there a better way to resolve this ?
I am using Elasitsearch to get logs from cloudwatch log group by subscribing a lambda to the log group. So whenever there is a log event pushed to the log group, my lambda will be triggered and it will save the log to Elasticsearch. Then I can search the log via Kibana dashboard.
I'd like to put the metrics data to Elasticsearch as well but I couldn't find a way to subscribe to metrics data.
You can use AWS Module in MetricBeat from the Elastic Beat's family. Note that pulling metrics from cloudwatch will result in chargeable API calls. So you should carefully consider the scraping frequency.
Thanks
I created Aurora Server Less DB Cluster (mysql5.10) on AWS.
then I enabled only slow query logs by setting keys in parameter cluster groups as -
slow_query_log : 1
long_query_time: 0.5
log_output: file
but In cluodWatch when I was looking for logs, I found logs-
My mysql user logs (which was expected)
But there are so many rdsadmin logs. Even these logs do not match the criteria (long_query_time: 0.5).
Please help me to find out, is there a way to disable logs of rdsadmin from cloudwatch??
I could not find a clear answer on AWS site explaining which parameters are supported for RDS Serverless v1. After some quick test, I cant make the long_query_time parameter work with Serverless. Maybe the cluster need to be restarted, or we need to recreate a parameter group ? https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html#aurora-serverless.parameter-groups
My solution is to enable slow_query log (it works), and stream the log to CloudWatch Logs.
Then I query the log with CloudWatch Logs Insights with the following query (assuming you have selected the slow_query log stream) :
parse #message "Query_time: * Lock_time: * Rows_sent: * Rows_examined: *\n*" as Query_time,Lock_time,Rows_sent,Rows_examined,q
| filter Query_time > 0.250
It will display all query slower than 250ms.
The AWS documentation indicates that multiple log event records are provided to Lambda when streaming logs from CloudWatch.
logEvents
The actual log data, represented as an array of log event
records. The "id" property is a unique identifier for every log event.
How does CloudWatch group these logs?
Time? Count? Randomly, from my perspective?
Currently you get one Lambda invocation for every PutLogEvents batch that CloudWatch Logs had received against that log group. However you should probably not rely on that because AWS could always change it (for example batch more, etc).
You can observe this behavior by running the CWL -> Lambda example in the AWS docs.
Some aws services allow you to configure the log intervals such as elastic load balancing. There's a choice between five and sixty minute log intervals. You may not see a specific increment or parameter in the docs because they are configurable based on each service.
So we have logs (apache, tomcat, etc) stored in Amazon CloudWatch Logs.
I'm trying to use Logstash to index from AWS and send them over to Elasticsearch/Kibana.
I can't seem to find a plugin to accomplish this.
Has anyone tried this and was successful?
I don't want the metrics, just the logs stored in AWS Logs.
Other posters have mentioned that CloudFormation templates are available that will stream your logs to Amazon Elasticsearch, but if you want to go through Logstash first, this logstash plugin may be of use to you:
https://github.com/lukewaite/logstash-input-cloudwatch-logs/
This plugin allows you to ingest specific CloudWatch Log Groups, or a series of groups that match a prefix into your Logstash pipeline, and work with the data as you will. It is published on RubyGems, and can be installed like a normal Logstash plugin: bin/logstash-plugin install logstash-input-cloudwatch_logs.
As already pointed out by BMW, AWS has just introduced a dedicated CloudWatch Logs Subscription Consumer, which provides one click access to a complete CloudWatch Logs + Elasticsearch + Kibana stack by means of a resp. AWS CloudFormation template, as further illustrated in the introductory blog post.
Given you seem to have an ELK stack readily available, it shouldn't be too complex to adjust the AWS sample template to target your own endpoints instead.
In order to use the CloudFormation template (as per BMW's answer) it needs to be customized, part of this would be providing your account ID and region as a CF Resource.
AWS: : AccountId and AWS: : Region are pseudo parameters that return the AWS account ID of the account in which the stack is being created, such as 123456789012, and a string representing the AWS Region in which the encompassing resource is being created, such as us-west-2. (http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html)