I am streaming AWSLogs to CLoudwatch and from there, I am streaming it on ElasticSearch domain. I can see in the overview, that I have a heck number of searchable documents, but I am not able to find the documents from many different log-streams when I am searching them in Kibana. It's working only for 2-3 out of 35 log streams. I can see all streams in CloudWatch logs, and on the right side, can also see that I am streaming to ElasticSearch instance. I will explain how I am doing it, maybe some idea what am I doing wrong. Thank you.
Installed AWSLogs service from here
Commands :
curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
sudo python ./awslogs-agent-setup.py --region OUR_REGION
Once that's done, I added in /var/awslogs/etc/awslogs.conf my log files in such manner :
[/var/www/html/var/log/exception.log]
datetime_format = %d/%b/%Y:%H:%M:%S
file = /var/www/html/var/log/exception.log
buffer_duration = 5000
log_stream_name = staging.1c.APP_NAME.exception.log
initial_position = end_of_file
log_group_name = staging.1c.APP_NAME.exception.log
After that, I logged into Kibana, and in index-patterns, defined an index-pattern as cwl-*.
Related
I have pyspark program running on AWS EMR cluster.
Cluster config is like this - emr-5.31.0, hadoop 2.10.0, hive 2.3.7, hue 4.7.1, pig 0.17.0.
Program processes some files on hdfs file system but at some moment it is getting errors.
In amazon console - YARN applications - application_XXX (Spark) - executors - driver - stderr:
'could not obtain block ... file=
A little before this message there is 'Task 0 in stage 35 failed 4 times. aborting job'
If i go to amazon console - YARN applications - application_XXX (Spark) - stages - 35 - tasks - 0 - stdout - i dont see anything bad at first glance except a lot of 'GC (allocation Failure)' messages.
In its stderr - there is a WARN - 'Could not obtain block XXX, file= No live nodes contain current block Block locations: Dead nodes: . Throwing a BlockMissingException.
If i go to monitoring tab - node status - i see that one node became unhealthy at that time and thats it. Number of nodes also changed at 'live data nodes', 'MR total nodes', 'MR active nodes', MR lost nodes' charts.
As i understand, task cannot find file on hdfs because node it was hosted on became unhealthy.
My question is where i can find the reasons node became unhealthy. I wasnt able to find any other logs on amazon console. May be there are some node-local places where this reason is stored?
Hi I launched a EMR myself some time ago, dont remember about the logs. But consulting the docs here:
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-web-log-files.html
It states that they are stored on the machines (which I assume you have the keys), they are also stored on S3 by default. Not sure in which bucket they will be created.
Best Regards :)
On the Summary page for your EMR cluster there is a section named "Configuration details".
Below that, there is a label named "Log URI". It points to an S3 URI, but, there is also a small folder icon.
Click on that icon and you can browse to the logs on the nodes for your EMR cluster.
Actually, for amazon there are more logs accessible via s3 location - there are logs for node boot and configuration part, and logs from running services on node - hdfs and yarn, which i was looking for. Path looks like this - s3 location/cluster id/node/node id/applications - here i was able to find hdfs and yarn logs.
I am trying to setup AWS cloud watch agent on one of our nodes in our cluster and unable to find the difference between start_of_file and end_of_file for initial_position configuration.
I created a log file tes1234.log and provided the below log configuration in awslogs.conf [/var/awslogs/etc/awslogs.conf] file
[test1234_log]
datetime_format = %Y-%m-%d %H:%M:%S
file = /var/xxx/log/test1234.log
buffer_duration = 5000
log_stream_name = test1234_log_stream
initial_position = start_of_file
log_group_name = xxx-test
After providing these information I started the agent and found that logstream test_1234 is created but when I change it to end_of_file I found that logstream is not getting created.
I unable to find the difference between start_of_file and end_of_fileand on which scenarios need to use what.Kindly help.
That setting lets you specify whether to consume the log file from the beginning, or whether to start from the end. This only applies to the very first time you start the agent, because once you start it the agent will save its own pointer on the file and will continue from that location if/when restarted.
You may want to choose "end_of_file" if you don't care about any old data at the time you install the agent for the very first time. If you'd like to upload all the data already accumulated in the file, then choose "start_of_file". The only downside of "start_from_file" is that the agent might take a while to upload the whole file and catch up to the tail.
I'm running my docker containers on CoreOS AWS instances and enabled aws log driver for the docker containers. Given below is my docker container run command.
docker run --log-driver=awslogs --log-opt awslogs-region=ap-southeast-1 --log-opt awslogs-group=stagingUrlMapperLogs --log-opt awslogs-datetime-format='\[%%b %%d, %%Y %%H:%%M:%%S\]' --log-opt tag="{{.Name}}/{{.ID}}" --net=host --name url-mapper url-mapper-example:latest
The issue is after some random period of time (1-2 days) on AWS CloudWatch side, no new log events are being recorded. After doing some research I came across this issue reported on AWS developer Forum. It says that adding file_fingerprint_lines option on CloudWatch config will solve the issue. But I didn't find any resources exaplaining how to set the file_fingerprint_lines command with docker run command.
Note - I'm running my servers in AWS autosacaling group which is connected to a launch configuration, so each time I scale up, new servers will spin up with the container running on it.
"But I didn't find any resources exaplaining how to set the file_fingerprint_lines command with docker run command."
I think that you have to set it in the CloudWatch Logs agent configuration file:
From the Amazon CloudWatch docs:
file_fingerprint_lines
Specifies the range of lines for identifying a file. The valid values
are one number or two dash delimited numbers, such as '1', '2-5'. The
default value is '1' so the first line is used to calculate
fingerprint. Fingerprint lines are not sent to CloudWatch Logs unless
all the specified lines are available
But, I think that the interesting point comes here:
What kinds of file rotations are supported?
The following file rotation mechanisms are supported:
Renaming existing log files with a numerical suffix, then re-creating
the original empty log file. For example, /var/log/syslog.log is
renamed /var/log/syslog.log.1. If /var/log/syslog.log.1 already exists
from a previous rotation, it is renamed /var/log/syslog.log.2.
Truncating the original log file in place after creating a copy. For
example, /var/log/syslog.log is copied to /var/log/syslog.log.1 and
/var/log/syslog.log is truncated. There might be data loss for this
case, so be careful about using this file rotation mechanism.
Creating a new file with a common pattern as the old one. For example,
/var/log/syslog.log.2014-01-01 remains and
/var/log/syslog.log.2014-01-02 is created.
The fingerprint (source ID) of the file is calculated by hashing the
log stream key and the first line of file content. To override this
behavior, the file_fingerprint_lines option can be used. When file
rotation happens, the new file is supposed to have new content and the
old file is not supposed to have content appended; the agent pushes
the new file after it finishes reading the old file.
And, how to override it:
You can have more than one [logstream] section, but each must have a
unique name within the configuration file, e.g., [logstream1],
[logstream2], and so on. The [logstream] value along with the first
line of data in the log file, define the log file's identity.
[general]
state_file = value
logging_config_file = value
use_gzip_http_content_encoding = [true | false]
[logstream1]
log_group_name = value
log_stream_name = value
datetime_format = value
time_zone = [LOCAL|UTC]
file = value
file_fingerprint_lines = integer | integer-integer
multi_line_start_pattern = regex | {datetime_format}
initial_position = [start_of_file | end_of_file]
encoding = [ascii|utf_8|..]
buffer_duration = integer
batch_count = integer
batch_size = integer
[logstream2]
...
I got a $1,200 invoice from Amazon for Cloudwatch services last month (specifically for 2 TB of log data ingestion in "AmazonCloudWatch PutLogEvents"), when I was expecting a few tens of dollars. I've logged into the Cloudwatch section of the AWS Console, and can see that one of my log groups used about 2TB of data, but there are thousands of different log streams in that log group, how can I tell which one used that amount of data?
On the CloudWatch console, use the IncomingBytes metrics to find the amount of data ingested by each log group for a particular time period in uncompressed bytes using Metrics page. Follow the below steps -
Go to CloudWatch metrics page and click on AWS namespace 'Logs' --> 'Log Group Metrics'.
Select the IncomingBytes metrics of the required log groups and click on 'Graphed metrics' tab to see the graph.
Change the start time and end time such that their difference is 30 days and change the period to 30 days. This way, we will get only one data point. Also changed the graph to Number and statistics to Sum.
This way, you will see the amount of data ingested by each log groups and get an idea about which log group is ingesting how much.
You can also achieve the same result using AWS CLI. An example scenario where you just want to know the total amount of data ingested by log groups for say 30 days, you can use get-metric-statistics CLI command-
sample CLI command -
aws cloudwatch get-metric-statistics --metric-name IncomingBytes --start-time 2018-05-01T00:00:00Z --end-time 2018-05-30T23:59:59Z --period 2592000 --namespace AWS/Logs --statistics Sum --region us-east-1
sample output -
{
"Datapoints": [
{
"Timestamp": "2018-05-01T00:00:00Z",
"Sum": 1686361672.0,
"Unit": "Bytes"
}
],
"Label": "IncomingBytes"
}
To find the same for a particular log group, you can change the command to accommodate dimensions like -
aws cloudwatch get-metric-statistics --metric-name IncomingBytes --start-time 2018-05-01T00:00:00Z --end-time 2018-05-30T23:59:59Z --period 2592000 --namespace AWS/Logs --statistics Sum --region us-east-1 --dimensions Name=LogGroupName,Value=test1
One by one, you can run this command on all log groups and check which log group is responsible for most of the bill for data ingested and take corrective measures.
NOTE: Change the parameters specific to your environment and requirement.
The solution provided by OP gives data for the amount of logs stored which is different from logs ingested.
What is the difference?
Data ingested per month is not same as Data storage bytes. After the data is ingested to CloudWatch, it is archived by CloudWatch which includes 26 bytes of metadata per log event and is compressed using gzip level 6 compression. So the Storage bytes refers to the storage space used by Cloudwatch to store the logs after they're ingested.
Reference : https://docs.aws.amazon.com/cli/latest/reference/cloudwatch/get-metric-statistics.html
We had a lambda logging GB of data of due to accidental check-in. Here's a boto3 based python script based on the info from the answers above that scans all log groups and prints out any group with logs greater than 1GB in the past 7 days. This helped me more than trying to use AWS dashboard which was slow to update.
#!/usr/bin/env python3
# Outputs all loggroups with > 1GB of incomingBytes in the past 7 days
import boto3
from datetime import datetime as dt
from datetime import timedelta
logs_client = boto3.client('logs')
cloudwatch_client = boto3.client('cloudwatch')
end_date = dt.today().isoformat(timespec='seconds')
start_date = (dt.today() - timedelta(days=7)).isoformat(timespec='seconds')
print("looking from %s to %s" % (start_date, end_date))
paginator = logs_client.get_paginator('describe_log_groups')
pages = paginator.paginate()
for page in pages:
for json_data in page['logGroups']:
log_group_name = json_data.get("logGroupName")
cw_response = cloudwatch_client.get_metric_statistics(
Namespace='AWS/Logs',
MetricName='IncomingBytes',
Dimensions=[
{
'Name': 'LogGroupName',
'Value': log_group_name
},
],
StartTime= start_date,
EndTime=end_date,
Period=3600 * 24 * 7,
Statistics=[
'Sum'
],
Unit='Bytes'
)
if len(cw_response.get("Datapoints")):
stats_data = cw_response.get("Datapoints")[0]
stats_sum = stats_data.get("Sum")
sum_GB = stats_sum / (1000 * 1000 * 1000)
if sum_GB > 1.0:
print("%s = %.2f GB" % (log_group_name , sum_GB))
Although the author of the question and other folks have answered the question in a good way, I will try to have a generic solution that could be applied without knowing the exact log-group-name which is causing too much of logs.
To do this, we can not use describe-log-streams function because this would need --log-group-name and as I said earlier I do not know the value of my log-group-name.
We can use describe-log-groups function because this function does not require any parameter.
Note that I am assuming that you have the required flag (--region) configured in ~/.aws/config file and your EC2 instance have the required permission to execute this command.
aws logs describe-log-groups
This command would list all the log groups in your aws account. The sample output of this would be
{
"logGroups": [
{
"metricFilterCount": 0,
"storedBytes": 62299573,
"arn": "arn:aws:logs:ap-southeast-1:855368385138:log-group:RDSOSMetrics:*",
"retentionInDays": 30,
"creationTime": 1566472016743,
"logGroupName": "/aws/lambda/us-east-1.test"
}
]
}
If you are interested in a specific prefix pattern only for the log group, you can use --log-group-name-prefix like this
aws logs describe-log-groups --log-group-name-prefix /aws/lambda
The output JSON of this command would also be similar to the above output.
If you have too many log groups in your account, analyzing the output of this becomes difficult and we need some command-line utility to give a brief insight into the result.
We will use the 'jq' command-line utility to get the desired thing. The intention is to get which log group has produced the most amount of log and hence caused more money.
From the output JSON, the fields which we need for our analysis would be "logGroupName" and "storedBytes". So taking these 2 fields in the 'jq' command.
aws logs describe-log-groups --log-group-name-prefix /aws/
| jq -M -r '.logGroups[] | "{\"logGroupName\":\"\(.logGroupName)\",
\"storedBytes\":\(.storedBytes)}"'
Using '\' in the command to do the escape because we want the output to be in the JSON format only to use the sort_by function of jq. The sample output of this would be something like below:
{"logGroupName":"/aws/lambda/test1","storedBytes":3045647212}
{"logGroupName":"/aws/lambda/projectTest","storedBytes":200165401}
{"logGroupName":"/aws/lambda/projectTest2","storedBytes":200}
Note that the output result would not be sorted on storedBytes, so we want to sort them in order to get which log group is the most problematic one.
we will use sort_by function of jq to accomplish this. The sample command would be like this
aws logs describe-log-groups --log-group-name-prefix /aws/
| jq -M -r '.logGroups[] | "{\"logGroupName\":\"\(.logGroupName)\",
\"storedBytes\":\(.storedBytes)}"'
| jq -s -c 'sort_by(.storedBytes) | .[]'
This would produce the below result for the above sample output
{"logGroupName":"/aws/lambda/projectTest2","storedBytes":200}
{"logGroupName":"/aws/lambda/projectTest","storedBytes":200165401}
{"logGroupName":"/aws/lambda/test1","storedBytes":3045647212}
The elements from the bottom of this list are the ones that have the most log associated with it. You may set the Expire Events After property to a finite period say 1 month to these log group.
If you want to know what is the sum of all the log byte then you can use the 'map' and 'add' function of jq like below.
aws logs describe-log-groups --log-group-name-prefix /aws/
| jq -M -r '.logGroups[] | "{\"logGroupName\":\"\(.logGroupName)\",
\"storedBytes\":\(.storedBytes)}"'
| jq -s -c 'sort_by(.storedBytes) | .[]'
| jq -s 'map(.storedBytes) | add '
The output of this command for the above sample output would be
3245812813
The answer has become lengthy but I hope it helps in figuring out the most problematic log group in cloudwatch.
You can also click the gear on the gear on the cloudwatch logs dashboard and choose the stored bytes column.
I also clicked anything that said 'never expire' and changed the logs to expire.
Use cloudwatch logs gear and select "Stored Bytes" column
*** UPDATE 20210907 - as #davur points out in one of the comments below, AWS deprecated storedBytes for individual LogStreams, so the method described in this answer no longer fulfils the requirement, although it might be interesting in other ways ***
Okay, I'm answering my own question here, but here we go (with all other answers welcome):
You can use a combination of AWS CLI tools, the csvfix CSV package and a spreadsheet to work this out.
Log into the AWS Cloudwatch Console and grab the name of the log group which has generated all the data. In my case it's called "test01-ecs".
Unfortunately in the Cloudwatch Console you can't sort the streams by "Stored Bytes" (which would tell you which ones are biggest). If there are too many streams in the log group to look through in the Console then you need to dump them somehow. For this you can use the AWS CLI tool:
$ aws logs describe-log-streams --log-group-name test01-ecs
The command above will give you JSON output (assuming your AWS CLI tool is set to JSON output - set it to output = json in ~/.aws/config if not) and it will look something like this:
{ "logStreams": [ { "creationTime": 1479218045690, "arn": "arn:aws:logs:eu-west-1:902720333704:log-group:test01-ecs:log-stream:test-spec/test-spec/0307d251-7764-459e-a68c-da47c3d9ecd9", "logStreamName": "test-spec/test-spec/0308d251-7764-4d9f-b68d-da47c3e9ebd8", "storedBytes": 7032 } ] }
Pipe this output to a JSON file - in my case the file was 31 MB in size:
$ aws logs describe-log-streams --log-group-name test01-ecs >> ./cloudwatch-output.json
Use the in2csv package (part of csvfix) to convert the JSON file to a CSV file which can easily be imported into a spreadsheet, making sure you define the logStreams key to be used to import on:
$ in2csv cloudwatch-output.json --key logStreams >> ./cloudwatch-output.csv
Import the resulting CSV file into a spreadsheet (I use LibreOffice myself as it seems great at dealing with CSV) making sure the storedBytes field is imported as an integer.
Sort the storedBytes column in the spreadsheet to work out which log stream or streams are generating the most data.
In my case this worked - it turned out one of my log streams (with logs from a broken TCP pipe in a redis instance) was 4,000 times the size of all the other streams combined!
An alternative to using the now deprecated storedBytes for log streams is to use Cloudwatch > Logs Insights and then run a query to count events by the log steam:
stats count(*) by #logStream
The log stream with the larger number of events will then probably be what is causing the high bill usage.
Is there any way to 1) filter and 2) retrieve the raw log data out of Cloudwatch via the API or from the CLI? I need to extract a subset of log events from Cloudwatch for analysis.
I don't need to create a metric or anything like that. This is for historical research of a specific event in time.
I have gone to the log viewer in the console but I am trying to pull out specific lines to tell me a story around a certain time. The log viewer would be nigh-impossible to use for this purpose. If I had the actual log file, I would just grep and be done in about 3 seconds. But I don't.
Clarification
In the description of Cloudwatch Logs, it says, "You can view the original log data (only in the web view?) to see the source of the problem if needed. Log data can be stored and accessed (only in the web view?) for as long as you need using highly durable, low-cost storage so you don’t have to worry about filling up hard drives." --italics are mine
If this console view is the only way to get at the source data, then storing logs via Cloudwatch is not an acceptable solution for my purposes. I need to get at the actual data with sufficient flexibility to search for patterns, not click through dozens of pages lines and copy/paste. It appears a better way to get to the source data may not be available however.
For using AWSCLI (plain one as well as with cwlogs plugin) see http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/SearchDataFilterPattern.html
For pattern syntax (plain text, [space separated] as as {JSON syntax}) see: http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html
For python command line utility awslogs see https://github.com/jorgebastida/awslogs.
AWSCLI: aws logs filter-log-events
AWSCLI is official CLI for AWS services and now it supports logs too.
To show help:
$ aws logs filter-log-events help
The filter can be based on:
log group name --log-group-name (only last one is used)
log stream name --log-stream-name (can be specified multiple times)
start time --start-time
end time --end-time (not --stop-time)
filter patter --filter-pattern
Only --log-group-name is obligatory.
Times are expressed as epoch using milliseconds (not seconds).
The call might look like this:
$ aws logs filter-log-events \
--start-time 1447167000000 \
--end-time 1447167600000 \
--log-group-name /var/log/syslog \
--filter-pattern ERROR \
--output text
It prints 6 columns of tab separated text:
1st: EVENTS (to denote, the line is a log record and not other information)
2nd: eventId
3rd: timestamp (time declared by the record as event time)
4th: logStreamName
5th: message
6th: ingestionTime
So if you have Linux command line utilities at hand and care only about log record messages for interval from 2015-11-10T14:50:00Z to 2015-11-10T15:00:00Z, you may get it as follows:
$ aws logs filter-log-events \
--start-time `date -d 2015-11-10T14:50:00Z +%s`000 \
--end-time `date -d 2015-11-10T15:00:00Z +%s`000 \
--log-group-name /var/log/syslog \
--filter-pattern ERROR \
--output text| grep "^EVENTS"|cut -f 5
AWSCLI with cwlogs plugin
The cwlogs AWSCLI plugin is simpler to use:
$ aws logs filter \
--start-time 2015-11-10T14:50:00Z \
--end-time 2015-11-10T15:00:00Z \
--log-group-name /var/log/syslog \
--filter-pattern ERROR
It expects human readable date-time and always returns text output with (space delimited) columns:
1st: logStreamName
2nd: date
3rd: time
4th till the end: message
On the other hand, it is a bit more difficult to install (few more steps to do plus current pip requires to declare the installation domain as trusted one).
$ pip install awscli-cwlogs --upgrade \
--extra-index-url=http://aws-cloudwatch.s3-website-us-east-1.amazonaws.com/ \
--trusted-host aws-cloudwatch.s3-website-us-east-1.amazonaws.com
$ aws configure set plugins.cwlogs cwlogs
(if you make typo in last command, just correct it in ~/.aws/config file)
awslogs command from jorgebastida/awslogs
This become my favourite one - easy to install, powerful, easy to use.
Installation:
$ pip install awslogs
To list available log groups:
$ awslogs groups
To list log streams
$ awslogs streams /var/log/syslog
To get the records and follow them (see new ones as they come):
$ awslogs get --watch /var/log/syslog
And you may filter the records by time range:
$ awslogs get /var/log/syslog -s 2015-11-10T15:45:00 -e 2015-11-10T15:50:00
Since version 0.2.0 you have there also the --filter-pattern option.
The output has columns:
1st: log group name
2nd: log stream name
3rd: message
Using --no-group and --no-stream you may switch the first two columns off.
Using --no-color you may get rid of color control characters in the output.
EDIT: as awslogs version 0.2.0 adds --filter-pattern, text updated.
If you are using the Python Boto3 library for extraction of AWS cloudwatch Logs. The function of get_log_events() accepts start and end time in milliseconds.
For reference: http://boto3.readthedocs.org/en/latest/reference/services/logs.html#CloudWatchLogs.Client.get_log_events
For this you can take a UTC time input and convert it into milliseconds by using the Datetime and timegm modules and you are good to go:
from calendar import timegm
from datetime import datetime, timedelta
# If no time filters are given use the last hour
now = datetime.utcnow()
start_time = start_time or now - timedelta(hours=1)
end_time = end_time or now
start_ms = timegm(start_time.utctimetuple()) * 1000
end_ms = timegm(end_time.utctimetuple()) * 1000
So, you can give inputs as stated below y using sys input as:
python flowlog_read.py '2015-11-13 00:00:00' '2015-11-14 00:00:00'
While Jan's answer is a great one and probably what the author wanted, please note that there is an additional way to get programmatic access to the logs - via subscriptions.
This is intended for always-on streaming scenarios where data is constantly fetched (usually into Kinesis stream) and then further processed.
Haven't used it myself, but here is an open-source cloudwatch to Excel exporter I came across on GitHub:
https://github.com/petezybrick/awscwxls
Generic AWS CloudWatch to Spreadsheet Exporter CloudWatch doesn't provide an Export utility - this does. awscwxls creates spreadsheets
based on generic sets of Namespace/Dimension/Metric/Statistic
specifications. As long as AWS continues to follow the
Namespace/Dimension/Metric/Statistic pattern, awscwxls should work for
existing and future Namespaces (Services). Each set of specifications
is stored in a properties file, so each properties file can be
configured for a specific set of AWS Services and resources. Take a
look at run/properties/template.properties for a complete example.
I think the best option to retrieve the data is provided as described in the API.