I am working on a project, where i am streaming my data to AWS CloudWatch logs, via AWS Cloudwatch logs agent. I can see that there are log-groups getting created and there are logs inside it. Still AWS Cloudwatch is showing 0 bytes stored. Why is that?
This is an issue, as I am also streaming the data to our ES domain, and this stored data might be stopping it. Or am I doing something wrong? Here is a screenshot of the info :
What am I doing wrong?
storedBytes -> (long)
The number of bytes stored.
IMPORTANT: Starting on June 17, 2019, this parameter will be
deprecated for log streams, and will be reported as zero. This change
applies only to log streams. The storedBytes parameter for log groups
is not affected.
Source
Related
We set up google cloud managed service for prometheus this week. While creating grafana dashboards I noticed that most metrics were ingested correctly, but some values were consistently incorrect.
The output of the metrics endpoint looks like this (truncated):
# HELP channel_socket_bytes_received_total Number of bytes received from clients # TYPE channel_socket_bytes_received_total counter # HELP event_collection_size Number of elements # TYPE event_collection_size gauge event_collection_size{name="interest"} 18 event_collection_size{name="rfq"} 362 event_collection_size{name="negotiation"} 12 # TYPE sq_firestore_read_total counter sq_firestore_read_total{collection="negotiation"} 12 sq_firestore_read_total{collection="rfq_interest"} 18 sq_firestore_read_total{collection="rfq"} 362
The output on this endpoint is generated by "prom-client": "14.1.0".
Google managed service for prometheus ingests these metrics. Almost all of them work as expected. But the sq_firestore_read_total metric is consistently wrong.
The google cloud metrics explorer shows these values:
Services were restarted a number of times. Once the value of one label reached 3, but more common is that the values of all 3 labels of the metric stay stuck at 0.
It seems to me that something goes wrong during the ingestion stage. Is this a bug in the google cloud managed service for prometheus?
Important to reiterate: The values I expect are 12, 16, and 362. The values that are ingested are either 0 and sometimes 3.
how can I get last 30 minutes AWS CloudWatch logs which are inserted to the specific LogStream using AWS Command ?
Can you describe what you already tried yourself and what you ran into? Looking at the AWS CLI command reference, it seems that you should be able to run "aws cloudwatch get-log-events ----log-stream-name <name of the stream> --start-time <timestamp>" to get a list of events, starting at given UNIX timestamp, calculating the timestamp should be fairly trivial.
Addition, based on your comment: you'll need to look into the AWS concept of pagination. Most/many AWS API calls (which the CLI also makes for you) retrieve a size/length limited set of data and return a token if there is more data present. You can then make a subsequent call passing that token, which tells the service to return data starting at that token. Repeat this process until you no longer get a token back, at which point you know you have iterated the full dataset.
For this specific CLI command, there is a flag.
--next-token (string)
The token for the next set of items to return. (You received this token from a previous call.
Hope this helps?
I do have a EC2 instance and a docker container running on it. Currently this docker container uses awslog driver to push logs to CloudWatch. If I go to CloudWatch console, I see a very large log stream (with container id as name) which contains all logs of last 16 days (since I've created the container). It almost seems like if I have this container running for 1 year, this log stream will keep all logs of 1 year. I am not quite sure what is the maximum size limit of a CloudWatch log stream, but most likely it will have a limit, at least I believe.
So my question is;
How to chunk this huge logstream? Hopefully by current date, smth like {{.ContainerId}}{{.CurrentDate}}
What is the maximum size limit of a CloudWatch log stream?
Is it a good practice to append onto a single huge log stream?
The following is the definition of Cloudwatch Log Stream as defined in the docs, here
Log streams
A log stream is a sequence of log events that share the same source. More specifically, a log stream is generally intended to represent the sequence of events coming from the application instance or resource being monitored. For example, a log stream may be associated with an Apache access log on a specific host. When you no longer need a log stream, you can delete it using the aws logs delete-log-stream command.
Unfortunately what you want is not possible at the moment. Not sure what exactly is your use-case but you can filter the logs streams using time, so separating them is not really necessary. See start-time and end-time in filter-log-events
You might want to define the following awslog driver options to get a better stream name.
awslogs-stream-prefix see docs
Ive been looking at AWS Kinesis analytics for some time now, and I struggle to make a following scenario work:
Say you have a Firehose that is connected to Kinesis Analytics as input stream.
This firehose is meant to output data every 60s.
I write data into this Firehose each 2s or so like:
{"value":1}.
Where the 1 is a random integer (can be 1, 5, 32 and so on).
How can I write my analytics to say find average value of all values reported in last 60s and pass it to a lambda?
I tried using various windows - however the Analytics seems to output the data every few seconds, instead of once 60s.
please have a look into the following links which have detailed examples how to achieve this.
https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-avg.html
https://docs.aws.amazon.com/kinesisanalytics/latest/sqlref/sql-reference-windowed-aggregation-stream.html
I have a Lambda function and its logs in Cloudwatch (Log group and Log Stream). Is it possible to filter (in Cloudwatch Management Console) all logs that contain "error"? For example logs containing "Process exited before completing request".
In Log Groups there is a button "Search Events". You must click on it first.
Then it "changes" to "Filter Streams":
Now you should just type your filter and select the beginning date-time.
So this is kind of a side issue, but it was relevant for us. (I posted this to another answer on StackOverflow but thought it would be relevant to this conversation too)
We've noticed that tailing and searching logs gets really slow after a log group has a lot of Log Streams in it, like when an AWS Lambda Function has had a lot of invocations. This is because "tail" type utilities and searching need to connect to each log stream to run. Log Events get expired and deleted due to the policy you set on the Log Group itself, but the Log Streams never get cleaned up. I made a few little utility scripts to help with that:
https://github.com/four43/aws-cloudwatch-log-clean
Hopefully that save you some agony over waiting for those logs to get searched.
You can also use CloudWatch Insights (https://aws.amazon.com/about-aws/whats-new/2018/11/announcing-amazon-cloudwatch-logs-insights-fast-interactive-log-analytics/) which is an AWS extension to CloudWatch logs that gives a pretty powerful query and analytics tool. However it can be slow. Some of my queries take up to a minute. Okay, if you really need that data.
You could also use a tool I created called SenseLogs. It downloads CloudWatch data to your browser where you can do queries like you ask about. You can use either full text and search for "error" or if your log data is structured (JSON), you can use a Javascript like expression language to filter by field, eg:
error == 'critical'
Posting an update as CloudWatch has changed since 2016:
In the Log Groups there is a Search all button for a full-text search
Then just type your search: