I'm trying to create an AWS dashboard visualization that displays the counts of cache hits vs. misses over a period of time. To do this, I'm setting up a log type dashboard with an insights query on the log. To be as simple as possible, my log is either:
{"cache.hit", true} or {"cache.hit", false}.
I would like for my dashboard to track both possibilities on the same graph, but it seems like I can't without breaking my log up into distinct rows for these values. For example, if my logs were simply:
{"cache.hit.true", true} or {"cache.hit.false", true}, then I could create 2 separate graphs to track these values independently in the dashboard, but that's not as clean.
To get them on one dash, I've tried this, but all it does is display the two fields, and the values for both display fields are the same, when they definitely shouldn't be:
fields #timestamp, #message, cache.hit as cache_hits
| filter cache_hits IN [0, 1]
| display cache_hits = 0 as in_cache_false
| display cache_hits = 1 as in_cache_true
| stat count (in_cache_true), count(in_cache_false) by bin(30s)
| sort #timestamp desc
| limit 20
This query below extracts out the cache hits and cache misses and then works out the cache hit percentage.
fields #timestamp, #message
| filter #message like /cache.hit/
| fields strcontains(#message, "true") as #CacheHit,
strcontains(#message, "false") as #CacheMiss
| stats sum(#CacheHit) as CacheHits, sum(#CacheMiss) as CacheMisses, sum(#CacheHit) / (sum(#CacheMiss) + sum(#CacheHit)) * 100 as HitPercentage by bin(30s)
| sort #timestamp desc
Related
Hope you're well. I've been trying to put together a CloudWatch Query that returns the first event in each contactId.
I thought I'd add a count stat, and then exclude all events that were equal to or greater than 2. I'm clearly not doing something right though. Although I am being provided with the count, it seems for some reason that the count is excluding other information from the query. The query returns almost no information on the event that it is counting. I'd like the count to be added, and also INCLUDE the information from the query.
Here is the query I am using:
fields #timestamp, #message
| sort number asc
| stats count(ContactId) as number by ContactId
| filter ContactFlowModuleType = 'SetLoggingBehavior' and Parameters.LoggingBehavior = 'Enable'
| fields #message
| display Results, ContactId, #timestamp, ContactFlowModuleType, number
With this query, it says that 'time stamp' is invalid. I believe the stats clause has something to do with it.
I'm looking to determine the sequence of events on a contactId basis, so that I can exclude all logged events after the initial event. For now, I'd just like to see a count on the basis of ContactId, so I can perform the exclusion myself.
Steve
I am trying to generate a graph that will display the success/failure rate of an operation. In my application I am pushing log events in the following format:
[loggingType] loggingMessage.
I want to create a pie chart that shows the ratio of success/failure but its not working. I am running the following:
filter #logStream="RunLogs"
| parse #message "[*] *" as loggingType, loggingMessage
| filter loggingType in ["pass","fail"]
| stats count(loggingType="pass")/count(loggingType="fail") as ratio by bin(12w)
It seems like the condition inside count does not work and grabs everything. It returns 1 every time :(
I came across a similar scenario; but, super weirdly I believe, if you change the query to use sum instead of count it works. Not sure why AWS query execution interprets in this way.
filter #logStream="RunLogs"
| parse #message "[*] *" as loggingType, loggingMessage
| filter loggingType in ["pass","fail"]
| stats sum(loggingType="pass")/sum(loggingType="fail") as ratio by bin(12w)
I'm trying to create a CloudWatch Insights query for Amazon Connect that will give me call counts by date. I'm able to get the number of log messages by date, however, I need to only count unique ContactId's. The query I have has many duplicated ContactId's since each time Connect logs to CloudWatch, it uses ContactId to tie all of the events related to a contact together. Is there a way to modify this query to only show the count of the unique ContactId?
filter #message like /ContactId/
| stats count(*) as callCount by toMillis(datefloor(1d))
| sort callCount desc
Embarassingly enough, almost immediately after posting this, I found my answer. count_distinct() gets me what I needed.
filter #message like /ContactId/
| stats count_distinct(ContactId) as callCount by toMillis(datefloor(1d))
| sort callCount desc
I have messages which are like below, the following message is one of the messages (have so many JSON formats which are not at all related to this)
request body to the server {'sender': '65ddd20eac244AAe619383e4d8cb558834', 'message': 'hello'}
I would like to group of these messages based on sender (alphanumeric value) which is enclosed in JSON.
CloudWatch Logs Insights query:
fields #message |
filter #message like 'request body to the server' |
parse #message "'sender': '*', 'message'" as sender |
stats count(*) by sender
Query results:
-------------------------------------------------
| sender | count(*) |
|------------------------------------|----------|
| 65ddd20eac244AAe619383e4d8cb558834 | 4 |
| 55ddd20eac244AAe619383e4d8cb558834 | 3 |
-------------------------------------------------
Screenshot:
you can use filter.
fields #timestamp, #message
| filter #message like "65ddd20eac244AAe619383e4d8cb558834"
| sort #timestamp desc
| limit 20
it will filter all the messages limit to 20 that send by 65ddd20eac244AAe619383e4d8cb558834.
update:
suppose the JSON log formate is this
{
"sender": "65ddd20eac244AAe619383e4d8cb558835",
"message": "Hi"
}
Now I want to count number of messages from 65ddd20eac244AAe619383e4d8cb558835
how many messages are coming from each user?
so simple you can run the query
stats count(sender) by sender |
# To filter only message the contain sender, to avoid lambda default logs
filter #message like "sender"
if you want to see messages as well then modify the query a bit
stats count(*) by sender, message |
filter #message like "sender"
Here #message refers to whole to index where message refer to the JSON object message.
count_distinct
Returns the number of unique values for the field. If the field has
very high cardinality (contains many unique values), the value
returned by count_distinct is just an approximation.
how many distinct users in the selected interval?
It will list distinct users in 3hr of interval
stats count_distinct(sender) as distinct_sender by bin(3hr) as interval
I typically run a query like
fields #timestamp, #message
| filter #message like /ERROR/
| sort #timestamp desc
| limit 20
Is there any way to get additional lines of context around the messages containing "ERROR"? Similar to the A, B, and C flags with grep?
Example
For example, if I have a given log with the following lines
DEBUG Line 1
DEBUG Line 2
ERROR message
DEBUG Line 3
DEBUG Line 4
Currently I get the following result
ERROR message
But I would like to get more context lines like
DEBUG Line 2
ERROR message
DEBUG Line 3
with the option to get more lines of context if I want.
You can actually query the #logStream as well, which in the results will be a link to the exact spot in the respective log stream of the match:
fields #timestamp, #message, #logStream
| filter #message like /ERROR/
| sort #timestamp desc
| limit 20
That will give you a column similar to the right-most one in this screenshot:
Clicking the link to the right will take you to and highlight the matching log line. I like to open this in a new tab and look around the highlighted line for context.
I found that the most useful solution is to do your query and search for errors and get the request id from the "requestId" field and open up a second browser tab. In the second tab perform a search on that request id.
Example:
fields #timestamp, #message
| filter #requestId like /fcd09029-0e22-4f57-826e-a64ccb385330/
| sort #timestamp asc
| limit 500
With the above query you get all the log messages in the correct order for the request where the error occurred. This is an example that works out of the box with lambda. But if you push logs to CloudWatch in a different way and there is no requestId i would suggest creating a requestId per request or another identifier that is more useful for you use case and push that with your log event.