I have a query to retrieve how much time an specific 'event' takes to finish:
fields #timestamp, #message
| parse #message "[Id: *] *" as eventID, loggingMessage
| stats sortsFirst(#timestamp) as date1, sortsLast(#timestamp) as date2 by eventID
this returns a table like
I can do things like | display (date2-date1) to make some calculations but what I would really like to do is to group all of them and calculate the avg(date2-date1). So only one result should appear.
I've tried what other posts recommend but
| stats sortsFirst(#timestamp) as date1, sortsLast(#timestamp) as date2 by eventID, avg(date2-date1)
Results in bad syntax due to the 'by eventID'. But if I remove this, my query is not being grouped by eventID.
How could I get around this?
Related
fields #timestamp, #message
| parse durationMs /(?<duration>[\d]+ )/
| parse message /(GET \/[^\s]+ [\d]+ )(?<responseTime>[\d]+)/
| display #timestamp, duration, responseTime
| sort #timestamp desc
This query works for me and fetches the values. The query is currently parsing the durationMs field and getting the value into duration field. Also parsing message field and getting the value into responseTime field.
I am looking for a way to parse durationMs and message fields and get the value into only one field. Is this possible? Please help.
coalesce function did the job for me.
fields #timestamp, #message
| parse durationMs /(?<duration>[\d]+ )/
| parse message /(GET \/[^\s]+ [\d]+ )(?<responseTime>[\d]+)/
| display #timestamp, coalesce(duration, responseTime) as response_time
| sort #timestamp desc
We want to find the missed utterance rate per day from Lex logs.
For example:
Day 1 - 10 total utterances, 1 missed utterance
Day 2 - 20 total utterances, 4 missed utterance
...
We want to be able to plot (missed utterances/total utterances x 100) per day (essentially, %) for one week, however we also need to include Lambda exceptions as part of our "missed utterances" count.
How do we calculate the total & missed utterance count and then obtain a %?
Is this possible in CloudWatch Insight Logs?
Expectd output is a graph for 7 days that has the percentage of missed utterances+exceptions to total utterances for the day.
<date 1> 1%
<date 2> 4%
...
One query we tried is:
fields #message
| filter #message like /Exception/ or missedUtterance=1
| stats count(*) as exceptionCount, count(#message) as messageCount by bin(1d)
| display exceptionCount, (exceptionCount/messageCount) * 100
| sort #timestamp desc
This is unfortunately not possible to do within CloudWatch Log Insights as you would need to have 2 filter & 2 stats commands.
One filter would be used for getting the total count & another for getting the exception + missed utterance count.
While you can filter after one another, you can't get the counts of the result of each filter as 2 stats commands are not supported from within Log Insights (yet).
The most you can do within CloudWatch is to create a dashboard (or 2 Log Insights) with the below queries and calculate the percentage yourself:
fields #message
| stats count(*) as totalUtteranceCount by bin(1d)
fields #message
| filter #message like /Exception/ or missedUtterance = 1
| stats count(*) as exceptionAndMissedUtteranceCount by bin(1d)
In an enterprise chatbot project that I was an engineer on, I configured logs to be exported to ElasticSearch (OpenSearch in AWS Console), which then opened a whole new world of data analysis & gave me the ability to run statistics like the above.
If this is a must, I would look to implementing a similar solution until AWS improves CloudWatch Log Insights or provides this statistic within Amazon Lex itself.
In the long run, I would go with the first option however Log Insights is not meant to be a full-blown data analysis tool & you'll need to carry out much more analysis on your data (missed utterances, intents etc.) anyway in order to be able to improve your bot.
Hopefully, something like this query works in the future!
fields #message
| stats count(*) as totalUtteranceCount by bin(1d)
| filter #message like /Exception/ or missedUtterance = 1
| stats count(*) as exceptionAndMissedUtteranceCount by bin(1d)
| display (exceptionAndMissedUtteranceCount/totalUtteranceCount) * 100
| sort #timestamp desc
We could get it working using the below query:
fields strcontains(#message, 'Error') as ErrorMessage
|fields strcontains(#message, '"missedUtterance":true') as #missedUtterance
| stats sum(ErrorMessage) as ErrorMessagCount , sum(missedUtterance) as missedCount,
count(#message) as messageCount , (((ErrorMessagCount) + (missedCount)) /messageCount * 100) by bin(1d)
Here, we are using strcontains instead of parse because if there are no missed utterance on a particular day, the calculation (ErrorMesageCount + missedCount)/messageCount * 100 was empty.
Answer is like:
I currently use two different cloud watch log insights queries one to get total request count and the other to get total error count. Below are the queries:
Total count:
fields #timestamp, #message
| filter #message like /reply.*MyAPI/
|parse #message '"reqID":*' as reqID
| stats count_distinct(reqID) as request_count by bin(1h) as hour
** Error count**
fields #timestamp, #message
| filter #message like /reply.*MyAPI.*Exception/
|parse #message '"reqID":*' as reqID
| stats count_distinct(reqID) as request_count by bin(1h) as hour
However I would like to calculate both total request counts and error request count in each bin and calculate error rates for each bin (error count/total request count) if possible with a single query. How would I go about this?
I have a log file which contains playerId values, some players have multiple entries in the file. I want to get an exact distinct count of to unique players, regardless of if they have 1 or multiple entries in the log file.
Using the query below it scans 497 records and finds 346 unique rows (346 is the number I want)
Query:
fields #timestamp, #message
| sort #timestamp desc
| filter #message like /(playerId)/
| parse #message "\"playerId\": \"*\"" as playerId
| stats count(playerId) as CT by playerId
If I change my query to use count_distinct instead, I get exactly what I want. Example below:
fields #timestamp, #message
| sort #timestamp desc
| filter #message like /(playerId)/
| parse #message "\"playerId\": \"*\"" as playerId
| stats count_distinct(playerId) as CT
The problem with count_distinct however is that as the query expands to a larger timeframe/more records the number of entries get into the thousands, and tens of thousands. This presents an issue as the numbers become approximations, due to the nature of Insights count_distinct behaviour...
"Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by count_distinct is just an approximation.".
Docs: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html
This is not acceptable, as I require exact numbers. Playing with the query a little, and sticking with count(), not count_distinct() I believe is the answer, however I've not been able to come to a single number... Examples which do not work... Any thoughts?
Ex 1:
fields #timestamp, #message
| sort #timestamp desc
| filter #message like /(playerId)/
| parse #message "\"playerId\": \"*\"" as playerId
| stats count(playerId) as CT by playerId
| stats count(*)
We are having trouble understanding the query.
To be clear, I'm looking for an exact count to be returned in a single row showing the number.
What if we introduce a dummy field that's hardcoded to "1"? The idea is to retrieve its min value so that it stays as a "1" even if the same playerId occurs more than once. And then we sum this field.
The log entry might look like this:
[1]"playerId": "1b45b168-00ed-42fe-a977-a8553440fe1a"
Query:
fields #timestamp, #message
| sort #timestamp desc
| filter #message like /(playerId)/
| parse #message "[*]\"playerId\": \"*\"" as dummyValue, playerId
| stats sum(min(dummyValue)) by playerId as CT
References used:
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_AggregationQuery.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CountOccurrencesExample.html
I have a lot of AWS Lambda logs which I need to query to find the relevant log stream name,
I am logging a particular string in the logs,
Which I need to do a like or exact query on.
The log format is something like this -
Request ID => 572bf6d2-e3ff-45dc-bf7d-c9c858dd5ccd
I am able to query the logs without the UUID string -
But if I mention the UUID in the query, it does not show results -
Queries used -
fields #timestamp, #message
| filter #message like /Request ID =>/
| sort #timestamp desc
| limit 20
fields #timestamp, #message
| filter #message like /Request ID => 572bf6d2-e3ff-45dc-bf7d-c9c858dd5ccd/
| sort #timestamp desc
| limit 20
Have you tried adding an additional filter on the message field to your first query to further narrow your results?
fields #timestamp, #message
| filter #message like /Request ID =>/
| filter #message like /572bf6d2-e3ff-45dc-bf7d-c9c858dd5ccd/
| sort #timestamp desc
| limit 20
Alternatively if all of your logs follow the same format you could use the parse keyword to split out your UUID field and search on it with something like
fields #timestamp, #message
| parse #message "* * Request ID => *" as datetime, someid, requestuuid
| filter uuid like /572bf6d2-e3ff-45dc-bf7d-c9c858dd5ccd/
| sort #timestamp desc
| limit 20
Also try widening your relative time range at the top right of the query, just in case the request you're looking for has dropped outside of the 1hr range since attempting the first query.
instead of using two like filters like in accepted answer, I would suggest using the in operator as follows. This way your code is shorter and cleaner.
fields #timestamp, #message
| filter #message in ["Request ID =>", "572bf6d2-e3ff-45dc-bf7d-c9c858dd5ccd"]
| sort #timestamp desc
| limit 20