I am wanting to parse some logs via Splunk that are in one event but multiple lines.
Value1: 1000 MS
Value2: 300 MS
Value3: 1500 MS
I am having a hard time looking through Splunk documentation on how to extract these fields from one event.
What I have so far only parses the first field and adding more fields does not seem to work.
rex field=_raw "(?<object1>\w+): (?<totalms1>\d+) MS" | table object1, totalms1
Related
I am creating a line chart in a CloudWatch dashboard. I can create a line representing the frequency with which one string appears the logs, using a filter. But, I don't know how to create two or more such lines.
After selecting a log group, I run this query:
filter name = "first log string"
| stats count(*) as firstString by bin(1hour)
This generates counts of results that, in the Visiulization tab, are displayed as a line chart with a single line.
Now I want to add another line representing "second log string" on the chart. I assume I will have to modify the filter in some way, or add a second filter.
Here are some things that don't work:
adding a second name to the filter
trying filter #message or #name as this documentation suggests
simply pasting in a modified duplicate of the query
Further, I seem to lack documentation that explains how filter is supposed to work. Search engines keep sending me back to this Filter And Pattern Syntax AWS document which dosen't appear to give any actual examples using filter.
To answer my own question, I had to use an array in my filter. I also used the sum() function instead of count().
filter name in ["first log string", "second log string"]
| fields name = "first log string" as #first_string, name = "second log string" as #second_string
| stats sum(#first_string) as first_string, sum(#second_string) as second_string by bin(1hour)
Now I have a line chart with two lines representing the frequency of two logged items
I'm running a CloudWatch log insights query on a single log stream that corresponds to a single Python AWS Lambda function. This function logs a unique line corresponding to the key in s3 that it is processing. It logs this line once at the beginning of the invocation. The only condition where it won't log this line is if it fails before it even reads the event.
The query is:
parse #message /(?<#unique_key>Processing key: \w+\/[\w=_-]+\/\w+\.\d{4}-\d{2}-\d{2}-\d{2}\.[\w-]+\.\w+\.\w+)/
| filter #message like /Processing key: \w+\/[\w=_-]+\/\w+\.\d{4}-\d{2}-\d{2}-\d{2}\.[\w-]+\.\w+\.\w+/
| stats count(#unique_key) - count_distinct(#unique_key) as #distinct_unique_keys_delta
by datefloor(#timestamp, 1d) as #_datefloor
| sort #_datefloor asc
The two regular expressions in this query will parse the full key of the s3 file being processed. In this particular problem and in general, my understanding is that the count(...) of any quantity minus the count_distinct(...) of the same quantity should always be greater than or equal to zero.
For several of the days in the results, it is a negative number.
I thought I might be misunderstanding the correct usage of datefloor(), so I tried running the following query:
parse #message /(?<#unique_key>Processing key: \w+\/[\w=_-]+\/\w+\.\d{4}-\d{2}-\d{2}-\d{2}\.[\w-]+\.\w+\.\w+)/
| filter #message like /Processing key: \w+\/[\w=_-]+\/\w+\.\d{4}-\d{2}-\d{2}-\d{2}\.[\w-]+\.\w+\.\w+/
| stats count(#unique_key) - count_distinct(#unique_key) as #distinct_unique_keys_delta
The result was -20,347.
At this point the only scenarios I can see are
Something wrong with the code executing the query.
I'm misunderstanding this tool.
I have discovered that the count_distinct function in AWS Log Insights queries doesn't really return a distinct count! As per the documentation
Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by count_distinct is just an approximation.
Apparently I can't just assume that a function returns an accurate result.
The documentation page.
I have two different log groups and am retrieving details in different ways.
Once per:
fields #timestamp, message.event.detail.myIdentifier as placeA.myIdentifier
And once with
fields #timestamp
| parse message.event.Records.0.body '"myIdentifier ": "*"' as placeB.myIdentifier
Is there a way to join these log entries by myIdentifier?
Ideally the goal would be to compare the timestamps of all places per identifier.
So e.g.:
myIdentifier
placeA-timestamp
placeB-timestamp
First
12:00:00
13:00:00
Second
12:05:00
13:05:00
Is there a way to achieve this with cloud watch log insights?
Thanks for your help!
I am trying to generate a graph that will display the success/failure rate of an operation. In my application I am pushing log events in the following format:
[loggingType] loggingMessage.
I want to create a pie chart that shows the ratio of success/failure but its not working. I am running the following:
filter #logStream="RunLogs"
| parse #message "[*] *" as loggingType, loggingMessage
| filter loggingType in ["pass","fail"]
| stats count(loggingType="pass")/count(loggingType="fail") as ratio by bin(12w)
It seems like the condition inside count does not work and grabs everything. It returns 1 every time :(
I came across a similar scenario; but, super weirdly I believe, if you change the query to use sum instead of count it works. Not sure why AWS query execution interprets in this way.
filter #logStream="RunLogs"
| parse #message "[*] *" as loggingType, loggingMessage
| filter loggingType in ["pass","fail"]
| stats sum(loggingType="pass")/sum(loggingType="fail") as ratio by bin(12w)
I typically run a query like
fields #timestamp, #message
| filter #message like /ERROR/
| sort #timestamp desc
| limit 20
Is there any way to get additional lines of context around the messages containing "ERROR"? Similar to the A, B, and C flags with grep?
Example
For example, if I have a given log with the following lines
DEBUG Line 1
DEBUG Line 2
ERROR message
DEBUG Line 3
DEBUG Line 4
Currently I get the following result
ERROR message
But I would like to get more context lines like
DEBUG Line 2
ERROR message
DEBUG Line 3
with the option to get more lines of context if I want.
You can actually query the #logStream as well, which in the results will be a link to the exact spot in the respective log stream of the match:
fields #timestamp, #message, #logStream
| filter #message like /ERROR/
| sort #timestamp desc
| limit 20
That will give you a column similar to the right-most one in this screenshot:
Clicking the link to the right will take you to and highlight the matching log line. I like to open this in a new tab and look around the highlighted line for context.
I found that the most useful solution is to do your query and search for errors and get the request id from the "requestId" field and open up a second browser tab. In the second tab perform a search on that request id.
Example:
fields #timestamp, #message
| filter #requestId like /fcd09029-0e22-4f57-826e-a64ccb385330/
| sort #timestamp asc
| limit 500
With the above query you get all the log messages in the correct order for the request where the error occurred. This is an example that works out of the box with lambda. But if you push logs to CloudWatch in a different way and there is no requestId i would suggest creating a requestId per request or another identifier that is more useful for you use case and push that with your log event.