I have filtered my logs to get ERROR like so
filter #message like /ERROR/
My logs look like so:
"2023-01-01 06:01:02.010 ERROR <details of the error up to 150 chars>
I wish to extract the output and group similar errors like below. I have mentioned up to 50 chars as the details in similar errors match for the first 50(or so) characters. How to do it in the logs insights query? Or using Python in case of boto3.
ERROR Type Count
ERROR <details up to 50 chars> 400 <- Error type x
ERROR <details up to 50 chars> 230 <- Error type y
Related
I have some logs comming into aws log group in messages(field), I want to write log insights query to get real time count of incomming logs,
But I have unique UUIDs in each log that needs to be replaced by '*' so that. similar logs can be grouped together for getting correct count.
I want the UUIDs in these logs to be replaced by '*' .
Sample logs in system.
messages(field)
v2/documents/0003cfad-c6ce-46f1-b617-9efd95d79b52
v2/documents/0003cfad-c6ce-46f1-b617-9efd95d79b52/status/004083f4-467e-4d25-9d71-baf7087acb2b/
v2/004083f4-467e-4d25-9d71-baf7087acb2b
v2/documents/004083f4-467e-4d25-9d71-baf7087acb2b/status
v2/0063891d-6822-493e-a650-31cc57989310/create/004083f4-467e-4d25-9d71-baf7087acb2b/
v2/documents/00ee9bb9-e21b-44c7-b437-d0c7dd1057f8
v2/documents/00ee9bb9-e21b-44c7-b437-d0c7dd1057f8/status/00ee9bb9-e21b-44c7-b437-d0c7dd1057f8/
v2/documents/00fcce48-1768-4e89-a58b-e699be061ae4/delete/00ee9bb9-e21b-44c7-b437-d0c7dd1057f8/
Expected result after replacement-
messages (field)
v2/documents/*
v2/documents/*/status/*/
v2/*
v2/documents/*/status
v2/*/create/*/
v2/documents/*
v2/documents/*/status/*/
v2/documents/*/delete/*/
Note:- All UUIDs can be unique but follow same pattern of ---
[varchar(8)-varchar(4)-varchar(4)-varchar(4)-varchar(12)]
I have tried to get desiered result using below query. This query uses parse function and relies completely on '/' as delimiter for creating new column, but this is not helping as all the logs do not follow same pattern.
Ps: each of these is error log with 503 status code.
fields #message
| filter #message like '" 503'
| parse #message "* * * * *" as a, b, c, uri, e
| parse uri "/*/*/*/*/*/*/*" as f1,f2,f3,f4,f5,f6,f7
| parse f7 "*?" as f8
| parse f7 "*/" as f9
| stats count(*)as Count by f5 as API, f9 as Call, f8 as call
Thanks in advance!!
I need to write the regex to fetch the details from the following data
Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) Avg(us) Std_dev Geo_dist
Period 5 145443 29088 22.4 37006 352 116302 6600 7692.04 4003.72
Global 10 281537 28153 23.2 41800 281 120023 6797 7564.64 4212.93
The above is the log which i get from a log file
I have tried writing the reg ex to get the details in the table format but could not get.
Below is the reg ex which i tried.
Type[\s+\S+].+\n(?<time>[\d+\S+\s+]+)[\s+\S+].*Period
When it comes to Period keyword the regex fails
If for some reason RichG's suggestion of using multikv doesn't work, the following should:
| rex field=_raw "(?<type>\w+)\s+(?<time>[\d\.]+)\s+(?<ops>[\d\.]+)\s+(?<tps>[\d\.]+)\s+(?<net>[\d\.]+)\s+(?<get_miss>[\d\.]+)\s+(?<min>[\d\.]+)\s+(?<max>[\d\.]+)\s+(?<avg>[\d\.]+)\s+(?<std_dev>[\d\.]+)\s+(?<geo_dist>[\d\.]+)"
Where is your data coming from?
i am working on Oracle to Snowflake migration.
while migrating oracle window functions to snowflake getting below error, could you let me know, alternate way for oracle function in snowflake.
SELECT
COL1,
COL2, ...,
SUM(SUM(TAB1.COL1)) OVER (PARTITION BY
TAB1.COL2,
TAB1.COL3,
TAB1.COL4,
TAB1.COL5,
TAB1.COL6,
TAB1.COL7,
TAB1.COL8,
TAB1.COL9,
TAB1.COL10,
ORDER BY MAX(CALENDAR_TAB.DATE_COLUMN) RANGE BETWEEN INTERVAL '21' DAY PRECEDING AND CURRENT ROW)/4 AS COLMN
FROM TAB1,CALENDAR_TAB
JOIN
GROUP BYCOL1,
COL2, ...
Below is the error message:
QL Error [1003] [42000]: SQL compilation error:
syntax error line 75 at position 60 unexpected 'INTERVAL'.
syntax error line 75 at position 78 unexpected 'PRECEDING'.
Per the documentation for Snowflake, here is the syntax:
https://docs.snowflake.com/en/sql-reference/functions-analytic.html#window-syntax-and-usage
slidingFrame ::=
{
ROWS BETWEEN <N> { PRECEDING | FOLLOWING } AND <N> { PRECEDING | FOLLOWING }
| ROWS BETWEEN UNBOUNDED PRECEDING AND <N> { PRECEDING | FOLLOWING }
| ROWS BETWEEN <N> { PRECEDING | FOLLOWING } AND UNBOUNDED FOLLOWING
}
It might not like the INTERVAL and the quoted number.
The Window frame document is a good place to start.
If I read the Oracle syntax correctly, the window frame your are using for the MAX is value based aka (interval '21' day) which Snowflake does not support, it only supports N rows based logic. If you have 1 row per day, and always 1 row, then you can use the row count logic, but otherwise this is not supported.
Which means you to join back to your own data tables and apply the prior time filter on the join.
I am trying to use the regex filter to parse my log
My regex expression and sample string are as follows
Regex:
/^(?<logtime>[^\|]*)\S +(?<cpu-usr>\S+) +(?<cpu-sys>\S+) +(?<cpu-idl>\S+) +(?<cpu-wai>\S+) +(?<cpu-hiq>\S+) +(?<cpu-siq>[^|]+)\S +(?<mem-used>\S+) +(?<mem-buff>\S+) +(?<mem-cach>\S+) +(?<mem-free>\S+)/
Sample string:
07-07 17:18:12| 0 0 100 0 0 0| 197M 48.9M 182M 546M
So I have written the following filter into my fluent.conf file
#filter dstat log into fields.
#----system---- ----total-cpu-usage---- ------memory-usage-----
# date/time |usr sys idl wai hiq siq| used buff cach free
<filter input.dstat>
#type parser
format /^(?<logtime>[^\|]*)\S +(?<cpu-usr>\S+) +(?<cpu-sys>\S+) +(?<cpu-idl>\S+) +(?<cpu-wai>\S+) +(?<cpu-hiq>\S+) +(?<cpu-siq>[^|]+)\S +(?<mem-used>\S+) +(?<mem-buff>\S+) +(?<mem-cach>\S+) +(?<mem-free>\S+)/
time_format %d-%m %H:%M:%S
time_key logtime
keep_time_key true
key_name log
</filter>
However, I get the following error for this filter.
fluentd_1 | 2017-07-07 05:42:36 +0000 [warn]: invalid time format: value = 07-0, error_class = ArgumentError, error = invalid strptime format - `%d-%m %H:%M:%S'
I have already checked the regex and time format using http://fluentular.herokuapp.com/ and it seems to be ok according to that. How do I proceed with this? My current assumption is that the error is due to %Y or year being missing from the time format. However my log does not contain the year information.
The problem was that I used '-' in the names instead of '_'
After replacing them, it works fine.
I have a very specific C++ project, and I use a NetBeans.
Reason for it is because we need to have a specific timestamps, and I found NetBeans templates a great tool for inserting an automatic header with all the relevant stuff.
I manage set everything up nicely, but I can't figure out how to set up the time format in the header template.
Currently it shows this:
Created on April 6, 2017, 2:18 PM
But since I work in Central Europe, I need a 24h hour format so I could have something like
Created on 06.04.2017. at 14:18
I found on how to change a date format here, but it doesn't work for times for some reason.
I even tried with using FreeMaker's template language reference, so I created a variable time that looks like this:
<#assign dateTime = .now>
<#assign time = dateTime?time>
${time?iso("Europe/Zagreb")}
But it still didn't change anything.
Now my template looks like this:
// -*- C++ -*-
<#assign licenseFirst = "/*">
<#assign licensePrefix = " * ">
<#assign licenseLast = " */">
<#assign aDateTime = .now>
<#assign time = aDateTime?time>
<#include "${project.licensePath}">
/*
* File: ${NAME}.${EXTENSION}
* Author: ${user}
*
* Created on ${DATE} at ${time?iso("Europe/Zagreb")}
*/
#ifndef ${GUARD_NAME}
#define ${GUARD_NAME}
#endif /* ${GUARD_NAME} */
Is this possible to be changed at all, and how?
Any help is appreciated.
In your question you link a resource (THANKS for that!!!) suggesting the following for dates:
${date?date?string("dd.MM.yyyy")}
I tried the same for time and it works well:
${time?time?string("HH.mm.ss")}
BTW I also tried producing some errors and got some nice error messages stating what NB expects and what it gets pretty clearly:
${date?time?string("hh.mm.ss")}
${time?date?string("hh.mm.ss")}
${time?datetime?string("hh.mm.ss")}
${date?datetime?string("hh.mm.ss")}
produced:
Error: on line 20, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "11-Jan-2018". The expected format was: "HH:mm:ss".
Error: on line 21, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "13:40:27". The expected format was: "dd-MMM-yyyy".
Error: on line 22, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "13:40:27". The expected format was: "dd-MMM-yyyy HH:mm:ss".
Error: on line 23, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "11-Jan-2018". The expected format was: "dd-MMM-yyyy HH:mm:ss".