fluentd filter regex pattern gives an invalid strptime format error - regex

I am trying to use the regex filter to parse my log
My regex expression and sample string are as follows
Regex:
/^(?<logtime>[^\|]*)\S +(?<cpu-usr>\S+) +(?<cpu-sys>\S+) +(?<cpu-idl>\S+) +(?<cpu-wai>\S+) +(?<cpu-hiq>\S+) +(?<cpu-siq>[^|]+)\S +(?<mem-used>\S+) +(?<mem-buff>\S+) +(?<mem-cach>\S+) +(?<mem-free>\S+)/
Sample string:
07-07 17:18:12| 0 0 100 0 0 0| 197M 48.9M 182M 546M
So I have written the following filter into my fluent.conf file
#filter dstat log into fields.
#----system---- ----total-cpu-usage---- ------memory-usage-----
# date/time |usr sys idl wai hiq siq| used buff cach free
<filter input.dstat>
#type parser
format /^(?<logtime>[^\|]*)\S +(?<cpu-usr>\S+) +(?<cpu-sys>\S+) +(?<cpu-idl>\S+) +(?<cpu-wai>\S+) +(?<cpu-hiq>\S+) +(?<cpu-siq>[^|]+)\S +(?<mem-used>\S+) +(?<mem-buff>\S+) +(?<mem-cach>\S+) +(?<mem-free>\S+)/
time_format %d-%m %H:%M:%S
time_key logtime
keep_time_key true
key_name log
</filter>
However, I get the following error for this filter.
fluentd_1 | 2017-07-07 05:42:36 +0000 [warn]: invalid time format: value = 07-0, error_class = ArgumentError, error = invalid strptime format - `%d-%m %H:%M:%S'
I have already checked the regex and time format using http://fluentular.herokuapp.com/ and it seems to be ok according to that. How do I proceed with this? My current assumption is that the error is due to %Y or year being missing from the time format. However my log does not contain the year information.

The problem was that I used '-' in the names instead of '_'
After replacing them, it works fine.

Related

my django project is throwing an value error

my django project is throwing a value error
the format of the date is correct but its showing that it does not match the error is :
time data 'Sept. 19, 2022' does not match format '%b. %d, %Y'
can anyone help me with this ?
just change sept to sep because DateTime uses normal months' names.

Not able to match the regex

I need to write the regex to fetch the details from the following data
Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) Avg(us) Std_dev Geo_dist
Period 5 145443 29088 22.4 37006 352 116302 6600 7692.04 4003.72
Global 10 281537 28153 23.2 41800 281 120023 6797 7564.64 4212.93
The above is the log which i get from a log file
I have tried writing the reg ex to get the details in the table format but could not get.
Below is the reg ex which i tried.
Type[\s+\S+].+\n(?<time>[\d+\S+\s+]+)[\s+\S+].*Period
When it comes to Period keyword the regex fails
If for some reason RichG's suggestion of using multikv doesn't work, the following should:
| rex field=_raw "(?<type>\w+)\s+(?<time>[\d\.]+)\s+(?<ops>[\d\.]+)\s+(?<tps>[\d\.]+)\s+(?<net>[\d\.]+)\s+(?<get_miss>[\d\.]+)\s+(?<min>[\d\.]+)\s+(?<max>[\d\.]+)\s+(?<avg>[\d\.]+)\s+(?<std_dev>[\d\.]+)\s+(?<geo_dist>[\d\.]+)"
Where is your data coming from?

Snowflake - getting 'Error parsing JSON' while using the Copy command from S3 to snowflake

i'm trying to copy gz files from my S3 directory to Snowflake.
i created a table in snowflake (notice that the 'extra' field is defined as 'Variant')
CREATE TABLE accesslog
(
loghash VARCHAR(32) NOT NULL,
logdatetime TIMESTAMP,
ip VARCHAR(15),
country VARCHAR(2),
querystring VARCHAR(2000),
version VARCHAR(15),
partner INTEGER,
name VARCHAR(100),
countervalue DOUBLE PRECISION,
username VARCHAR(50),
gamesessionid VARCHAR(36),
gameid INTEGER,
ingameid INTEGER,
machineuid VARCHAR(36),
extra variant,
ingame_window_name VARCHAR(2000),
extension_id VARCHAR(50)
);
i used this copy command in snowflake:
copy INTO accesslog
FROM s3://XXX
pattern='.*cds_201911.*'
CREDENTIALS = (
aws_key_id='XXX',
aws_secret_key='XXX')
FILE_FORMAT=(
error_on_column_count_mismatch=false
FIELD_OPTIONALLY_ENCLOSED_BY = '"'
TYPE = CSV
COMPRESSION = GZIP
FIELD_DELIMITER = '\t'
)
ON_ERROR = CONTINUE
I run it, and got this result (i got many error lines, this is an example to one)
snowflake result
snowflake result -more
a17589e44ae66ffb0a12360beab5ac12 2019-11-01 00:08:39 155.4.208.0 SE 0.136.0 3337 game_process_detected 0 OW_287d4ea0-4892-4814-b2a8-3a5703ae68f3 e9464ba4c9374275991f15e5ed7add13 765 19f030d4-f85f-4b85-9f12-6db9360d7fcc [{"Name":"file","Value":"wowvoiceproxy.exe"},{"Name":"folder","Value":"C:\\Program Files (x86)\\World of Warcraft\\_retail_\\Utils\\WowVoiceProxy.exe"}]
can you please tell me what cause this error?
thanks!
I'm guessing;
The 'Error parsing JSON' is certainly related to the extra variant field.
The JSON looks fine, but there are potential problems with the backslashes \.
If you look at the successfully loaded lines, have the backslashes been removed?
This can (maybe) happen if you have STAGE settings involving escape characters.
The \\Utils substring in the Windows path value can then trigger a Unicode decode error, eg.
Error parsing JSON: hex digit is expected in \U???????? escape sequence, pos 123
UPDATE:
It turns out you have to turn off escape char processing by adding the following to the FILE_FORMAT:
ESCAPE_UNENCLOSED_FIELD = NONE
The alternative is to doublequote fields or to doubly escape backslash, eg. C:\\\\Program Files.

RegEx to find numbers in a string using PowerShell

Imagine having 500 string like this with different dates:
The certificate has expired on 02/05/2014 15:43:01 UTC.
Given that this is a String and I'am using powershell. I need to treat the date (02/05/2014) as an object, so I can use operatators (-lt -gt).
Is the only way doing this is using RegEx, and in this case - can anyone help me finding the first 6 numbers (which change every time) using regEx.
>$regexStr = "(?<date>\d{2}\/\d{2}\/\d{4})"
>$testStr = "The certificate has expired on 02/05/2014 15:43:01 UTC."
>$testStr -match $regexStr
# $Matches will contain the regex group called "date"
>$Matches.date
02/05/2014
>$date = Get-Date ($Matches.date)
>$date
Wednesday, February 5, 2014 12:00:00 AM
If you need to parse the date string with another format you can do:
>$dateObj = [datetime]::ParseExact($Matches.date,”dd/MM/yyyy”,$null)
>$dateObj.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True DateTime System.ValueType
Hope that helps

How to change time format in NetBeans template

I have a very specific C++ project, and I use a NetBeans.
Reason for it is because we need to have a specific timestamps, and I found NetBeans templates a great tool for inserting an automatic header with all the relevant stuff.
I manage set everything up nicely, but I can't figure out how to set up the time format in the header template.
Currently it shows this:
Created on April 6, 2017, 2:18 PM
But since I work in Central Europe, I need a 24h hour format so I could have something like
Created on 06.04.2017. at 14:18
I found on how to change a date format here, but it doesn't work for times for some reason.
I even tried with using FreeMaker's template language reference, so I created a variable time that looks like this:
<#assign dateTime = .now>
<#assign time = dateTime?time>
${time?iso("Europe/Zagreb")}
But it still didn't change anything.
Now my template looks like this:
// -*- C++ -*-
<#assign licenseFirst = "/*">
<#assign licensePrefix = " * ">
<#assign licenseLast = " */">
<#assign aDateTime = .now>
<#assign time = aDateTime?time>
<#include "${project.licensePath}">
/*
* File: ${NAME}.${EXTENSION}
* Author: ${user}
*
* Created on ${DATE} at ${time?iso("Europe/Zagreb")}
*/
#ifndef ${GUARD_NAME}
#define ${GUARD_NAME}
#endif /* ${GUARD_NAME} */
Is this possible to be changed at all, and how?
Any help is appreciated.
In your question you link a resource (THANKS for that!!!) suggesting the following for dates:
${date?date?string("dd.MM.yyyy")}
I tried the same for time and it works well:
${time?time?string("HH.mm.ss")}
BTW I also tried producing some errors and got some nice error messages stating what NB expects and what it gets pretty clearly:
${date?time?string("hh.mm.ss")}
${time?date?string("hh.mm.ss")}
${time?datetime?string("hh.mm.ss")}
${date?datetime?string("hh.mm.ss")}
produced:
Error: on line 20, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "11-Jan-2018". The expected format was: "HH:mm:ss".
Error: on line 21, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "13:40:27". The expected format was: "dd-MMM-yyyy".
Error: on line 22, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "13:40:27". The expected format was: "dd-MMM-yyyy HH:mm:ss".
Error: on line 23, column 6 in Templates/Classes/Class.java
The string doesn't match the expected date/time format. The string to parse was: "11-Jan-2018". The expected format was: "dd-MMM-yyyy HH:mm:ss".