Okay, so I'm trying to find out if a ticket has been included in my release branch. The tickets are all built out of a project id and an id number, e.g. (PRO-123). I've tried this command:
git log --date=short --format="%h: %ad (%cn) %s" --abbrev-commit --grep='[A-Z]+-[0-9]+'
But it's not returning anything. If I take away the --grep part there's loads of matches to the pattern. For instance:
a6fdcd0: 2016-03-16 (ajfaraday) Merge remote-tracking branch 'origin/develop_5.2_customer' into release_5.2_customer
85d107a: 2016-03-16 (username) Merge pull request #477 from myapp/fix_CST-827_outline_method_in_use_check
6024bda: 2016-03-16 (Andrew Faraday) Merge pull request #473 from myapp/fix_CST-810_soap_container_create_bounds
eec2a61: 2016-03-16 (ajfaraday) added missing stubs
c03b3cb: 2016-03-15 (username) Merge pull request #472 from myapp/fix_CST-490_options_are_clickable_for_user_without_module_admin_rights
728539b: 2016-03-15 (username) Merge pull request #474 from myapp/fix_CST-873_hidden_error_on_pev_validation
4a11dd7: 2016-03-15 (username) Merge pull request #475 from myapp/fix_CST-854_copy_process_version_project_element_values
4a5af44: 2016-03-15 (ajfaraday) CST-854: fixed in-use check for methods
What am I doing wrong?
Okay, I think I've found the problem. It's some minor language difference in regexes (I'm usually writing them in my Ruby code).
For some reason [A-Z]+ wasn't matching but [A-Z]* is working fine. This line does what I wanted:
git log --date=short --format="%h: %ad (%cn) %s" --abbrev-commit --grep="[A-Z]*-[0-9]*"
Related
We are using Fluentbit as as Sidecar container in our ECS fargate Cluster which is running a dotnet application, initially we faced the issue of fluentbit sending the logs in multiline and we solved it using Fluentbit Multilne feature. Now the logs are being sent to Sumologic in Multiple however it is being sent as Json format whereas we just want fluentbit send only the raw log
Logs are currently
{
date:1675120653.269619,
container_id:"xvgbertytyuuyuyu",
container_name:"XXXXXXXXXX",
source:"stdout",
log:"2023-01-30 23:17:33.269Z DEBUG [.NET ThreadPool Worker] Connection.ManagedDbConnection - ComponentInstanceEntityAsync - Executing stored proc: dbo.prcGetComponentInstance"
}
We want only the line
2023-01-30 23:17:33.269Z DEBUG [.NET ThreadPool Worker] Connection.ManagedDbConnection - ComponentInstanceEntityAsync - Executing stored proc: dbo.prcGetComponentInstance
You need to modify Fluent Bit configuration to have the following filters and output configuration:
fluent.conf:
## prepare headers for Sumo Logic
[FILTER]
Name record_modifier
Match *
Record headers.content-type text/plain
## Set headers as headers attribute
[FILTER]
Name nest
Match *
Operation nest
Wildcard headers.*
Nest_under headers
Remove_prefix headers.
[OUTPUT]
Name http
...
# use log key as body
body_key $log
# use headers key as headers
headers_key $headers
That way, you are going to craft HTTP request manually. This is going to send request per log, which is not necessary a good idea. In order to mitigate that you can add the following parser and use it (flush_timeout may need an adjustment):
parsers.conf
# merge everything as one big log
[MULTILINE_PARSER]
name multiline-all
type regex
flush_timeout 500
#
# Regex rules for multiline parsing
# ---------------------------------
#
# configuration hints:
#
# - first state always has the name: start_state
# - every field in the rule must be inside double quotes
#
# rules | state name | regex pattern | next state
# ------|---------------|--------------------------------------------
rule "start_state" ".*" "cont"
rule "cont" ".*" "cont"
fluent.conf:
[INPUT]
name tail
...
multiline.parser multiline-all
I'm trying to parse some logs using split and regexes in powershell
Here's my code :
$string = "Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code. Test 229: Passed Test 260: Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1"
$string -Split '(Test (\d)+:)'
Result :
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
9
Passed
Test 260:
0
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
Expected result:
Starting ChromeDriver 78.0.3904.70Please protect ports used by ChromeDriver and related test frameworks to prevent access by malicious code.
Test 229:
Passed
Test 260:
Failed. Error message: Status: Test case failed. Steps: Navigate to: PurchReqTableListPage (purchreqpreparedbyme) Use the Quick Filter to find records. For example, filter on the Purchase requisition fION()</StackTrace> </Error> Playback results: Tests: 2 Passed: 1 Failed: 1
On this site : https://regexr.com/3c0lf I tried this regex and the groups captured were : Test 260: and Test 229: (which is exactly what I want)
I do not understand where the 0 and the 9 comes from.
Thanks a lot
Those are the last digits of the number. 0 from 26*0* and 9 from 22*9*.
You are seeing those because you've created an additional capturing group by putting parentheses around the digits. Just remove them like so:
$string -Split '(Test \d+:)
You probably don't even need those parentheses either, leaving just
$string -Split 'Test \d+:
I have prepared an alias to get a short log report in git
# excerpt from ~/.gitconfig
[alias]
lg = log --all --oneline --graph --decorate --pretty='%C(auto)%h %Cgreen%ai %C(reset)%C(auto)%s %d'
git lg generates one nice line per commit, but without information on the user:
* 623beff 2016-11-14 14:18:36 +0100 extended plotstyle option and automatic colors
or as screenshot:
But I want to see the initial letters of the committer real name (the full name is sometimes too long) in each line:
* 623beff 2016-11-14 14:18:36 +0100 (J.S.) extended plotstyle option and automatic colors
How can I get this result?
there is a way to do this to get the first letter of the first name, using %<(3,trunc)%cN:
git log --all --oneline --graph --decorate --pretty='%C(auto)%h %Cgreen%ai %C(reset)%C(auto)(%<(3,trunc)%cN) %s %d'
output:
* 8759307 2009-01-15 16:11:48 +0000 (S..) Remove spurious code trying to tag a branch root before the mark was created. (HEAD -> master, origin/master, origin/HEAD)
* 939f999 2008-12-11 13:41:37 +0000 (S..) When just writing output file, do not try to devise lock target with no repository.
We have recently setup AWS logs agent on one of our test servers. Our log files usually contain multi-line events. e.g one of our log event is:
[10-Jun-2016 07:30:16 UTC] SQS Post Response: Array
(
[Status] => 200
[ResponseBody] => <?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2009-02-01/"><SendMessageResult><MessageId>053c7sdf5-1e23-wa9d-99d8-2a0cf9eewe7a</MessageId><MD5OfMessageBody>8e542d2c2a1325a85eeb9sdfwersd58f</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>4esdfr30-c39b-526b-bds2-14e4gju18af</RequestId></ResponseMetadata></SendMessageResponse>
)
The log agent reference documentation says to use 'multi_line_start_pattern' option for such logs. Our AWS Log agent config is as follows:
[httpd_info.log]
file = /var/log/httpd/info.log*
log_stream_name = info.log
initial_position = start_of_file
log_group_name = test.server.name
multi_line_start_pattern = '(\[)+\d{2}-[a-zA-Z]{3}+-\d{4}'
However, the logs agent reporting breaks on aforementioned and similar events. The way it is being reported to CloudWatch Logs is as follows:
Event 1:
[10-Jun-2016 11:21:26 UTC] SQS Post Response: Array
Event 2:
( [Status] => 200 [ResponseBody] => <?xml version="1.0"?><SendMessageResponse xmlns="http://queue.amazonaws.com/doc/2009-02-01/"><SendMessageResult><MessageId>053c7sdf5-1e23-wa9d-99d8-2a0cf9eewe7a</MessageId><MD5OfMessageBody>8e542d2c2a1325a85eeb9sdfwersd58f</MD5OfMessageBody></SendMessageResult><ResponseMetadata><RequestId>4esdfr30-c39b-526b-bds2-14e4gju18af</RequestId></ResponseMetadata></SendMessageResponse>
Event 3:
)
Despite of the fact that its only a single event. Any clue whats going on here?
I think all you need to add is the following to your awslogs.conf
datetime_format = %d-%b-%Y %H:%M:%S UTC
time_zone = UTC
multi_line_start_pattern = {datetime_format}
http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
multi_line_start_pattern
Specifies the pattern for identifying the start of a log message. A log message is made of a line that matches the pattern and any following lines that don't match the pattern. The valid values are regular expression or {datetime_format}. When using {datetime_format}, the datetime_format option should be specified. The default value is ‘^[^\s]' so any line that begins with non-whitespace character closes the previous log message and starts a new log message.
If that datetime format didn't work, you would need to update your regex to actually match your specific datetime. I don't think the one you have listed above actually works for your given format.
You could try this for instance:
[\d{2}-[\w]{3}-\d{4}\s{1}\d{2}:\d{2}:\d{2}\s{1}\w+]
does match
[10-Jun-2016 11:21:26 UTC]
See here: http://www.regexpal.com/?fam=96811
Once completed, issue a restart of the service and check to see if its parsing correctly.
$ sudo service awslogs restart
I'm working with Splunk version 5.0.1.
I want to filter the logs being indexed.
I've added these lines to transforms.conf file:
[setparsing]
REGEX = log: myCompany|\[CRIT\]|\[ERR\]
DEST_KEY = queue
FORMAT = indexQueue
So what I want is to index all the log entries that have one of these strings.
But for some reason only logs with these entries are indexed:
log: myCompany
while log entries with the string "[CRIT]" or "[ERR]" aren't indexed.
What am I missing? Is there something wrong with the regex, because I checked many Perl examples. And that's how we write a regex for log: myCompany or [CRIT] or [ERR].