Regex and config.json -file - regex

I am building an Angular application and trying to figure out the way to write ngsw-config.json -file in order to define rules for service worker.
I assumed that regex would be recognized as regex in configuration file and not interpret as normal characters / text automatically, but it was not so. I have for example following piece of a code:
"name": "authentication",
"urls": [
"/login",
"/.*authentication.*"
],
part .* is not in my understanding recognized as regex (regex meaning in this case that any path that has text "authentication" would fall into this category, right?). This piece of a configuration tries to prevent service worker to take a lead in these two cases, it works with /login, but not with authentication part.
Question:
Can I somehow modify my file to make it recognize regex definitions?

According to the documentation at https://angular.io/guide/service-worker-config
you can use a limited glob format.
I don't know what kind of url you want to match.
Option: If you want to match a url like /foo/bar/authentication/foo2/bar2 you could use:
"name": "authentication",
"urls": [
"/login",
"/**/authentication/**/*"
],
Option: If you want to match a url like /foo/bar/something-authentication-otherthing/foo2/bar2 you could use:
"name": "authentication",
"urls": [
"/login",
"/**/*authentication*/**/*"
],

Related

How to exclude certain files from being piped to CloudWatch?

Given that I have the following files.
/tmp/example.txt
/tmp/example.txt_<date> (e.g. /tmp/example.txt_20230207)
/tmp/example.txt.lck
I want to pipe example.txt and /tmp/example.txt_<date> to CloudWatch and exclude example.txt.lck.
Following Glob syntax (as mentioned in the official document that it uses Glob), "/tmp/example.txt*(!(.lck))" should do the trick. (tested on DigitalOcean's online Glob tool)
However, when applied the configuration, it is not sending any of the above files to CloudWatch.
Is there any other methods or alternative to achieve the result?
I have tested with /tmp/example.txt* and it works, but it is sending all 3 files. So at the very least, I know that my CloudWatch Agent is working.
It is quite likely a mistake in my glob expression or it might not be supported at the moment (as we can see from this issue - https://github.com/aws/amazon-cloudwatch-agent/issues/328, "alternative brackets" (i.e. {}) is not supported, hence chances are I'm using unsupported glob expression too).
I think the syntax should be:
"/tmp/example.txt*[!(.lck)]"
If this doesn't work, you can hack your way around this by excluding "k" as the last character in the string.
"/tmp/example.txt*[!k]"
Or you can include the files separately in the config as follows:
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/tmp/example.txt_20230207",
"log_group_name": "",
"log_stream_name": "",
"timestamp_format": ""
},
{
"file_path": "/tmp/example.txt",
"log_group_name": "",
"log_stream_name": ""
}
]
},

logstash log parsing with regex and grok

Hello I have below logs
12-Apr-2021 16:11:41.078 WARNING [https-jsse-nio2-8443-exec-3] org.apache.catalina.realm.LockOutRealm.filterLockedAccounts An attempt was made to authenticate the locked user [uv19nb]
12-Apr-2021 16:01:01.505 FINE [https-jsse-nio2-8443-exec-8] org.apache.catalina.realm.CombinedRealm.authenticate Failed to authenticate user [uv19nb] with realm [org.apache.catalina.realm.JNDIRealm]
12-Apr-2021 17:12:45.289 FINE [https-jsse-nio2-8443-exec-5] org.apache.catalina.authenticator.FormAuthenticator.doAuthenticate Authentication of 'uv19nb' was successful
I am trying to build a pattern for these for logstash.
I have following
%{MY_DATE_PATTERN:timestamp}\s%{WORD:severity}\s\[%{DATA:thread}\]\s%{NOTSPACE:type_log}
which parses below
{
"timestamp": [
"12-Apr-2021 16:01:01.505"
],
"severity": [
"FINE"
],
"thread": [
"https-jsse-nio2-8443-exec-8"
],
"type_log": [
"org.apache.catalina.realm.CombinedRealm.authenticate"
]
}
and i would like to parse log as 2 parts as the bold ones and the user name what would you advise please?
An attempt was made to authenticate the locked user [uv19nb]
Failed to authenticate user [uv19nb] with realm [org.apache.catalina.realm.JNDIRealm]
Authentication of 'uv19nb' was successful
I have tried using (?<action>[^\[]*) and (?<action>[^']*) but they only capture if the next character is either [ or '.
I need some regex/grok pattern to catch all the sentence until any special character I believe and for user name I need to extract numbers and letters from [] and ''.
Provided the MY_DATE_PATTERN works well for you, you can use
%{MY_DATE_PATTERN:timestamp}\s+%{WORD:severity}\s+\[%{DATA:thread}\]\s+%{NOTSPACE:type_log}\s+(?<action>\w(?:[\w\s]*\w)?)
I added \s+(?<action>\w(?:[\w\s]*\w)?):
\s+ - one or more whitespaces
(?<action>\w(?:[\w\s]*\w)?) - Group "action":
\w - a word char followed with
(?:[\w\s]*\w)? - an optional occurrence of zero or more word and whitespace chars and then an obligatory word char.

grok parse optional field pattern doesn't work

I've got a log like this:
ERROR_MESSAGE:Invalid Credentials,THROTTLED_OUT_REASON:API_LIMIT_EXCEEDED
I'm trying to parse it with grok using grok debugger:
ERROR_MESSAGE:%{GREEDYDATA:errorMassage},THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason}
It works, but sometimes the log comes without THROTTLED_OUT_REASON field.
ERROR_MESSAGE:%{GREEDYDATA:errorMassage}
In that case I tried below code since THROTTLED_OUT_REASON is an optional field.
ERROR_MESSAGE:%{GREEDYDATA:errorMassage}(,THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason})?
So this should work for both cases. The given output for the log with optional field is:
{
"errorMassage": [
[
"Invalid Credentials,THROTTLED_OUT_REASON:API_LIMIT_EXCEEDED"
]
],
"throttledOutReason": [
[
null
]
]
}
But the expected output for the log with optional field:
{
"errorMassage": [
[
"Invalid Credentials"
]
],
"throttledOutReason": [
[
"API_LIMIT_EXCEEDED"
]
]
}
expected output for the log without optional field:
{
"errorMassage": [
[
"Invalid Credentials"
]
],
"throttledOutReason": [
[
null
]
]
}
Can anyone suggest a solution which gives correct output for both type of logs?
Since you use GREEDYDATA it "eats" as much as it can get in order to fill errormessage.
I do not know GROK enough to tell you what alternative defined patterns there are, but you should be able to use a custom pattern:
ERROR_MESSAGE:(?<errorMassage>.*?),THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason}
I got the answer using #Skeeve 's idea.
Here it is for anyone who would come up with a similar question:
I've used custom pattern in order to avoid excess eating of GREEDYDATA (for errorMessage field).
ERROR_MESSAGE:(?<errorMassage>([^,]*)?)(,THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason})?

Issues with regex in Kibana

I am having a hard time using a regex pattern inside Kibana/Elasticsearch version 6.5.4. The field I am searching for has the following mapping:
"field": {
"type": "text",
"analyzer": "custom_analyzer"
},
Regex searches in this field return several hits when requested straight to elasticsearch:
GET /my_index/_search
{
"query": {
"regexp":{
"field": "abc[0-9]{4}"
}
}
}
On the other hand, in Kibana's discover/dashboard pages all queries below return empty:
original query - field:/abc[0-9]{4}/
scaped query - field:/abc\[0\-9\]\{4\}/
desperate query - field:/.*/
Inspecting the request done by kibana to elasticsearch reveals the following query:
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "field:/abc[0-9]{4}/",
"analyze_wildcard": true,
"default_field": "*"
}
}
I expected kibana to understand the double forward slash syntax /my_query/ and make a ´regexp query´ instead of a ´query_string´. I have tried this with both query languages: "lucene", "kuery" and with the optional "experimental query features" enabled/disabled.
Digging further I found this old issue which says that elastic only runs regex into the now deprecated _all field. If this still holds true I am not sure how regex work in kibana/elastic 6.X.
What am I missing? Any help in clarifying the conditions to use regex in Kibana would be much appreciated
All other stack questions in this subject are either old or were related to syntax issues and/or lack of understanding of how the analyzer deals with whitespaces and did not provide me any help.
So I don't exactly have the answer on how to make Lucene work with Regexp search in Kibana. But I figured out a way to do this in Kibana.
Solution is to use Filter with custom DSL
Here is an example of what to put in Query JSON -
{
"regexp": {
"req.url.keyword": "/question/[0-9]+/answer"
}
}
Example Url I have in my data - /questions/432142/answer
Additional to this, you can write more filters using Kibana search (Lucene syntax)
It does the appropriate search, no escaping issue or any such thing.
Hope it helps.
Make sure Kibana hasn't got query feature turned on in top right.

Regex In body of API test

I'm testing API with https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects/lookup
The following brings a found result with data. I would like to use a regular expression with bring back all records with name having the number 100867. All my attempts result wit a missing result set.
i.e. change to "name": "/1000867.*/"
{
"keys": [
{
"path": [
{
"kind": "Job",
"name": "1000867:100071805:1"
}
]
}
]
}
The Google documentation for lookup key states that the name is a "string" and that
The name of the entity. A name matching regex __.*__ is reserved/read-only. A name must not be more than 1500 bytes when UTF-8 encoded. Cannot be "".
The regex part threw me off and the solution was to use runQuery!
Consider this closed.