Given that I have the following files.
/tmp/example.txt
/tmp/example.txt_<date> (e.g. /tmp/example.txt_20230207)
/tmp/example.txt.lck
I want to pipe example.txt and /tmp/example.txt_<date> to CloudWatch and exclude example.txt.lck.
Following Glob syntax (as mentioned in the official document that it uses Glob), "/tmp/example.txt*(!(.lck))" should do the trick. (tested on DigitalOcean's online Glob tool)
However, when applied the configuration, it is not sending any of the above files to CloudWatch.
Is there any other methods or alternative to achieve the result?
I have tested with /tmp/example.txt* and it works, but it is sending all 3 files. So at the very least, I know that my CloudWatch Agent is working.
It is quite likely a mistake in my glob expression or it might not be supported at the moment (as we can see from this issue - https://github.com/aws/amazon-cloudwatch-agent/issues/328, "alternative brackets" (i.e. {}) is not supported, hence chances are I'm using unsupported glob expression too).
I think the syntax should be:
"/tmp/example.txt*[!(.lck)]"
If this doesn't work, you can hack your way around this by excluding "k" as the last character in the string.
"/tmp/example.txt*[!k]"
Or you can include the files separately in the config as follows:
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/tmp/example.txt_20230207",
"log_group_name": "",
"log_stream_name": "",
"timestamp_format": ""
},
{
"file_path": "/tmp/example.txt",
"log_group_name": "",
"log_stream_name": ""
}
]
},
Related
I am building an Angular application and trying to figure out the way to write ngsw-config.json -file in order to define rules for service worker.
I assumed that regex would be recognized as regex in configuration file and not interpret as normal characters / text automatically, but it was not so. I have for example following piece of a code:
"name": "authentication",
"urls": [
"/login",
"/.*authentication.*"
],
part .* is not in my understanding recognized as regex (regex meaning in this case that any path that has text "authentication" would fall into this category, right?). This piece of a configuration tries to prevent service worker to take a lead in these two cases, it works with /login, but not with authentication part.
Question:
Can I somehow modify my file to make it recognize regex definitions?
According to the documentation at https://angular.io/guide/service-worker-config
you can use a limited glob format.
I don't know what kind of url you want to match.
Option: If you want to match a url like /foo/bar/authentication/foo2/bar2 you could use:
"name": "authentication",
"urls": [
"/login",
"/**/authentication/**/*"
],
Option: If you want to match a url like /foo/bar/something-authentication-otherthing/foo2/bar2 you could use:
"name": "authentication",
"urls": [
"/login",
"/**/*authentication*/**/*"
],
I am having a hard time using a regex pattern inside Kibana/Elasticsearch version 6.5.4. The field I am searching for has the following mapping:
"field": {
"type": "text",
"analyzer": "custom_analyzer"
},
Regex searches in this field return several hits when requested straight to elasticsearch:
GET /my_index/_search
{
"query": {
"regexp":{
"field": "abc[0-9]{4}"
}
}
}
On the other hand, in Kibana's discover/dashboard pages all queries below return empty:
original query - field:/abc[0-9]{4}/
scaped query - field:/abc\[0\-9\]\{4\}/
desperate query - field:/.*/
Inspecting the request done by kibana to elasticsearch reveals the following query:
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "field:/abc[0-9]{4}/",
"analyze_wildcard": true,
"default_field": "*"
}
}
I expected kibana to understand the double forward slash syntax /my_query/ and make a ´regexp query´ instead of a ´query_string´. I have tried this with both query languages: "lucene", "kuery" and with the optional "experimental query features" enabled/disabled.
Digging further I found this old issue which says that elastic only runs regex into the now deprecated _all field. If this still holds true I am not sure how regex work in kibana/elastic 6.X.
What am I missing? Any help in clarifying the conditions to use regex in Kibana would be much appreciated
All other stack questions in this subject are either old or were related to syntax issues and/or lack of understanding of how the analyzer deals with whitespaces and did not provide me any help.
So I don't exactly have the answer on how to make Lucene work with Regexp search in Kibana. But I figured out a way to do this in Kibana.
Solution is to use Filter with custom DSL
Here is an example of what to put in Query JSON -
{
"regexp": {
"req.url.keyword": "/question/[0-9]+/answer"
}
}
Example Url I have in my data - /questions/432142/answer
Additional to this, you can write more filters using Kibana search (Lucene syntax)
It does the appropriate search, no escaping issue or any such thing.
Hope it helps.
Make sure Kibana hasn't got query feature turned on in top right.
I'm testing API with https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects/lookup
The following brings a found result with data. I would like to use a regular expression with bring back all records with name having the number 100867. All my attempts result wit a missing result set.
i.e. change to "name": "/1000867.*/"
{
"keys": [
{
"path": [
{
"kind": "Job",
"name": "1000867:100071805:1"
}
]
}
]
}
The Google documentation for lookup key states that the name is a "string" and that
The name of the entity. A name matching regex __.*__ is reserved/read-only. A name must not be more than 1500 bytes when UTF-8 encoded. Cannot be "".
The regex part threw me off and the solution was to use runQuery!
Consider this closed.
My current routing tables is like below:
routing_table = [
("/api/ping", PingHandler),
("/css/(.*)", StaticFileHandler, {
"path": "my-website-path/css"
}),
("/js/(.*)", StaticFileHandler, {
"path": "my-website-path/js"
}),
("/fonts/(.*)", StaticFileHandler, {
"path": "my-website-path/fonts"
})
I would like to use only one regex to handle my static files.
Something like below ?
routing_table = [
("/api/ping", PingHandler),
("/(css|js|fonts)/(.*)", StaticFileHandler, {
"path": "my-website-path/$1"
})
How can I do that?
Thank you in advance.
A RequestHandler pass all matches as a positional arguments to the http-verb function. Since the StaticFileHandler extends it and you have 2 captured groups, your code won't work as expected. So the regex needs to be changed, step by step:
match entire path: /(.*)
first part should be fonts, js or css: ((jss|css|fonts)/.*
the inner group should not be captured - make use of ?:: ((?:jss|css|fonts)/.*
The code
routing_table = [
("/api/ping", PingHandler),
("/((?:css|js|fonts)/.*)", StaticFileHandler, {
"path": "my-website-path"
}
Keep in mind, that the StaitcFileHandler (as #cricket_007 mentioned)...
This handler is intended primarily for use in development and light-duty file serving; for heavy traffic it will be more efficient to use a dedicated static file server (such as nginx or Apache). We support the HTTP Accept-Ranges mechanism to return partial content (because some browsers require this functionality to be present to seek in HTML5 audio or video).
What is the default behavior for a regexp query against a non-analyzed field? Also, is that the same answer when dealing with .raw fields?
After everything i've read, i understand the following.
1. RegExp queries will work on analyzed and non-analyzed fields.
2. A regexp query should work across the entire phrase rather than just matching on a single token in non-analyzed fields.
Here's the problem though. I can not actually get this to work. I've tried it across multiple fields.
The setup i'm working with is a stock elk install and i'm dumping pfsense and snort logs into it with a basic parser. I'm currently on Kibana 4.3 and ES 2.1
I did a query to look at the mapping for one of the fields and it indicates it is not_analyzed, yet the regex does not work across the entire field.
"description": {
"type": "string",
"norms": {
"enabled": false
},
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 256
}
}
}
What am i missing here?
if a field is non-analyzed, the field is only a single token.
It's same answer when dealing with .raw fields, at least in my work.
You can use groovy script:
matcher = (doc[fields.raw].value =~ /${pattern}/ );
if(matcher.matches()) {
matcher.group(matchname)}
you can pass pattern and matchname in params.
What's meaning of tried it across multiple fields.? If your situation is more complex, maybe you could make a native java plugin.
UPDATE
{
"script_fields" : {
"regexp_field" : {
"script" : "matcher = (doc[fieldname].value =~ /${pattern}/ );if(matcher.matches()) {matcher.group(matchname)}",
"params" : {
"pattern" : "your pattern",
"matchname" : "your match",
"fieldname" : "fields.raw"
}
}
}
}