I have a question concerning log insights in aws. How is it possible to fetch all the occurrences in a log ? I tried with and without a regex and the parse will only fetch the first occurrence.
I have a log like this (and multiple entries of this kind of log):
[ERROR] - [{'id': 'id1'}, {'id': 'id2'}, {'id': 'id3'}]
And I want to extract all the ids, so I tried :
parse #message "id': '*'" as id
which return only id1 (the first occurrence) by log
and I also tried a regex :
parse #message /id': '(?<id>\S*)'/
which return only id1 (the first occurrence) as well by log
I expect something like [id1, id2, id3] or multiple line in the result (one by match).
I still haven't found a nice way to handle this, it seems like we can't get more than one result from one log
But maybe you can use the practice shared on the answer linked below and find how many items exist in each message
and you can also get list of the values by string manipulation
fields (strlen(#message)-strlen(replace(#message, "'id'", ""))) / strlen("'id'") as count,
replace(replace(replace(#message, "}", ""), "},", ","), "{'id': ", "") as list
# would return 3, ['id1', 'id2', 'id3']
https://stackoverflow.com/a/73254710/1762994
Related
I am creating a line chart in a CloudWatch dashboard. I can create a line representing the frequency with which one string appears the logs, using a filter. But, I don't know how to create two or more such lines.
After selecting a log group, I run this query:
filter name = "first log string"
| stats count(*) as firstString by bin(1hour)
This generates counts of results that, in the Visiulization tab, are displayed as a line chart with a single line.
Now I want to add another line representing "second log string" on the chart. I assume I will have to modify the filter in some way, or add a second filter.
Here are some things that don't work:
adding a second name to the filter
trying filter #message or #name as this documentation suggests
simply pasting in a modified duplicate of the query
Further, I seem to lack documentation that explains how filter is supposed to work. Search engines keep sending me back to this Filter And Pattern Syntax AWS document which dosen't appear to give any actual examples using filter.
To answer my own question, I had to use an array in my filter. I also used the sum() function instead of count().
filter name in ["first log string", "second log string"]
| fields name = "first log string" as #first_string, name = "second log string" as #second_string
| stats sum(#first_string) as first_string, sum(#second_string) as second_string by bin(1hour)
Now I have a line chart with two lines representing the frequency of two logged items
I am currently struggling with the following input:
# Time: 2022-06-01T20:00:00.000000Z
# User#Host: database[database] # [10.10.10.10] Id: 8888888
# Query_time: 0.000450 Lock_time: 0.000160 Rows_sent: 1 Rows_examined: 2
SET timestamp=1654715324;
SELECT id
FROM table_name
WHERE field = 'some-data' AND another_field != 'random-stuff'
ORDER BY field_2;
All my input data will look similar to this. Basically I want to check how many times a certain query shows up. Right now I am a little stuck because my regex cannot filter out the parameters between the single quotes.
I would like to match the following:
SELECT id
FROM table_name
WHERE field = '' AND another_field != ''
ORDER BY field_2;
I've managed to get the query from the input above with the following regExp, but right now this will only match the exact sql.
/(?<=\d;\n).+?(?=;)/gmi
I want to expand this regex so it will ignore anything between single quotes.
Help would be very much appreciated!
I want to retrieve the values of order numbers and store it in a list. But the problem is I am able to fetch one not the other
ID Order
test#xyz.com 1-1155945200890<<<<able to fetch this
test1#xyz.com Hi how are you? 1-1155945200890<<<not able to fetch
By using below I am able to retrive the values to those column which do not have the junk data with it like that of just '1-1155945200890' but not from 'Hi how are you? 1-1155945200890'
To define feedback:
for user in users:
intent_name=data_to_analyse2.loc[data_to_analyse2['ID'] == user]
intent_list=list(intent_name['INTENTNAME'])
feedback=list(intent_name['Input'])
to fetch the match pattern:
pattern=re.compile("1[\-][\d]{2,15}")
pattern_list=list(filter(pattern.match, feedback))
How can I get all the values to the matching expression even if it has junk data associated with it
So last week I was able to begin to stream my Appengine logs into BigQuery and am now attempting to pull some data out of the log entries into a table.
The data in protoPayload.resource is the page requested with the querystring paramters included.
The contents of protoPayload.resource looks like the following examples:
/service.html?device_ID=123456
/service.html?v=2&device_ID=78ec9b4a56
I am getting close, but when there is another entry before device_ID, I am not getting it. As you can see I am not great with Regex, but it is the only way I think I can parse the data in the query. To get just the device ID from the first example, I was able to use the following example. Works great. My next challenge is to the data when the second parameter exists. The device IDs can vary in length from about 10 to 26 characters.
SELECT
RIGHT(Regexp_extract(protoPayload.resource,r'[\?&]([^&]+)'),
length(Regexp_extract(protoPayload.resource,r'[\?&]([^&]+)'))-10) as Device_ID
FROM logs
What I would like is just the values from the querystring device_ID such as:
123456
78ec9b4a56
Assuming you have just 1 query string per record then you can do this:
SELECT REGEXP_EXTRACT(protoPayload.resource, r'device_ID=(.*)$') as device_id FROM mytable
The part within the parentheses will be captured and returned in the result.
If device_ID isn't guaranteed to be the last parameter in the string, then use something like this:
SELECT REGEXP_EXTRACT(protoPayload.resource, r'device_ID=([^\&]*)') as device_id FROM mytable
One approach is to split protoPayload.resource into multiple service entries, and then apply regexp - this way it will support arbitrary number of device_id, i.e.
select regexp_extract(service_entry, r'device_ID=(.*$)') from
(select split(protoPayload.resource, ' ') service_entry from
(select
'/service.html?device_ID=123456 /service.html?v=2&device_ID=78ec9b4a56'
as protoPayload.resource))
In Rails, I am trying to fetch data from mongodb using LIKE query by providing regular expression but even though not getting the correct output.
Model : User
_id, name, display_name, age, address, nick_name
a1, Johny, Johny K, 12, New York, John
b1, James, James Waltor, 15, New York, James
c1, Joshua, Joshua T, 13, California, Josh
Now I have 3 set of records.
Query 1 : Search User having 'Jo' as keyword in initial name
User.where(name: /^jo/i)
Output - Only One record - instead of two.
Query 2 :- Match the text with all column values
User.where($where: /^jo/i)
Not getting the proper output.
Ok on the Query 1, can you output the documents. I believe one of your records in 'name' has a character in front of it such as white space. I just run the same query locally and it pulled multiple records back.
Try this:
User.where(name/(.*)jo(.*)/i).count and see what that returns. It should match 2. If that works, then you'll need to look at what is incorrect with the store value.
On Query 2, where have you seen this syntax. The $where is expecting a string of a js function to execute to match records. In your case to match any field within the document with an expression you would need to do a recursive function across each field in each document.
For Query 2 to match against all fields
One solution, although inefficient, is to do it within the Rails app instead of Mongodb query.
e.g.
User.all.select do | user | user.attributes.values.grep(/^jo/i).any? end