DateTime format search in the splunk search query - amazon-web-services

I have "YYYY-MM-DD HH:MM:SS.QQ ERROR" in my splunk logs.
Now I want to search for similar date pattern along with Status like "2021-Apr-08 23:08:23.498 ERROR" in my splunk logs and create alert if the ERROR tag comes next to the date.
These date are changeable and are generated at run time.
Can any one suggest me how to check for Date time format along with Status in splunk query.

In the title you mentioned Amazon Web Services. If your events are actual AWS log data, you could install the Splunk Add-on for Amazon Web Services: https://splunkbase.splunk.com/app/1876/
The add-on comes with a lot of field extractions. After installing the add-on, all you need to do is have a look at your events to find out the correct field name for the status text and then search for status=ERROR.
Alternatively, you can create the field extraction yourself. This regular expression should do:
(?<date>\d\d\d\d-\w+-\d\d\s+\d\d:\d\d:\d\d\.\d\d\d)\s+(?<status>\w+)
You can test it here: https://regex101.com/r/pVg1Pm/1
Now use Splunk's rex command to do the field extraction at search time:
To have the field extraction done automatically, you can add new field extractions via Settings / Fields / Field extractions.

Related

GCP log explorer filter for list item count more than 1

I am trying to write a filter in GCP log explorer, which can look for a count of the values of an attribute.
Example:
I am trying to find the logs like below, which has two items for "referencedTables" attribute.
GCP Log Explorer Screenshot
I have tried below options which doesn't work -
protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.*.count>1
protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables.count>1
Also tried Regex looking for "tables" keyword occurrence twice -
protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTable=~"(\tables+::\tables+))"
Also tried Regex querying second item, which means there are more than one items -
protoPayload.metadata.jobChange.job.jobStats.queryStats.referencedTables1=~"^[A-Za-z0-9_.]+$"
Note that - these types of logs are BigQuery audit logs, that are logged in GCP logging service, when you run "insert into.. select" type of queries in BigQuery.
I think you can't use logging filters to filter across log entries only within a log entry.
One solution to your problem is log-based metrics where you'd create a metric by extracting values from logs but you'd then have to use MQL to query (e.g. count) the metric.
A more simple (albeit ad hoc) solution is to use use gcloud logging read to --filter the logs (possibly --format the results in JSON for easier processing) and then pipeline the results into a tool like jq where you could count the results.

Filter/Query Wildcard Issue - AWS OpenSearch

I'm looking into OpenSearch and trying out the dashboards & data querying but am having some difficulty with some specific queries I'd like to run.
The data I have streaming into OpenSearch is a set of custom error logs which the #message field contains json such as: ...{"code":400,"message":"Bad request","detail":"xxxx"}... (the entire message field is not valid json as it contains other data too)
I can specifically query for the code field with "code:400" but would like a more generic query to match all 4XX codes but adding any type of wildcard or range breaks the query and the quotes surrounding it are required otherwise the results include code OR 400
Is there any way to achieve this with the kind of data I have or is this a limitation in the querying syntax?

Create a report from GCP Cloud SQL logs

I have enabled logging on my GCP PostgreSQL 11 Cloud SQL database. The logs are being redirected to a bucket in the same project and they are in a JSON format.
The logs contain queries which were executed on the database. Is there a way to create a decent report from these JSON logs with a few fields from the log entries? Currently the log files are in JSON and not very reader friendly.
Additionally, if a multi-line query is run, then those many log entries are created for that query. If there is also a way to recognize logs which are belong to the same query, that will be helpful, too!
I guess the easiest way is using BigQuery.
BigQuery will import properly those jsonl files and will assign proper field names for the json data
When you have multiline-queries, you'll see that they appear as multiple log entries in the json files.
Looks like all entries from a multiline query have the same receiveTimestamp (which makes sense, since they were produced at the same time).
Also, the insertId field has a 's=xxxx' subfield that does not change for lines on the same statement. For example:
insertId: "s=6657e04f732a4f45a107bc2b56ae428c;i=1d4598;b=c09b782e120c4f1f983cec0993fdb866;m=c4ae690400;t=5b1b334351733;x=ccf0744974395562-0#a1"
The strategy to extract that statements in the right line order is:
Sort by the 's' field in insertId
Then sort by receiveTimestamp ascending (to get all the lines sent at once to the syslog agent in the cloudsql service)
And finally sort by timestamp ascending (to get the line ordering right)

Is there any way to track a job across services in stackdrive?

We use lots of components in Google Cloud, for example a job may start on App Engine, then do some work in Apache Airflow, then do some Dataflow work which will run a BigQuery insert.
Is there any way we can track the status of a job across all components using stack driver. For example tell stackdriver somehow a custom job id and query for it.
You can use advanced logs filters [1] to include log entries from various products. In the logging page search for your BigQuery Job ID. Click to the Job ID and select show matching entries. This will open advanced filter text box with the proper syntax. Then you can add more queries with an OR in between.

Google Analytic Filter not verifiying

I am setting up a custom exclude filter in Google Analytics doing this:
(4webmasters|best-seo-offer|buttons-for-your-website)\.[org|net|com]+
However when i click on verify, it says "This filter would not have changed your data. Either the filter configuration is incorrect, or the set of sampled data is too small."
When i run it through a regexr checker it works. What am i doing wrong, if anything?
There are 3 things to consider for the filter verification on Google Analytics,
First - The filter configuration: You should use Campaign source as filter field (don't use referral) and the expression which in your case seems right, although you can simplify it.
This should be enough 4webmasters|best-seo-offer|buttons-for-your-website
Second - The filter verification just takes a small sample of your data, the previous 7 days, not including the present day. So if there is no record in those 7 days of any of them, it won't work.
Third - If you try to verify with the filter previously saved, GA will consider that you are verifying a new filter with the same configuration and since it already exists, you will get the message that it won't change your data.