PCRE Regex: Is it possible for Regex to check for a pattern match within only the first X characters of a string, ignoring other parts of the string beyond that point?
My Regex:
I have a Regex:
/\S+V\s*/
This checks the string for non-whitespace characters whoich have a trailing 'V' and then a whitespace character or the end of the string.
This works. For example:
Example A:
SEBSTI FMDE OPORV AWEN STEM students into STEM
// Match found in 'OPORV' (correct)
Example B:
ARKFE SSETE BLMI EDSF BRNT CARFR (name removed) Academy Networking Event
//Match not found (correct).
Re: The capitalised text each letter and the letters placement has a meaning in the source data. This is followed by generic info for humans to read ("Academy Networking Event", etc.)
My Issue:
It can theoretically occur that sometimes there are names that involve roman numerals such as:
Example C:
ARKFE SSETE BLME CARFR Academy IV Networking Event
//Match found (incorrect).
I would like my Regex above to only check the first X characters of the string.
Can this be done in PCRE Regex itself? I can't find any reference to length counting in Regex and I suspect this can't easily be achieved. String lengths are completely arbitary. (We have no control over the source data).
Intention:
/\S+V\s*/{check within first 25 characters only}
ARKFE SSETE BLME CARFR Academy IV Networking Event
^
\- Cut off point. Not found so far so stop.
//Match not found (correct).
Workaround:
The Regex is in PHP and my current solution is to cut the string in PHP, to only check the first X characters, typically the first 20 characters, but I was curious if there was a way of doing this within the Regex without needing to manipulate the string directly in PHP?
$valueSubstring = substr($coreRow['value'],0,20); /* first 20 characters only */
$virtualCount = preg_match_all('/\S+V\s*/',$valueSubstring);
The trick is to capture the end of the line after the first 25 characters in a lookahead and to check if it follows the eventual match of your subpattern:
$pattern = '~^(?=.{0,25}(.*)).*?\K\S+V\b(?=.*\1)~m';
demo
details:
^ # start of the line
(?= # open a lookahead assertion
.{0,25} # the twenty first chararcters
(.*) # capture the end of the line
) # close the lookahead
.*? # consume lazily the characters
\K # the match result starts here
\S+V # your pattern
\b # a word boundary (that matches between a letter and a white-space
# or the end of the string)
(?=.*\1) # check that the end of the line follows with a reference to
# the capture group 1 content.
Note that you can also write the pattern in a more readable way like this:
$pattern = '~^
(*positive_lookahead: .{0,20} (?<line_end> .* ) )
.*? \K \S+ V \b
(*positive_lookahead: .*? \g{line_end} ) ~xm';
(The alternative syntax (*positive_lookahead: ...) is available since PHP 7.3)
You can find your pattern after X chars and skip the whole string, else, match your pattern. So, if X=25:
^.{25,}\S+V.*(*SKIP)(*F)|\S+V\s*
See the regex demo. Details:
^.{25,}\S+V.*(*SKIP)(*F) - start of string, 25 or more chars other than line break chars, as many as possible, then one or more non-whitespaces and V, and then the rest of the string, the match is failed and skipped
| - or
\S+V\s* - match one or more non-whitespaces, V and zero or more whitespace chars.
Any V ending in the first 25 positions
^.{1,24}V\s
See regex
Any word ending in V in the first 25 positions
^.{1,23}[A-Z]V\s
I want to write a regex pattern to match a string starting with "Z" and not containing the next 2 characters as "IU" followed by any other characters.
I am using this pattern but it is not working Z[^(IU)]+.*$
ZISADR - should match
ZIUSADR - should not match
ZDDDDR - should match
Try this regex:
^Z(?:I[^U]|[^I]).*$
Click for Demo
Explanation:
^ - asserts the start of the line
Z - matches Z
I[^U] - matches I followed by any character that is not a U
| - OR
[^I] - matches any character that is not a I
.* - matches 0+ occurrences of any character that is not a new line
$ - asserts the end of the line
When you want to negate certain characters in a string, you can use character class but when you want to negate more than one character in a particular sequence, you need to use negative look ahead and write your regex like this,
^Z(?!IU).*$
Demo
Also note, your first word ZISADR will match as Z is not followed by IU
Your regex, Z[^(IU)]+.*$ will match the starting with Z and [^(IU)]+ character class will match any character other than ( I U and ) one or more times further followed by .* means it will match any characters zero or more times which is not the behavior you wanted.
Edit: To provide a solution without look ahead
A non-lookahead based solution would be to use this regex,
^Z(?:I[^U]|[^I]U|[^I][^U]).*$
This regex has three main alternations which incorporate all cases needed to cover.
I[^U] - Ensures if second character is I then third shouldn't be U
[^I]U - Ensures if third character is U then second shouldn't be I
[^I][^U] - Ensures that both second and third characters shouldn't be I and U altogether.
Demo non-look ahead based solution
I'm trying to the smtp relay from the sendmail logs and to make it reliable I need to require multiple strings in the log entry. An example of a log file entry would be like this:
2018-02-20T19:35:35+00:00 mx01.example.org sendmail[12345]: v1k82343VJ8K: to=<user#foobar.com>, delay=00:00:01, xdelay=00:00:01, mailer=esmtp, tls_verify=OK, relay=mailserver1.foobar.com. [1.1.1.1], dsn=2.0.0, stat=Sent
I can't just key in on "relay=" because the particular relay name I need only appears in the log entry line that contains "to=" with it.
How do I write my regex so that:
The words "sendmail", followed by "to=", then followed by "relay=" all appear in the same log entry.
After "relay=" I match any letter, digit, and character until the comma.
The end result should be:
mailserver1.foobar.com. [1.1.1.1]
See regex in use here
^.*\bsendmail\b.*\bto=.*relay=\K[^,]*
^ Assert position at the start of the line
.* Match any character any number of times
\b Assert position as a word boundary
sendmail Match this literally
\b Assert position as a word boundary
.* Match any character any number of times
\b Assert position as a word boundary
to= Match this literally
.* Match any character any number of times
relay= Match this literallyl
\K Resets the starting point of the match. Any previously consumed characters are no longer included in the final match
[^,]* Match any character except , any number of times
Result: mailserver1.foobar.com. [1.1.1.1]
I recently started using regex, but I can't seem to figure out this problem:
https://xxxx.yyyy.com/en
For this URL I want to create a regex that is only valid when all conditions below are true:
does not contain 'xxxx'
does contain /en$ or /en/
I got to validate the 2 seperate conditions, but can't seem to put them together
\/en\/|\/en$|^(?!.*(xxxx)).*$
Can you please help?
Thanks!
You may use
/^(?!.*xxxx).*\/en(?:$|\/)/
See the regex demo
Details
^ - start of string
(?!.*xxxx) - there can't be xxxx after any 0+ chars other than line break chars
.* - any 0 or more chars other than line break chars, as many as possible
\/en - /en substring
(?:$|\/) - end of string or /
So, if you want to replace xxxx with more than one term, use
/^(?!.*(?:stage|acc)).*\/en(?:$|\/)/
Note that you may force the engine to match them as whole words if you add word boundaries:
/^(?!.*\b(?:stage|acc)\b).*\/en(?:$|\/)/
If you need a full string match, add .* at the end of the pattern.
Using only lookarounds:
^(?!.*xxxx)(?=.*\/en(?:$|\/)).*
^ // start of line
(?!.*xxxx) // look ahead and don't match anything then 'xxxx'
(?= // look ahead and match
.*\/en // anything then '/en'
(?:$|\/) // end of line OR a slash
) // end of look ahead
.* // match all (can be omitted if testing lines)
Flags: global, multiline
Steps: 188
Demo
I am looking for a regex that can be fed to a "create external table" statement of Hive QL in the form of
"input.regex"="the regex goes here"
The condition is that the logs in the files that the RegexSerDe must be reading are of the following form:
2013-02-12 12:03:22,323 [DEBUG] 2636hd3e-432g-dfg3-dwq3-y4dsfq3ew91b Some message that can contain any special character, including linebreaks. This one does not have a linebreak. It just has spaces on the same line.
2013-02-12 12:03:24,527 [DEBUG] 265y7d3e-432g-dfg3-dwq3-y4dsfq3ew91b Some other message that can contain any special character, including linebreaks. This one does not have one either. It just has spaces on the same line.
2013-02-12 12:03:24,946 [ERROR] 261rtd3e-432g-dfg3-dwq3-y4dsfq3ew91b Some message that can contain any special character, including linebreaks.
This is a special one.
This has a message that is multi-lined.
This is line number 4 of the same log.
Line 5.
2013-02-12 12:03:24,988 [INFO] 2632323e-432g-dfg3-dwq3-y4dsfq3ew91b Another 1-line log
2013-02-12 12:03:25,121 [DEBUG] 263tgd3e-432g-dfg3-dwq3-y4dsfq3ew91b Yet another one line log.
I am using the following create external table code:
CREATE EXTERNAL TABLE applogs (logdatetime STRING, logtype STRING, requestid STRING, verbosedata STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES
(
"input.regex" = "(\\A[[0-9:-] ]{19},[0-9]{3}) (\\[[A-Z]*\\]) ([0-9a-z-]*) (.*)?(?=(?:\\A[[0-9:-] ]{19},[0-9]|\\z))",
"output.format.string" = "%1$s \\[%2$s\\] %3$s %4$s"
)
STORED AS TEXTFILE
LOCATION 'hdfs:///logs-application';
Here's the thing:
It is able to pull all the FIRST LINES of each log. But not the other lines of logs that have more than one lines. I tried all links, replaced \z with \Z at the end, replaced \A with ^ and \Z or \z with $, nothing worked. Am I missing something in the output.format.string's %4$s? or am I not using the regex properly?
What the regex does:
It matches the timestamp first, followed by the log type (DEBUG or INFO or whatever), then the ID (mix of lower case alphabets, numbers and hyphens) followed by ANYTHING, till the next timestamp is found, or till the end of input is found to match the last log entry. I also tried adding the /m at the end, in which case, the table generated has all NULL values.
There seem to be a number of issues with your regex.
First, remove your double square brackets.
Second, \A and \Z/\z are to match the beginning and end of the input, not just a line. Change \A to ^ to match start-of-line but don't change \z to $ as you do actually want to match end-of-input in this case.
Third, you want to match (.*?), not (.*)?. The first pattern is ungreedy, whereas the second pattern is greedy but optional. It should have matched your entire input to the end as you allowed it to be followed by end-of-input.
Fourth, . does not match newlines. You can use (\s|\S) instead, or ([x]|[^x]), etc., any pair of complimentary matches.
Fifth, if it was giving you single line matches with \A and \Z/\z then the input was single lines also as you were anchoring the whole string.
I would suggest trying to match just \n, if nothing matches then newlines are not included.
You can't add /m to the end as the regex does not include delimiters. It will try to match the literal characters /m instead which is why you got no match.
If it was going to work the regex you want would be:
"^([0-9:- ]{19},[0-9]{3}) (\\[[A-Z]*\\]) ([0-9a-z-]*) ([\\s\\S]*?)(?=\\r?\\n([0-9:-] ){19},[0-9]|\\r?\\z)"
Breakdown:
^([0-9:- ]{19},[0-9]{3})
Match start of newline, and 19 characters that are digits, :, - or plus a comma, three digits and a space. Capture all but the final space (the timestamp).
(\\[[A-Z]*\\])
Match a literal [, any number of UPPERCASE letters, even none, a literal ] and a space. Capture all but the final space (the error level).
([0-9a-z-]*)
Match any number of digits, lowercase letters or - and a space. Capture all but the final space (the message id).
([\\s\\S]*?)(?=\\r?\\n([0-9:-] ){19},[0-9]|\\r?\\Z)
Match any whitespace or non-whitespace character (any character) but match ungreedy *?. Stop matching when a new record or end of input (\Z) is immediately ahead. In this case you don't want to match end of line as once again, you will only get one line in your output. Capture all but the final (the message text). The \r?\n is to skip the final newline at the end of your message, as is the \r?\Z. You could also write \r?\n\z Note: capital \Z includes the final newline at the end of the input if there is one. Lowercase \z matches at end of input only, not newline before end of input. I have added \z? just in case you have to deal with Windows line endings, however, I don't believe this should be necessary.
However, I suspect that unless you can feed the whole file in at once instead of line-by-line that this will not work either.
Another simple test you can try is:
"^([\\s\\S]+)^\\d"
If it works it will match any full line followed by a line digit on the next line (the first digit of your timestamp).
Following Java regex may help:
(\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2},\d{1,3})\s+(\[.+?\])\s+(.+?)\s+([\s\S\s]+?)(?=\d{4}-\d{1,2}-\d{1,2}|\Z)
Breakdown:
1st Capturing group (\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2},\d{1,3})
2nd Capturing group (\[.+?\])
3rd Capturing group (.+?)
4th Capturing group ([\s\S]+?).
(?=\d{4}-\d{1,2}-\d{1,2}|\Z) Positive Lookahead - Assert that the regex below can be matched.1st Alternative: \d{4}-\d{1,2}-\d{1,2}.2nd Alternative: \Z assert position at end of the string.
Reference http://regex101.com/
I don't know much about Hive, but the following regex, or a variation formatted for Java strings, might work:
(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d,\d+) \[([a-zA-Z_-]+)\] ([\w-]+) ((?:[^\n\r]+)(?:[\n\r]{1,2}\s[^\n\r]+)*)
This can be seen matching your sample data here:
http://rubular.com/r/tQp9iBp4JI
A breakdown:
(\d{4}-\d\d-\d\d \d\d:\d\d:\d\d,\d+) The date and time (capture group 1)
\[([a-zA-Z_-]+)\] The log level (capture group 2)
([\w-]+) The request id (capture group 3)
((?:[^\n\r]+)(?:[\n\r]{1,2}\s[^\n\r]+)*) The potentially multi-line message (capture group 4)
The first three capture groups are pretty simple.
The last one might is a little odd, but it's working on rubular. A breakdown:
( Capture it as one group
(?:[^\n\r]+) Match to the end of the line, dont capture
(?: Match line by line, after the first, but dont capture
[\n\r]{1,2} Match the new-line
\s Only lines starting with a space (this prevents new log-entries from matching)
[^\n\r]+ Match to the end of the line
)* Match zero or more of these extra lines
)
I used [^\n\r] instead of the . because it looks like RegexSerDe lets the . match new lines (link):
// Excerpt from https://github.com/apache/hive/blob/trunk/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/RegexSerDe.java#L101
if (inputRegex != null) {
inputPattern = Pattern.compile(inputRegex, Pattern.DOTALL
+ (inputRegexIgnoreCase ? Pattern.CASE_INSENSITIVE : 0));
} else {
inputPattern = null;
}
Hope this helps.