I am trying to parse out a specific number range, and can't seem to get it right. I am looking to extract specific browser versions from user agent strings. For example, I want to parse Chrome 1-42 and Firefox 1-40, but I can't figure out the syntax.
What I have so far is this, which kind of works, but it grabs the first number it sees and doesn't respect the 2 digit range:
Gecko..Chrome/([1-9].|[1-4][1-2].)
Sample:
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.1847.137 Safari/537.36
Firefox 29: Mozilla/5.0 (Android; Mobile; rv:29.0) Gecko/29.0 Firefox/23.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:29.0) Gecko/20100101 Firefox/29.0
Any ideas? TIA.
((?:(?:Mozilla\/(?:[1-9]|[1-3][0-9]|40))|(?:Chrome\/(?:[1-9]|[1-3][0-9]|4[0-3])))\.[^ ]+)
Is this what you would like? /Edited/
Demo:
https://regex101.com/r/gH1nU9/2
Because regex is text matching only and number are treated as text, to do something like 1 to 41 you would have to something like this:
\b[1-9]\b|\b[1-3][0-9]\b|4[0-2]\b
This is matching 1 to 9 or 10 to 39 or 40 to 42. I have added the boundries \b so that nothing except thes numbers are matched.
Related
I'm trying to parse an access log-file from Caddy with Powershell, and I have now gotten to the User-Agent string.
How would I go about getting the Browser and Operating System info out of the below string?
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36
Mozilla/5.0 (Linux; Android 13; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36
Home Assistant/2023.1.1-3124 (Android 13; SM-G991B)
This is a User-Agent string from my own computer, and I can't fathom why Safari is in there when I use Chrome to access a page.
I thought about parsing the string with RegEx, but my RegEx skills are barely existing.
I found a RegEx from https://regex101.com/r/2McsiK/1, but it captures a whole lot more than just the actual browser and OS
\((?<info>.*?)\)(\s|$)|(?<name>.*?)\/(?<version>.*?)(\s|$)
and it does not seem to work well with Powershell Match.
PS C:\> "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36" -match "\((?<info>.*?)\)(\s|$)|(?<name>.*?)\/(?<version>.*?)(\s|$)" | Out-Null
PS C:\> $Matches
Name Value
---- -----
version 5.0
name Mozilla
2
0 Mozilla/5.0
Any advice would be helpful.
See Mathias' comments on the question for the perils of user-agent sniffing (parsing the user-agent string) in general.
Regex-based PowerShell-only solution:
The following tries hard to extract the relevant information, but it's impossible to tell if it will work meaningfully across all platforms and browsers, given the lack of standardization of user-agent strings.
# Sample user-agent strings, spanning
# * Windows, macOS, Linux, iOS, and Android
# * Chrome, Safari, Edge, Firefox, Opera
$userAgentStrings = #(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15'
'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.41'
'Mozilla/5.0 (Linux; Android 13; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36'
'Home Assistant/2023.1.1-3124 (Android 13; SM-G991B)'
'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1'
'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0'
'Mozilla/5.0 (X11; Linux ppc64le; rv:75.0) Gecko/20100101 Firefox/75.0'
'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16.2'
)
$userAgentStrings | ForEach-Object {
if ($_ -match '^(?<browser1>.+?) \((?<os>.+?)\)(?: (?<engine>\S+(?: \(.+?\))?)(?: Version/(?<version>\S+(?: Mobile/\S+)?))?(?: (?<browser2>\S+))?(?: \S+? (?<browser4>\S+/\S+$))?)?') {
# Determine the true browser name and version.
$browser = if ($Matches.browser4) { $Matches.browser4 } elseif ($Matches.browser2) { $Matches.browser2 } else { $Matches.browser1 }
if ($Matches.version) {
$browser = ($browser -split '/')[0] + '/' + $Matches.version
}
# Output the captured substrings via a custom object.
[pscustomobject] #{
OS = $Matches.os
Browser = $browser
Engine = $Matches.engine
IsMobile = $Matches.os -match '\bAndroid\b' -or $Matches.version -match '\bMobile\b'
}
}
}
For an explanation of the regex and the ability to experiment with it, see this regex101.com page.
Output:
OS Browser Engine IsMobile
-- ------- ------ --------
Windows NT 10.0; Win64; x64 Chrome/110.0.0.0 AppleWebKit/537.36 (KHTML, like Gecko) False
Macintosh; Intel Mac OS X 10_15_7 Safari/16.2 AppleWebKit/605.1.15 (KHTML, like Gecko) False
Windows NT 10.0 Edg/110.0.1587.41 AppleWebKit/537.36 (KHTML, like Gecko) False
Linux; Android 13; SM-G991B Safari/537.36 AppleWebKit/537.36 (KHTML, like Gecko) True
Android 13; SM-G991B Home Assistant/2023.1.1-3124 True
iPhone; CPU iPhone OS 16_0_3 like Mac OS X Safari/16.0 Mobile/15E148 AppleWebKit/605.1.15 (KHTML, like Gecko) True
iPad; CPU OS 6_0 like Mac OS X Safari/6.0 Mobile/10A5355d AppleWebKit/536.26 (KHTML, like Gecko) True
Macintosh; Intel Mac OS X 10.15; rv:101.0 Firefox/101.0 Gecko/20100101 False
X11; Linux ppc64le; rv:75.0 Firefox/75.0 Gecko/20100101 False
X11; Linux i686; Ubuntu/14.10 Opera/12.16.2 Presto/2.12.388 False
More complete, web-service-based PowerShell solution:
https://useragentstring.com/pages/api.php offers an API that returns the parsed components as a JSON object, which a call via Invoke-RestMethod automatically converts to a PowerShell custom object.
While slower, this solution is more complete than the pure PowerShell solution, though it omits OS details.
$userAgentStrings = #(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15'
'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 Edg/110.0.1587.41'
'Mozilla/5.0 (Linux; Android 13; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Mobile Safari/537.36'
'Home Assistant/2023.1.1-3124 (Android 13; SM-G991B)'
'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1'
'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25'
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0'
'Mozilla/5.0 (X11; Linux ppc64le; rv:75.0) Gecko/20100101 Firefox/75.0'
'Opera/9.80 (X11; Linux i686; Ubuntu/14.10) Presto/2.12.388 Version/12.16.2'
)
$userAgentStrings |
ForEach-Object {
Invoke-RestMethod ('https://useragentstring.com?uas={0}&getJSON=all' -f $_)
} |
Format-Table
Output:
agent_type agent_name agent_version os_type os_name os_versionName os_versionNumber os_producer os_producerURL linux_distibution
---------- ---------- ------------- ------- ------- -------------- ---------------- ----------- -------------- -----------------
Browser Chrome 110.0.0.0 Windows Windows 10 Null
Browser Safari 16.2 Macintosh OS X 10_15_7 Null
Browser Chrome 110.0.0.0 Windows Windows 10 Null
Browser Android Webkit Browser -- Android Android 13 Null
unknown unknown Android Android 13 Null
Browser Safari 16.0 Macintosh iPhone OS 16_0_3 Null
Browser Safari 6.0 Macintosh iPhone OS 6_0 Null
Browser Firefox 101.0 Macintosh OS X 10.15 Null
Browser Firefox 75.0 Linux Linux Null
Browser Opera 12.16.2 Linux Linux Ubuntu
See following. You will need to trim spaces
$data = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36"
$pattern = "(?<browser>[^\s]+)\s*(\((?<comment>[^)]+)\))?"
$match = $data | select-string $pattern -AllMatches
foreach($m in $match.Matches)
{
$m
Write-Host "Browser = " $m.Groups['browser'], "Comment = " $m.Groups['comment']
}
Results
Groups : {0, 1, browser, comment}
Success : True
Name : 0
Captures : {0}
Index : 0
Length : 41
Value : Mozilla/5.0 (Windows NT 10.0; Win64; x64)
Browser = Mozilla/5.0 Comment = Windows NT 10.0; Win64; x64
Groups : {0, 1, browser, comment}
Success : True
Name : 0
Captures : {0}
Index : 42
Length : 38
Value : AppleWebKit/537.36 (KHTML, like Gecko)
Browser = AppleWebKit/537.36 Comment = KHTML, like Gecko
Groups : {0, 1, browser, comment}
Success : True
Name : 0
Captures : {0}
Index : 81
Length : 17
Value : Chrome/110.0.0.0
Browser = Chrome/110.0.0.0 Comment =
Groups : {0, 1, browser, comment}
Success : True
Name : 0
Captures : {0}
Index : 98
Length : 13
Value : Safari/537.36
Browser = Safari/537.36 Comment =
I have the following log message:
request="POST /api/settings/update HTTP/1.1\r\nHost: example.com\r\nConnection: keep-alive\r\nContent-Length: 601\r\nsec-ch-ua: %22 Not A;Brand%22;v=%2299%22, %22Chromium%22;v=%22101%22, %22Google Chrome%22;v=%22101%22\r\nAccept: application/json, text/plain, */*\r\nLang: ar\r\nsec-ch-ua-mobile: ?0\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\r\nsec-ch-ua-platform: %22Windows%22\r\nContent-Type: application/json;charset=UTF-8\r\nOrigin: https://example.com\r\nSec-Fetch-Site: same-origin\r\nSec-Fetch-Mode: cors\r\nSec-Fetch-Dest: empty\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en,ar;q=0.9,en-US;q=0.8\r\nCookie: .AspNetCore.Culture=c%3dh-AA%7fgujt%3Dar-AA; BPBBBBBBB=d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5d5; dm2=!v87cvbt78ffdv76tv87ht87vtsdb879bt879ftb7s6dbt87asdtf786astd7b6as76dftba87fa76sdfbt876asdtbf76sndtb7asdf76t7d6ft76dtbf/OUM=; .AspNetCore.Cookies=sdfgehth87s6dfh876sdfh765dfh675sfdsdf7g5fsrgerh76g-fy00-thrtPlNFc546v54eryrthrtjrtujrtuhv45634v5vkhrB-tyhtr6WRvdyhrth-_dfgtrhCFuQ05QmPch2trP-rtfoNhUrpl8C8xu-tdyhthrthyhth42i40S-hgjgsghjb-ghu56h_a9; BP40sdf215=408a4314117\r\nX-Forwarded-For: 5.65.57.55\r\n\r\n{%22confirmMessage%22:%22Thanks for your feedback%22}"
And I have the following Regex:
([^r]*(?:r(?!equest=")[^r]*)*request="(?<objectname>([^H]*(?:H(?!ost:)[^H]*)*Host:(?<dname>.*?)\\r\\n)?([^U]*(?:U(?!ser-Agent:)[^U]*)*User-Agent:(?<useragent>.*?)\\r\\n)?([^R]*(?:R(?!eferer:)[^R]*)*Referer:(?<object>[^\\])\\)?([^M]*(?:M(?!essage%22:%22)[^M]*)*Message%22:%22(?<subject>.*?)%22)??[^"]*)")?
The above is working fine, but sometimes the message fields are movable as the following:
request="POST /api/settings/update HTTP/1.1\r\nConnection: keep-alive\r\nContent-Length: 601\r\nsec-ch-ua: %22 Not A;Brand%22;v=%2299%22, %22Chromium%22;v=%22101%22, %22Google Chrome%22;v=%22101%22\r\nAccept: application/json, text/plain, */*\r\nLang: ar\r\nsec-ch-ua-mobile: ?0\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\r\nsec-ch-ua-platform: %22Windows%22\r\nHost: example.com\r\nContent-Type: application/json;charset=UTF-8\r\nOrigin: https://example.com\r\nSec-Fetch-Site: same-origin\r\nSec-Fetch-Mode: cors\r\nSec-Fetch-Dest: empty\r\nAccept-Encoding: gzip, deflate, br\r\nAccept-Language: en,ar;q=0.9,en-US;q=0.8\r\n"
As shown in above message the "User-Agent" came first and then "Host" and in the first message the Host was before User Agent.
Is there a way to match them regardless if their place are changed or not?
Regex Demo:
https://regex101.com/r/Iu7Db8/1
Thanks
The 'trick' here is to use look ahead. The look ahead starts from the beginning of the text every time.
I give you an example here with some of the header names. You can expand it with the names you want following this pattern.
Note that the 'test' name won't match so that group will be undefined:
/^(?=.*?request="(?<request>.*?(?=\\r\\n))|)(?=.*?Accept:(?<accept>.*?(?=\\r\\n))|)(?=.*?test:(?<test>.*?(?=\\r\\n))|)(?=.*?Connection:(?<connection>.*?(?=\\r\\n))|)/gm
Explanation:
^ - start from the beginning
(?= - look ahead for
.*?request=" - any characters zero or more times followed by 'request="'
(?<request> - group named 'request'
.*? - any characters zero or more times
(?=\\r\\n) - look ahead for '\r\n'
) - end group 'request'
| - OR empty
) - end look ahead
You can repeat this pattern with different group names as often as you need.
Now the regex don't care about the order.
Here's the link: Regex101
I have the following line from which I want to replace space with whitespace (tab) but want to keep the spaces within the double quotes as it is. I am on Notepad++.
[11/May/2020:10:10:20 -0400] "GET / HTTP/1.1" 302 523 52197 url.com - - TLSv1.2 19922 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" https://somelinkhere - -
Desired output:
[11/May/2020:10:10:20 -0400] "GET / HTTP/1.1" 302 523 52197 url.com - - TLSv1.2 19922 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36" https://somelinkhere - -
Through the following regex I was able to select the string under the double quotes, but it's of no use for me.
"([^"]*)"
Can you please help me how this can be achieved?
You can use
("[^"]*")|[ ]
Replace with (?1$1:\t).
Details:
("[^"]*") - Capturing group 1: a ", then zero or more chars other than " and then a "
| - or
[ ] - matches a space (you may remove [ and ] here , they are used to make the space pattern visible in the answer).
See the demo screenshot:
I have this log file that I'm currently trying to parse.
Jan 12 2019, 14:51:23, 117, 10.0.0.1, neil.armstrong, standard-users, -, TCP_Connect, "sports betting", -, 201, accept, GET, text, https, www.best-site.com, 443, /pages/home.php, ?user=narmstrong&team=wizards, -, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome Safari/537.36", 192.168.1.1, 1400, 1463, -, -, -
Jan 12 2019, 14:52:14, 86, 10.0.0.1, neil.armstrong, standard-users, -, TCP_Connect, "sports betting", -, 200, accept, POST, text, https, www.upload.best-site.com, 443, /, -, -, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537", 192.168.1.1, 230056, 600, -, -, -
Jan 12 2019, 14:52:54, 118, 10.0.0.1, neil.armstrong, standard-users, -, TCP_Connect, "sports betting", -, 200, accept, GET, text/javascript, http, google.fr, 80, /search, ?q=wizards, -, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537", 192.168.1.1, 1717, 17930, -, -, -
this is the regex that I'm currently using https://regex101.com/r/Asbpkx/3 it parses the log file fine until it reaches "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ Safari/537" then it splits at (KHTML, =like Gecko)
How can I complete the regex so that this does not happen?
I looked into this closer and the log file is not CSV format which is why the CSV parsing regex didn't work in my previous answer. (I also tried parsing it with excel and python csv, and both split at the comma after 'KHTML'.
Using a negative lookbehind makes the example you gave parse correctly.
(.+?)(?<!KHTML),
It looks like you are trying to parse csv using regex.
Use the regex described in this post:
https://stackoverflow.com/a/18147076/9397882
Regex: (?:^|,)(?=[^"]|(")?)"?((?(1)[^"]*|[^,"]*))"?(?=,|$)
Don't use regex for a CSV. Try these props.conf settings.
[mysourcetype]
INDEXED_EXTRACTIONS = CSV
FIELD_DELIMITED = ,
FIELD_QUOTE = "
FIELD_NAMES = Date, Time, Field3, IP_Addr, Field4, Field5, Field6
TIMESTAMP_FIELDS = Date, Time
Chrome use-mobile-user-agent not working
Running chrome from command line with flag --use-mobile-user-agent does not open the browser in mobile context (user-agent).
chrome --use-mobile-user-agent= true
Note:
passing user-agent option does work, but i feel its not the right way of doing things as chrome offers you this flag to boot in mobile context.
--user-agent= Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; ar) AppleWebKit/534.46.0 (KHTML, like Gecko) CriOS/19.0.1084.60 Mobile/9B206 Safari/7534.48.3
Chromium source code
reading some of the chromium source code, i see the following:
content_switches.cc
define kUseMobileUserAgent from "use-mobile-user-agent" flag:
Set when Chromium should use a mobile user agent.
const char kUseMobileUserAgent[] = "use-mobile-user-agent";
shell_content_client.cc
add "Mobile" to product if our variable switch is true/set.
std::string GetShellUserAgent() {
std::string product = "Chrome/" CONTENT_SHELL_VERSION;
base::CommandLine* command_line = base::CommandLine::ForCurrentProcess();
if (command_line->HasSwitch(switches::kUseMobileUserAgent))
product += " Mobile";
return BuildUserAgentFromProduct(product);
}
Extra detail (running from selenium)
As an extra detail, i run chrome in using selenium and pass the configurations:
...
"browserName": "chrome",
"chromeOptions": {
"args": [
"--user-agent= Mozilla/5.0 (iPhone; U; CPU iPhone OS 5_1_1 like Mac OS X; ar) AppleWebKit/534.46.0 (KHTML, like Gecko) CriOS/19.0.1084.60 Mobile/9B206 Safari/7534.48.3",
"--window-size=320,640",
"--disable-popup-blocking",
"--incognito",
"--test-type"
]
},
...
The string is built to "Chrome/53.0.2785.116 Mobile" in GetShellUserAgent, then in BuildUserAgentFromProduct, product is not used, and passed on to BuildUserAgentFromOSAndProduct, which is supposed to format a string as such;
"Mozilla/5.0 (%s) AppleWebKit/%d.%d (KHTML, like Gecko) %s Safari/%d.%d"
The product string is inserted into token four, where the fourth replacement token is before "Safari". Therefore "Chrome/53.0.2785.116 Mobile" should be placed there.
With and without the flag, my user agent is the same.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
So what does this mean, is it broken? Quite possibly.
In src/extensions/shell/common/shell_content_client.cc, BuildUserAgentFromProduct("Chrome/" PRODUCT_VERSION) is called in ShellContentClient::GetUserAgent. That just circumvents the call to GetShellUserAgent.
Well. There goes the mobile user agent flag. There's other places it's possible for the product to be replaced, but that's the one that sticks out as the culprit.