substring regex for first part of url - regex

I've got a large database of projects and issue trackers, some of which have urls.
I'd like to query it to figure out a list of urls for each project, but many have extra data I'd like to avoid.
I'd like to do something like this:
substring(tracker_extra_field_data.field_data FROM 'http://([^/]*).*')
Except some urls are https, and I'd like to capture that as well as the first sub directory.
For example, given the url:
https://dev.foo.com/bar/action/?param=val
I'd like the select to return:
https://dev.foo.com/bar/
Is there a semi-simple way to do this with substring/regex in pgsql?

try this:
select substring('https://dev.foo.com/bar/action/?param=val' from '(https?://([^/]*/){1,2})');
template1=# select substring('https://dev.foo.com/bar/action/?param=val' from '(https?://([^/]*/){1,2})');
substring
-------------------------
https://dev.foo.com/bar/
(1 row)
template1=# select substring('http://dev.foo.com/bar/action/?param=val' from '(https?://([^/]*/){1,2})');
substring
------------------------
http://dev.foo.com/bar/

Updated after I didn't read the Q properly at first.
Use the pattern
^https?://[^/]+(?:/[^/]+)?/?
^ .. start of string
? .. zero or one atoms
(?:) .. non-capturing parens
[^/]+ .. any character except /, 1 or more of them
This only accepts URLs starting with http:// or https:// (protocol header required).
->SQLfiddle with a bigger test case.

Related

How to match a regex with a fixed URL + variable slug

I am trying to write the following regexes for google analytics usage and so far I was unable to.
Case 1. to match with all the URLs containing /cms/en/product/{variable slug}/ which only contains one slug after the /product/. I mean something like the following:
/cms/en/product/firstslug/
Case 2. to match with all the URLs containing /cms/en/product/{variable slug1}/{variable slug2}/ which only contains two slugs after the /product/. I mean something like the following:
/cms/en/product/firstslug/secondslug/
Really appreciate anyone's help in advance.
I have already tried basics like the following and it doesn't work:
`/cms/en/product/.*/$
^/cms/en/product/.*/$
^/cms/en/product/.*/$
/cms/en/product/([^/]+)/?$
^/cms/en/product/([^/]+)/?$`
^/cms/en/product/[^/]+/$ matches "/cms/en/product/firstslug/"
^/cms/en/product/[^/]+/[^/]+/$ matches "/cms/en/product/firstslug/secondslug/"
^/cms/en/product/[^/]+/([^/]+/)?$ matches both "/cms/en/product/firstslug/" and "/cms/en/product/firstslug/secondslug/"
where
[^/]+ matches a single slug, i.e. one or more character(s) (+) which are not "/" ([^/])
([^/]+/)? matches an optional slug, i.e. an optional (?) group (()) of one or more character(s) (+) which are not "/" ([^/]) followed by a single "/"
Anyway: I would suggest using Content Grouping on collection.

How can I write a opposite regex to this regex?

this is a regex of a proxy, if I add this to my proxy:
(.*\.|)(abc|google)\.(org|net)
my proxy will not transmit the abc.org, abc.net, google.org, google.net's traffic.
how can I write a regex opposite to this regex? I mean only transmit the abc.org, abc.net, google.org, google.net's traffic.
EDIT-01
My thought is just want to transmit abc.org or www.abc.org, how can I do with that?
Try this:
^(?!(www\.)?(?:abc|google)\.(?:net|org)).*
Demo: https://regex101.com/r/WOnFx8/3/
I used ?! to reverse the matching of your regex. This way, it will match any domain except these specific 4 domains.
Another way to do it is by using this code to include anything before the desired domains:
^(?!(.*\.|)(?:abc|google)\.(?:net|org)).*
demo: https://regex101.com/r/WOnFx8/4/
Your regex you write
(.*\.|)(abc|google)\.(org|net)
mean any string is one of abc.org, gooogle.org, abc.net, google.net, with optional prefix string ends with dot (.)
Like: test.google.org, sub.abc.net,...
I think you want to match string like test.yahoo.com, but not test.google.org. If you can use negative look ahead, this is the answer:
^(.*\.|)(?!(abc|google)\.(org|net))\w+\.\w+$
Explain:
^ and $ to be sure your match is entire url string
Negative look ahead is to check the url is not something like abc.org, abc.net, google.org, google.net
And \w+\.\w+ to check the remain string is kind of URL type (something likes yahoo.com, etc...)
Im going to assume you have lookaheads, if so then you can simply use -
(^.*?\.(?!(abc|google))\w+\.(?:org|net)$)
Demo - https://regex101.com/r/5eC41R/3
What this does is -
Looks for the start of the url (till the first .)
Checks that next part is not abc or google
looks for the next section (till the next .)
Looks for a closing org or net
Note that since it is a lookahead it will be slow compared to other regex matches

Regex: start with something and ends with whatever except something

I need a regex for Url rewrite module, to validate urls in such way:
1) spa/ - match
2) spa/some/url - match
3) spa/some-url - match
4) spa/some.js - no match
5) spa/some.css - no match
So, it should match, if url
a) starts with "spa"
b) ends with whatever except ".js" or ".css"
What I tried to test is ^(spa/)((?!.js)|(?!.css))$
but it's not working.
Thank you and sorry if it's duplicated.
Try this regex:
^spa\/((.+)\/)*.*(?<!\.js|\.css)$
with g and m flags set.
Please note that this regex allows several characters that urls are not supposed to have. I have tried to keep it simple. So, you might want to tune it a bit before using it.
You need negative-lookbehind for this.
Try this (you may need to modify it slightly)
^spa.*(?<!(\.js|\.css))$
^spa : string beginning with spa
.* : followed by any character(s)
(?<!(\.js|\.css))$ : not ending with .js or .css

nginx regex - match variable number of fields

I have a route with urls that can have an optional extra field. It can be either of the form :
"/my-route/azezaezaeazeaze.123x456.jpg"
"/my-route/azezaezaeazeaze.123x456.6786786786.jpg"
with :
"azezaezaeazeaze" being a mongoId
123x456 two integers separated by "x"
6786786786 a unix timestamp
jpg an image extension (could be jpeg, png, gif...)
all those are separated by a "."
I would like to remove the optional part (the timestamp) from the request with the http rewrite module. So that the second url effectively becomes lie the first.
I made a small test on regex101 to get the groups, but :
- it doesn't seem to be the right syntax for nginx
- I do not see how it will allow me to remove the timestamp
How can I remove the timestamp from that url?
Starting from the right-hand end, you need to match a dot followed by anything
except a dot, so we have (\.[^.]*)$, then moving to the left, we want
to match a dot followed by only digits \.[0-9]*, which we dont want to
capture, and then to the left of that we want everything.
I ended up with something like this:
rewrite ^(.*)\.[0-9]*(\.[^.]*)$ $1$2 ;
Capitalizing on my first attempt and #meuh answer, I ended up with the following :
rewrite ^(/.*\..*)(\..*)(\..*)$ $1$3 last;
Now it works, but I would welcome any comment regarding the style/efficiency of this rewrite.

Regex to match anything after /

I'm basically not in the clue about regex but I need a regex statement that will recognise anything after the / in a URL.
Basically, i'm developing a site for someone and a page's URL (Local URL of Course) is say (http://)localhost/sweettemptations/available-sweets. This page is filled with custom post types (It's a WordPress site) which have the URL of (http://)localhost/sweettemptations/sweets/sweet-name.
What I want to do is redirect the URL (http://)localhost/sweettemptations/sweets back to (http://)localhost/sweettemptations/available-sweets which is easy to do, but I also need to redirect any type of sweet back to (http://)localhost/sweettemptations/available-sweets. So say I need to redirect (http://)localhost/sweettemptations/sweets/* back to (http://)localhost/sweettemptations/available-sweets.
If anyone could help by telling me how to write a proper regex statement to match everything after sweets/ in the URL, it would be hugely appreciated.
To do what you ask you need to use groups. In regular expression groups allow you to isolate parts of the whole match.
for example:
input string of: aaaaaaaabbbbcccc
regex: a*(b*)
The parenthesis mark a group in this case it will be group 1 since it is the first in the pattern.
Note: group 0 is implicit and is the complete match.
So the matches in my above case will be:
group 0: aaaaaaaabbbb
group 1: bbbb
In order to achieve what you want with the sweets pattern above, you just need to put a group around the end.
possible solution: /sweets/(.*)
the more precise you are with the pattern before the group the less likely you will have a possible false positive.
If what you really want is to match anything after the last / you can take another approach:
possible other solution: /([^/]*)
The pattern above will find a / with a string of characters that are NOT another / and keep it in group 1. Issue here is that you could match things that do not have sweets in the URL.
Note if you do not mind the / at the beginning then just remove the ( and ) and you do not have to worry about groups.
I like to use http://regexpal.com/ to test my regex.. It will mark in different colors the different matches.
Hope this helps.
I may have misunderstood you requirement in my original post.
if you just want to change any string that matches
(http://)localhost/sweettemptations/sweets/*
into the other one you provided (without adding the part match by your * at the end) I would use a regular expression to match the pattern in the URL but them just blind replace the whole string with the desired one:
(http://)localhost/sweettemptations/available-sweets
So if you want the URL:
http://localhost/sweettemptations/sweets/somethingmore.html
to turn into:
http://localhost/sweettemptations/available-sweets
and not into:
localhost/sweettemptations/available-sweets/somethingmore.html
Then the solution is simpler, no groups required :).
when doing this I would make sure you do not match the "localhost" part. Also I am assuming the (http://) really means an optional http:// in front as (http://) is not a valid protocol prefix.
so if that is what you want then this should match the pattern:
(http://)?[^/]+/sweettemptations/sweets/.*
This regular expression will match the http:// part optionally with a host (be it localhost, an IP or the host name). You could omit the .* at the end if you want.
If that pattern matches just replace the whole URL with the one you want to redirect to.
use this regular expression (?<=://).+