I have a couple of strings that end with a dot (.) at the end of the sentence which I need to remove in Yahoo Pipes.
Example:
example.com.
companywebsite.co.uk.
anothersite.co.
I've tried the following from a couple of posts here on SO but none have worked yet
/\.$/
or
^(.*)\\.(.*)$","$1!$2
Neither of these options have worked
I have tried a very simple find of
.com. and replace with .com
and
.co. to replace with .co
But the latter affects .com as well which is not ideal
EDIT: Here is a visual of what my pipe looks like.
If you can do something like this: ^(.*)\\.(.*)$","$1!$2, then doing this should work: "^(.+?)\.?$", $1. This should match the first part of the URL and leave out the period at the end, should it exist.
EDIT:
As per your image, you should place this: ^(.+?)\.?$ in your replace field and this: $1 in your with field. I do not know if you need to do any escaping, so you might have to use ^(.+?)\\.?$ instead of ^(.+?)\.?$.
Related
I am trying to create a regex in pcre, that is going to salinize URL with multiple slashes like the following:
https://www.domin.com/test1/////test2/somemoretests_67142 https://www.domin.com/test1/test2/somemoretests_67142///// https://www.domin.com/test1/test2///somemoretests_67142
So that I can replace it with the following: https://\2\4 and the link at the end of it looks: https://www.domin.com/test1/test2/somemoretests_67142
I have been struggling with it for the past couple of days, so any regex guru help is more than welcome :)
I have tried the following and more:
(http|https):\/\/(.*)(\/\/+)(.*)
(http|https):\/\/(.*)(\/\/){2,}(.*)
(http|https):\/\/(.*)(\/\/{2})(.*)
I am going to utilize these for Akamai to sanitize our URLs though cloudlet.
You can try:
(?<!https:\/)(?<!http:\/)(\/+$|(?<=\/)\/+)
And substitute the first group with empty string.
Regex demo.
This will produce this output:
https://www.domin.com/test1/test2/somemoretests_67142
https://www.domin.com/test1/test2/somemoretests_67142
https://www.domin.com/test1/test2/somemoretests_67142
I'm using the following regex to find URLs in a text file:
/http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+/
It outputs the following:
http://rda.ucar.edu/datasets/ds117.0/.
http://rda.ucar.edu/datasets/ds111.1/.
http://www.discover-earth.org/index.html).
http://community.eosdis.nasa.gov/measures/).
Ideally they would print out this:
http://rda.ucar.edu/datasets/ds117.0/
http://rda.ucar.edu/datasets/ds111.1/
http://www.discover-earth.org/index.html
http://community.eosdis.nasa.gov/measures/
Any ideas on how I should tweak my regex?
Thank you in advance!
UPDATE - Example of the text would be:
this is a test http://rda.ucar.edu/datasets/ds117.0/. and I want this to be copied over http://rda.ucar.edu/datasets/ds111.1/. http://www.discover-earth.org/index.html). http://community.eosdis.nasa.gov/measures/).
This will trim your output containing trail characters, ) .
import re
regx= re.compile(r'(?m)[\.\)]+$')
print(regx.sub('', your_output))
And this regex seems workable to extract URL from your original sample text.
https?:[\S]*\/(?:\w+(?:\.\w+)?)?
Demo,,, ( edited from https?:[\S]*\/)
Python script may be something like this
ss=""" this is a test http://rda.ucar.edu/datasets/ds117.0/. and I want this to be copied over http://rda.ucar.edu/datasets/ds111.1/. http://www.discover-earth.org/index.html). http://community.eosdis.nasa.gov/measures/). """
regx= re.compile(r'https?:[\S]*\/(?:\w+(?:\.\w+)?)?')
for m in regx.findall(ss):
print(m)
So for the urls you have here:
https://regex101.com/r/uSlkcQ/4
Pattern explanation:
Protocols (e.g. https://)
^[A-Za-z]{3,9}:(?://)
Look for recurring .[-;:&=+\$,\w]+-class (www.sub.domain.com)
(?:[\-;:&=\+\$,\w]+\.?)+`
Look for recurring /[\-;:&=\+\$,\w\.]+ (/some.path/to/somewhere)
(?:\/[\-;:&=\+\$,\w\.]+)+
Now, for your special case: ensure that the last character is not a dot or a parenthesis, using negative lookahead
(?!\.|\)).
The full pattern is then
^[A-Za-z]{3,9}:(?://)(?:[\-;:&=\+\$,\w]+\.?)+(?:\/[\-;:&=\+\$,\w\.]+)+(?!\.|\)).
There are a few things to improve or change in your existing regex to allow this to work:
http[s]? can be changed to https?. They're identical. No use putting s in its own character class
[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),] You can shorten this entire thing and combine character classes instead of using | between them. This not only improves performance, but also allows you to combine certain ranges into existing character class tokens. Simplifying this, we get [a-zA-Z0-9$-_#.&+!*\(\),]
We can go one step further: a-zA-Z0-9_ is the same as \w. So we can replace those in the character class to get [\w$-#.&+!*\(\),]
In the original regex we have $-_. This creates a range so it actually inclues everything between $ and _ on the ASCII table. This will cause unwanted characters to be matched: $%&'()*+,-./0123456789:;<=>?#ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_. There are a few options to fix this:
[-\w$#.&+!*\(\),] Place - at the start of the character class
[\w$#.&+!*\(\),-] Place - at the end of the character class
[\w$\-#.&+!*\(\),] Escape - such that you have \- instead
You don't need to escape ( and ) in the character class: [\w$#.&+!*(),-]
[0-9a-fA-F][0-9a-fA-F] You don't need to specify [0-9a-fA-F] twice. Just use a quantifier like so: [0-9a-fA-F]{2}
(?:%[0-9a-fA-F][0-9a-fA-F]) The non-capture group isn't actually needed here, so we can drop it (it adds another step that the regex engine needs to perform, which is unnecessary)
So the result of just simplifying your existing regex is the following:
https?://(?:[$\w#.&+!*(),-]|%[0-9a-fA-F]{2})+
Now you'll notice it doesn't match / so we need to add that to the character class. Your regex was matching this originally because it has an improper range $-_.
https?://(?:[$\w#.&+!*(),/-]|%[0-9a-fA-F]{2})+
Unfortunately, even with this change, it'll still match ). at the end. That's because your regex isn't told to stop matching after /. Even implementing this will now cause it to not match file names like index.html. So a better solution is needed. If you give me a couple of days, I'm working on a fully functional RFC-compliant regex that matches URLs. I figured, in the meantime, I would at least explain why your regex isn't working as you'd expect it to.
Thanks all for the responses. A coworker ended up helping me with it. Here is the solution:
des_links = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_#.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', des)
for i in des_links:
tmps = "/".join(i.split('/')[0:-1])
print(tmps)
I have a route with urls that can have an optional extra field. It can be either of the form :
"/my-route/azezaezaeazeaze.123x456.jpg"
"/my-route/azezaezaeazeaze.123x456.6786786786.jpg"
with :
"azezaezaeazeaze" being a mongoId
123x456 two integers separated by "x"
6786786786 a unix timestamp
jpg an image extension (could be jpeg, png, gif...)
all those are separated by a "."
I would like to remove the optional part (the timestamp) from the request with the http rewrite module. So that the second url effectively becomes lie the first.
I made a small test on regex101 to get the groups, but :
- it doesn't seem to be the right syntax for nginx
- I do not see how it will allow me to remove the timestamp
How can I remove the timestamp from that url?
Starting from the right-hand end, you need to match a dot followed by anything
except a dot, so we have (\.[^.]*)$, then moving to the left, we want
to match a dot followed by only digits \.[0-9]*, which we dont want to
capture, and then to the left of that we want everything.
I ended up with something like this:
rewrite ^(.*)\.[0-9]*(\.[^.]*)$ $1$2 ;
Capitalizing on my first attempt and #meuh answer, I ended up with the following :
rewrite ^(/.*\..*)(\..*)(\..*)$ $1$3 last;
Now it works, but I would welcome any comment regarding the style/efficiency of this rewrite.
suppose I have this url
url(r'^delete_group/(\w+)/', 'delete_group_view',name='delete_group')
In template
{%url 'delete_group' 'mwas'%} works but when I use
{%url 'delete_group' 'mwas 45'%} is not working. Any way to modify the url to accept both mwas and mwas 45
The issue might be your regex. The URL example you're showing has a space in it. \w won't match spaces. Try this instead: r'^delete_group/([\w\s]+)/ which allows either words or spaces in multiples.
However, know that spaces are not valid in URLs and will likely get converted to %20 or something similar. A best practice is to use hyphens where you would put a space.
I'd also point you at this answer to a similar question.
I want to filter out all URL's that contain certain words, for example:
I have a URL that looks like this:
www.google.com/&SaveThis=true&SaveType=VeryFast&Page=0
And sometimes the 'Save Type' might change to slow or something. So what I want to do is show all URL's that have the 'SaveType=VeryFast' sometimes this can be in the middle of a very long URL.
I tried this:
.*SaveType=VeryFast.*
But it didn't work!
Thanks
From Tip #4 on this page, it looks like you don't need the .* on either end. That is, without the ^ and $ anchors, using SaveType=VeryFast should match any URL that contains those exact characters. It does look like word boundary anchors (\b) are not supported, so you will likely also match any URL that contains e.g. OtherSaveType=VeryFast or SaveType=VeryFastly
Otherwise, I don't see anything wrong with your expression... (?)