Regex isn't recognized in a curl/grep combination - regex

Question

I try to get image urls from a list of html urls with following curl/grep/seed combination (with wget I fail with 403, but cUrl get the source code correctly):
curl -K "C:\urls.txt" | "C:\GnuWin32\bin\grep.exe" -o '(http[^\s]+(jpg|png|webp)\b)' | sed 's/\?.*//' > imglinks.txt
But I get an error The command "png" is either misspelled or could not be found.
Regex should be correct: https://regex101.com/r/Qk6A0Z/1/
How could this code be improved?
Edit: the source code of a single url from my list one can see running curl https://watchbase.com/sellita
The snippet, from where I want to get image urls looks like
<picture>
<source type="image/webp" data-srcset="https://cdn.watchbase.com/caliber/md/origin:png/sellita/sw200-1-bd.webp" srcset="https://assets.watchbase.com/img/FFFFFF-0.png" />
<img class="lazyload" data-src="https://cdn.watchbase.com/caliber/md/sellita/sw200-1-bd.png" src="https://assets.watchbase.com/img/FFFFFF-0.png" alt="Sellita caliber SW200-1"/>
</picture>
Expected output is a file with all image urls, even those from data-src and data-srcset.

Answer 1

You may try this xargs+curl+grep pipeline:
xargs -n 1 curl < "C:\urls.txt" | "C:\GnuWin32\bin\grep.exe" -Eo "http[^[:blank:]?'\"]+(jpe?g|png|gif|bmp|ico|tiff|webp)\b" > imglinks.txt

Answer 2

You can use
curl "https://watchbase.com/sellita" | "C:\GnuWin32\bin\grep.exe" -oE "http[^?[:space:]]+(jpg|png|webp)\b" > imglinks.txt
The 'png' is not recognized as an internal or external command, operable program or batch file issue is due to the use of single quotation marks. You should use double quotation marks in Windows grep.
To read all URLs from a file and process them, you may use
FOR /F %i in (C:\urls.txt) DO curl %i | "C:\GnuWin32\bin\grep.exe" -oP "http[^?\s]+(jpg|png|webp)\b" >> imglinks.txt

Answer 3

It's really bad practice trying to parse HTML with RegEX! And to see senior members even encouraging this really makes me want to cry. This way the constant flood of these questions will never end.
Please have a look at:
Why it's not possible to use regex to parse HTML/XML: a formal explanation in layman's terms
RegEx match open tags except XHTML self-contained tags
To parse HTML please use an HTML parser like xmllint, xmlstarlet, or xidel!
<picture>
<source type="image/webp" data-srcset="https://cdn.watchbase.com/caliber/md/origin:png/sellita/sw200-1-bd.webp" srcset="https://assets.watchbase.com/img/FFFFFF-0.png" />
<img class="lazyload" data-src="https://cdn.watchbase.com/caliber/md/sellita/sw200-1-bd.png" src="https://assets.watchbase.com/img/FFFFFF-0.png" alt="Sellita caliber SW200-1"/>
</picture>
https://assets.watchbase.com/img/FFFFFF-0.png is just 1 white pixel and returns in every single <picture>-node. So I'm going to assume you just want the attributes data-srcset and data-src.
xidel -s "https://watchbase.com/sellita" -e "//picture/(source/#data-srcset,img/#data-src)"
You can also use xidel (with just 1 invocation) to process the urls you have in "C:\urls.txt" (assuming they all have the same <picture>-nodes as https://watchbase.com/sellita).
xidel -s "C:\urls.txt" -e "for $url in x:lines($raw) return doc($url)//picture/(source/#data-srcset,img/#data-src)" > imglinks.txt
or
xidel -se "for $url in file:read-text-lines('C:\urls.txt') return doc($url)//picture/(source/#data-srcset,img/#data-src)" > imglinks.txt
If you're goal is to download all images from 'imglinks.txt', then xidel can do this too.
xidel -s "C:\urls.txt" -f "for $url in x:lines($raw) return doc($url)//picture/(source/#data-srcset,img/#data-src)" --download "."
or
xidel -s --xquery "for $url in file:read-text-lines('C:\urls.txt') for $img in doc($url)//picture/(source/#data-srcset,img/#data-src) return file:write-binary(tokenize($img,'/')[last()],string-to-base64Binary(x:request($img)/raw))"
xidel -s --xquery ^"^
for $url in file:read-text-lines('C:\urls.txt')^
for $img in doc($url)//picture/(source/#data-srcset,img/#data-src)^
return^
file:write-binary(^
tokenize($img,'/')[last()],^
string-to-base64Binary(x:request($img)/raw)^
)^
"