From past few days I'm trying to develop a regex that fetch all the external links from the web pages given to it using grep.
Here is my grep command
grep -h -o -e "\(\(mailto:\|\(\(ht\|f\)tp\(s\?\)\)\)\://\)\{1\}\(.*\?\)" "/mnt/websites_folder/folder_to_search" -r
now the grep seem to return everything after the external links in that given line
Example
if an html file contain something like this on same line
Googlehttps://yahoo.com'>Yahoo
then the given grep command return the following result
http://www.google.com">Google</a><p><a href='https://yahoo.com'>Yahoo</a></p>
the idea here is that if an html file contain more than one links(irrespective in a,img etc) in same line then the regex should fetch only the links and not all content of that line
I managed to developed the same in rubular.com
the regex is as follow
("|')(\b((ht|f)tps?:\/\/)(.*?)\b)("|')
with work with the above input
but iam not able to replicate the same in grep
can anyone help
I can't modify the html file so don't ask me to do that neither I can look for each specific tags and check their attributes to to get external links as it addup processing time and my application doesn't demand that
Thank You
Try this:
cat /path/to/file | egrep -o "(mailto|ftp|http(s)?://){1}[^'\"]+"
egrep -o "(mailto|ftp|http(s)?://){1}[^'\"]+" /path/to/file
Outputs one link per line. It assumes every link is inside single or double quotes. To exclude some certain domain links, use -v:
egrep -o "(mailto|ftp|http(s)?://){1}[^'\"]+" /path/to/file | egrep -v "yahoo.com"
By default grep prints the entire line a match was found on. The -o switch selects only the matched parts of a line. See the man page.
Related
I have multiple text files (let say a.txt) whose contents is as shown below. I need to grep ICV tool version (in this case it is ICV:2018.12-1) out of this files. Can any body help me to grep tool version after ICV as this tool version (2018.12-1) is changing from one text file to another text file but this format is remains same for all files like ;tool name:tool version;tool name:tool version;?
1) a.txt
setenv VDK_TOOL_VERS 'CDESIGNER:2014.12-SP2-2;CUSTOMCOMPILER:2018.09-SP1-1;HSPICE:2018.09-SP1-1;XA:2018.09-SP2;STARRCXT:2018.06-SP5;ICV:2018.12-1;PYLCC:2008.09-SP4-11;CALIBRE:2018.2-15.10;CIRANOVA:2012.12-1-gcc44x;HERCULES:2008.09-SP5-2;WAVEVIEW:2019.06';
Search for what is not a semicolon:
$ grep -o 'ICV:[^;]*' a.txt
ICV:2018.12-1
Only the version using lookbehinds:
$ grep -Po '(?<=ICV:)[^;]*' a.txt
2018.12-1
Lookahead & Lookbehind
I would like to use Vim to find certain string and replace it with another. For every replacements, it should ask for confirmation similar to what %s/foo/replace/gc does for a single file in Vim.
What have I tried?
sed: It doesn't do interactive replacements.
One of the comments in the following this link suggests vim -esnc '%s/foo/bar/g|:wq' file.txt. I tried vim -esnc '%s/foo/bar/gc|:wq' file.txt (used gc instead of g). Now the terminal gets stuck.
Emacs xah-find-replace package. Unfortunately it didn't do interactive replacements as promised in the link.
Combining :argdo with the substitute command would be the recommended way to do this.
You can populate the args by either opening all the files vim *.txt or manually populate this after opening vim using the command:
:args `find . -type f -name '*.txt'`
Now set hidden using the command:
:set hidden
this is required so that you're not prompted to save the file when switching from one buffer to the other. Refer, :h hidden for more information.
Now use the substitute command like you're used to, prefixing the argdo to perform this for every file in the argslist
:silent argdo %s/pattern/replace/gec
The silent is optional and just mutes the reporting. The e flag is to stop reporting the error no matches found message in some of the buffers
Now after replace, you can write the changes using the following command
:argdo update
This will write buffers that were modified only.
If you are looking for an interactive mode of replacement, it is easier to do it with vim.
vim -c '%s/PATTERN/REPLACEMENT/gc' -c 'wq' FILENAME
The stuck terminal in your case is due to piping the save command to the replacement string, as it does not allow the interactive mode to come in to action. And it is not a stuck terminal, if you type "yes" and press enter it should still show you the expected result.
In case multiple files are involved which is spread across multiple subdirectories, using find command with for loop will help as mentioned below:
for FILENAME in `find DIRECTORYPATH -type f -name *.txt`
do
vim -c '%s/PATTERN/REPLACEMENT/gc' -c 'wq' $FILENAME
done
In bash turn on double star to list all files in all subdirectories:
shopt -s globstar
Now start vim once with all files and run the substitute command for all files, then save and exit:
vim -c 'set nomore' -c 'argdo %s/foo/bar/gc' -c xa **/*.txt
In Unix, the command grep -o prints out only the matched string. This is very helpful when you are searching for a pattern using regular expression and only interested in what matched exactly and not the entire line.
However, I'm not able to find something similar for the Windows command findstr. Is there any substitute for printing only matched string in windows?
For example:
grep -o "10\.[0-9]+\.[0-9]+\.[0-9]+" myfile.txt
The above command prints only the IP address in myfile.txt of the form 10.*.*.* but not the entire lines which contain such IP adresses.
PowerShell:
select-string '10\.[0-9]+\.[0-9]+\.[0-9]+' myfile.txt | foreach-object {
$_.Matches[0].Groups[0].Value
}
Just use your familiar grep and other great Linux commands by downloading this UnxUtils (ready .exe binaries). Add it to your PATH environment variable for convenience
I have multiple XML files with strings that are all set to
<msg type="Status"
Some of these strings should be type "Warning" and I have a separate text file with the warning strings.
I can do a
grep -f strings.txt *.xml
this allows me to see which warning strings are incorrectly listed as status strings. What I would like to do is
grep -f strings.txt *.xml | sed 's/status/warning/'
This gives me my desired output, but it is only being displayed (not saved). There are multiple xml files so i can't just save the output to one file. I need sed to replace the string in the original .xml file it originated from. Thanks for your help.
you were close instead of grep -f strings.txt *.xml | sed 's/status/warning/'
do
grep -l strings.txt *.xml | xargs sed -i 's/status/warning/g'
you forgot to use xargs, read here for more info about xargs.
Recommend trying xmllint. Using the --pattern option, you can specify xpath search queries to search for exactly what you need it to find without complicated regular expressions.
There is an Perl script xml_grep using xpath as query language:
http://search.cpan.org/dist/XML-Twig/tools/xml_grep/xml_grep
xml_grep '//msg[#type="Warning"]' *.xml
I am currently trying to a make a script that would grep input to see if something is of a certain file type (zip for instance), although the text before the file type could be anything, so for instance
something.zip
this.zip
that.zip
would all fall under the category. I am trying to grep for these using a wildcard, and so far I have tried this
grep ".*.zip"
But whenever I do that, it will find the .zip files just fine, but it will still display output if there are additional characters after the .zip so for instance .zippppppp or .zipdsjdskjc would still be picked up by grep. Having said that, what should I do to prevent grep from displaying matches that have additional characters after the .zip?
Test for the end of the line with $ and escape the second . with a backslash so it only matches a period and not any character.
grep ".*\.zip$"
However ls *.zip is a more natural way to do this if you want to list all the .zip files in the current directory or find . -name "*.zip" for all .zip files in the sub-directories starting from (and including) the current directory.
On UNIX, try:
find . -type f -name \*.zip
You can also use grep to find all files with a specific extension:
find .|grep -e "\.gz$"
The . means the current folder.
If you want to specify a folder other than the current folder, just replace the . with the path of the folder.
Here is an example: Let's find all files that end with .gz and are in the folder /var/log
find /var/log/ |grep -e "\.gz$"
The output is something similar to the following:
✘ ⚙> find /var/log/ |grep -e "\.gz$"
/var/log//mail.log.1.gz
/var/log//mail.log.0.gz
/var/log//system.log.3.gz
/var/log//system.log.7.gz
/var/log//system.log.6.gz
/var/log//system.log.2.gz
/var/log//system.log.5.gz
/var/log//system.log.1.gz
/var/log//system.log.0.gz
/var/log//system.log.4.gz
The $ sign says that the file extension is ending with gz
I use this to get a listing of the file types inside a folder.
find . -type f | egrep -i -E -o "\.{1}\w*$" | sort -su
Outputs for example:
.DS_Store
.MP3
.aif
.aiff
.asd
.doc
.flac
.jpg
.m4a
.m4p
.m4r
.mp3
.pdf
.png
.txt
.wav
.wma
.zip
BONUS: with
find . -type f | egrep -i -E -o "\.{1}\w*$" | sort | uniq -c
You'll get the file count:
106 .DS_Store
35 .MP3
89 .aif
5 .aiff
525 .asd
1 .doc
60 .flac
48 .jpg
149 .m4a
11 .m4p
1 .m4r
12844 .mp3
1 .pdf
5 .png
9 .txt
108 .wav
44 .wma
2 .zip
You need to do a couple of things. It should look like this:
grep '.*\.zip$'
You need to escape the second dot, so it will just match a dot, and not any character. Using single quotes makes the escaping a bit easier.
You need the dollar sign at the end of the line to indicate that you want the "zip" to occur at the end of the line.
grep -r pattern --include="*.txt" /path/to/dir/
Try: grep -o -E "(\\.([A-z])+)+"
I used this to get multi-dotted/multiple extensions. So if the input was hello.tar.gz, then it would output .tar.gz.
For single dotted, use grep -o -E "\\.([A-z])+$".
Tested on Cygwin/MingW+MSYS.
One more fix/addon of the above example:
# multi-dotted/multiple extensions
grep -oEi "(\\.([A-z0-9])+)+" file.txt
# single dotted
grep -oEi "\\.([A-z0-9])+$" file.txt
This will get file extensions like '.mp3' and etc.
Just reviewing some of the other answers. The .* isn't necessary, and if you're looking for a certain file extension, it's best to include -i so that it's case-insensitive; in case the file is HELLO.ZIP, for example. I don't think the quotes are necessary, either.
grep -i \.zip$
If you just want to find in the current folder, why not with this simple command without grep ?
ls *.zip
Simply do :
grep ".*.zip$"
The "$" indicates the end of line