Regex to avoid all words between {{ and }} - regex

I am using https://github.com/tbroadley/spellchecker-cli.
I have a JSON file that I'd like to run spellChecker on and it looks like this:
{
"abc.editGroupsMaxLengthError": "Maximum {{charLen}} characters"
}
I would like to know how can all words between {{ and }} be ignored by the spellchecker.
I tried with
[A-Za-z]+}}
as documented here https://github.com/tbroadley/spellchecker-cli#ignore-regexes to ignore regex.
but it doesn't seem to use }} or {{ for some reason.
How can this be fixed?

You can wrap your {{...}} substrings with <!-- spellchecker-disable --> / <!-- spellchecker-enable --> tags, see this Github issue.
So, make sure your JSON looks like
{
"abc.editGroupsMaxLengthError": "Maximum <!-- spellchecker-disable -->{{charLen}}<!-- spellchecker-enable --> characters"
}
And the result will be
C:\Users\admin\Documents\1>spellchecker spellchecker -f spellchecker_test.json
Spellchecking 1 file...
spellchecker_test.json: no issues found
To wrap the {{...}} strings in a certain file in Windows you could use PowerShell, e.g., for a spellchecker_test.json file:
powershell -Command "& {(Get-Content spellchecker_test.json -Raw) -replace '(?s){{.*?}}','<!-- spellchecker-disable -->$&<!-- spellchecker-enable -->' | Set-Content spellchecker_test.json}"
In *nix, Perl is preferable:
perl -0777 -i -pe 's/\{\{.*?}}/<!-- spellchecker-disable -->$&<!-- spellchecker-enable -->/s' spellchecker_test.json

Related

Why is my regex failing to select the correct elements, when it works on the online regex tester

I have a number of xml files, that has HTML embedded in a node . I need capture everything that is not the tags, add some non HTML tags (for moodle) around the text.
I'm processing the files from the command line, using a bash script. I'm using xpath to get the content, piping through xargs to sneakily rip out newlines and then piping through sed.
Heres a sample of the tag:
xpath -q -e '/activity/page/content' page.xml|xargs
<content><h3 style=float:right><img
src=##PLUGINFILE##/consumables.png> </h3> <h3>TITLE</h3>
<p>In order to conduct an LE5 drug test you need a Druglizaer
(batch controlled) foil pouch that contains two items:</p>
<p></p> <ol> <li><span style=font-
weight:900>Druglizer Cartridge</span></li><li><span
style=font-weight:900>Druglizer Oral Fluid
Collector</span></li> </ol> <p></p></content>
On https://regex101.com/ I used \>(.*?)\< which is grouping the text as expected. but when I run with sed it isn't doing any substitutions.
#!/bin/bash
# get new name string
name=$(xpath -q -e '/activity/page/name' page.xml);
en=$(echo $name|sed -e 's/<[^>]*>//g');
vi=$(echo $en|trans -brief -t vi);
cn=$(echo $en|trans -brief -t zh-CN);
mlang_name=$(echo "{mlang en}$en{mlang}{mlang
vi}$vi{mlang}{mlang
zh_cn}$cn{mlang}")
# xmlstarlet to update node
# get new content string
content=$(xpath -q -e '/activity/page/content' page.xml);
# \>(.*?)\<
mlang_name=$(echo $content|sed -e 's/\>(.*?)\</\{mlang
en\}$1\{mlang\}\{mlang
vi\}#VI#\{mlang\}\{mlang
zh_cn\}#CN#\{mlang\}/g')
# xmlstarlet to update node
I need the replace to put {mlang en}TEXT{mlang} around the text.
I ended up using perl as it supports the non-greedy format i was using.
perl -pe 's/(.*?>)(.*?)(<.*?)/$1\{mlang en\}$2\{mlang\}$3/g'
With the above file, the full command I used was
content=$(xpath -q -e '/activity/page/content' page.xml);echo $content|xargs|sed -e 's/<|<content>//g'|sed -e 's|</content>||g' |perl -pe 's/(.*?>)(.*?)(<.*?)/$1\{mlang en\}$2\{mlang\}$3/g'|sed -e 's/{mlang en}[\ ]*{mlang}//g'|sed -e 's/<content>//g'
Which gave the following output
<h3 style=float:right><img src=##PLUGINFILE##/consumables.png></h3><h3>{mlang en}TITLE{mlang}</h3><p>{mlang en}In order to conduct an LE5 drug test you need a Druglizaer (batch controlled) foil pouch that contains two items:{mlang}</p><p></p><ol><li><span style=font-weight:900>{mlang en}Druglizer LE5 Cartridge{mlang}</span></li><li><span style=font-weight:900>{mlang en}Druglizer Oral Fluid Collector{mlang}</span></li></ol><p></p>
If there's a more elegant way feel free to let me know.

Strip everything except for the complete anchor tag - Perl

I am needing to parse an HTML file and remove everything except for the anchor tags in their entirety. So for example:
<html>
<body>
<p>boom</p>
Example
</body>
</html>
I only need to keep:
Example
I am using cURL to retrieve the html and a small snippet of code I found that strips everything but the anchor text of the tag. This is what I am using:
curl http://www.google.com 2>&1 | perl -pe 's/\<.*?\>//g'
Is there a simple command line way to do this? My end goal is to put this into a bash script and execute it. I am having a very difficult time understanding regular expressions and perl.
Using Mojolicious command line tool mojo:
mojo get http://www.google.com 'a'
Outputs:
<a class="gb1" href="http://www.google.com/imghp?hl=en&tab=wi">Images</a>
<a class="gb1" href="http://maps.google.com/maps?hl=en&tab=wl">Maps</a>
<a class="gb1" href="https://play.google.com/?hl=en&tab=w8">Play</a>
<a class="gb1" href="http://www.youtube.com/?tab=w1">YouTube</a>
<a class="gb1" href="http://news.google.com/nwshp?hl=en&tab=wn">News</a>
<a class="gb1" href="https://mail.google.com/mail/?tab=wm">Gmail</a>
<a class="gb1" href="https://drive.google.com/?tab=wo">Drive</a>
<a class="gb1" href="http://www.google.com/intl/en/options/" style="text-decoration:none"><u>More</u> »</a>
<a class="gb4" href="http://www.google.com/history/optout?hl=en">Web History</a>
<a class="gb4" href="/preferences?hl=en">Settings</a>
<a class="gb4" href="https://accounts.google.com/ServiceLogin?hl=en&continue=http://www.google.com/" id="gb_70" target="_top">Sign in</a>
Install Google Chrome
Advanced search
Language tools
Chromebook: For students
Advertising Programs
Business Solutions
+Google
About Google
Privacy & Terms
For a helpful 8 minute introductory video, check out: Mojocast Episode 5
Using Mojolicious, as #Miller above, but more exactly select the <a ... rel= :
If you have an html file
perl -Mojo -E 'say $_ for x(b("my.html")->slurp)->find("a[rel]")->each'
or for the online resource
perl -Mojo -E 'say $_ for g("http://example.com")->dom->find("a[rel]")->each'
#or
perl -Mojo -E 'g("http://example.com")->dom->find("a[rel]")->each(sub{say $_})'
If you want more granular control over your HTML, then you can use HTML::TagParser module available on CPAN.
use strict;
use warnings;
use HTML::TagParser;
my $html = HTML::TagParser->new( '<html>
<body>
<p>boom</p>
Example
</body>
</html>' );
my #list = $html->getElementsByTagName( "a" );
for my $elem ( #list ) {
my $name = $elem->tagName;
my $attr = $elem->attributes;
my $text = $elem->innerText;
print "<$name";
for my $key ( sort keys %$attr ) {
print " $key=\"$attr->{$key}\"";
}
print $text eq "" ? " />" : ">$text</$name>" , "\n";
}
Output:
Example
Ingy döt Net's pQuery deserves a mention:
perl -MpQuery -E 'pQuery("http://www.ubu.com/sound/barthes.html")
->find("a")->each(sub{say pQuery($_)->toHtml})'
Just the links:
perl -MpQuery -E 'pQuery("http://www.ubu.com/sound/barthes.html")
->find("a")->each(sub{say $_->{href}})'
Although - unlike mojo - there's no command line tool (i.e. not yet - it's not that kind of tool per se and is still "under construction"), it's a module to have on your watch list.

How to insert an arbitrary string after pattern with sed

It must be really easy, but somehow I don't get it… I want to process an HTML-file via a bash script and insert an HTML-String into a certain node:
org.html: <div id="wrapper"></div>
MYTEXT=$(phantomjs capture.js www.somesite.com)
# MYTEXT will look something like this:
# <div id="test" style="top: -1.9%;">Something</div>
sed -i "s/\<div id=\"wrapper\"\>/\<div id=\"wrapper\"\>$MYTEXT/" org.html
I always get this error: bad flag in substitute command: 'd' which is probably because sed interprets the content of $MYTEXT as a pattern as well – which is not what I want…
By the way: Duplicating \<div id=\"wrapper\"\> is probably also not necessary?
It seems the / in $MYTEXT's </div> part is interpreted indeed as the final / in the sed command. You can choose another delimiter, which does not appear in $MYTEXT, for instance:
sed -i "s|\<div id=\"wrapper\"\>|\<div id=\"wrapper\"\>$MYTEXT|" org.html

Perl script to search and replace multiple lines in multiple html files

I have many html files in a folder. I need to somehow remove a <div id="user-info" ...>...</div> from all of them. As far as I know I need to use a Perl script for that, but I don't know Perl to do that. Could someone get it for me?
Here is how the "bad" code looks like:
<div id="user-info" class="logged-in">
<a class="icon icon-key-delete" href="https://test.dev/login.php?0,logout=1">Log Out</a>
<a class="icon icon-user-edit" href="https://test.dev/control.php">Control Center</a>
</div> <!-- end of div id=user-info -->
Thank you in advance!
Using XML::XSH2:
for { glob '*.html' } {
open :F html (.) ;
delete //div[#id="user-info" and #class="logged-in"] ;
save :b ;
}
perl -0777 -i.withdiv -pe 's{<div[^>]+?id="user-info"[^>]*>.*?</div>}{}gsmi;' test.html
-0777 means split on nothing, so slurp in whole file (instead of line by line, the default for -p
-i.withdiv means alter files in place, leaving original with extension .withdiv (default for -p is to just print).
-p means pass line by line (except we are slurping) to passed code (see -e)
-e expects code to run.
man perlrun or perldoc perlrun for more info.
Here's another solution, which will be slightly more familiar to people that know jquery, as the syntax is similar. This uses Mojolicious' ojo module to load up the html content into a Mojo::DOM object, transform it, and then print that transformed version:
perl -Mojo -MFile::Slurp -E 'for (#ARGV) { say x(scalar(read_file $_))->at("#user-info")->replace("")->root; }' test.html test2.html test*.html
To replace content directly:
perl -Mojo -MFile::Slurp -E 'for (#ARGV) { write_file( $_, x(scalar(read_file $_))->at("#user-info")->replace("")->root ); }' test.html
Note, this won't JUST remove the div, it will also re-write the content based on Mojo's Mojo::DOM module, so tag attributes may not be in the same order. Specifically, I saw <div id="user-info2" class="logged-in"> rewritten as <div class="logged-in" id="user-info2">.
Mojolicious requires at least perl 5.10, but after that there's no non-core requirements.

Sed program - deleted strings reappearing?

I'm stumped. I have an HTML file that I'm trying to convert to plain text and I'm using sed to clean it up. I understand that sed works on the 'stream' and works one line at a time, but there are ways to match multiline patterns.
Here is the relevant section of my source file:
<h1 class="fn" id="myname">My Name</h1>
<span class="street-address">123 street</span>
<span class="locality">City</span>
<span class="region">Region</span>
<span class="postal-code">1A1 A1A</span>
<span class="email">my#email.ca</span>
<span class="tel">000-000-0000</span>
I would like this to be made into the following plaintext format:
My Name
123 street
City Region 1A1 A1A
my#email.ca
000-000-0000
The key is that City, Region, and Post code are all on one line now.
I use sed -f commands.sed file.html > output.txt and I believe that the following sed program (commands.sed) should put it in that format:
#using the '#' symbol as delimiter instead of '/'
#remove tags
s#<.*>\(.*\)</.*>#\1#g
#remove the nbsp
s#\( \)*##g
#add a newline before the address (actually typing a newline in the file)
s#\(123 street\)#\
\1#g
#and now the command that matches multiline patterns
#find 'City',read in the next two lines, and separate them with spaces
/City/ {
N
N
s#\(.*\)\n\(.*\)\n\(.*\)#\1 \2 \3#g
}
Seems to make sense. Tags are all stripped and then three lines are put into one.
Buuuuut it doesn't work that way. Here is the result I get:
My Name
123 street
City <span class="region">Region</span> <span class="postal-code">1A1 A1A</span>
my#email.ca
000-000-0000
To my (relatively inexperienced) eyes, it looks like sed is 'forgetting' the changes it made (stripping off the tags). How would I solve this? Is the solution to write the file after three commands and re-run sed for the fourth? Am I misusing sed? Am I misunderstanding the 'stream' part?
I'm running Mac OS X 10.4.11 with the bash shell and using the version of sed that comes with it.
I think you're confused. Sed operates line-by-line, and runs all commands on the line before moving to the next. You seem to be assuming it strips the tags on all lines, then goes back and runs the rest of the commands on the stripped lines. That's simply not the case.
See RegEx match open tags except XHTML self-contained tags ... and stop using sed for this.
Sed is a wonderful tool, but not for processing HTML. I suggest using Python and BeautifulSoup, which is basically built just for this sort of task.
If you have only one data block per php file, try the following (using sed)
kent$ cat t
<h1 class="fn" id="myname">My Name</h1>
<span class="street-address">123 street</span>
<span class="locality">City</span>
<span class="region">Region</span>
<span class="postal-code">1A1 A1A</span>
<span class="email">my#email.ca</span>
<span class="tel">000-000-0000</span>
kent$ sed 's/<[^>]*>//g; s/ //g' t |sed '1G;3{N;N; s/\n/ /g}'
My Name
123 street
City Region 1A1 A1A
my#email.ca
000-000-0000