lets say there is something like this
Lorem ipsum dolor sit amet, consectetur adipiscing elit. "Vestibulum interdum dolor nec sapien blandit a suscipit arcu fermentum. Nullam lacinia ipsum vitae enim consequat iaculis quis in augue. Phasellus fermentum congue blandit. Donec laoreet, ipsum et vestibulum vulputate, risus augue commodo nisi, vel hendrerit sem justo sed mauris." Phasellus ut nunc neque, id varius nunc. In enim lectus, blandit et dictum at, molestie in nunc. Vivamus eu ligula sed augue pretium tincidunt sit amet ac nisl. "Morbi eu elit diam, sed tristique nunc."
to be something like this
Lorem ipsum dolor sit amet, consectetur adipiscing elit. "Vestibulum interdum dolor nec sapien blandit a suscipit arcu fermentum[dot] Nullam lacinia ipsum vitae enim consequat iaculis quis in augue[dot] Phasellus fermentum congue blandit[dot] Donec laoreet, ipsum et vestibulum vulputate, risus augue commodo nisi, vel hendrerit sem justo sed mauris[dot]" Phasellus ut nunc neque, id varius nunc. In enim lectus, blandit et dictum at, molestie in nunc. Vivamus eu ligula sed augue pretium tincidunt sit amet ac nisl. "Morbi eu elit diam, sed tristique nunc[dot]"
i somehow found a regex to select all the "{sentence}" with "(.)+?" or use them like
regex('"(.)+?"','[sentence]')
but can we do something like replace the dots inside a group?. so i can get the output like above example?
I'm not sure regexps are able to suit your needs on their own.
You should implement an algorithm that replaces nested dots until the string doesn't contain nested dots anymore.
For example in PHP:
$string = 'He asked "Please." while she answered "No. Or maybe yes."';
var_dump($string);
while(preg_match('/"[^"]*\.[^"]*"/', $string)) {
$string = preg_replace('/("[^"]*)\.([^"]*")/', '$1[dot]$2', $string);
}
var_dump($string);
which prints:
string 'He asked "Please." while she answered "No. Or maybe yes."' (length=57)
string 'He asked "Please[dot]" while she answered "No[dot] Or maybe yes[dot]"' (length=69)
This is what I would do.
echo
preg_replace_callback('~(?<!\\\)"(.+?)((?<!\\\)")~',
/*
Pattern:
--------
(?<!\\\)" a double quote not preceded by a backward (escaping) slash
(.+?) anything (with min 1 char.) between condition above and below
((?<!\\\)") a double quote not preceded by a backward (escaping) slash
*/
// for anything that matches the above pattern
// the following function is called
create_function('$m',
'return preg_replace("~\.~","[dot]",$m[0]);'),
// which replaces each dot with [dot] and returns the match
$str);
EDIT: Added explanations in comments.
try this:
(\"[^\.]*)\.([^\"]*) to \1[dot]\2
works well in my editor, but sometimes $ is used instead of \ in replacement (e.g. in php)
With Javascript I would just do a basic replace:
str = str.replace(/".+?"/g,function(m) {
return m.replace(/\./g,'[dot]');
});
Related
I have an OCR text document where paragraphs have been broken into individual lines. I'd like to make them whole paragraphs on a single line again (as per the original PDF).
How can I use regex, or find and replace, to remove the line breaks between two lines of text and replace them with a space?
Eg:
Every line of text is on a newline. I'd like them to be whole paragraphs on a single line.
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam vehicula tellus faucibus metus consequat
scelerisque. Maecenas sit amet urna quis ipsum interdum consequat. Praesent elementum libero nec
velit suscipit placerat accumsan vitae lacus. Aliquam erat volutpat. Etiam egestas lectus sed orci
venenatis, ullamcorper gravida elit pulvinar. Pellentesque imperdiet, augue pulvinar sodales dapibus,
tortor magna rutrum nulla, vel ullamcorper mi purus a diam. Ut id odio sed arcu aliquet lobortis.
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Donec quam arcu, egestas feugiat eleifend blandit, vulputate non elit. Nulla a erat vel leo maximus
viverra at ac lorem. Nam non imperdiet lorem. Fusce tempor arcu massa, non commodo ligula lobortis
nec. Aliquam sit amet fringilla sapien, non euismod metus. Donec orci mi, sagittis vitae lobortis eu,
aliquet nec libero. Sed sodales magna lacus, pretium lobortis magna varius nec. Pellentesque quis
ipsum viverra orci lobortis egestas. Aliquam porttitor tincidunt ipsum, egestas placerat ante
consectetur in. Morbi porttitor lacus eu augue tincidunt, at aliquet lorem consectetur.
You might be looking for a programatic/dynamic approach for every new scan generated so I'm not sure if this answers your question, but since you have visual studio code in your tags I will answer how to do this in vscode.
Open keyboard shortcuts from File > Preferences > Keyboard shortcuts, and bind editor.action.joinLines to a shortcut of your choice like for example Ctrl + J.
Then go ahead and open the text you are looking to fix in vscode, select it and press that keybinding. You will notice everything will be in 1 line. I hope I helped!
I am using two regular expressions when removing linebreaks from OCR texts.
They can be used in the Find&Replace dialog from VS Code.
Remove linebreaks at lines ending with a hyphen: (?<=\w)- *\n *
Replace remaining linebreaks with whitespace, but keeping blank lines: (?<!\n) *\n *(?!\n).
Note that the * in the regular expression trims whitespace at the end and beginning of the lines.
There is also a Python tool based on Flair called dehyphen that does the job.
In my experience it produces useful results but may take quite long compared to replacing linebreaks with regular expressions.
Is it possible to find all punctuation marks of a given type, only when a key phrase exists?
For example:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer pulvinar ac augue nec auctor. Vestibulum eleifend, sem non placerat porttitor, urna neque pulvinar enim, ut ullamcorper massa libero nec tellus. Sed est massa, congue eu auctor gravida, efficitur sit amet lacus. Nullam tincidunt posuere sollicitudin. Sed ac ullamcorper risus, ac cursus justo. Phasellus vehicula quam nec libero venenatis venenatis. Donec metus erat: maximus in risus eu: imperdiet: dignissim mauris. Aliquam sit amet augue vel ex tincidunt convallis. Morbi a sem neque. Nam tellus dolor, congue in mi eu, laoreet sodales lectus. Fusce sed ullamcorper purus. Nulla facilisi.
For above, as long as "neque" is in the text, I want to find all occurrences of ":"
I've tried something like this without luck:
(.*\neque\b.*)(?!^)([:])
This works well in my system
explanation
extract the given phrase and store it in a variable.
if the phrase exists find the symbol and count its occurrences.
#!/bin/bash
a="Lorem ipsum dolor sit amet, consectetur adipiscing elit. Integer pulvinar ac augue nec auctor. Vestibulum eleifend, sem non placerat porttitor, urna neque pulvinar enim, ut ullamcorper massa libero nec tellus. Sed est massa, congue eu auctor gravida, efficitur sit amet lacus. Nullam tincidunt posuere sollicitudin. Sed ac ullamcorper risus, ac cursus justo. Phasellus vehicula quam nec libero venenatis venenatis. Donec metus erat: maximus in risus eu: imperdiet: dignissim mauris. Aliquam sit amet augue vel ex tincidunt convallis. Morbi a sem neque. Nam tellus dolor, congue in mi eu, laoreet sodales lectus. Fusce sed ullamcorper purus. Nulla facilisi."
b=$(echo "$a"| grep -o "neque"| head -1)
echo $b
if [ "$b" == "neque" ]
then
number_of_occurences=$(echo "$a"| grep -o ":"| wc -l)
echo "$number_of_occurences"
fi
Your desired action is not clear. What I can read from the highlights in your example, you want to find all words that end in :, but only if the word neque exists anywhere in the text. Assuming that is the case, you can use this regex:
/(?=.*\bneque\b)\w+:/g
Explanation:
(?=.*\bneque\b) - positive lookahead for neque with word boundaries, anywhere in the text; if this fails, the whole regex fails
\w+: - look for a word that is followed by :
use the g to find all occurrences of words followed by :
EDIT: After seeing that the bash tag has been added, here is a shell script version using a shortened string. The first example has the neque keyword, the second one not:
$ echo 'Urna neque metus erat: maximus in risus eu: imperdiet: dignissim.'\
> | egrep '\bneque\b' | egrep -o '\w+:'
erat:
eu:
imperdiet:
$
$ echo 'Urna metus erat: maximus in risus eu: imperdiet: dignissim.'\
> | egrep '\bneque\b' | egrep -o '\w+:'
$
Explanation:
use first egrep to filter by required keyword neque using word boundaries
use second 'egrep' with -o flag to extract words followed by :
I am trying to write an expression to take a block of text an return up until a full-stop before an ellipsis or three full-stops (... or …). So the idea is that the example text test string:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam felis nisi, eleifend ut quam eget, venenatis vestibulum turpis. Nam dignissim laoreet iaculis. Etiam sit amet rhoncus sem. Duis laoreet justo tellus, at volutpat risus molestie sed. Etiam posuere, arcu vitae faucibus hendrerit, lorem elit consequat urna, id congue eros felis in mauris. Donec non fermentum ipsum. Curabitur nec...
Would become:
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam felis nisi, eleifend ut quam eget, venenatis vestibulum turpis. Nam dignissim laoreet iaculis. Etiam sit amet rhoncus sem. Duis laoreet justo tellus, at volutpat risus molestie sed. Etiam posuere, arcu vitae faucibus hendrerit, lorem elit consequat urna, id congue eros felis in mauris. Donec non fermentum ipsum.
So far I have come up with this pathetic attempt. I keep getting right up until the last full-stop (because the quantifier consumes the previous two full-stops so there is nothing for the look ahead to fail on). I just can't seem to wrap my head around it:
Dim testText As String = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam felis nisi, eleifend ut quam eget, venenatis vestibulum turpis. Nam dignissim laoreet iaculis. Etiam sit amet rhoncus sem. Duis laoreet justo tellus, at volutpat risus molestie sed. Etiam posuere, arcu vitae faucibus hendrerit, lorem elit consequat urna, id congue eros felis in mauris. Donec non fermentum ipsum. Curabitur nec..."
Dim ellipsisExpression As String = "(.*\.(?!\.\.))"
Dim ellipsisMatch As Match
ellipsisMatch = Regex.Match(testText, ellipsisExpression)
If ellipsisMatch.Success Then
testText = ellipsisMatch.Groups(1).Value
End If
edit: I also need this expression to take any ... character in the text into account. for example the string:
`begin. this is a test... test complete. beginning shutdown... shutting down... `
should return
`begin. this is a test... test complete.`
The aim of this expression is to find the most flowing text before any truncation has occurred. A block of text with closure so it doesn't confuse readers expecting to be able to 'get more'.
You could replace [^.]*(?:\.{3}|…).* with an empty string to get the desired result.
For example:
result = Regex.Replace(input, "[^.]*(?:\\.{3}|…).*", "")
Use this:
result = Regex.Replace(input, "(.+\.).+(?:\.{3}|…)\s*", "$1")
Edit:
Use this regex instead:
(.+[^.]\.)(?:(?:[^.]{2})|$)
You could match that with:
.*(?<!\.)\.(?!\.)(?=(?:[^.]+|\.{3})*(?:\.{3}|…)$)
Or replace
(?<!\.)\.(?!\.)(?:[^.]+|\.{3})*(?:\.{3}|…)$
with a ..
I think I have come up with a solution that works for me. Thank you to everyone who answered previously but this expression seems to do what I need and doesn't execute as slowly as some of the other answers. It also takes other sentence terminating punctuation into account such as ! or ? and not just ..
(.*([^\.](?=\.|\?|!)(?!\.\.\.)).)
This get's the last sentence terminating character (defined with the lookahead). In this case they are ?, ! and . that isn't followed by .... This also solves the ellipsis character issue since it is effectively a sentence terminating white list. This expression succeeds in finding the largest block of text with closure.
Is there a regex to match "all characters including newlines"?
For example, in the regex below, there is no output from $2 because (.+?) doesn't include new lines when matching.
$string = "START Curabitur mollis, dolor ut rutrum consequat, arcu nisl ultrices diam, adipiscing aliquam ipsum metus id velit. Aenean vestibulum gravida felis, quis bibendum nisl euismod ut.
Nunc at orci sed quam pharetra congue. Nulla a justo vitae diam eleifend dictum. Maecenas egestas ipsum elementum dui sollicitudin tempus. Donec bibendum cursus nisi, vitae convallis ante ornare a. Curabitur libero lorem, semper sit amet cursus at, cursus id purus. Cras varius metus eu diam vulputate vel elementum mauris tempor.
Morbi tristique interdum libero, eu pulvinar elit fringilla vel. Curabitur fringilla bibendum urna, ullamcorper placerat quam fermentum id. Nunc aliquam, nunc sit amet bibendum lacinia, magna massa auctor enim, nec dictum sapien eros in arcu.
Pellentesque viverra ullamcorper lectus, a facilisis ipsum tempus et. Nulla mi enim, interdum at imperdiet eget, bibendum nec END";
$string =~ /(START)(.+?)(END)/;
print $2;
If you don't want add the /s regex modifier (perhaps you still want . to retain its original meaning elsewhere in the regex), you may also use a character class. One possibility:
[\S\s]
a character which is not a space or is a space. In other words, any character.
You can also change modifiers locally in a small part of the regex, like so:
(?s:.)
Add the s modifier to your regex to cause . to match newlines:
$string =~ /(START)(.+?)(END)/s;
Yeap, you just need to make . match newline :
$string =~ /(START)(.+?)(END)/s;
You want to use "multiline".
$string =~ /(START)(.+?)(END)/m;
I've struggled with regExp in Perl for some reason from the start and have a quick script i wrote here to count sentences in some text being inputted that won't work. I just get the number 1 back at the end and I know in the file specified there is several so the count should be higher. I can't see the issue...
#!C:\strawberry\perl\bin\perl.exe
#strict
#diagnostics
#warnings
$count = 0;
$file = "c:/programs/lorem.txt";
open(IN, "<$file") || die "Sorry, the file failed to open: $!";
while($line = <IN>)
{
if($line =~ m/^[A-Z]/)
{
$count++;
}
}
close(IN);
print("Sentances count was: ($count)");
The file lorem.txt is here......
Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc,
I don't know what's in your lorem.txt, but the code that you've given is not counting sentences. It's counting lines, and furthermore it's counting lines that begin with a capital letter.
This regex:
/^[A-Z]/
will only match at the beginning of a line, and only if the first character on that line is capitalized. So if you have a line that looks like it. And then we went... it will not be matched.
If you want to match all capital letters, just remove the ^ from the beginning of the regex.
This does not answer your specific question about regexp, but you could consider using a CPAN module: Text::Sentence. You can look at its source code to see how it defines a sentence.
use warnings;
use strict;
use Data::Dumper;
use Text::Sentence qw(split_sentences);
my $text = <<EOF;
One sentence. Here is another.
And yet another.
EOF
my #sentences = split_sentences($text);
print Dumper(\#sentences);
__END__
$VAR1 = [
'One sentence.',
'Here is another.',
'And yet another.'
];
A google search also turned up: Lingua::EN::Sentence
You are currently counting all lines that begin with a capital letter. Perhaps you intend to count all words that start with a capital letter? If so, try:
m/\W[A-Z]/
(Although this is not a robust count of sentences)
On another note, there is no need to do the file manipulation explicitly. perl does a really good job of that for you. Try this:
$ARGV[ 0 ] = "c:/programs/lorem.txt" unless #ARGV;
while( $line = <> ) {
...
If you do insist on doing an explicit open/close, it is considered bad practice to use raw filehandles. In other words, instead of "open IN...", do "open my $fh, '<', $file_name;"