Find combinations of two strings in multiple files - regex

I want to search through files in a folder and find the following strings in each file and I want to output it to a file. I would like to find a combination of 2 strings in the files no matter how it is written in the file. I should be able to find these combination of strings even if a carriage return exists in the middle of these 2 strings.
Here's the code I have so far:
$Path = "C:\Promotion\Scripts"
$txt_string1 = "CREATE"
$txt_string2 = "PROC"
$PathArray = #()
$Results = "C:\Promotion\Errors\Deployment_Errors.txt"
# This code snippet gets all the files in $Path that end in ".sql".
Get-ChildItem $Path -Filter "*.sql" |
Where-Object { $_.Attributes -ne "Directory"} |
ForEach-Object {
If (Get-Content $_.FullName | Select-String -Pattern $txt_string2) {
$PathArray += $_.FullName
}
}
$PathArray | ForEach-Object {$_} | Out-File $Results

for find more than one string in txt file You should Use like this method
"hello","guy","hello guy" | Select-String -Pattern '(hello.*guy)|(guy.*hello)'
the result :
hello guy
after you find strings you want out-file
like that:
"hello","guy","hello guy" | Select-String -Pattern '(hello.*guy)|(guy.*hello)' | Out-File -FilePath c:\test.txt
now we see in test.txt
PS C:\> Get-Content test.txt
hello guy

You can do this without loops. Define the combinations of your two search terms as alternatives in a regular expression with multiline support enabled ((?ms)).
$basepath = 'C:\Promotion\Scripts'
$results = 'C:\Promotion\Errors\Deployment_Errors.txt'
$term1 = 'CREATE'
$term2 = 'PROC'
$pattern = "(?ms)($term1.*$term2|$term2.*$term1)"
Get-ChildItem "$basepath\*.sql" |
? { Get-Content $_.FullName -Raw | Select-String -Pattern $pattern } |
select -Unique -Expand FullName |
Out-File $results
Note that this will report any file that contains both terms anywhere in it, no matter what other text is between them. If you want to find only files that contain combinations of the two terms either not separated (CREATEPROC or PROCCREATE) or separated nothing but whitespace, change the pattern to this:
$pattern = "(?ms)($term1\s*$term2|$term2\s*$term1)"
Depending on your search terms it may also be a good idea to escape them before building the regular expression, so that you don't get unwanted meta characters (not likely with the two string literals you have, but just to be on the safe side):
$term1 = [regex]::Escape('CREATE')
$term2 = [regex]::Escape('PROC')

Related

Print Powershell Regex captures to an output file

I have a file, input.txt, containing text like this:
GRP123456789
123456789012
GRP234567890
234567890123
GRP456789012
"A lot of text. More text. Blah blah blah: Foobar." (Source Error) (Blah blah blah)
GRP567890123
Source Error
GRP678901234
Source Error
GRP789012345
345678901234
456789012345
I'm attempting to capture all occurrences of "GRP#########" on the condition that at least one number is on the next line.
So GRP123456789 is valid, but GRP456789012 and GRP678901234 are not.
The RegEx pattern I came up with on http://regexstorm.net/tester is: (GRP[0-9]{9})\s\n\s+[0-9]
The PowerShell script I have so far, based off this site http://techtalk.gfi.com/windows-powershell-extracting-strings-using-regular-expressions/, is:
$input_path = 'C:\Users\rtaite\Desktop\input.txt'
$output_file = 'C:\Users\rtaite\Desktop\output.txt'
$regex = '(GRP[0-9]{9})\s\n\s+[0-9]'
select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Values } > $output_file
I'm not getting any output, and I'm not sure why.
Any help with this would be appreciated as I'm just trying to understand this better.
You need to turn the text input into a single string before passing it to Select-String, otherwise the cmdlet will operate on each line individually and thus never find a match.
Get-Content $input_path | Out-String |
Select-String $regex -AllMatches |
Select-Object -Expand Matches |
ForEach-Object { $_.Groups[1].Value } |
Set-Content $output_file
If you're using PowerShell v3 or newer you can replace Get-Content | Out-String with Get-Content -Raw.
To strip strings from a text file using a pattern, then the best tool for the job is the Select-String. This is also has a parameter called -Context which lets you capture lines before or after the matched line, ideal for just this problem.
So my solution would be something like this:
Select-String 'input.txt' -Pattern '^GRP[0-9]{9}' -Context 0, 1 | ? {
$_.Context.PostContext -match '\d'
} | Select -ExpandProperty line | Set-Content 'output_file.txt'
Using
[regex]::Matches($(Get-Content '.\Desktop\new 1.txt'), "GRP\d+(?=\s+\d)") |
% { $_.value | Out-File .\Desktop\new-1-matches.txt -Append }
I achieved the following output from your sample file:
GRP123456789
GRP234567890
GRP789012345

Using Select-String to extract all event titles, which are in parentheses, from a txt file

I'm trying to extract the titles of events in one of my logs, which is just a text file with lots of data. The filename is eventlog-1-5-2016.txt (date is always the current date). Each line in the file is one event like this:
1-1-16(Commodore Rally)Address|Time
1-2-16(Open House)Address|Time
I just want to go through the file and extract the title in parentheses, excluding the parentheses themselves, and output the list to the console, or a text file.
I've also tried output to a txt file but I'm missing something. Can you tell me why this doesn't work:
Console:
Select-String -Path c:\log\eventlog-1-5-2016.txt -Pattern '\(([^\)]+)\)' -AllMatches |
% { $_.Matches }
To File:
Select-string -Path c:\log\eventlog-1-5-2016.txt -Pattern '\(([^\)]+)\)' -AllMatches |
% { $_.Matches } | { $_.Value > C:\log\results.txt
or even a better way to do this if this wrong.
Bonus question, could the path auto calculate the current date and correct the file name for easy future pasting? (not major!)
The current date can be determined like this:
(Get-Date).ToString('d-M-yyyy')
Also, your regular expression could be simplified a little by using a non-greedy match:
`\((.+?)\)
If you want just the text between the parentheses you need the value of the captured group instead of the complete match:
$date = (Get-Date).ToString('d-M-yyyy')
Select-String -Path "C:\log\eventlog-$date.txt" -Pattern '\((.+?)\)' -AllMatches |
ForEach-Object { $_.Matches } |
ForEach-Object { $_.Groups[1] } |
ForEach-Object { $_.Value } |
Out-File 'C:\log\results.txt'
If you have PowerShell v3 or newer you could collapse the ForEach-Object statements:
Select-String -Path "C:\log\eventlog-$date.txt" -Pattern '\((.+?)\)' -AllMatches |
ForEach-Object { $_.Matches.Groups[1].Value } |
Out-File 'C:\log\results.txt'
Or you could use the -match operator:
Get-Content "C:\log\eventlog-$date.txt" |
Where-Object { $_ -match '\((.+?)\)' } |
ForEach-Object { $matches[1] } |
Set-Content 'C:\log\results.txt'
What happens is that Select-String doesn't quite do what you think it does. It will match patterns, but instead of returning the matched part, it will return you the whole matching string. Thus, the statement returns you the whole matched row instead of just substring in parenthesis. This is common a cause for confusion.
As a simple example in case of link rot,
[regex]$rx = '\(([^\)]+)\)'
cat C:\Temp\logfile.txt | % { $rx.Matches( $_ ).value }
(Commodore Rally)
(Open House)

PowerShell regex filter files

I am trying to filter files using PowerShell, and I need to insert a new line character in between </tr><tr> to break those into separate lines and then remove all the lines that match <tr> lots of characters BTE lots of characters </tr> and save the files in place.
Forgive me, as I am new to PowerShell, and this is simple in SED, but I must use PowerShell. This is what I have but could be completely wrong.
Get-Content *.htm | Foreach-Object {$_ -replace '</tr><tr>', '</tr>\r\n<tr>'; $_}f
Get-Content *.htm | Foreach-Object {$_ -replace '<tr>.*BTE.*</tr>', ''; $_}
So it just sounds like you need to save your changes back to the original files. Also we should just be able to make these changes in one pass instead of reading the files twice.
Get-ChildItem *.htm | Foreach-Object {
$singleFileName = $_.FullName
(Get-Content $singleFileName) -replace '</tr><tr>', "</tr>`r`n<tr>" -replace '<tr>.*BTE.*</tr>' | Set-Content $singleFileName
}
You can't read and write to the same file in the pipe. We place (Get-Content $singleFileName) in parenthesis so that the whole file is read at once.
Get-Content $singleFileName | Set-Content $singleFileName
As each line is passed down the pipe the file is left open so that Set-Content can't write to it.
I don't think you have to insert the line break if RegEx is able to capture the group like this.
Get-ChildItem *.htm | Foreach-Object {
$singleFileName = $_.FullName
([RegEx]::Matches((Get-Content $singleFileName),'<tr>.*?</tr>')).Value|?{$_ -notlike '<tr>*BTE*</tr>'} | Set-Content $singleFileName
}

PowerShell regex export match contents

I am learning regex and am trying to get a better understanding by using a text file with the value $100,000 in it. What I am trying to do is to search the text file for the string "$100,000" and if it is there export the value out into a new CSV. this is what I'm using so far.
[io.file]::readalltext("c:\utilities\notes_$datetime.txt") -match("[$][0-9][0-9][0-9],[0-9][0-9][0-9]") | Out-File C:\utilities\amount.txt -Encoding ascii -Force
Which returns true. Can someone point me in the right direction as to grabbing the string value that it finds into a new CSV?
many thanks!
You're reading the file into a single string, not an array of lines, so you should use the Select-String -AllMatches instead of the -match operator:
[IO.File]::ReadAllText("c:\utilities\notes_$datetime.txt") |
Select-String '\$\d{3},\d{3}' -AllMatches |
% { $_.Matches.Groups.Value } |
Out-File C:\utilities\amount.txt -Encoding ascii -Force
As a side note, using Get-Content -Raw would be slightly more PoSh than using .Net methods, although .Net methods provide better performance.
Get-Content "c:\utilities\notes_$datetime.txt" -Raw |
Select-String '\$\d{3},\d{3}' -AllMatches |
% { $_.Matches.Groups.Value } |
Out-File C:\utilities\amount.txt -Encoding ascii -Force
I prefer to use [regex]::match for that:
$x = 'text bla $100,000 text text'
[regex]::Match($x,"\$[\d]{3},[\d]{3}").Groups[0].Value
I also changed the expression a little bit ($ followed by 3 numbers, followed by a "," and another 3 numbers).
So your script could look like this:
$fileContent = Get-Content "c:\utilities\notes_$datetime.txt"
[regex]::Match($fileContent,"\$[\d]{3},[\d]{3}").Groups[0].Value | Out-File C:\utilities\amount.txt -Encoding ascii -Force
Why not use the Select-String cmdlet - far easier:
Select-String .\infile.csv -pattern '\$[\d]{3},[\d]{3}' | Select Line | Out-File outfile.txt
You can then process multiple files like so:
Get-Childitem *.csv | Select-String -pattern '\$[\d]{3},[\d]{3}' | Select Line | Out-File outfile.txt
The Select-String has the following properties:
Line - the line where the regex found a match
LineNumber - the line number in the file where the match was found
Filename - the name of the file the match was found in

How to find all regular expression matches in the file

I have a list of regular expressions(about 2000) and over a million html files. I want to check if each regular expression success on every file or not. How to do this on powershell?
Performance is important, so I don't want to loop through regular expressions.
I try
$text | Select-String -Pattern pattern1, pattern2,...
And it returns all matches, but I also want to find out, which pattern success which one not. I need to build a list of success regular expressions for each file
You could try something like this:
$regex = "^test","e2$" #Or use (Get-Content <path to your regex file>)
$ht = #{}
#Modify Get-Childitem to your criterias(filter, path, recurse etc.)
Get-ChildItem -Filter *.txt | Select-String -Pattern $regex | ForEach-Object {
$ht[$_.Path] += #($_ | Select-Object -ExpandProperty Pattern)
}
Test-output:
$ht | Format-Table -AutoSize
Name Value
---- -----
C:\Users\graimer\Desktop\New Text Document (2).txt {e2$}
C:\Users\graimer\Desktop\New Text Document.txt {^test, e2$}
You didn't specify how you wanted the output.
UPDATE: To match multiple patterns on a single line, try this(mjolinor's answer is probably faster then this).
$regex = "^test","e2$" #Or use (Get-Content <path to your regex file>)
$ht = #{}
#Modify Get-Childitem to your criterias(filter, path, recurse etc.)
$regex | ForEach-Object {
$pattern = $_
Get-ChildItem -Filter *.txt | Select-String -Pattern $pattern | ForEach-Object {
$ht[$_.Path] += #($_ | Select-Object -ExpandProperty Pattern)
}
}
UPDATE2: I don't have enough samples to try it, but since you have such a huge amount of files, you migh want to try reading the file into memory before looping through the patterns. It may be faster.
$regex = "^test","e2$" #Or use (Get-Content <path to your regex file>)
$ht = #{}
#Modify Get-Childitem to your criterias(filter, path, recurse etc.)
Get-ChildItem -Filter *.txt | ForEach-Object {
$text = $_ | Get-Content
$filename = $_.FullName
$regex | ForEach-Object {
$text | Select-String -Pattern $_ | ForEach-Object {
$ht[$filename] += #($_ | Select-Object -ExpandProperty Pattern)
}
}
}
I don't see any way around doing a foreach through the regex collection.
This is the best I could come up with performance-wise:
$regexes = 'pattern1','pattern2'
$files = get-childitem -Path <file path> |
select -ExpandProperty fullname
$ht = #{}
foreach ($file in $files)
{
$ht[$file] = New-Object collections.arraylist
foreach ($regex in $regexes)
{
if (select-string $regex $file -Quiet)
{
[void]$ht[$file].add($regex)
}
}
}
$ht
You could speed up the process by using background jobs and dividing up the file collection among the jobs.