PowerShell: Pull table from a string - regex
I have a command that I ran on PowerShell 2.0 that gives me the output below.
Everything in the screenshot above is one giant string. I want to pull out the table part of the string so that I can potentially format it as a list. Ultimately, I want to output to look like:
INSTANCE_NAME: Sample Name
STATUS: MOUNTED
DATABASE_STATUS: ACTIVE
My first thought was the use regex to pull out the table. I thought something like this might work, but I've so far been unsuccessful.
$tabletext = [regex]::match($rawtext, "(INSTANCE_NAME(.+\r\n)+)")
EDIT:
Here is the text as a string.
SQL*Plus: Release 12.1.0.1.0 Production on Wed Apr 20 16:34:57 2016
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
SQL>
INSTANCE_NAME STATUS DATABASE_STATUS
---------------- ------------ -----------------
sample_name OPEN ACTIVE
SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
I've done something very similar to parse Firebird sql output.
Here's a script that works on your sample data:
function parse-headers($lines) {
$h = $lines[0]
$headers = $h.Split(" ", [System.StringSplitOptions]::RemoveEmptyEntries)
$headers = $headers | % {
new-object pscustomobject -property #{
Length = $_.Length
Offset = $h.IndexOf($_)
Text = $_
}
}
for($i = 1; $i -lt $headers.length; $i++) {
$headers[$i-1].Length = $headers[$i].Offset - $headers[$i-1].Offset - 1
}
$headers[$header.length-1].Length = $h.length - $headers[$header.length-1].Offset
return $headers
}
function parse-sqloutput($lines) {
$headers = parse-headers $lines
$result = #()
for($l = 2; $l -lt $lines.Length; $l++) {
$line = $lines[$l]
$headernames = $headers | % { $h = #{} } { $h[$_.Text] = $null } { $h }
$r = New-Object -type pscustomobject -Property $headernames
for($i = 0; $i -lt $headers.length; $i++) {
try {
$h = $headers[$i]
$name = $h.text
if ($i -eq $headers.length - 1) {
$value = $line.Substring($h.Offset).Trim()
}
else {
$value = $line.Substring($h.Offset, $h.Length).Trim()
}
$r.$name = $value
} catch {
Write-Warning "failed to parse line $l col $i"
throw
}
}
$result += $r
}
return $result
}
function get-sqltable($sqlout) {
#find sql table output
$startidx = -1
$endidx = -1
for($i = 0; $i -lt $sqlout.Length; $i++) {
$line = $sqlout[$i]
if ($line -match "^\s*([\-]+\s*)+$") {
$startidx = $i - 1
}
if ($startidx -ge 0 -and $line -match "^\s*$") {
$endidx = $i
}
if ($startidx -ge 0 -and $endidx -ge 0) { break }
}
$sqlout = $sqlout | select -Skip $startidx -First ($endidx-$startidx)
return $sqlout
}
$sqlout = get-content "sqlout.txt" | out-string
#split giant string into lines
$sqlout = $sqlout | Split-String "`r`n"
$sqlout = get-sqltable $sqlout
#we should now have only sql table in $sqlout
$result = parse-sqloutput $sqlout
$result | Format-List
The idea is:
Find a line that contains only strings of - chars and assume it marks header row.
Look for first empty line after header row and assume it is the end of the table output.
Parse the header row, get columns' names and lengths.
Get values basing on parsed column lengths.
Related
Replace next two lines after string match
I´m currently working on a script that should based on user´s choice replace two lines in a file after a matching string. The file I want to edit looks like this: [default] string_a=sadasdasdas string_b=dasdasdasdas [profile1] string_a=xxxxxx string_b=xsaassaasas [profile2] string_a=yyyyyyy string_b=yaayayayaya I want always to override string_a & string_b after [default]. Note that [default] could also be at the very bottom of the file, therefore I cannot just count lines an do it that static. The user can pick between (in this case) profile 1 & profile 2. After he picked e.g profile 2, string_a & string_b of profile2 should be replaced with string_a & string_b of default. My current code like like this: $filePath = './credentials' $fileContent = Get-Content $filePath $profiles = [regex]::Matches($fileContent, '\[(.*?)\]') |ForEach-Object { $_.Groups[1].Value } Write-Host "Following profiles found: " for ($i=0; $i -lt $profiles.length; $i++){ Write-Host $i"." $profiles[$i] } $userInput = Read-Host "Which profile set to default? " Write-Host $profiles[$userInput] $fileContent | Select-String $profiles[$userInput] -Context 1,2 | ForEach-Object { $stringA = $_.Context.PostContext[0] $stringB = $_.Context.PostContext[1] #At this point I have access to the both string´s I want to replace the string´s of the default profile # I could do this, but then I still have the old lines in the file... # So the following code is not an option. $NewContent = Get-Content -Path $filePath | ForEach-Object { # Output the existing line to pipeline in any case $_ # If line matches regex if($_ -match ('^' + [regex]::Escape('[default]'))) { # Add output additional line $stringA $stringB } } # Write content of $NewContent varibale back to file $NewContent | Out-File -FilePath $filePath -Encoding Default -Force } Example output file, in case the user picked profile1 as the new default [default] string_a=xxxxxx string_b=xsaassaasas [profile1] string_a=xxxxxx string_b=xsaassaasas [profile2] string_a=yyyyyyy string_b=yaayayayaya Hope this is not obvious, but as it is my first real powershell script I was not able to find a solution for my problem yet. Any help would be great! Thanks
Example: # This is sample data $lines = #(#' [default] string_a=sadasdasdas string_b=dasdasdasdas [profile1] string_a=xxxxxx string_b=xsaassaasas [profile2] string_a=yyyyyyy string_b=yaayayayaya '# -split "`r`n") # In real world use: # $encoding = [System.Text.Encoding]::ASCII # $lines = [System.IO.File]::ReadAllLines($path, $encoding) #Read file $lines = $lines | ForEach-Object { $_.Trim()} # Trim spaces $sections = #{} $currentSection = $null $hasErrors = $false $lines | ForEach-Object { if ([String]::IsNullOrWhiteSpace($_)) { #ignore } elseif ($_.StartsWith('[') -and $_.EndsWith(']') ) { $currentSection = $_.Substring($_.IndexOf('[') + 1, $_.LastIndexOf(']') - 1) $sections[$currentSection] = #{} } elseif ($sections.ContainsKey($currentSection)) { $PVPair = [String[]]$_.Split('=',2) if ($PVPair.Count -eq 2) { $sections[$currentSection][$PVPair[0]] = $PVPair[1] } else { Write-Warning -Message "Wrong line format [$($_)]" $hasErrors = $true } } else { Write-Warning -Message "Unexpected behaviour on section $currentSection, line $($_)" $hasErrors = $true } } if ($hasErrors) { Write-Error -Message 'Errors occured' return } # Choice $choice = $null $choiceVariants = #($sections.Keys | Where-Object { $_ -ne 'default' }) while ($choiceVariants -notcontains $choice) { Write-Host "Choose between $($choiceVariants -join ',')" $choice = $choiceVariants | Out-GridView -Title 'Choose variant' -OutputMode Single #Alternative: $choice = Read-Host -Prompt "Your choice" } Write-Host -ForegroundColor Yellow "You choose $($choice)" # Change $sections[$choice]['string_a'] = $sections['default']['string_a'] $sections[$choice]['string_b'] = 'newXSAA' # Output $outputLines = $sections.Keys | ForEach-Object { $sectionName = $_ Write-Output "[$sectionName]" $sections[$sectionName].Keys | ForEach-Object { Write-Output "$_=$($sections[$sectionName][$_])" } } # This is sample output $outputLines | % { Write-Host $_ -f Magenta } # In Real world: # [System.IO.File]::WriteAllLines($path, $outputLines, $encoding)
Skip Header Row in a High Performance Powershell Regex Script Block
I received some amazing help from Stack Overflow ... however ... it was so amazing I need a little more help to get to closer to the finish line. I'm parsing multiple enormous 4GB files 2X per month. I need be able to be able to skip the header, count the total lines, matched lines, and the not matched lines. I'm sure this is super-simple for a PowerShell superstar, but at my newbie PS level my skills are not yet strong. Perhaps a little help from you would save the week. :) Data Sample: ID FIRST_NAME LAST_NAME COLUMN_NM_TOO_LON5THCOLUMN 10000000001MINNIE MOUSE COLUMN VALUE LONGSTARTS 10000000002MICKLE ROONEY MOUSE COLUMN VALUE LONGSTARTS Code Block (based on this answer): #$match_regex matches each fixed length field by length; the () specifies that each matched field be stored in a capture group: [regex]$match_regex = '^(.{10})(.{50})(.{50})(.{50})(.{50})(.{3})(.{8})(.{4})(.{50})(.{2})(.{30})(.{6})(.{3})(.{4})(.{25})(.{2})(.{10})(.{3})(.{8})(.{4})(.{50})(.{2})(.{30})(.{6})(.{3})(.{2})(.{25})(.{2})(.{10})(.{3})(.{10})(.{10})(.{10})(.{2})(.{10})(.{50})(.{50})(.{50})(.{50})(.{8})(.{4})(.{50})(.{2})(.{30})(.{6})(.{3})(.{2})(.{25})(.{2})(.{10})(.{3})(.{4})(.{2})(.{4})(.{10})(.{38})(.{38})(.{15})(.{1})(.{10})(.{2})(.{10})(.{10})(.{10})(.{10})(.{38})(.{38})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})(.{10})$' Measure-Command { & { switch -File $infile -Regex { $match_regex { # Join what all the capture groups matched with a tab char. $Matches[1..($Matches.Count-1)].Trim() -join "`t" } } } | Out-File $outFile }
You only need to keep track of two counts - matched, and unmatched lines - and then a Boolean to indicate whether you've skipped the first line $first = $false $matched = 0 $unmatched = 0 . { switch -File $infile -Regex { $match_regex { if($first){ # Join what all the capture groups matched with a tab char. $Matches[1..($Matches.Count-1)].Trim() -join "`t" $matched++ } $first = $true } default{ $unmatched++ # you can remove this, if the pattern always matches the header $first = $true } } } | Out-File $outFile $total = $matched + $unmatched
Using System.IO.StreamReader reduced the processing time to about 20% of what it had been. This was absolutely needed for my requirement. I added logic and counters without sacrificing much on performance. The field counter and row by row comparison is particularly helpful in finding bad records. This is a copy/paste of actual code but I shortened some things, made some things slightly pseudo code, so you may have to play with it to get things working just so for yourself. Function Get-Regx-Data-Format() { Param ([String] $filename) if ($filename -eq 'FILE NAME') { [regex]$match_regex = '^(.{10})(.{10})(.{10})(.{30})(.{30})(.{30})(.{4})(.{1})' } return $match_regex } Foreach ($file in $cutoff_files) { $starttime_for_file = (Get-Date) $source_file = $file + '_' + $proc_yyyymm + $source_file_suffix $source_path = $source_dir + $source_file $parse_file = $file + '_' + $proc_yyyymm + '_load' +$parse_target_suffix $parse_file_path = $parse_target_dir + $parse_file $error_file = $file + '_err_' + $proc_yyyymm + $error_target_suffix $error_file_path = $error_target_dir + $error_file [regex]$match_data_regex = Get-Regx-Data-Format $file Remove-Item -path "$parse_file_path" -Force -ErrorAction SilentlyContinue Remove-Item -path "$error_file_path" -Force -ErrorAction SilentlyContinue [long]$matched_cnt = 0 [long]$unmatched_cnt = 0 [long]$loop_counter = 0 [boolean]$has_header_row=$true [int]$field_cnt=0 [int]$previous_field_cnt=0 [int]$array_length=0 $parse_minutes = Measure-Command { try { $stream_log = [System.IO.StreamReader]::new($source_path) $stream_in = [System.IO.StreamReader]::new($source_path) $stream_out = [System.IO.StreamWriter]::new($parse_file_path) $stream_err = [System.IO.StreamWriter]::new($error_file_path) while ($line = $stream_in.ReadLine()) { if ($line -match $match_data_regex) { #if matched and it's the header, parse and write to the beg of output file if (($loop_counter -eq 0) -and $has_header_row) { $stream_out.WriteLine(($Matches[1..($array_length)].Trim() -join "`t")) } else { $previous_field_cnt = $field_cnt #add year month to line start, trim and join every captured field w/tabs $stream_out.WriteLine("$proc_yyyymm`t" + ` ($Matches[1..($array_length)].Trim() -join "`t")) $matched_cnt++ $field_cnt=$Matches.Count if (($previous_field_cnt -ne $field_cnt) -and $loop_counter -gt 1) { write-host "`nError on line $($loop_counter + 1). ` The field count does not match the previous correctly ` formatted (non-error) row." } } } else { if (($loop_counter -eq 0) -and $has_header_row) { #if the header, write to the beginning of the output file $stream_out.WriteLine($line) } else { $stream_err.WriteLine($line) $unmatched_cnt++ } } $loop_counter++ } } finally { $stream_in.Dispose() $stream_out.Dispose() $stream_err.Dispose() $stream_log.Dispose() } } | Select-Object -Property TotalMinutes write-host "`n$file_list_idx. File $file parsing results....`nMatched Count = $matched_cnt UnMatched Count = $unmatched_cnt Parse Minutes = $parse_minutes`n" $file_list_idx++ $endtime_for_file = (Get-Date) write-host "`nEnded processing file at $endtime_for_file" $TimeDiff_for_file = (New-TimeSpan $starttime_for_file $endtime_for_file) $Hrs_for_file = $TimeDiff_for_file.Hours $Mins_for_file = $TimeDiff_for_file.Minutes $Secs_for_file = $TimeDiff_for_file.Seconds write-host "`nElapsed Time for file $file processing: $Hrs_for_file`:$Mins_for_file`:$Secs_for_file" } $endtime = (Get-Date -format "HH:mm:ss") $TimeDiff = (New-TimeSpan $starttime $endtime) $Hrs = $TimeDiff.Hours $Mins = $TimeDiff.Minutes $Secs = $TimeDiff.Seconds write-host "`nTotal Elapsed Time: $Hrs`:$Mins`:$Secs"
Loop through a text file and Extract a set of 100 IP's from a text file and output to separate text files
I have a text file that contains around 900 IP's. I need to create batch of 100 IP's from that file and output them into new files. That would create around 9 text files. Our API only allows to POST 100 IP's at a time. Could you please help me out here? Below is the format of the text file 10.86.50.55,10.190.206.20,10.190.49.31,10.190.50.117,10.86.50.57,10.190.49.216,10.190.50.120,10.190.200.27,10.86.50.58,10.86.50.94,10.190.38.181,10.190.50.119,10.86.50.53,10.190.50.167,10.190.49.30,10.190.49.89,10.190.50.115,10.86.50.54,10.86.50.56,10.86.50.59,10.190.50.210,10.190.49.20,10.190.50.172,10.190.49.21,10.86.49.18,10.190.50.173,10.86.49.49,10.190.50.171,10.190.50.174,10.86.49.63,10.190.50.175,10.13.12.200,10.190.49.27,10.190.49.19,10.86.49.29,10.13.12.201,10.86.49.28,10.190.49.62,10.86.50.147,10.86.49.24,10.86.50.146,10.190.50.182,10.190.50.25,10.190.38.252,10.190.50.57,10.190.50.54,10.86.50.78,10.190.50.23,10.190.49.8,10.86.50.80,10.190.50.53,10.190.49.229,10.190.50.58,10.190.50.130,10.190.50.22,10.86.52.22,10.19.68.61,10.41.43.130,10.190.50.56,10.190.50.123,10.190.49.55,10.190.49.66,10.190.49.68,10.190.50.86,10.86.49.113,10.86.49.114,10.86.49.101,10.190.50.150,10.190.49.184,10.190.50.152,10.190.50.151,10.86.49.43,10.190.192.25,10.190.192.23,10.190.49.115,10.86.49.44,10.190.38.149,10.190.38.151,10.190.38.150,10.190.38.152,10.190.38.145,10.190.38.141,10.190.38.148,10.190.38.142,10.190.38.144,10.190.38.147,10.190.38.143,10.190.38.146,10.190.192.26,10.190.38.251,10.190.49.105,10.190.49.110,10.190.49.137,10.190.49.242,10.190.50.221,10.86.50.72,10.86.49.16,10.86.49.15,10.190.49.112,10.86.49.32,10.86.49.11,10.190.49.150,10.190.49.159,10.190.49.206,10.86.52.28,10.190.49.151,10.190.49.207,10.86.49.19,10.190.38.103,10.190.38.101,10.190.38.116,10.190.38.120,10.190.38.102,10.190.38.123,10.190.38.140,10.190.198.50,10.190.38.109,10.190.38.108,10.190.38.111,10.190.38.112,10.190.38.113,10.190.38.114,10.190.49.152,10.190.50.43,10.86.49.23,10.86.49.205,10.86.49.220,10.190.50.230,10.190.192.238,10.190.192.237,10.190.192.239,10.190.50.7,10.190.50.10,10.86.50.86,10.190.38.125,10.190.38.127,10.190.38.126,10.190.50.227,10.190.50.149,10.86.49.59,10.190.49.158,10.190.49.157,10.190.44.11,10.190.38.124,10.190.50.153,10.190.49.40,10.190.192.235,10.190.192.236,10.190.50.241,10.190.50.240,10.86.46.8,10.190.38.234,10.190.38.233,10.86.50.163,10.86.50.180,10.86.50.164,10.190.49.245,10.190.49.244,10.190.192.244,10.190.38.130,10.86.49.142,10.86.49.102,10.86.49.141,10.86.49.67,10.190.50.206,10.190.192.243,10.190.192.241 I tried looking online to come up with a bit of working code but can't really think what would best work in this situation $IP = 'H:\IP.txt' $re = '\d*.\d*.\d*.\d*,' Select-String -Path $IP -Pattern $re -AllMatches | Select-Object -Expand Matches | ForEach-Object { $Out = 'C:\path\to\out.txt' -f | Set-Content $clientlog }
This will do what you are after $bulkIP = (get-content H:\IP.txt) -split ',' $i = 0 # Created loop Do{ # Completed an action every 100 counts (including 0) If(0 -eq $i % 100) { # If the array is a valid entry. Removing this will usually end up creating an empty junk file called -1 or something If($bulkIP[$i]) { # outputs 100 lines into a folder with the starting index as the name. # Eg. The first 1-100, the file would be called 1.txt. 501-600 would be called 501.txt etc $bulkIP[$($i)..$($i+99)] | Out-File "C:\path\to\$($bulkip.IndexOf($bulkip[$($i)+1])).txt" } } $i++ }While($i -le 1000)
what this does ... calculates the number of batches calcs the start & end index of each batch creates a range from the above creates a PSCustomObject to hold each batch creates an array slice from the range sends that out to the collection $Var shows what is in the collection & in the 1st batch from that collection here's the code ... # fake reading in a raw text file # in real life, use Get-Content -Raw $InStuff = #' 10.86.50.55,10.190.206.20,10.190.49.31,10.190.50.117,10.86.50.57,10.190.49.216,10.190.50.120,10.190.200.27,10.86.50.58,10.86.50.94,10.190.38.181,10.190.50.119,10.86.50.53,10.190.50.167,10.190.49.30,10.190.49.89,10.190.50.115,10.86.50.54,10.86.50.56,10.86.50.59,10.190.50.210,10.190.49.20,10.190.50.172,10.190.49.21,10.86.49.18,10.190.50.173,10.86.49.49,10.190.50.171,10.190.50.174,10.86.49.63,10.190.50.175,10.13.12.200,10.190.49.27,10.190.49.19,10.86.49.29,10.13.12.201,10.86.49.28,10.190.49.62,10.86.50.147,10.86.49.24,10.86.50.146,10.190.50.182,10.190.50.25,10.190.38.252,10.190.50.57,10.190.50.54,10.86.50.78,10.190.50.23,10.190.49.8,10.86.50.80,10.190.50.53,10.190.49.229,10.190.50.58,10.190.50.130,10.190.50.22,10.86.52.22,10.19.68.61,10.41.43.130,10.190.50.56,10.190.50.123,10.190.49.55,10.190.49.66,10.190.49.68,10.190.50.86,10.86.49.113,10.86.49.114,10.86.49.101,10.190.50.150,10.190.49.184,10.190.50.152,10.190.50.151,10.86.49.43,10.190.192.25,10.190.192.23,10.190.49.115,10.86.49.44,10.190.38.149,10.190.38.151,10.190.38.150,10.190.38.152,10.190.38.145,10.190.38.141,10.190.38.148,10.190.38.142,10.190.38.144,10.190.38.147,10.190.38.143,10.190.38.146,10.190.192.26,10.190.38.251,10.190.49.105,10.190.49.110,10.190.49.137,10.190.49.242,10.190.50.221,10.86.50.72,10.86.49.16,10.86.49.15,10.190.49.112,10.86.49.32,10.86.49.11,10.190.49.150,10.190.49.159,10.190.49.206,10.86.52.28,10.190.49.151,10.190.49.207,10.86.49.19,10.190.38.103,10.190.38.101,10.190.38.116,10.190.38.120,10.190.38.102,10.190.38.123,10.190.38.140,10.190.198.50,10.190.38.109,10.190.38.108,10.190.38.111,10.190.38.112,10.190.38.113,10.190.38.114,10.190.49.152,10.190.50.43,10.86.49.23,10.86.49.205,10.86.49.220,10.190.50.230,10.190.192.238,10.190.192.237,10.190.192.239,10.190.50.7,10.190.50.10,10.86.50.86,10.190.38.125,10.190.38.127,10.190.38.126,10.190.50.227,10.190.50.149,10.86.49.59,10.190.49.158,10.190.49.157,10.190.44.11,10.190.38.124,10.190.50.153,10.190.49.40,10.190.192.235,10.190.192.236,10.190.50.241,10.190.50.240,10.86.46.8,10.190.38.234,10.190.38.233,10.86.50.163,10.86.50.180,10.86.50.164,10.190.49.245,10.190.49.244,10.190.192.244,10.190.38.130,10.86.49.142,10.86.49.102,10.86.49.141,10.86.49.67,10.190.50.206,10.190.192.243,10.190.192.241 '# $SplitInStuff = $InStuff.Split(',') $BatchSize = 25 $BatchCount = [math]::Truncate($SplitInStuff.Count / $BatchSize) + 1 $Start = $End = 0 $Result = foreach ($BC_Item in 1..$BatchCount) { $Start = $End if ($BC_Item -eq 1) { $End = $Start + $BatchSize - 1 } else { $End = $Start + $BatchSize } $Range = $Start..$End [PSCustomObject]#{ IP_List = $SplitInStuff[$Range] } } $Result '=' * 20 $Result[0] '=' * 20 $Result[0].IP_List.Count '=' * 20 $Result[0].IP_List screen output ... IP_List ------- {10.86.50.55, 10.190.206.20, 10.190.49.31, 10.190.50.117...} {10.86.49.18, 10.190.50.173, 10.86.49.49, 10.190.50.171...} {10.86.50.80, 10.190.50.53, 10.190.49.229, 10.190.50.58...} {10.190.49.115, 10.86.49.44, 10.190.38.149, 10.190.38.151...} {10.86.49.32, 10.86.49.11, 10.190.49.150, 10.190.49.159...} {10.86.49.23, 10.86.49.205, 10.86.49.220, 10.190.50.230...} {10.190.50.240, 10.86.46.8, 10.190.38.234, 10.190.38.233...} ==================== {10.86.50.55, 10.190.206.20, 10.190.49.31, 10.190.50.117...} ==================== 25 ==================== 10.86.50.55 10.190.206.20 10.190.49.31 10.190.50.117 10.86.50.57 10.190.49.216 10.190.50.120 10.190.200.27 10.86.50.58 10.86.50.94 10.190.38.181 10.190.50.119 10.86.50.53 10.190.50.167 10.190.49.30 10.190.49.89 10.190.50.115 10.86.50.54 10.86.50.56 10.86.50.59 10.190.50.210 10.190.49.20 10.190.50.172 10.190.49.21 10.86.49.18
try this $cpt=0 $Rang=1 #remove old file Get-ChildItem "H:\FileIP_*.txt" -file | Remove-Item -Force (Get-Content "H:\IP.txt") -split ',' | %{ if (!($cpt++ % 100)) {$FileResult="H:\FileIP_{0:D3}.txt" -f $Rang++} # build filename if cpt divisile by 100 $_ | Out-File $FileResult -Append }
Powershell log tailer issues
I have written a log tailer with Powershell, the tailer loads in an xml file which contains configuration information regarding when to report on a word match in the log tail (basically if certain patterns occur X amount of times in the tail). At the moment the tailer is not returning matches for many of the lines that contain matches. For example we are retrieving a log file with many INFO lines, if I check for the word INFO nothing is detected, however if I look for the work shutdown it returns matches (the line with shutdown also contains INFO on the line). The really strange thing is that using the same log file and same Powershell script seems to produce perfectly accurate results on my own machine but behaves strangely on the server. I suspect that this might be an issue with the version of Powershell that is running on the server, so I was hoping someone here might know of issues that can come up with different versions. I have also noticed that when I print out the number of matches, if nothing is found the output is blank, perhaps this should be 0 and is causing some weird issue to trigger? function Main() { #### GLOBAL SETTINGS $DebugPreference = "Continue" $serverName = $env:COMPUTERNAME $scriptPath = Split-Path $script:MyInvocation.MyCommand.Path $logConfigPath = "$scriptPath/config.xml" #### VARIABLES RELATING TO THE LOG FILE #contains the log path and log file mask $logPaths = #() $logFileMasks = #() # the total number of lines grabbed from the end of the log file for evaluation $numLinesToTail = 1000 # key value pair for the strings to match and the max count of matches before they are considered an issue $keywords = #() $maxCounts = #() #### VARIABLES RELATING TO THE EMAIL SETTINGS $smtpServer = "mail server" $emailSubject = "$serverName log report" $toEmailAddress = "email accounts" $fromEmailAddress = "" # any initial content you want in the email body should go here (e.g. the name of the server that this is on) $htmlBodyContent = "<p><h3>SERVER $serverName : </h3></p><p>Items that appear in red have exceeded their match threshold and should be investigated.<br/>Tail Lines: $numLinesToTail</p>" #### FUNCTION CALLS LoadLogTailerConfig $logConfigPath ([ref]$logPaths) ([ref]$logFileMasks) ([ref]$keywords) ([ref]$maxCounts) for ($i = 0; $i -lt $logPaths.Count; $i++) { $tail = GetLogTail $numLinesToTail $logPaths[$i] $logFileMasks[$i] $tailIssueTable = CheckForKeywords $tail $keywords[$i] $maxCounts[$i] if ($tailIssueTable -ne "") { $htmlBodyContent += "<br/>Logs scanned: " + (GetLatestLogFileFullName $logPaths[$i] $logFileMasks[$i]) + "<br/><br/>" + $tailIssueTable SendIssueEmail $smtpServer $emailSubject $toEmailAddress $ccEmailAddress $fromEmailAddress $htmlBodyContent } } } # Loads in configuration data for the utility to use function LoadLogTailerConfig($logConfigPath, [ref]$logPaths, [ref]$logFileMasks, [ref]$keywords, [ref]$maxCounts) { Write-Debug "Loading config file data from $logConfigPath" [xml]$configData = Get-Content $logConfigPath foreach ($log in $configData.Logs.Log) { $logPaths.Value += $log.FilePath $logFileMasks.Value += $log.FileMask $kwp = #() $kwc = #() foreach ($keywordSet in $log.Keywords.Keyword) { $kwp += $keywordSet.Pattern $kwc += $keywordSet.MaxMatches } $keywords.Value += #(,$kwp) $maxCounts.Value += #(,$kwc) } } # Gets a string containing the last X lines of the most recent log file function GetLogTail($numLinesToTail, $logPath, $logFileMask) { $logFile = GetLatestLogFileFullName $logPath $logFileMask #Get-ChildItem $logPath -Filter $logFileMask | sort LastWriteTime | select -Last 1 Write-Debug "Getting $numLinesToTail line tail of $logFile" $tail = Get-Content "$logFile" | select -Last $numLinesToTail return $tail } function GetLatestLogFileFullName($logPath, $logFileMask) { $logFile = Get-ChildItem $logPath -Filter $logFileMask | sort LastWriteTime | select -Last 1 return "$logPath$logFile" } # Returns body text for email containing details on keywords in the log file and their frequency function CheckForKeywords($tail, $keywords, $maxCounts) { $issuesFound = 0 $htmlBodyContent += "<table><tr><th style=""text-align : left;"">Keyword</th><th>Max Count Value</th><th>Count Total<th></tr>" for ($i = 0; $i -lt $keywords.Count; $i++) { $keywordCount = ($tail | Select-String $keywords[$i] -AllMatches).Matches.Count Write-Debug (("Match count for {0} : {1}" -f $keywords[$i], $keywordCount)) if ($keywordCount -gt $maxCounts[$i]) { # style red if the count threshold has been exceeded $htmlBodyContent += "<tr style=""color : red;""><td>" + $keywords[$i] + "</td><td>" + $maxCounts[$i] + "</td><td>" + $keywordCount + "</td></tr>" $issuesFound = 1 } else { # style green if the count threshold has not been exceeded $htmlBodyContent += "<tr style=""color : green;""><td>" + $keywords[$i] + "</td><td>" + $maxCounts[$i] + "</td><td>" + $keywordCount + "</td></tr>" } } $htmlBodyContent += "</table>" if ($issuesFound -eq 1) { return $htmlBodyContent } return "" } # Sends out an email to the specified email address function SendIssueEmail($smtpServer, $subject, $toAddress, $ccAddress, $fromAddress, $bodyContent) { Write-Debug "Sending email with subject: $subject, To: $toAddress, via SMTP ($smtpServer)" Send-MailMessage -SmtpServer $smtpServer -Subject $subject -To $toAddress -From $fromAddress -BodyAsHtml $bodyContent } cls Main And a XML config example: <Logs> <Log> <FilePath>C:/Some/Path</FilePath> <FileMask>log.*</FileMask> <Keywords> <Keyword> <Pattern>NullReferenceException</Pattern> <MaxMatches>10</MaxMatches> </Keyword> <Keyword> <Pattern>Exception</Pattern> <MaxMatches>10</MaxMatches> </Keyword> </Keywords> </Log> <Log> <FilePath>C:/Some/Path</FilePath> <FileMask>test.*</FileMask> <Keywords> <Keyword> <Pattern>NullReferenceException</Pattern> <MaxMatches>100</MaxMatches> </Keyword> </Keywords> </Log> </Logs> EDIT : The server that is having the issues is running Powershell V 1.0, however the test servers are also running the same version perfectly fine...
Your function GetLatestLogFileFullName is one problem. It can and will generate invalid paths. function GetLatestLogFileFullName($logPath, $logFileMask) { $logFile = Get-ChildItem $logPath -Filter $logFileMask | sort LastWriteTime | select -Last 1 return "$logPath$logFile" } Use this instead: return $logfile.FullName And you should also check for cases where there is no valid log file: if ($logfile) { return $logfile.FullName } else { return $null } The second problem will be your Select-String usage. $keywordCount = ($tail | Select-String $keywords[$i] -AllMatches).Matches.Count In PowerShell v1 Select-String does not have -AllMatches parameter. PS> Get-Help Select-String NAME Select-String SYNOPSIS Identifies patterns in strings. SYNTAX Select-String [-pattern] <string[]> -inputObject <psobject>[-include <string[]>] [-exclude <string[]>] [-simpleMatch] [-caseSensitive] [-quiet] [-list] [<CommonParameters>] Select-String [-pattern] <string[]> [-path] <string[]> [-include<string[]>] [-exclude <string[]>] [-simpleMatch] [-caseSensitive] [-quiet] [-list] [<CommonParameters>] Check the PowerShell versions on your servers using the $PSVersionTable variable. Do not rely on the version displayed in the title bar! If the variable does not exist you have Version 1.
Special characters passed as parameters in URL
Requirement: Need way to handle Special characters like % and &. Need to tweak code below so that Special characters which come via $Control file are treated as it is. For example: I have one of entry in $control file as 25% Dextrose(25ml). I need a way so that $ie.Navigate should simply navigate to https://www.xxxy.com/search/all?name=25% Dextrose(25ml). Currently it gets routed to https://www.xxxy.com/search/all?name=25%% Dextrose(25ml) (note a extra % in URL) and thus does not find that web-page. **Few examples of special characters that need to be tackled:** '/' - 32care Mouth/Throat '%' - 3d1% Gel(30g) '&' - Accustix Glucose & Protein '/' - Ace Revelol(25/(2.5mg) function getStringMatch { # Loop through all 2 digit combinations in the $path directory foreach ($control In $controls) { $ie = New-Object -COMObject InternetExplorer.Application $ie.visible = $true $site = $ie.Navigate("https://www.xxxy.com/search/all?name=$control") $ie.ReadyState while ($ie.Busy -and $ie.ReadyState -ne 4){ sleep -Milliseconds 100 } $link = $null $link = $ie.Document.get_links() | where-object {$_.innerText -eq "$control"} $link.click() while ($ie.Busy -and $ie.ReadyState -ne 4){ sleep -Milliseconds 100 } $ie2 = (New-Object -COM 'Shell.Application').Windows() | ? { $_.Name -eq 'Windows Internet Explorer' -and $_.LocationName -match "^$control" } # NEED outerHTML of new page. CURRENTLY it is working for some. $ie.Document.body.outerHTML > d:\med$control.txt } } $controls = "Sporanox" getStringMatch
You want to URL encode the URI. Add this at the very start: Add-Type -AssemblyName 'System.Web' And then encode the URL like this: $controlUri = [System.Web.HttpUtility]::UrlEncode($control) $site = $ie.Navigate("https://www.xxxy.com/search/all?name=$controlUri")
As Biffen pointed out, Web servers will treat special characters as codes. So in your case, $control needs to be modified so that the Web server understands where you want to go. One way to fix it is the look for specific characters in the original product name you are looking for, and replace them with something that the server will understand: Here is the entire code: function getStringMatch { # Loop through all 2 digit combinations in the $path directory foreach ($control In $controls) { $ie = New-Object -COMObject InternetExplorer.Application $ie.visible = $true $s = $control -replace '%','%25' $s = $s -replace ' ','+' $s = $s -replace '&','%26' $s = $s -replace '/','%2F' $site = $ie.Navigate("https://www.xxxy.com/search/all?name=$s") while ($ie.Busy -and $ie.ReadyState -ne 4){ sleep -Milliseconds 100 } $link = $null $link = $ie.Document.get_links() | where-object {if ($_.innerText){$_.innerText.contains($control)}} $link.click() while ($ie.Busy){ sleep -Milliseconds 100 } $ie.Document.body.outerHTML > d:\TEMP\med$control.txt } } $controls = "Accustix Glucose & Protein" getStringMatch I tried with the following strings: "3d1% Gel(30g)" "Ace Revelol(25/2mg)" "Accustix Glucose & Protein" "32care Mouth/Throat"