Powershell: apply multiple queries to a foreach on logs - regex

I need to get the total data transferred value from multiple log files. The data value is shown under the total column in the bytes row.
It is also respectively delimited with a g, m, t or k to show gb/mb/tb/kb.
Total Copied Skipped Mismatch FAILED Extras
Bytes : 54.414 g 54.414 g 0 0 0 0
Currently I have this script which can go through all the files and extract the value between Bytes: and the g but I need to be able to add more queries and to the `foreach file and sum them all to one consistent value.
This is what I have currently but only outputs the kb.
$pattern = "(?<=.*Bytes :.*)\w.+?(?= g.*)"
$pattern1 = "(?<=.*Bytes :.*)\w.+?(?= m.*)"
$pattern2 = "(?<=.*Bytes :.*)\w.+?(?= k.*)"
Get-ChildItem "C:\Users\logs" -Filter "BFR*" | ForEach-Object {
Get-Content "C:\Users\logs\*.log" | where-Object {$_ -match $pattern } | ForEach-Object {
[double] ($matches[0])
} | Measure-Object -Sum | Select-Object -ExpandProperty sum
} | ForEach-Object {
Get-Content "C:\Users\logs\*.log" | where-Object {$_ -match $pattern1 } | ForEach-Object {
[double] ($matches[0])
} | Measure-Object -Sum | Select-Object -ExpandProperty sum
} | ForEach-Object {
Get-Content "C:\Users\logs\*.log" | where-Object {$_ -match $pattern2 } | ForEach-Object {
[double] ($matches[0])
} | Measure-Object -Sum | Select-Object -ExpandProperty sum
}

Here is one possibility. (Sorry #JamesC, it's a 1-liner :-)):
Get-ChildItem .\LogFolder\*.log |
ForEach-Object {$totalBytes = 0}{
Get-Content $_ | Select-String -Pattern "^Bytes\s+:\s+(?<size>\d+\.\d+) (?<units>[tgmk]).*$" |
Foreach-Object {
$size = $_.Matches.Groups[1].Value
switch ($_.Matches.Groups[2].Value)
{
t {$totalbytes += (1tb * $size)}
g {$totalbytes += (1gb * $size)}
m {$totalbytes += (1mb * $size)}
k {$totalbytes += (1kb * $size)}
}
}
} {"Total Bytes: $totalBytes"}

Related

Keep the $character in regular expression replace

Two problems of regular replace
1.need to keep the front $character in the replacement result
2.Skipping the first two lines and the last line is not valid
Code:
$str = #'
#$start1 Random characters
#$start2 Random characters
$p1.AppendBreak($BreakType.LineBreak)
$doc.Protect($ProtectionType.AllowOnlyRevisions, "123")
$footerPara.AppendField("page", $FieldType.FieldPage)
$footerParagraph.AppendField("number of pages", $FieldType.FieldSectionPages)
$txtWatermark.Layout = $WatermarkLayout.Diagonal
$tr1.CharacterFormat.Border.BorderType = $BorderStyle.DashDotStroker
$stri.CharacterFormat.TextBackgroundColor = $Color.LightGray
$document.LoadFromFile(".\Template_HtmlFile.html", $FileFormat.Html, $XHTMLValidationType.None)
$docObject.DocumentObjectType -eq $DocumentObjectType.Picture
$document.Sections[0].Paragraphs[0].InsertSectionBreak($SectionBreakType.NoBreak)
$footerParagraph.Format.HorizontalAlignment = $Spire.Doc.Documents.HorizontalAlignment.Right
#end Random characters
'#
$str | Foreach-Object {
$_ -replace '\$\w+\.(\w+)', '"$1"'
} | Set-Content .\ok.txt
<# -Skip -SkipLast not valid
$str | Foreach-Object {
$_ -replace '\$\w+\.(\w+)', '"$1"'
} | Select-Object -Skip 2 | Select-Object -SkipLast 1 | Set-Content .\ok.txt
#>
Expected results:
At least for your example here string, you need to break it into a string array. Then for the replacement I was only successful when capturing both the beginning and the desired changed text.
$str -split '\r?\n' | Select-Object -Skip 2 |
Select-Object -SkipLast 1 | Foreach-Object {
$_ -replace '(^.+?)\$.+\.(\w+)', '$1"$2"'
} | Set-Content .\ok.txt
Contents of ok.txt
$p1.AppendBreak("LineBreak")
$doc.Protect("AllowOnlyRevisions", "123")
$footerPara.AppendField("page", "FieldPage")
$footerParagraph.AppendField("number of pages", "FieldSectionPages")
$txtWatermark.Layout = "Diagonal"
$tr1.CharacterFormat.Border.BorderType = "DashDotStroker"
$stri.CharacterFormat.TextBackgroundColor = "LightGray"
$document.LoadFromFile(".\Template_HtmlFile.html", "None")
$docObject.DocumentObjectType -eq "Picture"
$document.Sections[0].Paragraphs[0].InsertSectionBreak("NoBreak")
$footerParagraph.Format.HorizontalAlignment = "Right"

How to recursively edit a lot of files adding part of their own path to some line?

On a Win7 PC i have a deep but recursive directory structure
Drive
|
+--a_Dir/.../MainDir
|
+--MainSubDir
| |
| +--George
| | |
| | +--AFile.edt
| | |
| | +--AnotherFile.edt
| | |
| | +--...
| |
| +--Mildred
| | |
| | +--... .edt
| |
| +--OtherPeople....
| |
| +--... .edt
|
|
+--ReferenceSubDir
|
+--George
| |
| +--ARefFile.ref
| |
| +--AnotherRefFile.ref
| |
| +--...
|
+--Mildred
| |
| +--... .ref
|
+--OtherPeople....
|
+--... .ref
A .edt file MAY contain one or more lines like this:
reference = SomeReferenceFile.ref
I need to change that lines adding the appropriate path after the "=" sign.
As example, suppose that the file George/AFile.edt contains
reference = ARefFile.ref
I need to change it like this:
reference = MainDir/ReferenceSubDir/George/ARefFile.ref
So, i need to find all the .edt which contains "reference = ", know the directory containing each of the found files, and edit each line in that way.
I tried with the search/replace functionality in SublimeText3 but i don't have (found) a variable containing the path i could use for "mass" replacing.
I don't care what to use (batch files, free utilities) as this will be the only time i'll do this job,
Thanks
Ended up with a powershell script.
Here it is, in case someone else find it useful:
$include = #("*.edt")
$FolderToSearch = "x:\a_Dir\aSubDir\MainDir\MainSubDir\"
$FindExpr = "(Reference = )(?!ReferenceSubDir\\)(.*\.ref)"
$files = Get-ChildItem -Path $FolderToSearch -Recurse -Include $include
$opts = #()
$opts += "IgnoreCase"
$regex = new-object Text.RegularExpressions.Regex $FindExpr, $opts
foreach ($file in $files){
if (Select-String -Quiet $file -Pattern $FindExpr ) {
$Percorso = Split-Path -Path $file
$ReplaceExpr = $Percorso.Replace($FolderToSearch,'$1ReferenceSubDir\') + '\$2'
Write-Host $ReplaceExpr
try {
$text = [IO.File]::ReadAllText($file.FullName)
}
catch [Management.Automation.MethodInvocationException] {
write-error $ERROR[0]
continue
}
$replace = $regex.Replace($text, $ReplaceExpr)
try {
[IO.File]::WriteAllText($file, $replace)
}
catch [Management.Automation.MethodInvocationException] {
write-error $ERROR[0]
continue
}
}
}

Parse CSV and decode base64

The goal of this script is supposed to be to find any base64 encoded strings within a particular row of a CSV, grab only the base64, add the correct amount of padding charcters to it, decode it, and write it to an output.
function Get-FileName($initialDirectory) {
[System.Reflection.Assembly]::LoadWithPartialName("System.Sindows.Forms") | Out-Null
$OpenFileDialog = New-Object System.Windows.Forms.OpenFileDialog
$OpenFileDialog.InitialDirectory = $initialDirectory
$OpenFileDialog.Filter = "CSV (*.csv)| *.csv"
$OpenFileDialog.ShowDialog() | Out-Null
$OpenFileDialog.Filename
}
$inputfile = Get-FileName "C:\My Documents\"
$inputdata = Import-Csv $inputfile
$output = "C:\My Documents\output.csv"
$regex = [regex]::New('(?<=-[Ee]ncoded[Cc]ommand\s"??)[\w\d=\/!]{0,792}')
$csv1 = $inputdata | select -ExpandProperty 'NameOfRow' | ? {$_ -ne ""}
| Select-String -Pattern $regex -Allmatches | % {$_.Matches}
| ? {$_.Value -ne ""} | % {$_.Value}
foreach ($line in $csv1) {
$csvL = $line.Length
$csvM = $csvL %= 4
if ($csvM | % {$_ -eq 1}) {
$line | % {$_ + "==="}
} elseif ($csvM | % {$_ -eq 2}) {
$line | % {$_ + "=="}
} elseif ($csvM | % {$_ -eq 3}) {
$line | % {$_ + "="}
}
$decode = [System.Text.Encoding]::ASCII.GetString([System.Convert]::FromBase64String($line))
$decode > $output
}

Get the numbers after ":" and count them with the help of powershell

Could someone please help me with extracting and counting the numbers from a text file with PowerShell?
Example: c:\temp\1.txt is some text with semicolon and numbers after them. I need to sum all of these numbers.
blablabl:5 dzfdsfdsfsdfsf:10
sdfsdfsdfdffs:8sdfsfsfdsfdsf:111
5+10+8+111...
What I've tried so far:
$LogText = "C:\temp\1.txt"
[regex]$Regex = "\. (\d+):[1]"
$Matches = $Regex.Matches($LogText)
$Matches | ForEach-Object {
Write-Host $Matches
}
#$array = #()
#$array = new-object collections.arraylist
$array = while ($Matches.Success) {
Write-Host $array[i++]
}
# -------------------------------------------------------------------
$text = Get-Content "C:\temp\1.txt"
[regex]$Regex = "\d"
$Matches = $Regex.Matches($text)
# -------------------------------------------------------------------
$pos = $text.IndexOf(":")
$rightPart = $text.Substring($pos+1)
Write-Host $rightPart
Use Select-String to extract the matches from the file and Measure-Object to do the calculation.
Select-String -Path 'C:\temp\1.txt' -Pattern '(?<=:)\d+' -AllMatches |
Select-Object -Expand Matches |
Select-Object -Expand Value |
Measure-Object -Sum |
Select-Object -Expand Sum
(?<=:) is a positive lookbehind assertion to match the colon preceding the number without making it part of the match.
Try it like that:
$txt=
#"
blablabl:5 dzfdsfdsfsdfsf:10
sdfsdfsdfdffs:8sdfsfsfdsfdsf:111
"#
[regex]$Regex = '\d+'
$sum=0;
$Regex.Matches($txt) | ForEach-Object {
$val = [int]$_.Value
$val
$sum+=$val
}
$sum

Powershell parsing xml logfile & get currently parsed filename

I'm new with powershell and in need of guidance. Been scouring the site for answers and coming up blank, decided to ask instead. If this has been answered please refer me to the link.
I have an application log (xml format) like below:
<log><identifier>123axr4x5</identifier><login>USER1</login><source>Order-Management</source><AddlInfo>Execution Time : 20ms</AddlInfo><Exception></Exception><timestamp>01/01/2015:22:00:00</timestamp><serverticks>643670855</serverticks><PID>1234</PID><Machine>PRD01X12mm</Machine></log>
<log><identifier>dd8jksl3g</identifier><login>USER2</login><source>Service-Assurance</source><AddlInfo>Execution Time : 80ms</AddlInfo><Exception></Exception><timestamp>01/01/2015:22:00:00</timestamp><serverticks>643680865</serverticks><PID>1234</PID><Machine>PRD01X12mm</Machine></log>
: and so on
I am creating a log parser that will scan a folder and its subfolder for matching regex pattern, and based on certain threshold, output into gridview/export to CSV. I am almost done, however i'm unable to solve 1 problem, which is to get the filename currently being parsed, to be displayed on the gridview.
Basically i am using piped Get-ChildItem as below
Get-ChildItem $Dir -recurse -Filter *logging*.txt|
Sort-Object LastWriteTime |
?{$_.LastWriteTime -gt (Get-Date).AddMinutes(-60)}|
Select-String -Pattern $Text |
Select-String -Pattern $Text3 |
Select-String -Pattern $Text2 -allmatches |
Foreach-Object {
$information = $_|Select-Object -Property API, Duration,DataRetrieved, ServerTime, ServerTicks , Identifier, Filename
$information.Filename = $_.Name
#$information.Filename = $_.FullName
} |
Out-GridView
Below is the full code:
$Dir = "C:\log\"
$threshold = 1 + 0
$StartTime = (Get-Date).ToString();
$EndTime = (Get-Date).ToString();
$Text = "abc"
$Text2 = "def"
$Text3 = "ghi"
$OutFile = "result"
$OutPath = $Dir + $OutFile + ".txt"
#ExtractionParameters
$AddlInnfoTagBegin = "AddlInfo"
$AddlInnfoTagEnd = "/AddlInfo"
$ServerTimeOfLogTagBegin = "ServerTimeOfLog"
$ServerTimeOfLogTagEnd = "/ServerTimeOfLog"
$ServerTicksTagBegin = "ServerTicks"
$ServerTicksTagEnd = "/ServerTicks"
$IdentifierTagBegin = "Identifier"
$IdentifierTagEnd = "/Identifier"
#parse file in folders
Get-ChildItem $Dir -recurse -Filter *logging*.txt|
Sort-Object LastWriteTime |
#?{$_.LastWriteTime -gt (Get-Date).AddMinutes(-60)}|
Select-String -Pattern $Text |
Select-String -Pattern $Text3 |
Select-String -Pattern $Text2 -allmatches |
Foreach-Object {
# take line and split it at tabulators
$parts = $_.Line
#write $parts
$indexOfAddlInfoBegin = $parts.IndexOf($AddlInnfoTagBegin) + $AddlInnfoTagBegin.Length +1
$indexOfAddlInfoEnd = $parts.IndexOf($AddlInnfoTagEnd) -1
$AddlInfoData = $parts.Substring($indexOfAddlInfoBegin, $indexOfAddlInfoEnd - $indexOfAddlInfoBegin)
$AddlInfoReplaced = $AddlInfoData.Replace(" seconds ","#")
$AddlInfoSplit = $AddlInfoReplaced.Split('#')
$information = $_|Select-Object -Property API, Duration,DataRetrieved, ServerTime, ServerTicks , Identifier, Filename
#get filename, which does not work
$information.Filename = $_.Name
#$information.Filename = $_.FullName
$information.API = $AddlInfoSplit[0].Split(':')[0]
$information.DataRetrieved = $AddlInfoSplit[1]
$information.Duration = $AddlInfoSplit[0].Split(':')[1]
$information.Duration = $information.Duration.Replace("Execution Time = ","")
$indexOfServerTimeBegin = $parts.IndexOf($ServerTimeOfLogTagBegin) + $ServerTimeOfLogTagBegin.Length +1
$indexOfServerTimeEnd = $parts.IndexOf($ServerTimeOfLogTagEnd) -1
$ServerTimeData = $parts.Substring($indexOfServerTimeBegin, $indexOfServerTimeEnd - $indexOfServerTimeBegin)
$information.ServerTime = $ServerTimeData
$indexOfServerTicksBegin = $parts.IndexOf($ServerTicksTagBegin) + $ServerTicksTagBegin.Length +1
$indexOfServerTicksEnd = $parts.IndexOf($ServerTicksTagEnd) -1
$ServerTickData = $parts.Substring($indexOfServerTicksBegin, $indexOfServerTicksEnd - $indexOfServerTicksBegin)
$information.ServerTicks = $ServerTickData
$indexOfIdentifierBegin = $parts.IndexOf($IdentifierTagBegin) + $IdentifierTagBegin.Length +1
$indexOfIdentifierEnd = $parts.IndexOf($IdentifierTagEnd) -1
$IdentifierData = $parts.Substring($indexOfIdentifierBegin, $indexOfIdentifierEnd - $indexOfIdentifierBegin)
$information.Identifier = $IdentifierData
$DurationAsInt = 0 + $information.Duration
if($DurationAsInt -gt $threshold) {
write $information
}
} |
Out-GridView
#Out-File -FilePath $OutPath -Append -Width 200
Any help is appreciated, thanks!!
-CL
The property you are looking for is "FileName".
$information.Filename = $_.FileName
Powershell provides a cmdlet "Get-Member" which would list all available properties/methods. You could enumerate the members to console and inspect what is available
Write-Host ( $_ | Get-Member)