Add row to column to CSV from variable list - list

I've been fighting for over an hour for something that seems simple to me ... But I can not do it
Explanation :
I import a CSV who look like this, he's empty :
DAYS | PC1 | PC2
And i have a Variable "$Days" who look like this :
PS C:\Users> $Days
12/02/2019
13/02/2019
14/02/2019
15/02/2019
16/02/2019
All i want is to add every line to the "Days" column...
Like :
DAYS | PC1 | PC2
12/02/2019 | |
13/02/2019 | |
14/02/2019 | |
15/02/2019 | |
16/02/2019 | |
I try something like :
Foreach ($row in $Days){
$CSV.Days = $row
}
But he tell me that the property "Days" is not found.
i also try something like this :
$CSV | Select-Object #{Name='Days';Expression={$forearch ($row in $Days){$row}}},*
But no result again, i don't know what i'm doing wrong...
Thank you in advance for your help.

The problem with your approach is that the file is empty thus no properties are generated for $CSV
A simple solution is generating the $CSV by iterating $Days and building a [PSCustomObject]
$Days = #'
12/02/2019
13/02/2019
14/02/2019
15/02/2019
16/02/2019
'# -split '\r?\n'
$CSV = $Days | ForEach-Object{
[PSCustomObject]#{
DAYS = $_
PC1 = $NULL
PC2 = $NULL
}
}
$CSV
$CSV | Export-Csv .\YouNameIt.csv -NoTypeInformation

Or, a shorter version:
$CSV-out = $csv | ForEach-Object{[PSCustomObject]#{DAYS = $_}}|Select days, PC1, PC2
$csv is a list of values. Similar to
$csv = #("12/02/2019","13/02/2019","14/02/2019","15/02/2019","16/02/2019")
If you then do a $CSV | FT
DAYS PC1 PC2
---- --- ---
12/02/2019
13/02/2019
14/02/2019
15/02/2019
16/02/2019
You don't have to include a custom object just to add fields. You can do a
select
and it will add them for you as well.

Related

Powershell file property listing

I'm trying to collect some file properties using PowerShell within Win 2008. To do so, I've created the following script.
# BASIC PARAMETER IF NOT SET
param(
$REGEX='.*'
)
# CURRENT DATE FROM 00H
$DATAATUAL = Get-Date -Hour 0 -Minute 0 -Second 0 -Day 1 # DAY 1 FOR TESTING ONLY
# APPLICATION'S FILE PATH
$PATH = "C:\FTP\import\"
# FILE LIST WITH SELECTED FILES FROM REGULAR EXPRESSION
$FILELIST = Get-ChildItem -Path $PATH | Where-Object { ($_.LastWriteTime -ge $DATAATUAL) -and ($_.Name -cmatch "$REGEX") } | Select-Object -ExpandProperty Name
# OUTPUT IN A SORT OF CSV FORMAT
if ($FILELIST -ne $null) {
Write-Host "name;suffix;fileprocstart;filesize;filewrite"
ForEach ($FILE in $FILELIST) {
# FILE NAME PREFFIX AND SUFFIX
$FILENAME = Select-String -InputObject $FILE -CaseSensitive -Pattern "(^\d+)_($REGEX)"
# FILE TIMESTAMP CONVERTION TO EPOCH UTC-0
$FILEPROCSTART = $FILENAME.Matches.Groups[1].value
$FILEPROCSTART = [datetime]::ParseExact($FILEPROCSTART,"yyyyMMddHHmmss",$null) | Get-Date -UFormat "%s"
$FILEPROCSTART = $($FILEPROCSTART -as [long]) + 10800 # TIMEZONE CORRECTION - ADDED 3H TO BECOME UTC-0
$FILESUFFIX = $FILENAME.Matches.Groups[2].value
# FILE SIZE AND WRITE TIME
$FILESIZE = Get-ChildItem -Path $PATH -Filter $FILE | Select-Object -ExpandProperty Length
$FILEWRITE = Get-ChildItem -Path $PATH -Filter $FILE | Select-Object -ExpandProperty LastWriteTime | Get-Date -UFormat "%s"
# OUTPUT
Write-Host "$FILENAME;$FILESUFFIX;$FILEPROCSTART;$FILESIZE;$FILEWRITE"
}
}
# NO FILES FOUND
Else {
Write-Host "Empty"
}
I can start it like so:
script.ps1 -REGEX 'pattern'
It results in a list like this:
name;suffix;fileprocstart;filesize;filewrite
20220709101112_cacs1_v83.txt;cacs1_v83.txt;1657361472;5;1657397022,47321
20220709101112_cacs1_v83.txt.log;cacs1_v83.txt.log;1657361472;5;1657397041,83271
20220709101112_cacs2_v83.txt;cacs2_v83.txt;1657361472;5;1657397039,70775
20220709101112_cacs3_v83.txt.log;cacs3_v83.txt.log;1657361472;5;1657397038,03647
20220709101112_cakauto4.txt;cakauto4.txt;1657361472;5;1657397037,48906
20220709111112_coord_multicanal.txt.log;coord_multicanal.txt.log;1657365072;5;1657398468,95865
All files are generated on a daily basis and have a format similar to this:
20220709101112_cacs1_v83.txt
20220709101112_cacs1_v83.txt.log
20220709101112_cacs2_v83.txt
20220709101112_cacs3_v83.txt.log
20220709101112_cakauto4.txt
20220709101112_coord_multicanal.txt.log
Basically, the script outputs the file name, file suffix (no timestamp), file timestamp (unix format), file size and Last Write time (unix format), all in a sort of CSV format. It is meant to be started by another system to collect those properties.
It kind of works, but I can't help thinking there must be a better way to do that.
Any considerations on how to improve it?
I'm not sure if I got it right but if I understand this right:
Basically, the script outputs the file name, file suffix, file name timestamp, file size and Last Write time, all in a sort of CSV format. It is meant to be started by another system to collect those properties.
This should be all you need to start with:
$ComputerName = 'RemoteW2K8Computer'
Invoke-Command -ComputerName $ComputerName -ScriptBlock {
Get-ChildItem -Path 'C:\FTP\import' |
Select-Object -Property BaseName,Extension,Length,LastWriteTime,
#{Name = 'FileNameTimeStamp'; Expression = {($_.BaseName -split '_')[0]}}
}
Using #Olaf great tips, I've rewritten the script this way.
param($REGEX='.*')
$DATAATUAL = Get-Date -Hour 0 -Minute 0 -Second 0 -Day 1 # DAY 1 FOR TESTING ONLY
$PATH = "C:\FTP\import"
$TZ = [TimeZoneInfo]::FindSystemTimeZoneById("E. South America Standard Time")
$FILELIST = Get-ChildItem -Path $PATH |
Where-Object { ($_.LastWriteTime -ge $DATAATUAL) -and ($_.Name -cmatch "$REGEX") } |
Select-Object -Property Name,Length,
#{Name = 'Suffix'; Expression = { ($_.Name -split '_',2)[1] } },
#{Name = 'ProcStart'; Expression = {
$PROCSTART = ($_.Name -split '_')[0];
$PROCSTART = [datetime]::ParseExact($PROCSTART,"yyyyMMddHHmmss",$null);
[TimeZoneInfo]::ConvertTimeToUtc($PROCSTART, $TZ) | Get-Date -UFormat "%s";
} },
#{Name = 'FileWrite' ; Expression = {
$WRITETIME = $_.LastWriteTime;
[TimeZoneInfo]::ConvertTimeToUtc($WRITETIME) | Get-Date -UFormat "%s";
} }
if ($FILELIST -ne $null) {
Write-Host "name;suffix;procstart;filesize;filewrite"
# $FILELIST | ConvertTo-Csv -Delimiter ';' -NoTypeInformation
ForEach ($FILE in $FILELIST) {
$FILENAME = $FILE.Name
$FILESUFFIX = $FILE.Suffix
$FILESIZE = $FILE.Length
$FILEPROCSTART = $FILE.ProcStart
$FILEWRITE = $FILE.FileWrite
Write-Host "$FILENAME;$FILESUFFIX;$FILESIZE;$FILEPROCSTART;$FILEWRITE"
}
}
Else {
Write-Host "Empty"
}
As said, the output is in a CSV format.
name;suffix;procstart;filesize;filewrite
20220709101112_cacs1_v83.txt;cacs1_v83.txt;5;1657361472;1657397022,47321
20220709101112_cacs1_v83.txt.log;cacs1_v83.txt.log;5;1657361472;1657397041,83271
If I use ConvertTo-Csv (much simpler) instead of ForEach, the output would also be a CSV.
However, it places quotation marks that mess up other conversions to JSON elsewhere (maybe I can improve that later).
# $FILELIST | ConvertTo-Csv -Delimiter ';' -NoTypeInformation
"Name";"Length";"Suffix";"ProcStart";"FileWrite"
"20220709101112_cacs1_v83.txt";"5";"cacs1_v83.txt";"1657361472";"1657397022,47321"
"20220709101112_cacs1_v83.txt.log";"5";"cacs1_v83.txt.log";"1657361472";"1657397041,83271"
The other system convert it to this (I can't use ConvertTo-Json in Win2008 :-/):
{
"\"Name\"": "\"20220709101112_cacs1_v83.txt\"",
"\"Length\"": "\"5\"",
"\"Suffix\"": "\"cacs1_v83.txt\"",
"\"ProcStart\"": "\"1657361472\"",
"\"FileWrite\"": "\"1657397022,47321\""
}
Therefore, I find that writing the values with ForEach gives me a cleaner output.
Also, for fun, measuring with Measure-Command, I found that the new script is a bit faster.
The previous script takes about 24 milliseconds to complete while using a small dataset.
Now, the new one takes about 13 milliseconds with the same dataset.
All in all, a small, but good improvement, I guess.
Cheers to #Olaf for pointing to a better script and for his patience. Thank you.

Retrieving multiple strings from one line of a log

I have a huge ACS.txt report created in Kiwi, and I'd like to:
ID particular lines which have a set string "RADIUS Accounting" then ....
...from those lines take two values "User-ID=XXXXXXXX#domain.com" and "MAC=xx-xx-xx-xx-xx-xx-xx-xx", then output that in a txt.
This is what I have right now
Get-Content C:ACS.txt | ForEach-Object {
$null = $_ -match "RADIUS Accounting",\s.*User-Name=(?<user>[0-9]+#domain.com).*Calling-Station-ID=(?<mac>([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})).*"; $matches.user; $matches.mac
}
I think it's giving me what I want, it's just in one long list, rather than user/mac per line.
What about using Select-String and iterate the found (sub)matches
building a new PSCustomObject you can export or view
$UserMac = Select-String -Path C:ACS.txt -Pattern "RADIUS Accounting.*User-Name=([0-9]+#domain.com).*Calling-Station-ID=(([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2}))" |
ForEach-Object{
[PSCustomObject]#{
User = $matches.Groups[1].Value
Mac = $matches.Groups[2].Value
}
}
# $UserMac
$UserMac | Out-Gridview
# $UserMac | Export-Csv .\UserMac.csv -NoType

Capturing multiple points of data from a text file using Regex in powershell

So I got this regex expression to work in Regex101 and it captures exactly what I want to capture. https://regex101.com/r/aJ1bZ4/3
But when I try the same thing in powershell all I get is the first set of matches. I've tried using the (?s:), the (?m:) but none of these modifiers seem to do the job. Here is my powershell script.
$reportTitleList = type ReportExecution.log | Out-String |
where {$_ -match "(?<date>\d{4}\/\d{2}\/\d{2}).*ID=(?<reportID>.*):.*Started.*Title=(?<reportName>.*)\[.*\n.*Begin ....... (?<reportHash>.*)"} |
foreach {
new-object PSObject -prop #{
Date=$matches['date']
ReportID=$matches['reportID']
ReportName=$matches['reportName']
ReportHash=$matches['reportHash']
}
}
$reportTitleList > reportTitleList.txt
What am I doing wrong? Why am I not getting all the matches as the regex101 example?
-match only find the first match. To use a global search you need to use [regex]::Matches() or Select-String with the -AllMatches switch. Ex:
#In PoweShell 3.0+ you can replace `Get-Content | Out-String` with `Get-Content -Raw`
$reportlist = Get-Content -Path ReportExecution.log | Out-String |
Select-String -Pattern $pattern -AllMatches |
Select-Object -ExpandProperty Matches |
Select-Object #{n="Date";e={$_.Groups["date"]}},
#{n="ReportID";e={$_.Groups["reportID"]}},
#{n="ReportName";e={$_.Groups["reportName"]}},
#{n="ReportHash";e={$_.Groups["reportHash"]}}
#Show output
$reportlist
Output:
Date ReportID ReportName ReportHash
---- -------- ---------- ----------
2015/03/23 578 Calendar Day Activity/Calendar Day Activity 38C19F4E790446709B8C7A32FF97BC...
2015/03/23 861 Program Format Report/Program Format Report 3C9CB2150AF14B15A1B361729C007B...
2015/03/23 1077 Multi-Station Program Availability/Multi-Station Program Availability 52526430EE4E401BA4376B38A2D88B...
2015/03/23 1299 Program Audit Trail/Program Audit Trail FDD1B7D9F34E46549A377A17B9A7A1...
2015/03/23 1541 Program Availability/Program Availability 843B44F4475C4950A7784C8961B642...
2015/03/23 1756 Program Description Export/Program Description Export E5800A76C68E4D5281B8D680DB2E93...
-match returns as soon as it finds a match (they should have a -matches operator right?). If you want multiple matches, use:
$mymatches = [regex]::matches($input,$pattern)
output will be different than -match, however, and you'll have to massage it a bit, something like: (see here for another example of conversion)
$mymatches | ForEach-Object { if ( $_.Success) { echo $_.value}}

RegEx required for search-replace using PowerShell

I'm trying to load up a file from a PS script and need to search replace on the basis of given pattern and new values. I need to know what the pattern would be. Here is an excerpt from the file:
USER_IDEN;SYSTEM1;USERNAME1;
30;WINDOWS;Wanner.Siegfried;
63;WINDOWS;Ott.Rudolf;
68;WINDOWS;Waldera.Alicja;
94;WINDOWS;Lanzl.Dieter;
98;WINDOWS;Hofmeier.Erhard;
ReplacerValue: "#dummy.domain.com"
What to be replaced: USERNAME1 column
Expected result:
USER_IDEN;SYSTEM1;USERNAME1;
30;WINDOWS;Wanner.Siegfried#dummy.domain.com;
63;WINDOWS;Ott.Rudolf#dummy.domain.com;
68;WINDOWS;Waldera.Alicja#dummy.domain.com;
94;WINDOWS;Lanzl.Dieter#dummy.domain.com;
98;WINDOWS;Hofmeier.Erhard#dummy.domain.com;
Also, the file can be like this as well:
USER_IDEN;SYSTEM1;USERNAME1;SYSTEM2;USERNAME2;SYSTEM3;USERNAME3;
30;WINDOWS;Wanner.Siegfried;WINDOWS2;Wanner.Siegfried;LINUX;Dev-1;LINUX2;QA1
63;WINDOWS;Ott.Rudolf;WINDOWS2;Ott.Rudolf;LINUX;Dev-2
68;WINDOWS;Waldera.Alicja;
94;WINDOWS;Lanzl.Dieter;WINDOWS4;Lanzl.Dieter;WINDOWS3;Lead1
98;WINDOWS;Hofmeier.Erhard;
In the above examples, I want to seek the values under USERNAMEn columns but there is a possibility that the column row may not be present but the CSV (;) and the pairs will remain same and also the first value is the identifier so it's always there.
I have found the way to start but need to get the pattern:
(Get-Content C:\script\test.txt) |
Foreach-Object {$_ -replace "^([0-9]+;WINDOWS;[^;]+);$", '$#dummy.domain.com;'} |
Set-Content C:\script\test.txt
Edit
I came up with this pattern: ^([0-9]+;WINDOWS;[^;]+);$
It is very much fixed to this particular file only with no more than one Domain-Username pair and doesn't depend on the columns.
I think that using a regex to do this is going about it the hard way. Instead of using Get-Content use Import-Csv which will split your columns for you. You can then use Get-Memeber to identify the USERNAME columns. Something like this:
$x = Import-Csv YourFile.csv -Delimiter ';'
$f = #($x[0] | Get-Member -MemberType NoteProperty | Select name -ExpandProperty name | ? {$_ -match 'USERNAME'})
$f | % {
$n = $_
$x | % {$_."$n" = $_."$n" + '#dummy.domain.com'}
}
$x | Export-Csv .\YourFile.csv -Delimiter ';' -NoTypeInformation

compare files in order to extract text from multiline regex:compare-object,hastables or psobject?

The beginning of my code is the following:
$file1 = "G:\test_powershell_subtitle\The Big Bang Theory - 08x06 - french.srt"
$file2 = "G:\test_powershell_subtitle\The Big Bang Theory - 08x06 - english.srt"
$text1 =get-content($file1) -Raw
$text2 =get-content($file2) -Raw
$regex = [regex]'(?m)(?<sequence>\d+)\r\n(?<timecode>\d{2}:\d{2}:\d{2},\d{3} --> \d{2}:\d{2}:\d{2},\d{3})\r\n(?<text>[\s\S]*?\r\n\r\n)'
$matches = $regex.Matches($text1)
$matches |% {
if ($_ -match $regex){
new-object psobject -property #{
sequence = $matches['sequence']
timecode = $matches['timecode']
text = $matches['text']
}
}
}
The output:
timecode sequence text
---- -------- ----
00:00:02,880 --> 00:00:04,146 1 I like your suit....
00:00:04,148 --> 00:00:06,699 2 Oh, thanks. Got a ...
00:00:06,701 --> 00:00:08,651 3 How does it feel knowing...
00:00:08,653 --> 00:00:10,786 4 is to go out...
.
My goal is to merge the subtitles of different languages ​​according to the timecode to a single file.
What is the best way to proceed? compare-object,hastables or psobject?
Thank you for help.
You will have some more work ahead of you but this should be enough to satisfy the question at hand. Group-Object is the way to go I think for this one.
function Convert-SubtitlesToObject{
param(
[parameter(Mandatory=$true)]
[ValidateScript({Test-Path $_})]
[String]
$Path
)
$regex = [regex]'(?m)(?<sequence>\d+)\r\n(?<timecode>\d{2}:\d{2}:\d{2},\d{3} --> \d{2}:\d{2}:\d{2},\d{3})\r\n(?<text>[\s\S]*?\r\n\r\n)'
$text = Get-Content($Path) -Raw
$matches = $regex.Matches($text)
$matches | Where-Object{$_ -match $regex} | ForEach-Object{
[PSCustomObject][ordered]#{
sequence = $matches['sequence']
timecode = $matches['timecode']
text = $matches['text']
}
}
}
$englishSubs = Convert-SubtitlesToObject -Path 'C:\temp\put\The Big Bang Theory - 8x06 - The Expedition Approximation.HDTV.LOL.HI.en.srt'
$frenchSubs = Convert-SubtitlesToObject -Path 'C:\temp\put\The Big Bang Theory - 8x06 - The Expedition Approximation.HDTV.LOL.fr.srt'
$collection = #()
$collection += $englishSubs
$collection += $frenchSubs
$sequence = 0
$collection | Group-Object timecode | Select-Object Name,#{l="Text";e={$_.Group.Text}} | ForEach-Object{
$sequence++
Write-Output "$sequence`r`n$($_.Name)`r`n$($_.Text)"
}
I converted you code into a function since you will be calling that for all files. Run this for both english and french subs and put them into the larger $collection. Call Group-Object on that collection and group them by the timecode. Take that data and expand the text into a single field. After all that take the collect the output to best imitate a subtitle file. You need to watch out for timecodes that dont match but I leave that to you as to what you will do in that situation.
Here is some sample output which you could pipe into Out-File or Add-Content:
1
00:00:00,000 --> 00:00:01,800
English Subtitles (HI)
[MP4] The Big Bang Theory S08E06 (720p) The Expedition Approximation HDTV [KoTuWa]
2
00:00:02,880 --> 00:00:04,146
I like your suit.
J'aime ton tailleur.
3
00:00:04,148 --> 00:00:06,699
Oh, thanks. Got a couple
new outfits for work.
Merci.
J'en ai acheté pour le boulot.
Disclaimer
I know some PowerShell. I know jack about subtitle file formats.
Sorry for my late reply . I tried to find a solution by myself. It is not complete , especially if the timecode are not identical . Yours is better.
Here is my solution.
function Convert-SubtitlesToObject{
param(
[parameter(Mandatory=$true)]
[ValidateScript({Test-Path $_})]
[String]
$Path
)
$regex = [regex]'(?m)(?<sequence>\d+)\r\n(?<timecode>(?<t1>\d{2}:\d{2}:\d{2},\d{3}) --> \d{2}:\d{2}:\d{2},\d{3})\r\n(?<text>[\s\S]*?\r\n\r\n)'
$text = Get-Content($Path) -Raw
$matches = $regex.Matches($text)
$matches | Where-Object{$_ -match $regex} | ForEach-Object{
[PSCustomObject][ordered]#{
sequence = $matches['sequence']
timecode = $matches['timecode']
text = $matches['text']
}
}
}
$englishSubs = Convert-SubtitlesToObject -Path 'G:\test_powershell_subtitle\The Big Bang Theory - 08x06 - english.srt'
$frenchSubs = Convert-SubtitlesToObject -Path 'G:\test_powershell_subtitle\The Big Bang Theory - 08x06 - french.srt'
$temp = Compare-Object $frenchSubs $englishSubs -property sequence,timecode,text
$subtitles=$temp | Group-Object -Property timecode| % {
[PSCustomObject][ordered] #{
seq=$_.group[1].sequence;
time=$_.name;
string=$_.group[0].text+$_.group[1].text}}
#Construct an out-array to use for data export
$OutArray = #()
$Outarray +=$subtitles.psobject.properties | % {$_.value} # each object's fields
# get the index for element that is -eq to SyncRoot
# The SyncRoot is returning the collection
$index = 0..($outarray.psobject.properties.name.length - 1) | ? {$outarray.psobject.properties.name[$_] -eq "SyncRoot"}
for($i = $index; $i -le $OutArray.matches.count; $i++){
Write-Output "$($outarray[$i].seq)`r`n$($outarray[$i].time)`r`n$($outarray[$i].string)`r`n"
}