The CSV file only contains a partial entry from the last regex match.
I've used the ISE debugger and can verify it's finding matches.
$h = #{}
$a = #()
Get-ChildItem C:\Users\speterson\Documents\script\*.kiy | foreach {
Get-Content $_ | foreach {
if ($_ -match 'IF Ingroup\s+\(\s+\"(..+?)\"\s+\)') {
$h.Group = $matches[1]
}
if ($_ -match 'use\s+([A-Za-z]):"(\\\\..*?\\..*)\"')) {
$h.DriveLetter = $matches[1].ToUpper()
$h.Path = $matches[2]
}
}
$a += New-Object PSCustomObject -Property $h
}
$a | Export-Csv c:\temp\Whatever.csv -NoTypeInfo
The input files look like this, but have 1000+ lines in them:
IF Ingroup ( "RPC3WIA01NT" )
use v: /del
ENDIF
IF Ingroup ( "JWA03KRONOSGLOBAL" )
use v:"\\$homesrvr\$dept"
ENDIF
IF Ingroup ( "P-USERS" )
use p:'\\PServer\PDRIVE
ENDIF
CSV file only shows:
GROUP
P-USERS
I want to ignore the drive letters with the /del.
I'm trying to get a CSV file that shows
Group Drive Path
JWA03KRONOSGLOBAL V \\$homesrvr\$dept
P-USERS P \\PServer\PDRIVE
Your code has two loops, one nested in the other. The outer loop processes each file from the Get-ChildItem call. The inner loop processes the content of the current file of the outer loop. However, since you're creating your objects after the inner loop finished you're only getting the last result from each processed file. Move object creation into the inner loop to get all results from all files.
I'd also recommend not re-using a hashtable. Re-using objects always bears the risk of having data carried over somewhere undesired. Hashtable creation is so inexpensive that running that risk is never worth it.
On top of that your processing of the files' content is flawed, because the inner loop processes the content one line at a time, but both of your conditionals match on different lines and are not linked to each other. If you created a new object with every iteration that would give you incorrect results. Read the file as a whole and then use Select-String with a multiline regex to extract the desired information.
Another thing to avoid is appending to an array in a loop (that's a slow operation because it involves re-creating the array and copying elements over and over). Since you're using ForEach-Object you can pipe directly into Export-Csv.
Something like this should work:
$re = 'IF Ingroup\s+\(\s+"(.+?)"\s+\)\s+' +
"use\s+([a-z]):\s*[`"'](\\\\[^`"'\s]+)"
Get-ChildItem 'C:\Users\speterson\Documents\script\*.kiy' | ForEach-Object {
Get-Content $_.FullName |
Out-String |
Select-String $re -AllMatches |
Select-Object -Expand Matches |
ForEach-Object {
New-Object -Type PSObject -Property #{
'Group' = $_.Groups[1].Value
'DriveLetter' = $_.Groups[2].Value
'Path' = $_.Groups[3].Value
}
}
} | Export-Csv 'C:\path\to\output.csv' -NoType
Related
I have a file with lines that i wish to remove like the following:
key="Id" value=123"
key="FirstName" value=Name1"
key="LastName" value=Name2"
<!--key="FirstName" value=Name3"
key="LastName" value=Name4"-->
key="Address" value=Address1"
<!--key="Address" value=Address2"
key="FirstName" value=Name1"
key="LastName" value=Name2"-->
key="ReferenceNo" value=765
have tried the following: `
$values = #('key="FirstName"','key="Lastname"', 'add key="Address"');
$regexValues = [string]::Join('|',$values)
$lineprod = Get-Content "D:\test\testfile.txt" | Select-String $regexValues|Select-Object -
ExpandProperty Line
if ($null -ne $lineprod)
{
foreach ($value in $lineprod)
{
$prod = $value.Trim()
$contentProd | ForEach-Object {$_ -replace $prod,""} |Set-Content "D:\test\testfile.txt"
}
}
The issue is that only some of the lines get replaced and or removed and some remain.
The output should be
key="Id" value=123"
key="ReferenceNo" value=765
But i seem to get
key="Id" value=123"
key="ReferenceNo" value=765
<!--key="Address" value=Address2"
key="FirstName" value=Name1"
key="LastName" value=Name2"-->
Any ideas as to why this is happening or changes to the code above ?
Based on your comment, the token 'add key="Address"' should be changed for just 'key="Address"' then the concatenating logic to build your regex looks good. You need to use the -NotMatch switch so it matches anything but those values. Also, Select-String can read files, so, Get-Content can be removed.
Note, the use of (...) in this case is important because you're reading and writing to the same file in the same pipeline. Wrapping the statement in parentheses ensure that all output from Select-String is consumed before passing it through the pipeline. Otherwise, you would end up with an empty file.
$values = 'key="FirstName"', 'key="Lastname"', 'key="Address"'
$regexValues = [string]::Join('|', $values)
(Select-String D:\test\testfile.txt -Pattern $regexValues -NotMatch) |
ForEach-Object Line | Set-Content D:\test\testfile.txt
Outputs:
key="Id" value=123"
key="ReferenceNo" value=765
I write a Powershell script and regex to search two configs text files to find matches for Management Vlan. For example, each text file has two Management vlan configured as below:
Config1.txt
123 MGMT_123_VLAN
234 MGMT_VLAN_234
Config2.txt
890 MGMT_VLAN_890
125 MGMT_VLAN_USERS
Below is my script. It has several problems.
First, if I ran the script with the $Mgmt_vlan = Select-String -Path $File -Pattern $String -AllMatches then the screen output shows the expected four (4) Mgmt vlan, but in the CSV file output shows as follow
Filename Mgmt_vlan
Config1.txt System.Object[]
Config2.txt System.Object[]
I ran the script the output on the console screen shows exactly four (4) Management vlans that I expected, but in the CSV file it did not. It shows only these vlans
Second, if I ran the script with $Mgmt_vlan = Select-String -Path $File -Pattern $String | Select -First 1
Then the CSV shows as follows:
Filename Mgmt_vlan
Config1.txt 123 MGMT_123_VLAN
Config2.txt 890 MGMT_VLAN_890
The second method Select -First 1 appears to select only the first match in the file. I tried to change it to Select -First 2 and then CSV shows column Mgmt_Vlan as System.Object[].
The result output to the screen shows exactly four(4) Mgmt Vlans as expected.
$folder = "c:\config_folder"
$files = Get-childitem $folder\*.txt
Function find_management_vlan($Text)
{
$Vlan = #()
foreach($file in files) {
Mgmt_Vlan = Select-String -Path $File -Pattern $Text -AllMatches
if($Mgmt_Vlan) # if there is a match
{
$Vlan += New-Object -PSObject -Property #{'Filename' = $File; 'Mgmt_vlan' = $Mgmt_vlan}
$Vlan | Select 'Filename', 'Mgmt_vlan' | export-csv C:\documents\Mgmt_vlan.csv
$Mgmt_Vlan # test to see if it shows correct matches on screen and yes it did
}
else
{
$Vlan += New-Object -PSObject -Property #{'Filename' = $File; 'Mgmt_vlan' = "Mgmt Vlan Not Found"}
$Vlan | Select 'Filename', 'Mgmt_vlan' | Export-CSV C:\Documents\Mgmt_vlan.csv
}
}
}
find_management_vlan "^\d{1,3}\s.MGMT_"
Regex correction
First of all, there are a lot of mistakes in this code.
So this is probably not code that you actually used.
Secondly, that pattern will not match your strings, because if you use "^\d{1,3}\s.MGMT_" you will match 1-3 numbers, any whitespace character (equal to [\r\n\t\f\v ]), any character (except for line terminators) and MGMT_ chars and anything after that. So not really what you want. So in your case you can use for example this: ^\d{1,3}\sMGMT_ or with \s+ for more than one match.
Code Correction
Now back to your code... You create array $Vlan, that's ok.
After that, you tried to get all strings (in your case 2 strings from every file in your directory) and you create PSObject with two complex objects. One is FileInfo from System.IO and second one is an array of strings (String[]) from System. Inside the Export-Csv function .ToString() is called on every property of the object being processed. If you call .ToString() on an array (i.e. Mgmt_vlan) you will get "System.Object[]", as per default implementation. So you must have a collection of "flat" objects if you want to make a csv from it.
Second big mistake is creating a function with more than one responsibility. In your case your function is responsible for gathering data and after that for exporting data. That's a big no no. So repair your code and move that Export somewhere else. You can use for example something like this (i used get-content, because I like it more, but you can use whatever you want to get your string collection.
function Get-ManagementVlans($pattern, $files)
{
$Vlans = #()
foreach ($file in $files)
{
$matches = (Get-Content $file.FullName -Encoding UTF8).Where({$_ -imatch $pattern})
if ($matches)
{
$Vlans += $matches | % { New-Object -TypeName PSObject -Property #{'Filename' = $File; 'Mgmt_vlan' = $_.Trim()} }
}
else
{
$Vlans += New-Object -TypeName PSObject -Property #{'Filename' = $File; 'Mgmt_vlan' = "Mgmt Vlan Not Found"}
}
}
return $Vlans
}
function Export-ManagementVlans($path, $data)
{
#do something...
$data | Select Filename,Mgmt_vlan | Export-Csv "$path\Mgmt_vlan.csv" -Encoding UTF8 -NoTypeInformation
}
$folder = "C:\temp\soHelp"
$files = dir "$folder\*.txt"
$Vlans = Get-ManagementVlans -pattern "^\d{1,3}\sMGMT_" -files $files
$Vlans
Export-ManagementVlans -path $folder -data $Vlans```
Summary
But in my opinion in this case is overprogramming to create something like you did. You can easily do it in oneliner (but you didn't have information if the file doesn't include anything). The power of powershell is this:
$pattern = "^\d{1,3}\s+MGMT_"
$path = "C:\temp\soHelp\"
dir $path -Filter *.txt -File | Get-Content -Encoding UTF8 | ? {$_ -imatch $pattern} | select #{l="FileName";e={$_.PSChildName}},#{l="Mgmt_vlan";e={$_}} | Export-Csv -Path "$path\Report.csv" -Encoding UTF8 -NoTypeInformation
or with Select-String:
dir $path -Filter *.txt -File | Select-String -Pattern $pattern -AllMatches | select FileName,#{l="Mgmt_vlan";e={$_.Line}} | Export-Csv -Path "$path\Report.csv" -Encoding UTF8 -NoTypeInformation
Using powershell, I am trying to determine which perl scripts in a directory are not called from any other script. In my Select-String I am grouping the matches because there is some other logic I use to filter out results where the line is commented, and a bunch of other scenarios I want to exclude(for simplicity I excluded that from the code posted below). My main problem is in the "-notin" part.
I can get this to work if I remove the grouping from Select-string and only match the filename itself. So this works.
$searchlocation = "C:\Temp\"
$allresults = Select-String -Path "$searchlocation*.pl" -Pattern '\w+\.pl'
$allperlfiles = Get-Childitem -Path "$searchlocation*.pl"
$allperlfiles | foreach-object -process{
$_ | where {$_.name -notin $allresults.matches.value} | Select -expandproperty name | Write-Host
}
However I cannot get the following to work. The only difference between this and above is the value for the "-Pattern" and the value after "-notin". I'm not sure how to use "notin" along with matching groups.
$searchlocation = "C:\Temp\"
$allresults = Select-String -Path "$searchlocation*.pl" -Pattern '(.*?)(\w+\.pl)'
$allperlfiles = Get-Childitem -Path "$searchlocation*.pl"
$allperlfiles | foreach-object -process{
$_ | where {$_.name -notin $allresults.matches.groups[2].value} | Select -expandproperty name | Write-Host}
At a high level the code should search all perl scripts in a directory for any lines that execute any other perl script. With that I now have $allresults which basically gives me a list of all perl scripts called from other files. To get the inverse of that(files that are NOT called from any other file) I get a list of all perl scripts in the directory, cycle through those and list out the ones that DONT show up in $allresults.
When you select a grouping you need to do so using a Select statement, or iteratively in a loop, otherwise you are only going to select the value from the Nth match.
IE if your $Allresults object contains
File.pl, File 2.pl, File 3.pl
Then $allresults.Matches.Groups[2].value Only Returns File2.pl
Instead, you need to select those values!
$allresults | select #{N="Match";E={ $($_.Matches.Groups[2].value) } }
Which will return:
Match
-----
File1.pl
File2.pl
File3.pl
In your specific example, each match has three sub-items, the results will be completely sequential, so what you would term "match 1, group 1" is groups[0] while "match 2, group 1" is groups[3]
This means the matches you care about (those with grouping 2) are in the array values contained in the set {2,5,8,11,...,etc.} or can be described as (N*3-1) Where N is the number of the match. So For Match 1 = (1*3)-1 = [2]; while For Match 13 = (13*3)-1 = [38]
You can iterate through them using a loop to check:
for($i=0; $i -le ($allresults.Matches.groups.count-1); $i++){
"Group[$i] = ""$($allresults.Matches.Groups[$i].value)"""
}
I noticed that you took the time to avoid loops in collecting your data, but then accidentally seem to have fallen prey to using one in matching your data.
Not-In and other compares when used by the select and where clauses don't need a loop structure and are faster if not looped, so you can forego the Foreach-object loop and have a better process just by using a simple Where (?).
$SearchLocation = "C:\Temp\"
$FileGlob = "*.pl"
$allresults = Select-String -Path "$SearchLocation$FileGlob" -Pattern '(.*?)([\w\.]+\.bat)'
$allperlfiles = Get-Childitem -Path "$SearchLocation$FileGlob"
$allperlfiles | ? {
$_.name -notin $(
$allresults | select #{N="Match";E={ $($_.Matches.Groups[2].value) } }
)
} | Select -expandproperty name | Write-Host
Now, that should be faster and simpler code to maintain, but, as you may have noticed, it still has some redundancies now that you are not looping.
As you are piping it all into a Select which can do the work of the where, and what's more you only are looking to match the NAME property here so you can either for-go the last select by only piping the name of the file in the first place, or you can forgo the where and select exactly what you want.
I think the former is far simpler, and the latter is useful if you are going to actually do something with those other values inside the loop that we don't know yet.
Finally, Write-host is likely redundant as any object output will echo to the console.
Here is that version which incorporates the removal of the unnecessary loops and removes redundancies related to the output of the info you wanted, all together.
$SearchLocation = "C:\Temp\"
$FileGlob = "*.pl"
$allresults = Select-String -Path "$SearchLocation$FileGlob" -Pattern ('(.*?)([\w\.]+\'+$FileGlob+')')
$allperlfiles = Get-Childitem -Path "$SearchLocation$FileGlob"
$allperlfiles.name | ? {
$_ -notin $(
$allresults | select #{
N="Match";E={
$($_.Matches.Groups[2].value)
}
}
)
}
I am simply trying to create a powershell script that will change number values in a set of text files. The data in the text files are separated by semi-colons. The values I want to change are always the 2nd and 3rd tokens on each line of the text file.
An example of a line in one of the files:
"Bridge_Asphalt_F";202498.396728;1104.362183;9.721280;0.000000;0.000000;1.000000;-1.299559;
I want to allow the user of the script to enter values to be added to(or subtracted from) the 2nd and 3rd values in all the lines of all the text files in the current directory.
I have a very basic understanding of scripting, but I've been searching around for hours trying to wrap my head around how this would be accomplished.
This is what I have so far but I'm sure I'm getting a few things wrong:
$east = Read-Host 'Easting?'
$north = Read-Host 'Northing?'
Get-ChildItem *.txt |
Foreach-Object {
$c = ($_ | Get-Content)
$c = $c -replace $regexB,$regexB+$east
$c = $c -replace $regexC,$regexC+$north
[IO.File]::WriteAllText($_.FullName, ($c -join "`r`n"))
}
The values determine an object's location on a map (for a game) and I want to be able to move all objects on the entire map by a certain distance on both x and y axis.
Assuming that each line in the file has the same format as your example, then you can treat the file as a CSV and update it like this:
$offset2 = 100
$offset3 = 100
Import-Csv .\data.txt -Delimiter ';' -Header (1 .. 9) |
ForEach-Object {
$_.2 = ([double]$_.2) + $offset2
$_.3 = ([double]$_.3) + $offset3
$_
} | ConvertTo-Csv -NoTypeInformation -Delimiter ';' |
Select-Object -Skip 1 |
Add-Content .\updated.txt
Note:
ConvertTo-Csv surrounds each item with quotes, so you end up with something like this:
"Bridge_Asphalt_F";"202198.396728";"1104.362183";"9.721280";"0.000000";"0.000000";"1.000000";"-1.299559"
This may cause problems if this isn't expected by your game. If so, then some more processing on the pipeline could be done to strip it out.
Also, I've had issues in the past with trying to import and export to the same CSV file, hence my code outputs to a different file. Test it yourself and if it works with the same file, great, otherwise, copy my example, then add a line to replace the existing file with the new one (e.g. using Move-Item).
I guess that's what you need:
cls
cd C:\Users\dandraka\Desktop\test #or whereever
$eastStr = Read-Host 'Easting?'
$northStr = Read-Host 'Northing?'
# convert input to number
$east = [decimal]::Parse($eastStr)
$north = [decimal]::Parse($northStr)
# loop through files
$files = Get-ChildItem *.txt
$files | Foreach-Object {
$fileName = $_.FullName # just for clarity
Write-Host $fileName
$newLines = New-Object System.Collections.ArrayList
# loop through lines of each file
$lines = Get-Content -Path $fileName
$lines | ForEach-Object {
$line = $_.ToString() # just for clarity
$lineItems = $line -split ';'
$pointName = $lineItems[0]
$latitudeStr = $lineItems[1]
$longitudeStr = $lineItems[2]
# convert to number
$latitude = [decimal]::Parse($latitudeStr)
$longitude = [decimal]::Parse($longitudeStr)
Write-Host "$pointName latitude $latitude , longitude $longitude"
# do the math
$newLatitude = $latitude + $north
$newLongitude = $longitude + $east
Write-Host "$pointName new latitude $newLatitude , new longitude $newLongitude"
# recontruct the line
$newLine = ""
for($i=0; $i -lt $lineItems.Count; $i++) {
if ($i -eq 1) {
$newLine += "$newLatitude;"
continue
}
if ($i -eq 2) {
$newLine += "$newLongitude;"
continue
}
# this if fixes a small bug, without it there are two ; at the end of each line
if ($lineItems[$i].Length -gt 0) {
$newLine += "$($lineItems[$i]);"
}
}
Write-Host "Old line $line"
Write-Host "New line $newLine"
$newLines.Add($newLine) | Out-Null
}
# write file
$newFilename = $fileName.Replace(".txt", ".dat")
[System.IO.File]::WriteAllLines($newFilename, $newLines)
Write-Host "File $newFilename written"
}
A few things to note here:
As you mention that you're starting with powershell, I've written the code more verbose than I would for, say, a seasoned developer. But that actually doesn't hurt.
For the same reason, the code is sub-optimal on purpose (makes for easier to read code). But for better performance and large files (say, a few 10s of MB or more) you need to do things differently, e.g. avoid strings and use string builder instead.
Obviously you can comment out all the Write-Host statements, they're there just to help you make sure the code is working properly.
Hope that helps!
Jim
If your game cannot handle the quoted coordinate values you get when using ConvertTo-Csv or Export-Csv, this should update the values while leaving the quotes off:
$eastOffset = 100
$northOffset = -200
(Get-Content 'D:\coordinates.txt') | ForEach-Object {
$fields = $_ -split ';'
[double]$fields[1] += $eastOffset
[double]$fields[2] += $northOffset
# write the updated stuff to file
Add-Content -Path 'D:\newcoordinates.txt' -Value ($fields -join ';')
}
this content
"Bridge_Asphalt_F";202498.396728;1104.362183;9.721280;0.000000;0.000000;1.000000;-1.299559;
"Road_Asphalt_F";202123.396728;1104.362456;9.721280;0.000000;0.000000;1.000000;-1.299559;
would become
"Bridge_Asphalt_F";202598.396728;904.362183;9.721280;0.000000;0.000000;1.000000;-1.299559;
"Road_Asphalt_F";202223.396728;904.362456;9.721280;0.000000;0.000000;1.000000;-1.299559;
I can't seem to figure out how to simply export formatted information to a CSV unless I iterate through each item in the object and write to the CSV line by line, which takes forever. I can export values instantly to the CSV, it's just when using the properties dictionary I run into issues.
The TestCSV file is formatted with a column that has IP addresses.
Here's what I have:
$CSV = "C:\TEMP\OutputFile.csv"
$RX = "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(?:\.|dot|\[dot\]|\[\.\])){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
$TestCSV = "C:\TEMP\FileWithIPs.csv"
$spreadsheetDataobject = import-csv $TestCSV
$Finding = $spreadsheetDataObject | Select-String $RX
$Props = #{ #create a properties dictionary
LineNumber = $finding.LineNumber
Matches = $finding.Matches.Value
}
$OBJ = New-Object -TypeName psobject -Property $Props
$OBJ | Select-Object Matches,LineNumber | Export-Csv -Path $CSV -Append -NoTypeInformation
This isn't going to work as written. You are using Import-CSV which creates an array of objects with properties. The Select-String command expects strings as input, not objects. If you want to use Select-String you would want to simply specify the file name, or use Get-Content on the file, and pass that to Select-String. If what you want is the line number, and the IP I think this would probably work just as well if not better for you:
$CSV = "C:\TEMP\OutputFile.csv"
$RX = "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(?:\.|dot|\[dot\]|\[\.\])){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"
$TestCSV = "C:\TEMP\FileWithIPs.csv"
$spreadsheetDataobject = import-csv $TestCSV
$spreadsheetDataobject |
Where{$_.IP -match $RX} |
Select-Object #{l='Matches';e={$_.IP}},#{l='LineNumber';e={[array]::IndexOf($spreadsheetDataobject,$_)+1}} |
Export-Csv -Path $CSV -Append -NoTypeInformation
Edit: wOxxOm is quite right, this answer has considerably more overhead than parsing the text directly like he does. Though, for somebody who's new to PowerShell it's probably easier to understand.
In regards to $_.IP, since you use Import-CSV you create an array of objects. Each object has properties associated with it based on the header of the CSV file. IP was listed in the header as one of your columns, so each object has a property of IP, and the value of that property is whatever was in the IP column for that record.
Let me explain the Select line for you, and then you'll see that it's easy to add your source path as another column.
What I'm doing is defining properties with a hashtable. For my examples I will refer to the first one shown above. Since it is a hashtable it starts with #{ and ends with }. Inside there are two key/value pairs:
l='Matches'
e={$_.IP}
Essentially 'l' is short for Label, and 'e' is short for Expression. The label determines the name of the property being defined (which equates to the column header when you export). The expression defines the value assigned to the property. In this case I am really just renaming the IP column to Matches, since the value that I assign for each row is whatever is in the IP field. If you open the CSV in Excel, copy the entire IP column, paste it in at the end, and change the header to Matches, that is basically all I'm doing. So to add the file path as a column we can add one more hashtable to the Select line with this:
#{
l='FilePath'
e={$CSV}
}
That adds a third property, where the name is FilePath, and the value is whatever is stored in $CSV. That updated Select line would look like this:
Select-Object #{l='Matches';e={$_.IP}},#{l='LineNumber';e={[array]::IndexOf($spreadsheetDataobject,$_)+1}},#{l='FilePath'e={$CSV}} |
Any code based on the built-in CSV cmdlets is extremely slow because objects are created for each field on each line, and it's noticeable on large files (for example, code from the other answer takes 900 seconds to process a 9MB file with 100k lines).
If your input CSV file is simple, you can process it as text in less than a second for a 100k lines file:
$CSV = .......
$RX = .......
$TestCSV = .......
$line = 0 # header line doesn't count
$lastMatchPos = 0
$text = [IO.File]::ReadAllText($TestCSV) -replace '"http.+?",', ','
$out = New-Object Text.StringBuilder
ForEach ($m in ([regex]"(?<=,""?)$RX(?=""?,)").Matches($text)) {
$line += $m.index - $lastMatchPos -
$text.substring($lastMatchPos, $m.index-$lastMatchPos).Replace("`n",'').length
$lastMatchPos = $m.Index + $m.length
$out.AppendLine('' + $line + ',' + $m.value) >$null
}
if (!(Test-Path $CSV)) {
'LineNumber,IP' | Out-File $CSV -Encoding ascii
}
$out.ToString() | Out-File $CSV -Encoding ascii -Append
The code zaps quoted URLs fields just in the unlikely but possible case those contain a matching IP.