Match and $Matches in Powershell via RegEx - regex

I have a little issue with a Powershell project, I've been working on for some time now.
The basic idea is that six iPerf speed measurements will be executed.
A logfile is created to have some data which can be displayed to the user.
But there's some issue with the match and variable Matches in Powershell to display multiple values..
There's the code, I've been working on..
$1 = (Get-Content -Path 'iperf3.txt' -TotalCount 398)[-1] # Fetch details about speed measurements for download
$2 = (Get-Content -Path 'iperf3.txt' -TotalCount 796)[-1] # Fetch details about speed measurements for upload
$3 = (Get-Content -Path 'iperf3.txt' -TotalCount 1195)[-1] # Same as above
$4 = (Get-Content -Path 'iperf3.txt' -TotalCount 1593)[-1] # Same as above
$5 = (Get-Content -Path 'iperf3.txt' -TotalCount 1992)[-1] # Same as above
$6 = (Get-Content -Path 'iperf3.txt' -TotalCount 2390)[-1] # Same as above
Output via Get-Content and TotalCount
[SUM] 0.00-30.00 sec 1.09 GBytes 313 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 311 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
Afterwards, I use the RegEx and Variable to output the numbers before Mbits/sec and include Mbits/sec from this line of code..
$1 -match '\d+\sMbits[/]sec'
$2 -match '\d+\sMbits[/]sec'
etc.
I do variable Matches to validate that the Variable is True, and receive the output of 312 Mbits/sec, but nothing more.
Now this is where I cannot see the fault in the code. The variable passed and it's true, but I only have one Value as 313 Mbits/sec via Value.
I figured that I would see both 313 Mbits/sec and 312 Mbits/sec in the output/value prompt.
Did I do something wrong while using the match/Matches variable/function?
Any feedback and/or suggestions will be appreciated.

The automatic $Matches variable only ever reflects the results of the most recent -match operation - and then only if (a) the matching was successful and (b), fundamentally, only if the LHS was a single string - if the LHS was a collection (array), -match acts as a filter, returning the subarray of matching elements, and does not populate $Matches.
However, your command can be greatly streamlined:
Use a single Get-Content call
Use a Select-Object call with the -Index parameter to extract the lines of interest (indices are 0-based).
Use the -replace operator instead of -match in order to directly extract the substrings of interest:
(
Get-Content 'iperf3.txt' | Select-Object -Index 397,795,1194,1592,1991,2389
) -replace '.+\b(\d+\sMbits/sec).+', '$1'
Taking a step back:
Instead of selecting the lines of interest by fixed indices (line numbers), select them by regexes too, which allows you to use a single Select-String call:
Select-String -LiteralPath 'iperf3.txt' -Pattern '\s*\[SUM].+\b(\d+\sMbits/sec).+' |
ForEach-Object {
$_.Matches.Groups[1].Value
}

You didn't do anything wrong it's just the default behavior for the -match operator and how the $Matches automatic variable is populated.
Here is an extract from Matching operators that explains very well how it works:
It is important to note that the $Matches hashtable contains only the first occurrence of any matching pattern.
You have 2 workarounds, the first one could be using Regex.Matches Method to find all appearances of the matched pattern:
$string = #'
[SUM] 0.00-30.00 sec 1.09 GBytes 313 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 311 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
'#
[regex]::Matches($string, '\d+\sMbits[/]sec').Value
Note that, in above example, $string is a multi-line string, however in the example it will be an array since it requires a loop.
$string = #'
[SUM] 0.00-30.00 sec 1.09 GBytes 313 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 311 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.11 GBytes 317 Mbits/sec receiver
[SUM] 0.00-30.00 sec 1.09 GBytes 312 Mbits/sec receiver
'# -split '\r?\n'
foreach($line in $string) {
if($line -match '\d+\sMbits[/]sec') {
$Matches[0]
}
}
GitHub issue #7867 proposes to add a -matchall operator to PowerShell, if you believe it would be helpful consider up-voting it.

Related

The throughput of ip_reasseble is low

The throughput of ip_reasseble is low, ip_reassembly example was used to test it. Another PC2 run iperf -c to send ip fragments packets, the maximum throughput is about 900Mbps.
Testing setups:
PC1(dpdk) --- PC1(iperf -c)
PC1: 10Gb/s NIC
PC2: 1Gb/s NIC, MTU: 1500
#./build/ip_reassembly -l 2 -- -p 0x1 &
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:00:1f.6 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:15b7 net_e1000_em
EAL: PCI device 0000:04:00.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:04:00.1 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 8086:10fb net_ixgbe
IP_RSMBL: Creating LPM table on socket 0
IP_RSMBL: Creating LPM6 table on socket 0
USER1: rte_ip_frag_table_create: allocated of 25165952 bytes at socket 0
Initializing port 0 ... Port 0 modified RSS hash function based on hardware support,requested:0xa38c configured:0x8104
Address:00:1B:21:C1:E9:C6
txq=2,0,0
IP_RSMBL: Socket 0: adding route 100.10.0.0/16 (port 0)
IP_RSMBL: Socket 0: adding route 100.20.0.0/16 (port 1)
IP_RSMBL: Socket 0: adding route 100.30.0.0/16 (port 2)
IP_RSMBL: Socket 0: adding route 100.40.0.0/16 (port 3)
IP_RSMBL: Socket 0: adding route 100.50.0.0/16 (port 4)
IP_RSMBL: Socket 0: adding route 100.60.0.0/16 (port 5)
IP_RSMBL: Socket 0: adding route 100.70.0.0/16 (port 6)
IP_RSMBL: Socket 0: adding route 100.80.0.0/16 (port 7)
IP_RSMBL: Socket 0: adding route 0101:0101:0101:0101:0101:0101:0101:0101/48 (port 0)
IP_RSMBL: Socket 0: adding route 0201:0101:0101:0101:0101:0101:0101:0101/48 (port 1)
IP_RSMBL: Socket 0: adding route 0301:0101:0101:0101:0101:0101:0101:0101/48 (port 2)
IP_RSMBL: Socket 0: adding route 0401:0101:0101:0101:0101:0101:0101:0101/48 (port 3)
IP_RSMBL: Socket 0: adding route 0501:0101:0101:0101:0101:0101:0101:0101/48 (port 4)
IP_RSMBL: Socket 0: adding route 0601:0101:0101:0101:0101:0101:0101:0101/48 (port 5)
IP_RSMBL: Socket 0: adding route 0701:0101:0101:0101:0101:0101:0101:0101/48 (port 6)
IP_RSMBL: Socket 0: adding route 0801:0101:0101:0101:0101:0101:0101:0101/48 (port 7)
Checking link status
done
Port0 Link Up. Speed 10000 Mbps - full-duplex
IP_RSMBL: entering main loop on lcore 2
IP_RSMBL: -- lcoreid=2 portid=0
run iperf -c on PC2
# iperf -c 192.168.10.157 -i 1 -u -t 30 -p 2152 -b 900M -l 1600
------------------------------------------------------------
Client connecting to 192.168.10.157, UDP port 2152
Sending 1600 byte datagrams, IPG target: 13.56 us (kalman adjust)
UDP buffer size: 958 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.10.100 port 37771 connected with 192.168.10.157 port 2152
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 113 MBytes 944 Mbits/sec
[ 3] 1.0- 2.0 sec 112 MBytes 944 Mbits/sec
[ 3] 2.0- 3.0 sec 112 MBytes 944 Mbits/sec
[ 3] 3.0- 4.0 sec 113 MBytes 944 Mbits/sec
[ 3] 4.0- 5.0 sec 112 MBytes 944 Mbits/sec
[ 3] 5.0- 6.0 sec 112 MBytes 944 Mbits/sec
[ 3] 6.0- 7.0 sec 113 MBytes 944 Mbits/sec
[ 3] 7.0- 8.0 sec 112 MBytes 944 Mbits/sec
[ 3] 8.0- 9.0 sec 113 MBytes 944 Mbits/sec
[ 3] 9.0-10.0 sec 112 MBytes 944 Mbits/sec
[ 3] 10.0-11.0 sec 112 MBytes 944 Mbits/sec
[ 3] 11.0-12.0 sec 112 MBytes 944 Mbits/sec
[ 3] 12.0-13.0 sec 112 MBytes 944 Mbits/sec
[ 3] 13.0-14.0 sec 112 MBytes 944 Mbits/sec
[ 3] 14.0-15.0 sec 113 MBytes 944 Mbits/sec
[ 3] 15.0-16.0 sec 112 MBytes 944 Mbits/sec
[ 3] 16.0-17.0 sec 113 MBytes 944 Mbits/sec
[ 3] 17.0-18.0 sec 112 MBytes 944 Mbits/sec
[ 3] 18.0-19.0 sec 112 MBytes 944 Mbits/sec
[ 3] 19.0-20.0 sec 112 MBytes 944 Mbits/sec
[ 3] 20.0-21.0 sec 113 MBytes 944 Mbits/sec
[ 3] 21.0-22.0 sec 112 MBytes 944 Mbits/sec
[ 3] 22.0-23.0 sec 112 MBytes 944 Mbits/sec
[ 3] 23.0-24.0 sec 112 MBytes 944 Mbits/sec
[ 3] 24.0-25.0 sec 112 MBytes 944 Mbits/sec
[ 3] 25.0-26.0 sec 113 MBytes 944 Mbits/sec
[ 3] 26.0-27.0 sec 112 MBytes 944 Mbits/sec
[ 3] 27.0-28.0 sec 112 MBytes 944 Mbits/sec
[ 3] 28.0-29.0 sec 113 MBytes 944 Mbits/sec
[ 3] 29.0-30.0 sec 113 MBytes 944 Mbits/sec
[ 3] WARNING: did not receive ack of last datagram after 10 tries.
[ 3] 0.0-30.0 sec 3.30 GBytes 944 Mbits/sec
[ 3] Sent 2211842 datagrams
the result of ip fragments reassembled:
# ps
PID TTY TIME CMD
335 pts/9 00:02:35 ip_reassembly
535 pts/9 00:00:00 ps
25304 pts/9 00:00:00 su
25306 pts/9 00:00:00 zsh
# kill -SIGUSR1 335
-- lcoreid=2 portid=0 frag tbl stat:
max entries: 4096;
entries in use: 4088;
finds/inserts: 4344078;
entries added: 883521;
entries deleted by timeout: 837;
entries reused by timeout: 0;
total add failures: 2581961;
add no-space failures: 2581961;
add hash-collisions failures: 0;
TX bursts: 0
TX packets _queued: 0
TX packets dropped: 0
TX packets send: 0
RX gtpu packets: 872727
I add some stats of udp port 2152 in example of ip_reassembly to show the successful reassembled
packets. According to the result, the PC2 send 2211842 datagrams while only 872727 packets were reassembled by ip_reassembly. When I low down the sending speed of iperf to 800Mbps, no drops print.
I don't find the throughput description in DPDK guide https://doc.dpdk.org/guides-22.07/prog_guide/ip_fragment_reassembly_lib.html
has anyone met the same questions?

How to match a regular expression to a text file?

I want to read from a file as text by data=fileread('channelresult') function. Then the data is used for regular expression matching. The channelresult file content is:
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-0.10 sec 9.24 MBytes 774 Mbits/sec
[ 4] 0.10-0.20 sec 14.8 MBytes 1.24 Gbits/sec
[ 4] 0.20-0.30 sec 15.0 MBytes 1.27 Gbits/sec
[ 4] 0.30-0.40 sec 17.6 MBytes 1.48 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.74 GBytes 1.49 Gbits/sec 1005 sender
[ 4] 0.00-10.00 sec 1.74 GBytes 1.49 Gbits/sec receiver
The regular expression I use is
pattern=number_str+'\s+sec\s+'+number_str+'\s+\w+\s+'+number_str+'\s+(\w)\w+/\w+\s+(\d+)\s+'+number_str+'\s(\w)'
And number_str='(\d*\.\d+|\d+)'. When I use out = regexp(data,pattern,'match') the variable out does not contain anything. It's a 0 by 0 cell array.

SAS ARIMA modelling for 5 different variables

I am trying do a ARIMA model estimation for 5 different variables. The data consists of 16 months of Point of Sales. How do I approach this complicated ARIMA modelling?
Furthermore I would like to do:
A simple moving average of each product group
A Holt-Winters
exponential smoothing model
Data is as follows with date and product groups:
Date Gloves ShoeCovers Socks Warmers HeadWear
apr-14 11015 3827 3465 1264 772
maj-14 11087 2776 4378 1099 1423
jun-14 7645 1432 4490 674 670
jul-14 10083 7975 2577 1558 8501
aug-14 13887 8577 6854 1305 15621
sep-14 9186 5213 5244 1183 6784
okt-14 7611 4279 4150 977 6191
nov-14 6410 4033 2918 507 8276
dec-14 4856 3552 3192 450 4810
jan-15 17506 7274 3137 2216 3979
feb-15 21518 5672 8848 1838 2321
mar-15 17395 5200 5712 1604 2282
apr-15 11405 4531 5185 1479 1888
maj-15 11509 5690 4370 1145 2369
jun-15 9945 2610 4884 882 1709
jul-15 8707 5658 4570 1948 6255
Any skilled forecasters out there willing to help? Much appreciated!

AWS intermittent slow ssh and GET requests

we have few EC2 server on AWS but one one particular server we are noticing odd issues. First of all, SSH takes tens of seconds but only happens intermittently. Secondly, get request to the server also takes tens of seconds, but again it happens intermittently. We checked the stats on the server and everything looks fine. CPU average is around 7% and 25GB of free ram.
Here is the response from ss -s:
Total: 2553 (kernel 0)
TCP: 12015 (estab 2342, closed 9189, orphaned 478, synrecv 0, timewait 9188/0), ports 0
Transport Total IP IPv6
* 0 - -
RAW 0 0 0
UDP 4 4 0
TCP 2826 2765 61
INET 2830 2769 61
FRAG 0 0 0
And breakdown of current connection status:
1 established)
1 Foreign
6 LISTEN
62 LAST_ACK
136 SYN_RECV
155 CLOSING
251 FIN_WAIT1
1078 FIN_WAIT2
2197 ESTABLISHED
8229 TIME_WAIT
We have ruled out DNS issue, as it happens when we try to access it via it's hostname or the ip address. There are no load balancer in front of that server. We do use Route53 for routing purposes, but I don't see any issue with that.
Using AB to do get requests:
Run #1
Server Software: nginx/1.6.1
Server Hostname: my.hidden.com
Server Port: 443
SSL/TLS Protocol: TLSv1,DHE-RSA-AES256-SHA,2048,256
Document Path: /debug
Document Length: 43 bytes
Concurrency Level: 1
Time taken for tests: 104.415 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 162304 bytes
HTML transferred: 21500 bytes
Requests per second: 4.79 [#/sec] (mean)
Time per request: 208.829 [ms] (mean)
Time per request: 208.829 [ms] (mean, across all concurrent requests)
Transfer rate: 1.52 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 134 160 35.7 154 709
Processing: 39 49 8.2 48 127
Waiting: 39 49 8.2 48 127
Total: 178 209 38.6 203 809
Percentage of the requests served within a certain time (ms)
50% 203
66% 209
75% 212
80% 215
90% 225
95% 244
98% 273
99% 302
100% 809 (longest request)
Run #2
Server Software: nginx/1.6.1
Server Hostname: my.hidden.com
Server Port: 443
SSL/TLS Protocol: TLSv1,DHE-RSA-AES256-SHA,2048,256
Document Path: /debug
Document Length: 43 bytes
Concurrency Level: 1
Time taken for tests: 515.608 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 162284 bytes
HTML transferred: 21500 bytes
Requests per second: 0.97 [#/sec] (mean)
Time per request: 1031.216 [ms] (mean)
Time per request: 1031.216 [ms] (mean, across all concurrent requests)
Transfer rate: 0.31 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 136 980 1251.8 164 5575
Processing: 39 51 33.0 48 730
Waiting: 39 51 33.0 48 730
Total: 180 1031 1258.7 216 6306
Percentage of the requests served within a certain time (ms)
50% 216
66% 243
75% 2839
80% 2850
90% 2874
95% 2903
98% 2935
99% 2992
100% 6306 (longest request)
Run #3
Server Software: nginx/1.6.1
Server Hostname: my.hidden.com
Server Port: 443
SSL/TLS Protocol: TLSv1,DHE-RSA-AES256-SHA,2048,256
Document Path: /debug
Document Length: 43 bytes
Concurrency Level: 1
Time taken for tests: 417.639 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 162320 bytes
HTML transferred: 21500 bytes
Requests per second: 1.20 [#/sec] (mean)
Time per request: 835.279 [ms] (mean)
Time per request: 835.279 [ms] (mean, across all concurrent requests)
Transfer rate: 0.38 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 137 781 1140.0 162 5053
Processing: 38 54 56.6 49 1281
Waiting: 38 54 56.6 49 1281
Total: 179 835 1141.2 213 5104
Percentage of the requests served within a certain time (ms)
50% 213
66% 233
75% 334
80% 2835
90% 2872
95% 2915
98% 3030
99% 3137
100% 5104 (longest request)
Run #4
Server Software: nginx/1.6.1
Server Hostname: my.hidden.com
Server Port: 443
SSL/TLS Protocol: TLSv1,DHE-RSA-AES256-SHA,2048,256
Document Path: /debug
Document Length: 43 bytes
Concurrency Level: 1
Time taken for tests: 104.806 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 162250 bytes
HTML transferred: 21500 bytes
Requests per second: 4.77 [#/sec] (mean)
Time per request: 209.611 [ms] (mean)
Time per request: 209.611 [ms] (mean, across all concurrent requests)
Transfer rate: 1.51 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 135 160 13.8 157 227
Processing: 39 50 8.8 48 159
Waiting: 39 50 8.8 48 159
Total: 179 209 17.7 206 331
Percentage of the requests served within a certain time (ms)
50% 206
66% 212
75% 216
80% 219
90% 230
95% 240
98% 266
99% 275
100% 331 (longest request)
Run #5
Server Software: nginx/1.6.1
Server Hostname: my.hidden.com
Server Port: 443
SSL/TLS Protocol: TLSv1,DHE-RSA-AES256-SHA,2048,256
Document Path: /debug
Document Length: 43 bytes
Concurrency Level: 1
Time taken for tests: 110.983 seconds
Complete requests: 500
Failed requests: 0
Total transferred: 162282 bytes
HTML transferred: 21500 bytes
Requests per second: 4.51 [#/sec] (mean)
Time per request: 221.967 [ms] (mean)
Time per request: 221.967 [ms] (mean, across all concurrent requests)
Transfer rate: 1.43 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 132 170 149.3 159 3460
Processing: 38 52 25.8 49 589
Waiting: 38 52 25.8 49 589
Total: 177 222 152.6 208 3532
Percentage of the requests served within a certain time (ms)
50% 208
66% 217
75% 222
80% 227
90% 240
95% 257
98% 282
99% 336
100% 3532 (longest request)

GREP values from a column in txt file

I have a txt file with 1200 entries in this way (iPerf output by the way)
1 [ 4] 0.0- 1.0 sec 10.6 MBytes 89.1 Mbits/sec
2 [ 4] 1.0- 2.0 sec 13.5 MBytes 113 Mbits/sec
3 [ 4] 2.0- 3.0 sec 9.50 MBytes 79.7 Mbits/sec
4 [ 4] 3.0- 4.0 sec 9.00 MBytes 75.5 Mbits/sec
How can I get ONLY the second values expressed in Mbits/sec using grep ?
Output example:
89.1
113
79.7
75.5
awk '{print $9}' your-file.txt
will do it for you. For example:
$ cat ~/test.txt
1 [ 4] 0.0- 1.0 sec 10.6 MBytes 89.1 Mbits/sec
2 [ 4] 1.0- 2.0 sec 13.5 MBytes 113 Mbits/sec
3 [ 4] 2.0- 3.0 sec 9.50 MBytes 79.7 Mbits/sec
4 [ 4] 3.0- 4.0 sec 9.00 MBytes 75.5 Mbits/sec
$ awk '{print $9}' ~/test.txt
89.1
113
79.7
75.5
Another way to tackle this is:
awk -F 'MBytes' '{print $2}' test.txt | awk -F 'Mbits' '{print $1}' | tr -d " "
In the above method we are:
Splitting each line by MBytes.
That gives us 2 parts: $1 is everything before MBytes. $2 is everything after MBytes
We choose everything after MBytes and split it further by Mbits
That gives us two parts again and we choose everything before Mbits
If there is white space before and after the numbers, we use tr to remove white space
So we get
$ cat test.txt
1 [ 4] 0.0- 1.0 sec 10.6 MBytes 89.1 Mbits/sec
2 [ 4] 1.0- 2.0 sec 13.5 MBytes 113 Mbits/sec
3 [ 4] 2.0- 3.0 sec 9.50 MBytes 79.7 Mbits/sec
4 [ 4] 3.0- 4.0 sec 9.00 MBytes 75.5 Mbits/sec
awk -F 'MBytes' '{print $2}' test.txt | awk -F 'Mbits' '{print $1}' | tr -d " "
Result:
89.1
113
79.7
75.5
if your data is fixed length format you can always use cut
cut -c38-41 data
if you know that the values are 4 chars wide.