Extract data from one file and save to another - regex

I'm running ansible playbook task using shell command to extract data from file based on regex match and save it to another file.
I tried using awk and sed but not able to get the regex working.
awk '$NF == "-m.comment.*\""' iptable.txt" > file1.txt
sed 's/\/.*' iptable.txt > file2.txt
I need to save any content from -m comment till the double quotes. to file1.txt and remaining content to file2.txt. If the line doesnot have comment field, then it should be saved only to file2.txt.
-P INPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p icmp -m state --state NEW -m comment --comment "Echo Request" -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m comment --comment "tcp" -j ACCEPT
Expected output:
cat file1.txt
-m comment --comment "Echo Request"
-m comment --comment "tcp"
cat file2.txt
-P INPUT ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p icmp -m state --state NEW -j ACCEPT
-A INPUT -p tcp -m state --state NEW -j ACCEPT

With GNU awk for the 3rd arg to match():
awk 'match($0,/(.*)(-m.comment.*")(.*)/,a) {
print a[2] " > foo"
print a[1] a[3] " > bar"
}' file
-m comment --comment "Echo Request" > foo
-A INPUT -p icmp -m state --state NEW -j ACCEPT > bar
With any awk:
awk 'match($0,/-m.comment.*"/) {
print substr($0,RSTART,RLENGTH) " > foo"
print substr($0,1,RSTART-1) substr($0,RSTART+RLENGTH) " > bar"
}' file
Just change " > foo" to > "foo" and ditto for bar to really write to new files.
If that's not exactly what you need then edit your question to clarify your requirements and provide more truly representative sample input/output.
Oh and when you wrote:
$NF == "-m.comment.*\""
There's nothing wrong with -m.comment.*\" as a regexp per-se BUT == is telling awk to do a literal string comparison, not a regexp comparison.
Given your updated question, just tweak the above to:
awk 'match($0,/(.*)(-m.comment.*")(.*)/,a) {
print a[2] " > foo"
print a[1] a[3] " > bar"
next
}
{ print $0 " > bar" }' file
-P INPUT ACCEPT > bar
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT > bar
-m comment --comment "Echo Request" > foo
-A INPUT -p icmp -m state --state NEW -j ACCEPT > bar
-m comment --comment "tcp" > foo
-A INPUT -p tcp -m state --state NEW -j ACCEPT > bar

Related

Hard regex with sed

In a script.sh file, I have the following line:
ExecStart=ssh -nN -R 46:192.168.0.1:56 192.168.0.2
I try to replace with sed the second port (56 here) knowing that its value can vary between 1 and 65535.
So I tried that without success :
sed -i -e "s/:.*[[:space/]]/other port number/2g' script.sh
Could you help me solve my regex?
You may use:
sed -i "s/:[0-9]\{1,5\} /:other port number /" script.sh
$ other_port_number="123"
$ echo "ExecStart=ssh -nN -R 46:192.168.0.1:56 192.168.0.2" | sed "s/:[0-9]\{1,5\} /:$other_port_number /"
ExecStart=ssh -nN -R 46:192.168.0.1:123 192.168.0.2

Parse the output of my bash script and save as CSV

I have a bash script that SSH'es to a list of servers (given a .txt file), runs another script inside each server, and shows the results. But I need to parse the verbose data from output, and eventually save some meaningful results as a .CSV file.
Here is my main script:
set +e
while read line
do
ssh myUser#"$line" -t 'sudo su /path/to/script.sh' < /dev/null
done < "/home/listOfServers.txt"
where the listOfServers.txt is like
server1
server2
server3
The output of running my script looks like this, showing the results for each servers one after another.
SNAME:WORKFLOW_APS_001 |10891 | Alive:2018-06-18:06:54 |TCP
SNAME:WORKFLOW_APSWEB_001 |11343 | Alive:2018-06-18:06:54 |TCP
Processes in Instance: WORKFLOW_OHS_002
WORKFLOW_OHS_002 | OHS | 8925 | Alive | 852960621 | 1367120 | 510:11:51 | http:9881
Processes in Instance: WORKFLOW_OHS_003
WORKFLOW_OHS_003 | OHS | 9187 | Alive | 2041606684 | 1367120 | 510:11:51 | http:9883
SNAME:WORKFLOW_RPSF_001 |10431 | Alive:2018-06-18:06:55 |TCP
SNAME:WORKFLOW_SCPTL_001 |9788 | Alive:2018-06-18:06:55 |TCP
...
From this output, I only need the OHS names and their status, and save along with the original server's name as a CSV. The pattern to me looks like this: I need to look at each line, and if the line doesn't contain "Processes in Instance" or "SNAME", then split based on space, and grab the 1st (OHS name) and 4th field (status). So my CSV will look like:
server1, WORKFLOW_OHS_002, Alive
server1, WORKFLOW_OHS_003, Alive
server2, .....
...
How can I modify my bash to do this?
You can use awk:
while read -r line; do
ssh myUser#"$line" -t 'sudo su /path/to/script.sh' < /dev/null |
awk -v s="$line" -F '|' -v OFS=', ' '!/^[[:blank:]]*SNAME:/ && NF>2 {
gsub(/^[[:blank:]]+|[[:blank:]]+$/, "");
gsub(/[[:blank:]]*\|[[:blank:]]*/, "|");
print s, $1, $4
}'
done < "/home/listOfServers.txt"
EDIT: As per your comment below, you can do this to handle error conditions:
while read -r line; do
out=$(ssh myUser#"$line" -t 'sudo su /path/to/script.sh' < /dev/null 2>&1)
if [[ -z $out ]]; then
echo "$line, NULL, NULL"
elif [[ $out == *"timed out"* ]]; then
echo "$line, FAIL, FAIL"
else
awk -v s="$line" -F '|' -v OFS=', ' '!/^[[:blank:]]*SNAME:/ && NF>2 {
gsub(/^[[:blank:]]+|[[:blank:]]+$/, "");
gsub(/[[:blank:]]*\|[[:blank:]]*/, "|");
print s, $1, $4
}' <<< "$out"
fi
done < "/home/listOfServers.txt"
Something to try - and good luck.
while read line
do ssh myUser#"$line" -t 'sudo su /path/to/script.sh' < /dev/null |
sed -E "/Processes in Instance/d; /SNAME/d;
s/^ *([^| ]*) *[|][^|]*[|][^|]*[|] *([^| ]*).*/$line,\1,\2/;"
done < "/home/listOfServers.txt"
You ought to be able to improve on that. :)

Bash for loop prints unexpected output, why is that?

I wrote a script which queries a dns zone on AWS Route53 and returns IP's of specific servers.
The aim is to run this script after starting a CloudFormation stack and that the script will update my local ~/.ssh/config file with the updated servers IP's.
I want to "catch" the line in the config which starts with HostName but related to the hostname of the server I would like to edit and so I've ran the following command:
$ grep -A2 dev-api2.company-private ~/.ssh/config
Host dev-api2.company-private
User ec2-user
HostName 5.5.5.5
Next, I've ran:
$ for line in $(grep -A2 dev-api2.company-private ~/.ssh/config); do echo $line ; done
Host
dev-api2.company-private
User
ec2-user
HostName
5.5.5.5
For some reason the output is printed not as I expected, it broke each word to a separate line, but why does it happen?
Edit #1:
My idea was to do something like that:
for line in $(grep -A2 dev-api2.company-private ~/.ssh/config)
do
if [[ $line == "^HostName.*" ]]
then
sed -i 's!$line!HostName 1.1.1.1!g' ~/.ssh/config
fi
done
Edit #2:
Ok, so the relevant portion of the ~/.ssh/config file looks like that:
Host as-bastion
User ec2-user
HostName ec2-34.us-west-2.compute.amazonaws.com
IdentityFile ~/.ssh/company/bastion.pem
Host dev-api1.company-private
User ec2-user
HostName 172.51.24.171
IdentityFile ~/.ssh/company/company.pem
ProxyCommand ssh ec2-user#as-bastion -W %h:%p
Host dev-api2.company-private
User ec2-user
HostName 172.51.25.248
IdentityFile ~/.ssh/company/company.pem
ProxyCommand ssh ec2-user#as-bastion -W %h:%p
The file contains many blocks like the ones above, but I want to sed only the "HostName" lines of blocks which contain the string I've used in the grep (dev-api2.company-private).
#anubhava: The code in your answer changed lines containing the string "HostName" even in blocks which are not related to "dev-api2.company-private".
How can it be done?
You should be using this while read loop using process substitution instead of a for loop using command substitution:
while IFS= read -r line; do
echo "$line"
done < <(grep -A2 dev-api2.company-private ~/.ssh/config)
Based on your comments I suggest it is better to use awk like this:
awk -v ORS='\n\n' -v RS= -v api='^Host dev-api2' -v ip='1.1.1.1' '
$0 ~ api "\\.company-private$" {sub(/\nHostName [^\n]+/, "\nHostName " ip)} 1' ~/.ssh/config
To save changes use this awk command:
awk -v ORS='\n\n' -v RS= -v api='^Host dev-api2' -v ip='1.1.1.1' '
$0 ~ api "\\.company-private$" {sub(/\nHostName [^\n]+/, "\nHostName " ip)} 1
' ~/.ssh/config > $$.tmp && mv $$.tmp ~/.ssh/config
Here is one liner sed to do the same job:
sed -i.bak '/dev-api2\.company-private/{N;p;N;s/.*/HostName 1.1.1.1/;}' ~/.ssh/config

Maintain $ (dollar sign) when performing find and replace

My script appends a block of text to the end of a file.
### Start My Block
$IPT -A INPUT -s 8.8.8.8 -j ACCEPT
$IPT -A INPUT -s 8.8.8.4 -j ACCEPT
### End My Block
If my text block doesn't exists, it successfully appends the file as shown above using the following:
HEAD="### Start My Block"
TAIL="### End My Block"
REPLACEMENT_CONTENT="
\$IPT -A INPUT -s 8.8.8.8 -j ACCEPT
\$IPT -A INPUT -s 8.8.8.4 -j ACCEPT"
echo -e "$HEAD" >> $FILE
echo -e "$REPLACEMENT_CONTENT" >> $FILE
echo -e "$TAIL" >> $FILE
If my text block exists, it attempts to replace the block's contents using the following:
HEAD="### Start My Block"
TAIL="### End My Block"
REPLACEMENT_CONTENT="
\$IPT -A INPUT -s 4.4.4.4 -j ACCEPT
\$IPT -A INPUT -s 4.4.2.2 -j ACCEPT"
perl -0777 -i -pe "s/($HEAD).*(\\n$TAIL)/\$1$REPLACEMENT_CONTENT\\$2/s" $FILE
The problem is I lose the "$IPT" and end up using the following:
### Start My Block
-A INPUT -s 4.4.4.4 -j ACCEPT
-A INPUT -s 4.4.2.2 -j ACCEPT
### End My Block
When I output the above via ">>", with the dollar sign escaped \$, the $IPT is included. When I revisit the file to replace existing content between the $HEAD and $TAIL and use the perl function with an updated block, the escaped dollar sign doesn't seem to hold. My guess is it's being interpreted and I'm not sure how to prevent that if escaping isn't enough.
Does anyone know how I can modify the command above to maintain the "$IPT"?
Yes, you are right your $IPT get interpreted in your regex inside perl.
So, using \$IPT this you are escaping $ for the shell. But you also need to escape $ for perl, so that it shouldn't interpret. For that you should do like this:
REPLACEMENT_CONTENT="
\\\$IPT -A INPUT -s 8.8.8.8 -j ACCEPT
\\\$IPT -A INPUT -s 8.8.8.4 -j ACCEPT"
This escapes \ once and then $ itself sending perl \$ and thus it did not get interpreted.
OR
Just wrap REPLACEMENT_CONTENT in single quotes.
REPLACEMENT_CONTENT='
\$IPT -A INPUT -s 8.8.8.8 -j ACCEPT
\$IPT -A INPUT -s 8.8.8.4 -j ACCEPT'
And after that perl part:
perl -0777 -i -pe "s/($HEAD).*($TAIL)/\$1$REPLACEMENT_CONTENT\n\$2/s" $FILE

Using the eval command to create a command alias

For the life of me I cannot get the bash script to execute the alias command to set the hostname of a workstation the alias name to the WOL (Wakeup On Lan) equivalent command. I figure there must be an issue with quoting somewhere that I am missing.
#!/bin/bash
WOLHosts=`nvram get wol_hosts`
WOLList=($(echo "$WOLHosts" | grep -o '[A-F0-9]\{2\}:[A-F0-9]\{2\}:[A-F0-9]\{2\}:[A-F0-9]\{2\}:[A-F0-9]\{2\}:[A-F0-9]\{2\}=[^=]*=[0-9]*[.][0-9]*[.][0-9]*[.][0-9]*' ))
if [ "${#WOLList[#]}" -gt 0 ]
then
for Match in ${WOLList[#]}
do
Command=`echo "$Match" | sed -r "s/([A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2})=([^=]*)=([0-9]*[.][0-9]*[.][0-9]*[.][0-9]*)/alias \2='\/usr\/sbin\/wol -i \3 \1'/"`
Name=`echo "$Match" | sed -r "s/([A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2})=([^=]*)=([0-9]*[.][0-9]*[.][0-9]*[.][0-9]*)/\2/"`
Com=`echo "$Match" | sed -r "s/([A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2}:[A-F0-9]{2})=([^=]*)=([0-9]*[.][0-9]*[.][0-9]*[.][0-9]*)/\/usr\/sbin\/wol -i \3 \1/"`
alias $Name="$Com"
eval $Command
echo "$Command"
done
fi
exit 0
Here is some sample data and output that I am currently receiving with the script:
Input (into WOLHosts):
00:1F:D0:26:72:53=Justin-PC=192.168.1.255 00:16:17:DD:12:7B=Justin-HTPC=192.168.1.255 00:1C:25:BC:C3:85=justinlaptop=192.168.1.255
The output produced by the vi WOecho "$Command" is:
alias Justin-PC='/usr/sbin/wol -i 192.168.1.255 00:1F:D0:26:72:53'
alias Justin-HTPC='/usr/sbin/wol -i 192.168.1.255 00:16:17:DD:12:7B'
alias justinlaptop='/usr/sbin/wol -i 192.168.1.255 00:1C:25:BC:C3:85'
Since you appear to be running this as a script, your current shell will not receive the aliases -- the aliases will disappear then the bash process driving the script ends.
Try: . script.sh or source script.sh