I've had a bit of shell scripting practice reading piped input from other programs, but am unsure how to approach this problem.
THE BACKSTORY
A program robinbotter whose internals I can't really fix/modify takes its input from files equities.sym and blacklist.sym, each simple text files containing one ticker symbol per line.
When it runs okay, its output produces:
...
Downloading instruments: [ AFL KELYB LFUS ]
...
When it breaks due to internal bugs,
...
Downloading instruments: [ AFL KELYB LFUS LNVGY
and halts there, with no further output, yielding exit code 0 like in the okay case (unfortunately).
The ticker symbols are printed out with slight delay--no newlines in between--while the program is processing them.
When it hits LNVGY or unpredictably any other many possibilities, somehow it can't handle or at least skip them, instead crashing with no proper exception nor error code.
THE QUESTION
I'm trying to write a minimalistic wrapper script in BASH (eg. retryRB.sh ./robinbotter) which:
Somehow monitors the live unbuffered output of robinbotter, using a regex or other method to detect when output of a line containing "Downloading instruments: [ " doesn't end with "]" before the program ends. In which case:
Take the last symbol printed out (eg. LNVGY) which crashes the program, and append it to the bottom of file blacklist.sym. Like with
echo $lastSymbol >> blacklist.sym
Restart the program robinbotter, retaining its original command-line parameters: $#
I am familiar with tools like awk and sed, and would be open to building a short solution in Ruby if Bash doesn't cut it.
Here you have a Bash version of a code that imitates what your binary does.
then you have a wrapper which logs when the apps successfully completes, and also, when it fails. On failure, it also appends the last item printed, as you can see in the images below ( in this case Im hard-coding a failure on Bomb! and Boom! but you get the idea):
main.sh
#!/bin/bash
some=('Pera' 'Manzana' 'Frutilla' 'Durazno' 'Banana' 'Lechuga' 'Sandia' 'Papa' 'Melon' 'Milanesa' 'Bomb!' 'Boom!')
printf 'Downloading instruments: [ '
for (( i=1 ; i < 5 ; i++ )) {
item=${some[$( shuf -i 0-$(( ${#some[#]} - 1 )) -n 1 )]}
printf "$item"
[[ $item == 'Bomb!' || $item == "Boom!" ]] && exit || printf "$item"
[[ $i -lt 4 ]] && printf ' '
}
printf ' ]'
wrapper.sh
#!/bin/bash
while :
do
res=$( ./main.sh )
[[ ! "$res" =~ \[[^]]*\] ]] && printf "Failure : ${res##*[\[ ]}" || printf "Success"
printf '\n'
sleep 1
done
You can test these scripts and then put your binary in place of main.sh.
Regards!
Related
This question already has answers here:
How to parse $QUERY_STRING from a bash CGI script?
(16 answers)
Closed 3 years ago.
All right, folks, you may have seen this infamous quirk to get hold of those values:
query=`echo $QUERY_STRING | sed "s/=/='/g; s/&/';/g; s/$/'/"`
eval $query
If the query string is host=example.com&port=80 it works just fine and you get the values in bash variables host and port.
However, you may know that a cleverly crafted query string will cause an arbitrary command to be executed on the server side.
I'm looking for a secure replacement or an alternative not using eval. After some research I dug up these alternatives:
read host port <<< $(echo "$QUERY_STRING" | tr '=&' ' ' | cut -d ' ' -f 2,4)
echo $host
echo $port
and
if [[ $QUERY_STRING =~ ^host=([^&]*)\&port=(.*)$ ]]
then
echo ${BASH_REMATCH[1]}
echo ${BASH_REMATCH[2]}
else
echo no match, sorry
fi
Unfortunately these two alternatives only work if the pars come in the order host,port. But they could come in the opposite order.
There could also be more than 2 pars, and any order is possible and allowed. So how do you propose to get the values into the
appropriate bash vars? Can the above methods be amended? Remember that with n pars there are n! possible orders. With 2 pars
there are only 2, but with 3 pars there are already 3! = 6.
I returned to the first method. Can it be made safe to run eval? Can you transform $QUERY_STRING with sed in a way that
makes it safe to do eval $query ?
EDIT: Note that this question differs from the other one referred to and is not a duplicate. The emphasis here is on using eval in a safe way. That is not answered in the other thread.
This method is safe. It does not eval or execute the QUERY_STRING. It uses string manipulation to break up the string into pieces:
QUERY_STRING='host=example.com&port=80'
declare -a pairs
IFS='&' read -ra pairs <<<"$QUERY_STRING"
declare -A values
for pair in "${pairs[#]}"; do
IFS='=' read -r key value <<<"$pair"
values["$key"]="$value"
done
echo do something with "${values[host]}" and "${values[port]}"
URL "percent decoding" left as an exercise.
You must avoid executing strings at all time when they come from untrusted sources. Therefore I would strongly suggest never to use eval in Bash do something with a string.
To be really save, I think I would echo the string into a file, use grep to retrieve parts of the string and remove the file afterwards. Always use a directory out of the web root.
#! /bin/bash
MYFILE=$(mktemp)
QUERY_STRING='host=example.com&port=80&host=lepmaxe.moc&port=80'
echo "${QUERY_STRING}" > ${MYFILE}
TMP_ARR=($(grep -Eo '(host|port)[^&]*' ${MYFILE}))
[ ${#TMP_ARR} -gt 0 ] || exit 1
[ $((${#TMP_ARR} % 2)) -eq 0 ] || exit 1
declare -A ARRAY;
for ((i = 0; i < ${#TMP_ARR[#]}; i+=2)); do
tmp=$(echo ${TMP_ARR[#]:$((i)):2})
port=$(echo $tmp | sed -r 's/.*port=([^ ]*).*/\1/')
host=$(echo $tmp | sed -r 's/.*host=([^ ]*).*/\1/')
ARRAY[$host]=$port
done
for i in ${!ARRAY[#]}; do
echo "$i = ${ARRAY[$i]}"
done
rm ${MYFILE}
exit 0
This produces:
lepmaxe.moc = 80
example.com = 80
My pattern defined to match in if-else block is :
pat="17[0-1][0-9][0-9][0-9].AUG"
nln=""
In my script, I'm taking user input which needs to be matched against the pattern, which if doesn't match, appropriate error messages are to be shown. Pretty simple, but giving me a hard time though. My code block from the script is this:
echo "How many days' AUDIT Logs need to be searched?"
read days
echo "Enter file name(s)[For multiple files, one file per line]: "
for(( c = 0 ; c < $days ; c++))
do
read elements
if [[ $elements =~ $pat ]];
then
array[$c]="$elements"
elif [[ $elements =~ $nln ]];
then
echo "No file entered.Run script again. Exiting"
exit;
else
echo "Invalid filename entered: $elements.Run script again. Exiting"
exit;
fi
done
The format I want from the user for filenames to be entered is this:
170402.AUG
So basically yymmdd.AUG (where y-year,m-month,d-day), with trailing or leading spaces is fine. Anything other than that should throw "Invalid filename entered: $elements.Run script again. Exiting" message. Also I want to check if if it is a blank line with a "Enter" hit, it should give an error saying "No file entered.Run script again. Exiting"
However my code, even if I enter something like "xxx" as filename, which should be throwing "Invalid filename entered: $elements.Run script again. Exiting", is actually checking true against a blank line, and throwing "No file entered.Run script again. Exiting"
Need some help with handling the regular expressions' check with user input, as otherwise rest of my script works just fine.
I think as discussed in the comments you are confusing with the glob match and a regEx match, what you have defined as pat is a glob match which needs to be equated with the == operator as,
pat="17[0-1][0-9][0-9][0-9].AUG"
string="170402.AUG"
[[ $string == $pat ]] && printf "Match success\n"
The equivalent ~ match would be to something as
pat="17[[:digit:]]{4}\.AUG"
[[ $string =~ $pat ]] && printf "Match success\n"
As you can see the . in the regex syntax has been escaped to deprive of its special meaning ( to match any character) but just to use as a literal dot. The POSIX character class [[:digit:]] with a character count {4} allows you to match 4 digits followed by .AUG
And for the string empty check do as suggested by the comments from Cyrus, or by Benjamin.W
[[ $elements == "" ]]
(or)
[[ -z $elements ]]
I would not bug the user with how many days (who want count 15 days or like)? Also, why only one file per line? You should help the users, not bug them like microsoft...
For the start:
show_help() { cat <<'EOF'
bla bla....
EOF
}
show_files() { echo "${#files[#]} valid files entered: ${files[#]}"; }
while read -r -p 'files? (h-help)> ' line
do
case "$line" in
q) echo "quitting..." ; exit 0 ;;
h) show_help ; continue;;
'') (( ${#files} )) && show_files; continue ;;
l) show_files ; continue ;;
p) (( ${#files} )) && break || { echo "No files enterd.. quitting" ; exit 1; } ;; # go to processing
esac
# select (grep) the valid patterns from the entered line
# and append them into the array
# using the -P (if your grep know it) you can construct very complex regexes
files+=( $(grep -oP '17\d{4}.\w{3}' <<< "$line") )
done
echo "processing files ${files[#]}"
Using such logic you can build really powerful and user-friendly app. Also, you can use -e for the read enable the readline functions (cursor keys and like)...
But :) Consider just create a simple script, which accepts arguments. Without any dialogs and such. example:
myscript -h
same as above, or some longer help text
myscript 170402.AUG 170403.AUG 170404.AUG 170405.AUG
will do whatever it should do with the files. Main benefit, you could use globbing in the filenames, like
myscript 1704*
and so on...
And if you really want the dialog, it could show it when someone runs the script without any argument, e.g.:
myscript
will run in interactive mode...
What am I'm doing wrong over here?
The script by default enters this IF statement & displays the echo statement to exit.
#!/bin/ksh
server=$1
dbname=$2
IFS="
"
if [[ "${dbname}" != "abc_def_data" || "${dbname}" != "abc_def01_data" ]]; then
echo "Msg: Triggers can only be applied to CMS_JAD:abc_def_data/abc_def01_data!"
exit 0
fi
chaining of != conditions requires some inversion of thinking.
I much prefer a clearer path to testing these conditions by using the case ... esac structure.
case "${dbname}" in
abc_def_data|abc_def01_data )
#dbg echo "matched, but for real code replace with just a ':' char"
:
;;
* )
echo "didn_t match any expected values for \$dbname"
echo exit 1
;;
esac
Note that as you're really trying to find the *) case, the actions for the abc_def_data (etc) match can be anything, but to just skip to the next section of code, you would only need the shell's null cmd : .
Edit 1
Note that I have echo exit 1, just so if you copy/paste this to a command line, your shell won't exit. In real code, remove the echo and expect the exit to work.
Edit 2
Also, note that the | char in the case match (abc_def_data**|**abc_def01_data) is essentially an OR (I think it is called something else in the "case match" context).
IHTH
Did you, by any chance, meant to write this?
if [[ "${dbname}" != "abc_def_data" && "${dbname}" != "abc_def01_data" ]]; then
echo "Msg: Triggers can only be applied to CMS_JAD:abc_def_data/abc_def01_data!"
exit 0
fi
try this man, it should work just fine you should have seperated the conditions with "[ ]" and used -o instead of ||....
btw it worked for me fine...
server=$1
dbname=$2
IFS=""
if [ "${dbname}" != "abc_def_data" ] -o [ "${dbname}" != "abc_def01_data" ]
then
echo "Msg: Triggers can only be applied to CMS_JAD:abc_def_data/abc_def01_data!"
exit 0
fi
Currently, I'm trying to convert my Bash script to a C++ executable, but I am stuck with the sed command.
Here is my Bash script:
unset WIFIMAC
unset BTMAC
# Skip processing if MAC addresses are already written
if [ -f /data/.mac.info -a -f /data/.bt.info ]; then
echo "MAC addresses already found."
fi
# Wait until Samsung's RIL announces MAC addresses
until [ $(expr length "$WIFIMAC") == 17 ]; do
WIFIMAC=`getprop ril.wifi_macaddr`
sleep 1
done
until [ $(expr length "$BTMAC") == 12 ]; do
BTMAC=`getprop ril.bt_macaddr`
sleep 1
done
# Set WiFi MAC address
echo $WIFIMAC >/data/.mac.info
# Convert BT MAC address to proper format
echo $BTMAC | sed 's!^M$!!;s!\-!!g;s!\.!!g;s!\(..\)!\1:!g;s!:$!!' >/data/.bt.info
exit
Here my try to convert it to C++ (I put comments next to Bash commands):
# This script will read the MAC addresses from Samsung's RIL.
unset WIFIMAC ---> char wifimac....
unset BTMAC ---> char btmac...
# Skip processing if MAC addresses are already written
if [ -f /data/.mac.info -a -f /data/.bt.info ]; then ----> create file_exist(); function with fd = open... and put a smiply if return block
echo "MAC addresses already found."
fi
# Wait until Samsung's RIL announces MAC addresses
until [ $(expr length "$WIFIMAC") == 17 ]; do -----> while strlen(wifimac) == 17 blah blah blah....
WIFIMAC=`getprop ril.wifi_macaddr` -----> property_get function in cutils.h
sleep 1 -----> mdelay(1) if ım not wrong huh?.....
done
until [ $(expr length "$BTMAC") == 12 ]; do
BTMAC=`getprop ril.bt_macaddr` -----> SAME COMMANDS ABOVE
sleep 1
done
# Set WiFi MAC address
echo $WIFIMAC >/data/.mac.info -----> create write_string_to_path(); function with write(fd, ...)
# Convert BT MAC address to proper format
echo $BTMAC | sed 's!^M$!!;s!\-!!g;s!\.!!g;s!\(..\)!\1:!g;s!:$!!' >/data/.bt.info -----> ********HERE İS THE COMMAND "sed" *********
exit
I know a little bit about sed, but I don't know what it is doing here, thus I don't know which command in C++ does the same thing.
sed reads its input (in your example the string in $BTMAC which is echoed) and based on the expression passed to it, replaces some parts of the input data with something else (again according to the expression) and writes the result into the stdout or output file (in your case: /data/.bt.info file)
For example:
$ sed 's/replaceThis/WithThis/'
will replace word replaceThis with word WithThis in its input data and output the result. While both the above strings can be regular expressions.
If you are unfamiliar with it, you have to read the manual to first understand what's going on in your sed command. no one here is going to do that for you.
Now, in C++11, we have regex library that you can and I guess should use. If you can't use C++11 stuff, you should seek out other libraries which provide working with regex like Qt.
So in regex library of C++11, you can use regex_replace() function, or search with regex_search() and replace manually.
I have a text file named raw.txt with something like the following:
T DOTTY CRONO 52/50 53/40 54/30 55/20 RESNO NETKI
U CYMON DENDU 51/50 52/40 53/30 54/20 DOGAL BEXET
V YQX KOBEV 50/50 51/40 52/30 53/20 MALOT GISTI
W VIXUN LOGSU 49/50 50/40 51/30 52/20 LIMRI XETBO
X YYT NOVEP 48/50 49/40 50/30 51/20 DINIM ELSOX
Y DOVEY 42/60 44/50 47/40 49/30 50/20 SOMAX ATSUR
Z SOORY 43/50 46/40 48/30 49/20 BEDRA NERTU
A DINIM 51/20 52/30 50/40 47/50 RONPO COLOR
B SOMAX 50/20 51/30 49/40 46/50 URTAK BANCS
C BEDRA 49/20 50/30 48/40 45/50 VODOR RAFIN
D ETIKI 48/15 48/20 49/30 47/40 44/50 BOBTU JAROM
E 46/40 43/50 42/60 DOVEY
F 45/40 42/50 41/60 JOBOC
G 43/40 41/50 40/60 SLATN
I'm reading it into an array:
while read line; do
set $line
IFS=' ' read -a array <<< "$line"
done < raw.txt
I'm trying to replace all occurrences of [A-Z]{5} with an curl result where the match of [A-Z]{5} is fed as a variable into the curl call.
First match to be replaced would be DOTTY. The call looks similar to curl -s http://example.com/api_call/DOTTY and the result is something like -55.5833 50.6333 which should replace DOTTY in the array.
I was so far unable to correctly match the desired string and feed the match into curl.
Your help is greatly appreciated.
All the best,
Chris
EDIT:
Solution
Working solution based on #Kevin extensive answer and #Floris hint about a possible carriage return in the curl result. This was indeed the case. Thank you! Combined with some tinkering on my side I now got it to work.
#!/bin/bash
while read line; do
set $line
IFS=' ' read -a array <<< "$line"
i=0
for str in ${array[#]}; do
if [[ "$str" =~ [A-Z]{5} ]]; then
curl_tmp=$(curl -s http://example.com/api_call/$str)
# cut off line break
curl=${curl_tmp/$'\r'}
# insert at given index
declare array[$i]="$curl"
fi
let i++
done
# write to file
for index in "${array[#]}"; do
echo $index
done >> $WORK_DIR/nats.txt
done < raw.txt
I didn't change anything about your script except add the matching part, since it seems that's what you're needing help on:
#!/bin/bash
while read line; do
set $line
IFS=' ' read -a array <<< "$line"
for str in ${array[#]}; do
if [[ "$str" =~ [A-Z]{5} ]]; then
echo curl "http://example.com/api_call/$str"
fi
done
done < raw.txt
EDIT: added in the url example you provided with the variable in the URI. You can do whatever you need with the fetched output by changing it to do_something "$(curl ...)"
EDIT2: Since you're wanting to maintain the bash array you create from each line, how about this:
I'm not great at bash when it comes to arrays, so I expect someone to call me out on it, but this should work.
I've left some echos there so you can see what it's doing. The shift commands are to push the array index from the current location when the regex matches. The tmp variable to hold your curl output could probably be improved, but this should get you started, I hope.
removed temporarily to avoid confusion
EDIT3: Oops the above didn't actually work. My mistake. Let me try again here.
EDIT4:
#!/bin/bash
while read line; do
set $line
IFS=' ' read -a array <<< "$line"
i=0
# echo ${array[#]} below is just so you can see it before processing. You can remove this
echo "Array before processing: ${array[#]}"
for str in ${array[#]}; do
if [[ "$str" =~ [A-Z]{5} ]]; then
# replace the echo command below with your curl command
# ie - curl="$(curl http://example.com/api_call/$str)"
curl="$(echo 1234 -1234)"
if [[ "$flag" = "1" ]]; then
array=( ${adjustedArray[#]} )
push=$(( $push + 2 ));
let i++
else
push=1
fi
adjustedArray=( ${array[#]:0:$i} ${curl[#]} ${array[#]:$(( $i + $push)):${#array[#]}} )
#echo "DEBUG adjustedArray in loop: ${adjustedArray[#]}"
flag=1;
fi
let i++
done
unset flag
echo "final: ${adjustedArray[#]}"
# do further processing here
done < raw.txt
I know there's a smarter way to do this than the above, but we're getting into areas in bash where I'm not really suited to give advice. The above should work, but I'm hoping someone can do better.
Hope it helps, anyway
ps - You should probably not use a shell script for this unless you really need to. Perl, php, or python would make the code simple and readable
Since I misread the first time:
How about just using sed?
sed "s/\([A-Z]\{5\}\)/$(echo curl http:\\/\\/example.com\\/api_call\\/\\1)/g" /tmp/raw.txt
Try that, then try removing the echo. I'm not 100% on this since I can't run it on the real domain
EDIT: And just so I'm clear, the echo is just there so you can see what it will do with the echo removed
create a file cmatch:
#!/bin/bash
while read line
do
echo $line
a=`echo $line | egrep -o '\b[A-Z]{5}\b'`
for v in $a
do
echo "doing curl to replace $v in $line"
r=`curl -s http://example.com/api_call/$v`
r1=`echo $r | xargs echo`
line=`echo $line | sed 's/'$v'/'$r1'/'`
done
done
then call it with
chmod 755 cmatch
./cmatch < inputfile.txt > outputfile.txt
It will do what you asked
Notes:
the \b before and after the [A-Z]{5} ensures that ABCDEFG (which is not a five letter word) will not match.
using egrep -o produces an array of matches
I loop over this array to allow the replacement of multiple matches in a line
I update the line for each match found using the result of the curl call
to keep code clean, I assign the result of the curl to an intermediate variable
edit Just saw the comments about arrays. I suggest to take the output of this script and convert it to an array if you want to do further manipulation...
more edits If your curl command returns a multi-line string (which would explain the error you see), you can use the new line I introduced in the script to remove the newlines (essentially stringing all the arguments together):
echo $r | xargs echo
calls echo with one line at a time as argument, and without the carriage returns. It's a fun way of getting rid of carriage returns.
#!/bin/bash
while read line;do
set -- $line
echo "second parm is $2"
echo "do your curl here"
done < afile.txt