I'm a bit new to bash scripting, I have a C++ program communicating back and forth with this bash script through some named pipes. I used inotifywait to watch a folder for new files and when a new file is added (ending in .job) sending it to through the pipe.
I'm having the C++ program pipe back the result, and if the result is 'quit', I want to bash script to quit execution.
I was trying to accomplish this with exit 1 as seen below, but that doesn't seem to exit the entire script. Instead after that exit is ran, when I drop another file in the watch folder the script ends.
I read a bit about subshells, and am wondering if this has something to do with them and any suggestions on how to exit the entire script.
DROP_FOLDER="$1"
DATA_FOLDER="$2"
OUTPUT_FOLDER="$3"
PATH_TO_EXECS="./tmp/"
PATH_TO_COMPLETED="../completed/"
# create pipes
read_pipe=/tmp/c_to_bash
write_pipe=/tmp/bash_to_c
if [[ ! -p $read_pipe ]]; then
mkfifo $read_pipe
fi
if [[ ! -p $write_pipe ]]; then
mkfifo $write_pipe
fi
# start c++ program
./tmp/v2 $DATA_FOLDER $OUTPUT_FOLDER $PATH_TO_EXECS "${write_pipe}" "${read_pipe}" &
# watch drop folder
inotifywait -m $DROP_FOLDER -e create -e moved_to |
while read path action file; do
# ends in .tga
if [[ "$file" =~ .*tga$ ]]; then
# move to image dir
mv "${DROP_FOLDER}${file}" "${DATA_FOLDER}${file}"
fi
# ends in .job
if [[ "$file" =~ .*job$ ]]; then
# pipe to dispatcher
echo "${DROP_FOLDER}${file}" > $write_pipe
# wait for result from pipe
if read line <$read_pipe; then
echo $line
# check for quit result
if [[ "$line" == 'quit' ]]; then
# move job file to completed
mv "${DROP_FOLDER}${file}" "${PATH_TO_COMPLETED}${file}"
# exit
exit 1
fi
# check for continue result
if [[ "$line" == 'continue' ]]; then
# move job file to completed
mv "${DROP_FOLDER}${file}" "${PATH_TO_COMPLETED}${file}"
fi
fi
fi
done
The problem is that exit only exits the current subshell, which in your case is your while loop due to the pipeline.
Bash still waits for inotifywait to exit, which it won't do until it tries to write another value and detects that the pipe is broken.
To work around it, you can use process substitution instead of a pipe:
while read path action file; do
...
done < <(inotifywait -m $DROP_FOLDER -e create -e moved_to)
This works because the loop is not executed in a subshell, and therefore an exit statement will exit the whole script. Additionally, bash doesn't wait for process substitutions to exit, so while it may hang around until the next time it tries to write, it won't stop the script from exiting.
In general, you can use kill "$$" from a subshell in order to terminate the main script ($$ will expand to the pid of the main shell even in subshells, and you can set a TERM trap in order to catch that signal).
But it looks that you actually want to terminate the left side of a pipeline from its right side -- i.e. cause inotifywait to terminate without waiting until it's writing something to the orphan pipe and is killed by SIGPIPE. For that you can kill just the inotifywait process explicitly with pkill:
inotifywait -m /some/dir -e create,modify |
while read path action file; do
pkill -PIPE -P "$$" -x inotifywait
done
pkill -P selects by parent; $$ should be the PID of your script. This solution is of course, not fool-proof. Also have a look at this.
Related
I need to go another server and perform a word count. Based on the count variable I will perform a if else logic.
However i am unable to do a word count and further unable to compare the variable value in if condition.
Error:
wc: cannot open the file v.txt
Script:
#!/bin/bash
ssh u1#s1 "cd ~/path1/ | fgrep-f abc.csv xyz.csv > par.csv | a=$(wc -l par.csv)| if ["$a" == "0"];
then echo "success"
fi"
First, although the wc program is named for 'word count', wc -l actually counts lines not words. I assume that is what you want even though it isn't what you said.
A shell pipline one | two | three runs things in parallel with (only) their stdout and stdin connected; thus your command runs one subshell that changes directory to ~/path1 and immediately exits with no effect on anything else, and at the same time tries to run fgrep-f (see below) in a different subshell which has not changed the directory and thus probably can't find any file, and in a third subshell does the assignment a= (see below) which also immediately exits so it cannot be used for anything.
You want to do things sequentially:
ssh u#h 'cd path1; fgrep -f abc.csv xyz.csv >par.csv; a=$(wc -l par.csv); if [ "$a" == "0" ] ...'
# you _might_ want to use && instead of ; so that if one command fails
# the subsequent ones aren't attempted (and possibly go further wrong)
Note several other important changes I made:
the command you give ssh to send the remote must be in singlequotes ' not doublequotes " if it contains any dollar as yours does (or backtick); with " the $(wc ...) is done in the local shell before sending the command to the remote
you don't need ~/ in ~/path1 because ssh (or really sshd) always starts in your home directory
there is no common command or program fgrep-f; I assume you meant the program fgrep with the flag -f, which must be separated by a space. Also fgrep although traditional is not standard (POSIX); grep -F is preferred
you must have a space after [ and before ]
However, this won't do what you probably want. The value of $a will be something like 0 par.csv or 1 par.csv or 999 par.csv; it will never equal 0 so your "success" branch will never happen. In addition there's no need to do these in separate commands: if your actual goal is to check that there are no occurrences in xyz.csv of the (exact/non-regexp) strings in abc.csv both in path1, you can just do
ssh u#h 'if ! grep -qFf path1/abc.csv path1/xyz.csv; then echo success; fi'
# _this_ case would work with " instead of ' but easier to be consistent
grep (always) sets its exit status to indicate whether it found anything or not; flag -q tells it not to output any matches. So grep -q ... just sets the status to true if it matched and false otherwise; using ! inverts this so that if grep does not match anything, the then clause is executed.
If you want the line count for something else as well, you can do it with a pipe
'a=$( fgrep -Ff path1/abc.csv path1/xyz.csv | wc -l ); if [ $a == 0 ] ...'
Not only does this avoid the temp file, when the input to wc is stdin (here the pipe) and not a named file, it outputs only the number and no filename -- 999 rather than 999 par.csv -- thus making the comparison work right.
I am trying to create a for loop with two list files that will basically echo the script to change the dbowner of multiple databases. The list files contain multiple servers and the login name list contain multiple login names. But they are line separated in order to match each database with the login name.
This is what I have so far but it is obviously taking the first server name and looping it through each login name and then moves onto the next server name.
for servername in $(cat servername.list); do
for loginname in $(cat loginname.list); do
echo "USE $servername"
echo "go"
echo "EXEC sp_changedbowner '$loginname'"
echo "go"
echo "USE master"
echo "go"
echo ""
done
done
I want the output to be this:
USE server1
go
EXEC sp_changedbowner 'login1'
go
USE master
go
USE server2
go
EXEC sp_changedbowner 'login2'
go
USE master
go
You can do it like this:
while read -r; do
server="$REPLY"
read -r <&3 || break
login="$REPLY"
echo \
"USE $server
go
EXEC sp_changedbowner '$login'
go
USE master
go
"
done <servername.list 3<loginname.list
[Live demo]
Using input redirection you can open both files at the same time for the while loop. File descriptor 0 (standard input) is available, but 1 and 2 (standard output / standard error, respectively) are taken. The next free file descriptor is 3.
The loop then first reads a line from 0 (standard input, now connected to servername.list), then a line from 3 (now connected to loginname.list). read places the input in a variable called REPLY, which we copy over to server and login, respectively. These variables are used in the echo string to produce the desired output.
This repeats until one of the files runs out of lines.
By the way, you should not attempt to read lines with for.
I would like to set a script in order to continuously parse a specific marker in a xml file.
The script contains the following while loop:
function scan_t()
{
INPUT_FILE=${1}
while : ; do
if [[ -f "$INPUT_FILE" ]]
then
ret=`cat ${INPUT_FILE} | grep "<data>" | awk -F"=|>" '{print $2}' | awk -F"=|<" '{print $1}'`
if [[ "$ret" -ne 0 ]] && [[ -n "$ret" ]]
then
...
fi
fi
done
}
scant_t "/tmp/test.xml"
The line format is :
<data>0</data> or <data>1</data> <data>2</data> ..
Even if the condition if [[ -f "$INPUT_FILE" ]] has been added to the script, sometimes I get:
cat: /tmp/test.xml: No such file or directory.
Indeed, the $INPUT_FILE is normally consumed by an other process which is charged to suppress the file after reading.
This while loop is only used for test, the cat error doesn't matter but I would like to hide this return because it pollutes the terminal a lot.
If some other process can also read and remove the file before this script sees it, you've designed your system with a race condition. (I assume that "charged to suppress" means "designed to unlink"...)
If it's optional for this script to see every input file, then just redirect stderr to /dev/null (i.e. ignore errors when the race condition bites). If it's not optional, then have this script rename the input file to something else, and have the other process watch for that. Check for that file existing before you do the rename, to make sure you don't overwrite a file the other process hasn't read yet.
Your loop has a horrible design. First, you're busy-waiting (with no sleep at all) on the file coming into existence. Second, you're running 4 programs when the input exists, instead of 1.
The busy-wait can be avoided by using inotifywait to watch the directory for changes. So the if [[ -f $INPUT_FILE ]] loop body only runs after a modification to the directory, rather than as fast as a CPU core can run it.
The second is simpler to address: never cat file | something. Either something file, or something < file if something doesn't take filenames on its command line, or behaves differently. cat is only useful if you have multiple files to concatenate. For reading a file into a shell variable, use foo=$(<file).
I see from comments you've already managed to turn your whole pipeline into a single command. So write
INPUT_FILE=foo;
inotifywait -m -e close_write -e moved_to --format %f . |
while IFS= read -r event_file;do
[[ $event_file == $INPUT_FILE ]] &&
awk -F '[<,>]' '/data/ {printf "%s ",$3} END {print ""}' "$INPUT_FILE" 2>/dev/null
# echo "$event_file" &&
# date;
done
# tested and working with the commented-out echo/date commands
Note that I'm waiting for close_write and moved_to, rather than other events, to avoid jumping the gun and reading a file that's not finished being written. Put $INPUT_FILE in its own directory, so you don't get false-positive events waking up your loop for other filenames.
To also implement the rename-to-input-for-next-stage suggestion, you'd put a while [[ -e $INPUT2 ]]; do sleep 0.2; done; mv -n "$INPUT_FILE" "$INPUT2" busy-wait loop after the awk.
An alternative would be to run inotifywait once per loop iteration, but that has the potential for you to get stuck with $INPUT_FILE created before inotifywait started watching. So the producer would be waiting for the consumer to consume, and the consumer wouldn't see the event.
# Race condition with an asynchronous producer, DON'T USE
while inotifywait -qq -e close_write -e moved_to; do
[[ $event_file == $INPUT_FILE ]] &&
awk -F '[<,>]' '/data/ {printf "%s ",$3} END {print ""}' "$INPUT_FILE" 2>/dev/null
done
There doesn't seem to be a way to specify the name of a file that doesn't exist yet, even as a filter, so the loop body needs to test for the specific file existing in the dir before using.
If you don't have inotifywait available, you could just put a sleep into the loop. GNU sleep supports fractional seconds, like sleep 0.5. Busybox probably doesn't. You might want to write a tiny trivial C program anyway, which keeps trying to open(2) the file in a loop that includes a usleep or nanosleep. When open succeeds, redirect stdin from that, and exec your awk program. That way, there's no race possible between a stat and an open.
#include <unistd.h> // for usleep/dup2
#include <sys/types.h> // for open
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h> // for perror
void waitloop(const char *path)
{
const char *const awk_args[] = { "-F", "[<,>]",
"/data/ {printf \"%s \",$3} END {print \"\"}",
path
};
while(42) {
int fd = open(path, O_RDONLY);
if (-1 != fd) {
// if you fork() here, you can avoid the shell loop too.
dup2(fd, 0); // redirect stdin from fd. In theory should check for error here, too.
close(fd); // and do this in the parent after fork
execv("/usr/bin/awk", (char * const*)awk_args); // execv's prototype doesn't prevent it from modifying the strings?
} else if(errno != ENOENT) {
perror("opening the file");
} // else ignore ENOENT
usleep(10000); // 10 milliseconds.
}
}
// optional TODO: error-check *all* the system calls.
This compiles, but I haven't tested it. Looping inside a single process doing open / usleep is much lighter weight than running a whole process to do sleep 0.01 from a shell.
Even better would be to use inotify to watch for directory events to detect the file appearing, instead of usleep. To avoid a race, after setting up the inotify watch, do another check for the file existing, in case it got created after your last check, but before the inotify watch became active.
I don't do much shell scripting but I want to essentially do this:
run the command "grunt check" about 30 times (the process takes 60 seconds).
Do a regex on the output of that command for "Some random error message. Failed." Where "Failed" is the thing I'm searching for but I want to capture the whole sentence.
Write the associated line to a file.
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 30 ]; do
command grunt check
// ERROR = regex(/\/Failed./)
// WRITE ERROR TO FILE
let COUNTER=COUNTER+1
done
for ((cr=0; cr<30; cr++))
do
grunt check | grep Failed
done > outfile.txt
counter=0
while [ $counter -lt 30 ]; do
grunt check | grep Failed
let counter=counter+1
done > some file
The above uses a pipeline to capture the output of the grunt command and sent it to grep. grep searches through the output and prints any lines that contain the word Failed. Any such lines are then sent to a file named somefile.
As a minor point, I have converted COUNTER to lower case. This is because the system uses upper case environment variables. If you make a practice of using lower case ones then you won't accidentally overwrite one. (In this particular case, there is no system variable named COUNTER, so you are safe.)
Another method for counting to 30:
You might find this simpler:
for counter in {1..30}; do
grunt check | grep Failed
done > somefile
The {1..30} notation provides the numbers from one to thirty. It is a bash feature so don't try to use it on a bare-bones POSIX shell.
To get more context
If you would like to see more context around the error message, grep offers several options to help. To see both the line matching "Failed" and the line before, use -B:
for counter in {1..30}; do
grunt check | grep -B 1 Failed
done >somefile
Similarly, -A can be used to display lines after the match. -C will display lines both before and after the match.
I am running a shell script on windows with cygwin in which I execute a program multiple times with different arguments each time. Sometimes, the program generates segmentation fault for some input arguments. I want to generate a text file in which the shell script can write for which of the inputs, the program failed. Basically I want to check return value of the program each time it runs. Here I am assuming that when program fails, it returns a different value from that when it succeeds. I am not sure about this. The executable is a C++ program.
Is it possible to do this? Please guide. If possible, please provide a code snippet for shell script.
Also, please tell what all values are returned.
My script is .sh file.
The return value of the last program that finished is available in the environment variable $?.
You can test the return value using shell's if command:
if program; then
echo Success
else
echo Fail
fi
or by using "and" or "or" lists to do extra commands only if yours succeeds or failed:
program && echo Success
program || echo Fail
Note that the test succeeds if the program returns 0 for success, which is slightly counterintuitive if you're used to C/C++ conditions succeeding for non-zero values.
if it is bat file you can use %ERRORLEVEL%
Assuming no significant spaces in your command line arguments:
cat <<'EOF' |
-V
-h
-:
-a whatnot peezat
!
while read args
do
if program $args
then : OK
else echo "!! FAIL !! ($?) $args" >> logfile
fi
done
This takes a but more effort (to be polite about it) if you must retain spaces. Well, a bit more effort; you probably use an eval in front of the 'program'.