I am running C++ program on Linux [Red Hat], This program creates temporary files on the Hard Disk to compute its result. I need to know how much space does this program use from the Disk while it is running. I could not able to change the source code to keep the files so I can subtract the size of the Program folder before and after producing the results. is there any profiling tool or command line that I can use to help me in this situation.
You can use du -h /path/todir before and after the program run from the terminal, I suppose.
du - estimate file space usage
-h, --human-readable
print sizes in human readable format (e.g., 1K 234M 2G)
If you need further options just take a look at man du
You may make use of stats maintained in /proc/[pid]/io .
From /proc man page :
This file contains I/O statistics for the process, for
example:
# cat /proc/3828/io
rchar: 323934931
wchar: 323929600
syscr: 632687
syscw: 632675
read_bytes: 0
write_bytes: 323932160
cancelled_write_bytes: 0
write_bytes: bytes written
Attempt to count the number of bytes which this process
caused to be sent to the storage layer.
You may write script something like ,
#!/bin/bash
c=0
echo $1
rm /tmp/writtenBytes.txt
psout=`ps aux | grep "$1" | grep -v $0 | grep -v grep `
while [[ "$psout" != "" ]]
do
pid=`echo $psout | awk '{print $2}'`
cat /proc/$pid/io | grep write | grep -v cancelled >> /tmp/writtenBytes.txt
psout=`ps aux | grep "$1" | grep -v $0 | grep -v grep`
echo $psout
sleep 1
done
Run this script as
bash -x getIO.sh "postgres: stats collector"
This script will create file /tmp/writtenBytes.txt, containing number of bytes written by processes named "postgres: stats collector"
Related
I need to go another server and perform a word count. Based on the count variable I will perform a if else logic.
However i am unable to do a word count and further unable to compare the variable value in if condition.
Error:
wc: cannot open the file v.txt
Script:
#!/bin/bash
ssh u1#s1 "cd ~/path1/ | fgrep-f abc.csv xyz.csv > par.csv | a=$(wc -l par.csv)| if ["$a" == "0"];
then echo "success"
fi"
First, although the wc program is named for 'word count', wc -l actually counts lines not words. I assume that is what you want even though it isn't what you said.
A shell pipline one | two | three runs things in parallel with (only) their stdout and stdin connected; thus your command runs one subshell that changes directory to ~/path1 and immediately exits with no effect on anything else, and at the same time tries to run fgrep-f (see below) in a different subshell which has not changed the directory and thus probably can't find any file, and in a third subshell does the assignment a= (see below) which also immediately exits so it cannot be used for anything.
You want to do things sequentially:
ssh u#h 'cd path1; fgrep -f abc.csv xyz.csv >par.csv; a=$(wc -l par.csv); if [ "$a" == "0" ] ...'
# you _might_ want to use && instead of ; so that if one command fails
# the subsequent ones aren't attempted (and possibly go further wrong)
Note several other important changes I made:
the command you give ssh to send the remote must be in singlequotes ' not doublequotes " if it contains any dollar as yours does (or backtick); with " the $(wc ...) is done in the local shell before sending the command to the remote
you don't need ~/ in ~/path1 because ssh (or really sshd) always starts in your home directory
there is no common command or program fgrep-f; I assume you meant the program fgrep with the flag -f, which must be separated by a space. Also fgrep although traditional is not standard (POSIX); grep -F is preferred
you must have a space after [ and before ]
However, this won't do what you probably want. The value of $a will be something like 0 par.csv or 1 par.csv or 999 par.csv; it will never equal 0 so your "success" branch will never happen. In addition there's no need to do these in separate commands: if your actual goal is to check that there are no occurrences in xyz.csv of the (exact/non-regexp) strings in abc.csv both in path1, you can just do
ssh u#h 'if ! grep -qFf path1/abc.csv path1/xyz.csv; then echo success; fi'
# _this_ case would work with " instead of ' but easier to be consistent
grep (always) sets its exit status to indicate whether it found anything or not; flag -q tells it not to output any matches. So grep -q ... just sets the status to true if it matched and false otherwise; using ! inverts this so that if grep does not match anything, the then clause is executed.
If you want the line count for something else as well, you can do it with a pipe
'a=$( fgrep -Ff path1/abc.csv path1/xyz.csv | wc -l ); if [ $a == 0 ] ...'
Not only does this avoid the temp file, when the input to wc is stdin (here the pipe) and not a named file, it outputs only the number and no filename -- 999 rather than 999 par.csv -- thus making the comparison work right.
In Windows, I would have done a search for finding a folders name using findsr Similarly, I want to get a specific folder using grep
In windows, I'm using svnlook tree -t [repos_path] | findstr (13\.9\.[0-9]+\/)
In Ec2 Maiche (Linux) svnlook tree /var/www/svn/ILS | grep -Eo '(13\.9\.[0-9]+\/)'
and I got the repos that I need
13.9.4/
13.9.5/
13.9.6/
13.9.7/
my problem is the grep line in Linux doesn't want to stop (exit) it's still running.
how could I stop it after matching?
You can specify the -m: maximal number of counts. After the specified number of matching lines, grep will stop.
After ^Z the svnlook is paused. You kan kill (^C) the program, send it to the background (bg) or continue (fg).
When you want to interrupt you can use ^C or start the grep with a -m option.
I am working on a C++ Project. To fulfill one of the requirement, I need to check if a port is available for using in my application anytime. To fulfill this , I have come to this following solution.
#include <iostream>
#include <cstdlib>
#include <stdexcept>
#include <string>
#include <stdio.h>
std::string _executeShellCommand(std::string command) {
char buffer[256];
std::string result = "";
const char * cmd = command.c_str();
FILE* pipe = popen(cmd, "r");
if (!pipe) throw std::runtime_error("popen() failed!");
try {
while (!feof(pipe))
if (fgets(buffer, 128, pipe) != NULL)
result += buffer;
} catch (...) {
pclose(pipe);
throw;
}
pclose(pipe);
return result;
}
bool _isAvailablePort(unsigned short usPort){
char shellCommand[256], pcPort[6];
sprintf(shellCommand, "netstat -lntu | awk '{print $4}' | grep ':' | cut -d \":\" -f 2 | sort | uniq | grep %hu", usPort);
sprintf(pcPort, "%hu", usPort);
std::string output = _executeShellCommand(std::string(shellCommand));
if(output.find(std::string(pcPort)) != std::string::npos)
return false;
else
return true;
}
int main () {
bool res = _isAvailablePort(5678);
return 0;
}
Here Basically the _executeShellCommand function can excute any shell command anytime and can return the stdout output as return string.
And I am executing the following shell command in that function.
netstat -lntu | awk '{print $4}' | grep ':' | cut -d \":\" -f 2 | sort | uniq | grep portToCheck
So, if the port is already in use, the _executeShellCommand will return the PortValue itself, else it will return Blank. So, checking the returned string, I can decide.
So far so good.
Now, I want to be make my Project completely Crash-proof. So, before firing the netstat command, I want to make sure if it really exists or not. I want help in this case. I know, It's kind of stupid to doubt the availability of netstat command in a linux machine. I am just thinking of some user who removed netstat binary from his machine for some reason.
N.B. : I don't want make a bind() call to chack if the port is available or not. Also, it will be best if I can check if netstat command is available without calling _executeShellCommand for another time (i.e. without executing another Shell Command).
An even better idea is to make your code work completely without netstat altogether.
On Linux, all that netstat does (for your use case) is read the contents of /proc/net/tcp, which enumerates all ports in use.
All you have to do is open /proc/net/tcp yourself, and parse it. This becomes just an ordinary, boring, file parsing code. Can't get much more "crash-proof" than that.
You will find the documentation of the format of /proc/net/tcp in Linux manual pages.
In the unlikely event that you need to check UDP ports, this would be /proc/net/udp.
Of course, there is a race window between the time you check /proc/net/tcp, where someone can grab the port. But that's also true with netstat as well, and since that's going to be a much slower process, this will actually be an improvement, and reduce the race window significantly.
Since you're asking for a way to check if netstat command is available, I won't try to suggest the other ways in C++. The shell way is checking return code of the following command:
command -v netstat
If netstat binary is available in $PATH, then the command returns 0. In Bash it usually looks like this:
command -v netstat
if [ $? -eq 0 ]; then
netstat # ...
else
echo >&2 "Error: netstat is not available"
fi
Or simply
command -v netstat >/dev/null && netstat # ...
I would like to set a script in order to continuously parse a specific marker in a xml file.
The script contains the following while loop:
function scan_t()
{
INPUT_FILE=${1}
while : ; do
if [[ -f "$INPUT_FILE" ]]
then
ret=`cat ${INPUT_FILE} | grep "<data>" | awk -F"=|>" '{print $2}' | awk -F"=|<" '{print $1}'`
if [[ "$ret" -ne 0 ]] && [[ -n "$ret" ]]
then
...
fi
fi
done
}
scant_t "/tmp/test.xml"
The line format is :
<data>0</data> or <data>1</data> <data>2</data> ..
Even if the condition if [[ -f "$INPUT_FILE" ]] has been added to the script, sometimes I get:
cat: /tmp/test.xml: No such file or directory.
Indeed, the $INPUT_FILE is normally consumed by an other process which is charged to suppress the file after reading.
This while loop is only used for test, the cat error doesn't matter but I would like to hide this return because it pollutes the terminal a lot.
If some other process can also read and remove the file before this script sees it, you've designed your system with a race condition. (I assume that "charged to suppress" means "designed to unlink"...)
If it's optional for this script to see every input file, then just redirect stderr to /dev/null (i.e. ignore errors when the race condition bites). If it's not optional, then have this script rename the input file to something else, and have the other process watch for that. Check for that file existing before you do the rename, to make sure you don't overwrite a file the other process hasn't read yet.
Your loop has a horrible design. First, you're busy-waiting (with no sleep at all) on the file coming into existence. Second, you're running 4 programs when the input exists, instead of 1.
The busy-wait can be avoided by using inotifywait to watch the directory for changes. So the if [[ -f $INPUT_FILE ]] loop body only runs after a modification to the directory, rather than as fast as a CPU core can run it.
The second is simpler to address: never cat file | something. Either something file, or something < file if something doesn't take filenames on its command line, or behaves differently. cat is only useful if you have multiple files to concatenate. For reading a file into a shell variable, use foo=$(<file).
I see from comments you've already managed to turn your whole pipeline into a single command. So write
INPUT_FILE=foo;
inotifywait -m -e close_write -e moved_to --format %f . |
while IFS= read -r event_file;do
[[ $event_file == $INPUT_FILE ]] &&
awk -F '[<,>]' '/data/ {printf "%s ",$3} END {print ""}' "$INPUT_FILE" 2>/dev/null
# echo "$event_file" &&
# date;
done
# tested and working with the commented-out echo/date commands
Note that I'm waiting for close_write and moved_to, rather than other events, to avoid jumping the gun and reading a file that's not finished being written. Put $INPUT_FILE in its own directory, so you don't get false-positive events waking up your loop for other filenames.
To also implement the rename-to-input-for-next-stage suggestion, you'd put a while [[ -e $INPUT2 ]]; do sleep 0.2; done; mv -n "$INPUT_FILE" "$INPUT2" busy-wait loop after the awk.
An alternative would be to run inotifywait once per loop iteration, but that has the potential for you to get stuck with $INPUT_FILE created before inotifywait started watching. So the producer would be waiting for the consumer to consume, and the consumer wouldn't see the event.
# Race condition with an asynchronous producer, DON'T USE
while inotifywait -qq -e close_write -e moved_to; do
[[ $event_file == $INPUT_FILE ]] &&
awk -F '[<,>]' '/data/ {printf "%s ",$3} END {print ""}' "$INPUT_FILE" 2>/dev/null
done
There doesn't seem to be a way to specify the name of a file that doesn't exist yet, even as a filter, so the loop body needs to test for the specific file existing in the dir before using.
If you don't have inotifywait available, you could just put a sleep into the loop. GNU sleep supports fractional seconds, like sleep 0.5. Busybox probably doesn't. You might want to write a tiny trivial C program anyway, which keeps trying to open(2) the file in a loop that includes a usleep or nanosleep. When open succeeds, redirect stdin from that, and exec your awk program. That way, there's no race possible between a stat and an open.
#include <unistd.h> // for usleep/dup2
#include <sys/types.h> // for open
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h> // for perror
void waitloop(const char *path)
{
const char *const awk_args[] = { "-F", "[<,>]",
"/data/ {printf \"%s \",$3} END {print \"\"}",
path
};
while(42) {
int fd = open(path, O_RDONLY);
if (-1 != fd) {
// if you fork() here, you can avoid the shell loop too.
dup2(fd, 0); // redirect stdin from fd. In theory should check for error here, too.
close(fd); // and do this in the parent after fork
execv("/usr/bin/awk", (char * const*)awk_args); // execv's prototype doesn't prevent it from modifying the strings?
} else if(errno != ENOENT) {
perror("opening the file");
} // else ignore ENOENT
usleep(10000); // 10 milliseconds.
}
}
// optional TODO: error-check *all* the system calls.
This compiles, but I haven't tested it. Looping inside a single process doing open / usleep is much lighter weight than running a whole process to do sleep 0.01 from a shell.
Even better would be to use inotify to watch for directory events to detect the file appearing, instead of usleep. To avoid a race, after setting up the inotify watch, do another check for the file existing, in case it got created after your last check, but before the inotify watch became active.
I don't do much shell scripting but I want to essentially do this:
run the command "grunt check" about 30 times (the process takes 60 seconds).
Do a regex on the output of that command for "Some random error message. Failed." Where "Failed" is the thing I'm searching for but I want to capture the whole sentence.
Write the associated line to a file.
#!/bin/bash
COUNTER=0
while [ $COUNTER -lt 30 ]; do
command grunt check
// ERROR = regex(/\/Failed./)
// WRITE ERROR TO FILE
let COUNTER=COUNTER+1
done
for ((cr=0; cr<30; cr++))
do
grunt check | grep Failed
done > outfile.txt
counter=0
while [ $counter -lt 30 ]; do
grunt check | grep Failed
let counter=counter+1
done > some file
The above uses a pipeline to capture the output of the grunt command and sent it to grep. grep searches through the output and prints any lines that contain the word Failed. Any such lines are then sent to a file named somefile.
As a minor point, I have converted COUNTER to lower case. This is because the system uses upper case environment variables. If you make a practice of using lower case ones then you won't accidentally overwrite one. (In this particular case, there is no system variable named COUNTER, so you are safe.)
Another method for counting to 30:
You might find this simpler:
for counter in {1..30}; do
grunt check | grep Failed
done > somefile
The {1..30} notation provides the numbers from one to thirty. It is a bash feature so don't try to use it on a bare-bones POSIX shell.
To get more context
If you would like to see more context around the error message, grep offers several options to help. To see both the line matching "Failed" and the line before, use -B:
for counter in {1..30}; do
grunt check | grep -B 1 Failed
done >somefile
Similarly, -A can be used to display lines after the match. -C will display lines both before and after the match.