A program is massively reading from the disk but I don't know which file it is reading nor where in the code it is reading.
Is there any kind of tools on linux to monitor this ?
Related question (windows) : Disk IO profiler for existing applications
So, you can use:
/proc/PID/fd
or
lsof -p PID
to know which file your process use.
for example, with lsof -p 27666 (assume 27666 is the PID of a.out program) you can see this:
./a.out 22531 me 9w REG 8,5 131072 528280 /home/me/tmp/test.db
./a.out 22531 me 9r REG 8,5 131072 528280 /home/me/tmp/test2.db
If the system is really busy with IO, just look at top and you'll see the IO-bound process usually stuck in a D-state.
strace -c myprog is my best friend for a first attempt at all generic 'what is my application doing/where is it spending most time' questions. Strace can also attach to running processes, so you can observe the program as it's running.
Another good strace trick is to output it (with strace -o myprogrun.log) to a log file , then view it with a modern vim as it does a very nice job syntax highlighting the log. It's a lot easier to find things this way, as the default strace output is not very human-readable.
Important thing to remember is to log to another partition/set of disks than where the IO problem is! Do not induce extra IO problems as strace can generate a lot of output. I like to use a TmpFS or ZRAM RAM-disks for such occasions.
Related
I've been searching for a solution on Linux to find out what thread first aquired a fd but no luck for now.
/proc/pid/task/
shows the fd to be available to each thread, which makes sense since descriptors are available troughout the whole process space.
lsof
is of course not of much help either for this usecase.
The program is very complex, and strace or gdb won't help either, there are tons of closed source libraries used. File path is known but does not help since I don't have access to the code in the libaries. I suspect the fd leak is due to some race condition that occurrs very very rarely and I need to trace the thread that did opened the file.
One solution that would be easy to implement would be for me to add a log in the kernel file open handler or in the c library, but for good reasons I'm not able to alter nor the kernel neither the standard library.
Some suggestions?
If you have kernel symbols available use SystemTap:
sudo stap -e 'probe syscall.open.return { \
printf("tid=%d, fd=%d\n", tid(), $return) }'
I've been trying to profile my C++ application in Linux by following this article on perf record. My understanding is all I need to do is run perf record program [program_options], where program is the program executable and [program options] are the arguments I want to pass to the program. However, when I try to profile my application like this:
perf record ./csvJsonTransducer -enable-AVX-deletion test.csv testout.json
perf returns almost immediately with a report. It takes nearly 30 seconds to run./csvJsonTransducer -enable-AVX-deletion test.csv testout.json without perf, though, and I want perf to monitor my program for the entirety of its execution, not return immediately. Why is perf returning so quickly? How can I make it take the entire run of my program into account?
Your commands seems ok. Try change the paranoid level at /proc/sys/kernel/perf_event_paranoid. Setting this parameter to -1 (as root) should solve permission issues:
echo "-1" > /proc/sys/kernel/perf_event_paranoid
You can also try to set the event that you want to monitor with perf record. The default event is cycles (if supported). Check man perf-list.
Try the command:
perf record -e cycles ./csvJsonTransducer -enable-AVX-deletion test.csv testout.json
to force the monitoring of cycles.
how can I measure memory consumed by process? process quits really quickly so utilities like top are useless. I tried using massif by valgrind, but it measures only memory allocated via malloc/new + stack, and not static variables for example. --pages-as-heap doesn't help as well because it shows mapped memory as well.
Something that might work for you is using a script that will repeatedly run 'ps' immediately after your program starts. I've written up the following script that should work for your purposes, just replace the variables at the top with your specific details. It currently runs 'netstat' in the background (notice the & symbol) and samples the memory 10 times with 0.1 second intervals between the samples, writing the results of the memory checking to a file as it goes. I've run this on cygwin and it works (minus the -o rss,vsz parameters), I don't have access to a linux machine at the moment but it should be simple to adapt if for some reason it doesn't immediately work.
#! /bin/bash
saveFileName=saveFile.txt
userName=jacob
programName=netstat
numberOfSamples="10"
delayBetweenSamples="0.1"
saveFileName=saveFile
i="0"
$programName &
while [ $i -lt $numberOfSamples ]
do
ps -u $userName -o rss,vsz | grep $programName >> $saveFileName
i=$[$i+1]
sleep $delayBetweenSamples
done
If your program completes so fast that the delay between executing it and running ps in the script is too long you might consider running your program with a delay and using a very high sample frequency to try and catch it. You can do that by using 'sleep' and two ampersands like sleep 2 && netstat . That will wait 2 seconds and then run netstat.
If none of this sounds good to you, perhaps try running your program within a debugger. I believe gdb has some memory tracking options you could look into.
I want to write a program like system monitor.
I want to have a list of programs with their process ID and usage of CPU and RAM.
I know Linux writes this information in the /proc folder but somebody told me that I can use some functions to get it too. For example a program that will return a list like:
name PID RAM
sh 3904 72KIB
And I want to code in C++.
Why don't you look at the source code for top, which displays these and many more process statistics?
Here is the busybox version, which is comparatively short and simple. It gets the information by reading the proc filesystem, that logic is here.
I have an application (the source for which I don't have), which can be invoked from command line like this
$ ./notmyapp
I want to know all the locations where the application is writing to. It outputs some files in the directory it is being called from, but I need to make sure that those are the only files that are created.
So, I need to isolate the application to find out which all files it created/edited while it was running.
How can I do this?
Some way using Perl or C or C++? Do any of the standard libraries in these languages have ways to do this?
strace, ktrace/kdump, truss, dtruss, or whatever other program your platform provides for tracing system calls is probably what you're looking for.
Expect lots of output from any of those. To figure out what files the application is reading and writing to, you might want to limit the output to just a few syscalls. strace -eopen ./notmyapp, for example.
The application might also fork off child processes to do some of its work. With most system call tracers, you'll have to be specific about tracing those child processes as well. With strace, that'd be strace -f ./notmyapp.
In unix systems, you can use strace to print out a trace of all the system calls made and signals received by a process:
$ strace ./notmyapp
grep can be used to limit the output to subset of system calls:
$ strace ./notmyapp 2>&1 | egrep '(open|write)'
you could use strace.
You say in response to rafl's answer that notmyapp is supposed to produce a prompt and wait for [...] inputs before doing something.
Put your inputs in advance into a plain text file (say, responses.txt), one input per line. Then use strace, as suggested to track calls to open() or write() piping in the contents of responses.txt:
$ strace -eopen -ewrite ./notmyapp < responses.txt
If you're expecting a lot of file access, then you may want to pipe the output to your favourite pager or editor:
$ strace -eopen -ewrite ./notmyapp < responses.txt | vim -R -
strace is a powerful tool. For more information, consult man strace.
You could try running it as a user which has no rights to write anywhere on any drive. Then you get an error message when it tries to create/write the first file. Log that directory/file and give write rights to it, then repeat until there are no more error messages.