I'm wondering how you can implement a program similar to tail -f in C/C++, a program that watches for and processes new lines added to a log file?
You can use fseek() to clear the eof condition on the stream. Essentially, read to the end of the file, sleep for a while, fseek() (without changing your position) to clear eof, the read to end of file again. wash, rinse, repeat. man fseek(3) for details.
Here's what it looks like in perl. perl's seek() is essentially a wrapper for fseek(3), so the logic is the same:
wembley 0 /home/jj33/swap >#> cat p
my $f = shift;
open(I, "<$f") || die "Couldn't open $f: $!\n";
while (1) {
seek(I, 0, 1);
while (defined(my $l = <I>)) {
print "Got: $l";
}
print "Hit EOF, sleeping\n";
sleep(10);
}
wembley 0 /home/jj33/swap >#> cat tfile
This is
some
text
in
a file
wembley 0 /home/jj33/swap >#> perl p tfile
Got: This is
Got: some
Got: text
Got: in
Got: a file
Hit EOF, sleeping
Then, in another session:
wembley 0 /home/jj33/swap > echo "another line of text" >> tfile
And back to the original program output:
Hit EOF, sleeping
Got: another line of text
Hit EOF, sleeping
I think what you're looking for is the select() call in c/c++. I found a copy of the man page here: http://www.opengroup.org/onlinepubs/007908775/xsh/select.html. Select takes file descriptors as arguments and tells you when one of them has changed and is ready for reading.
The tail program is open source, so you could reference that. I wondered the same thing and looked at the code a while back, thinking it would be pretty simple, but I was surprised at how complex it was. There are lots of gotchas that have to be taken into account.
See here
You could either call out to tail and retrieve the stream back into your app, or as it's open source, maybe try to pull it into your own code.
Also, it is possible in C++ iostream to open a file for viewing only and just read to the end, while buffering the last 10-20 lines, then output that.
Related
I have a program that writes data to a file. Normally, the file is on disk, but I am experimenting with writing to /dev/stdout. Currently, when I do this, the program will not exit until I press Ctrl-C . Is there a way for the program to signal that the output is done ?
Edit:
Currently, a disk file is opened via fopen(FILE_NAME), so I am now trying to pass in /dev/stdout as FILE_NAME.
Edit 2:
command line is
MY_PROGRAM -i foo -o /dev/stdout > dump.png
Edit 3:
It looks like the problem here is that stdout is already open, and I am opening it a second time.
The EOF condition on a FIFO (which, if you're piping from your program into something else, is what your stdout is) is set when no file handles are still open for write.
In C, the standard-library is fclose(stdout), whereas the syscall interface is close(1) -- if you're using fopen(), you'll want to pair it with fclose().
If you're also doing a separate outFile = fopen("/dev/stdout", "w") or similar, then you'll need to close that copy as well: fclose(outFile) as well as fclose(stdout).
I'm writing a code in which I check a given directory for new files or directories. New in my situation is regarded to last time the code has been run in that directory. So I create a log file and then I acquire the log in a string vector. The code is the following:
ifstream Finp;
string directory;
vector <string> newfilelist;
system( ("ls -B "+directory+" > "+directory+"filelist.log").c_str() );
Finp.open( (directory+"filelist.log").c_str() );
while ( true ) {
string stmp;
Finp >> stmp;
if( Finp.eof() ) break;
newfilelist.push_back( stmp );
}
Now what's happening is the following:
1) if the log "filelist.log" already exists, everything runs smoothly
2) if the log "filelist.log" does not exist, it is correctly created but when the code opens the file and starts acquiring it, it gets stuck in a loop and the stmp string is endlessly empty (as if the file has no eof() and yet is empty!). what is intresting is the fact that if I place a random system command before opening the log everything runs smoothly!
What am I missing?
I think that, the only thing you would need to change is that nasty while(true) loop to while(file<< to_variable). This would only read data IF there is some.
The eof() is actually quite deceiving. You would guess that it is called right at the end of the file. Though true, when you try to read from the file at the >> operation the pointer in the stream will jump back before the EOF and try to read what is there.
There is quite the few threads here discussing just EOF and using that as a condition for a loop.
If the file does not exist, Finp.open() will basically fail, and checking Finp.eof() is meaningless. Before even attempt to enter a loop to read or check eof, you need to check the Finp status with Finp.good() and then proceed with reading only if this method returns true.
I am in a bind right now and the most frustrating thing about this is that I know what the problem is but, I cannot fix it :(...
My goal is to ultimately use getline to read lines of strings from redirected input (from a text file) and keep going until EOF is reached.
Example text file (contents):
Hello World!
Good Bye.
My source code(only includes the section where it will not work):
while (!(getline(std::cin, s_array)).eof()){ // it won't read second line
//do some awesome stuff to the first line read!
}
As far as I know, getline reads everything upto the newline and stops so how do we get it to keep reading because it always stops at Hello World!.
Use while (getline(std::cin, s_array)) { } instead.
std::getline() returns istream&, and istream::operator void*() makes it evaluated as false whenever any error flag is set.
You should definitely read Joseph Mansfield's blog post titled "Don't condition input on eof()" which describes this pitfall in details and provides a well justified guideline.
I have captured some application data in wireshark, (FIX messages) , and I am wondering if there is a way to export just the tcp data layer of each packet into a separate file, one row/line per packet.
I thought there used to be a function called tcp.data but this doesn't seem to exist in the version of wireshark I'm using (v1.10.1).
I had been planning on adding an extra column in Wireshark preferences, and setting it to type "custom" then putting tcp.data into the field. Then exporting this column to a text file.
Thanks in advance for any ideas or suggestions.
PS. the other idea i had was to write a script to parse the capture file and effectively chop off the leading headers in front of the tcp layer data, but this will take some time - hopefully a way exists already to do this within wireshark itself.
UPDATE 1:
Extending Isaac's solution, I have come up with the following, however this is actually printing the entire tcp segment, not just the data from within the segment. I've also tried tcp.segment_data but this also results in the same issue where more than the tcp data payload is getting outputted. Unfortunately, at the moment the best option looks like manually parsing the pcap file. Does anyone else have any suggestions, or perhaps spot what I've got wrong in the tshark command syntax?
tshark -r capture_chopped.pcap -c4 -2 -R "(tcp.len > 0)" -T fields -d tcp.port==2634,echo -e tcp.segment -w - > output.2
UPDATE 2 - ISSUE RESOLVED:
I found that every option with tshark didn't provide the entire info I needed, so I went with creating my own Perl script to fish out the FIX message strings from the pcap file directly. The Perl script is included below in case it is helpful to anyone else in a similar situation with a PCAP file.
You don't need a script, you can use the built-in wireshark tool called tshark. It is usually located at: c:\Program Files\Wireshark if you installed wireshark in the default folder.
Then you use the following command line and it will do the trick:
tshark -r c:\captures\your_file.cap -R "(tcp.len > 0)" -T fields -d tcp.port=3868,echo -e echo.data
Few things to note about the above:
It filters tcp packets that have no payload, remove it if you want to identify the empty ones
It assumes you know the protocol port that your file contain which is usually a reasonable assumption. In the above 3868, replace it with the protocol you are using.
Then redirect the output to a file and you are done.
In the end I found that creating a perl script to parse the PCAP file was sufficient for my needs - mainly due the other options of using tshark would have still needed some extra cleanup / manipulation (eg. converting the HEX into binary) so the script seemed like the best approach to me.
The perl script is included here in case it is useful for anyone else.
use strict;
use warnings;
my $in_file = shift || die "must give input pcap file as argument here. \n";
my $out_file = 'fix_output.txt';
open (my $out_fh, ">", $out_file) || die "error opening output file: [$out_file]. $! \n";
open (my $in_fh , "<", $in_file) || die "error opening input file: [$in_file]. $! \n";
my $pcap_blob = do {
local $/ = undef;
<$in_fh>;
};
my #FIX_strings = $pcap_blob =~ m/(8=FIX\.4\.0.*10=\d\d\d)/g;
print "found " . scalar #FIX_strings . " FIX messages\n";
for (#FIX_strings){
print {$out_fh} $_ , "\n";
}
print "finished writing FIX messages to $out_file\n";
close $out_fh;
close $in_fh;
print "all done\n";
exit;
I'm writing a linux shell for a custom scripting language, and I want to print a "... " before each follow-up line a user enters for a single statement, with the first line having a ">>> " printed, then waiting for input. Following is an example:
>>> void f() {
... "this is a test"
... }
>>>
I'm reading the line with fgets, and after I read it completely, I print the "... ", and repeat using another call to fgets. This works fine for moderately fast interactive input. But if I paste code containing newlines into the terminal, what I get is the following
>>> void f() {
"this is a test"
}
... ... >>>
The "... "'es are printed too late, even though I emit a fflush call after I print them to stdout. Does anyone know whether there is anything special to do to make this work?
If you turn off echo (see stty(1) -echo) for the terminal, then you are in complete control over when the input is printed to the screen.
My assumption is the paste is causing all lines to be written to the terminal at once, and your program will never get an opportunity to send output to the terminal when necessary. So, if you turn off echo and print the user's input as they enter it, you can perform your special handling when you see the newline chars.
You can see that the irb program does something very similar by running strace on it:
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
...
ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig -icanon -echo ...}) = 0
There's not really a simple way to do it using stdio -- you'll need to use something like ncurses to control the terminal. The issue is that when you copy and paste multiple lines like that, they all get pulled into stdin's read buffer in a single call to read(2), so stdout has no chance to intervene and print the intervening prompts.