streaming fast changing data between servers - c++

Question

I have server 1 which is generating a large amount of data, e.g there are files that are constantly being updated, on the time scale of milliseconds.
I would like to get these files onto another server, using C++ or standard Linux methods.
Currently, I have been doing this by compressing the files every second and using scp to transfer them, and unpacking on the other server.
However, the latency of this is very high and I can't break sub 1 second times with this.
Can anybody suggest the methods I can use to move the data with lower latency?

Answer 1

Just an idea, i don't know if it'll work for your particular situation:
Write two programs. One that runs on the server on which you files are being updated, and it monitors the changes with inotify. The other program runs on the second server and maintains a tcp connection with the first one. Whenever the first program detects a change, it sends the changed part of the file to the second program, which can apply the change to its own copy of the file.
Also, if the first server is not actually generating data for those file, but is reading it from the newtwork, it would be a good idea to just multicast the stream of data to both servers.

Answer 2

On Linux you can use DRBD and a cluster file system like GFS2 to have some partition transparently replicated between the two servers.
Another option, would be to use rsync.

Answer 3

A Perl script that uses inotify to detect changes on the filesystem and rsync over SSH to resynchronize the remote copies:
#!/usr/bin/perl
use strict;
use warnings;
use Net::OpenSSH;
use Linux::Inotify2;
use Time::HiRes qw(sleep);
my $usage = "Usage:\n $0 local_dir [user\#]host remote_dir\n\n";
#ARGV == 3 or die $usage;
my ($local, $host, $remote) = #ARGV;
-d $local or die $usage;
my $ssh = Net::OpenSSH->new($host);
$ssh->error and die "unable to connect to remote host: " . $ssh->error;
my $inotify = Linux::Inotify2->new;
$inotify->watch ($local, IN_MODIFY|IN_MOVED_TO);
$ssh->rsync_put({verbose => 1, glob => 1}, "$local/*", $remote);
while (1) {
my #events = $inotify->read or die "read error: $!";
my %changed;
$changed{"$local/$_->{name}"} = 1 for #events;
$ssh->rsync_put({verbose => 1}, keys %changed, $remote);
sleep 0.1;
}
update: #user788171: in response to your question:
It may be or may be not, there are too many unknowns:
how many files do actually change
their size
how large are the changes relative to full file size
network bandwidth and latency
available CPU power in both sides
etc.
But trying it is cheap so I suggest you do it, if it is not enough, then you can try to identify bottlenecks and try to eliminate them.
For instance, rsync is a talky protocol, very sensible to network latency, so, if your files are small, scp may produce better results. Or you could keep a local copy of the last version transmitted locally for every file and send just deltas. If CPU is the bottleneck, rewrite it in C++, eliminate SSH, etc.
And if anyway, that approach turns to be a dead end, then, you can still...
Do it at the OS level, using DRDB or some other transparent replication mechanism. You can even try to implement it yourself using FUSE.
Modify your main application to write a log of the changes that can be streamed easyly to the other side.