I have two programs that transmits UDP messages between them. The first program is a simulator and the other program is a controller.
Both simulator and controller is written i C++. I want the simulator to transmit the simulation state to the controller that in turn sends a control signal to the simulator. All messages will be under 1kb.
I have now used UDP for the ethernet connection. The speed is good for the first 2-3 seconds, and then it drops significantly to 1/10th of the original speed. I suspect this is due to network congestion.
The problem:
I though that UDP was faster than TCP, but do you think that TCP will be faster due to the congestion? Is there anything I can do to increase the speed?
EDIT: How I measure the speed
I have just done some simulation with varying length
Simulating 5 seconds: 1.10s
Simulating 50 seconds: 26s
Simulating 100 seconds: 61s
You can see that the speed drops when the simulation is run for longer period.
EDIT2: blocking/non-blocking
I am using non-blocking sending and blocking receiving. The simulation is written using Simulink, and receive and send is implemented as two C++-blocks.
My guess is that the sending happens first, and then the receive-block is run.
There are no fixed intervals. The sending happens when the calculations are finished. It's all in one thread.
Related
I have an Multi threaded C++ network application which listens for UDP packets as input - this data then hops through the application/processed via various queues and threads and finally pushed out on a TCP socket.
What I am seeing is that if inputs come in slow lets say 5/sec, the total response time (in-to-out) is slow (lets say 100ms) and if the inputs come in fast e.g. 20/sec, the response time is also fast (~50ms). This observation is really weird. And also messes up the response time in the fast case because the 1st response is always slow. Just to make sure - the application is doing exactly the same amount of work in both slow and fast cases.
Things that have been tried to investigate this -
Its a dual Xeon box running Linux 2.6 kernel - disabled Turbo boost, made sure processors are in C0 states.
eliminated network causes. the root cause is within the box. ( in software or hardware )
I have a fake input going through the system from input to output on a timer to keep the application "warm" - no effect. (the application
's worker threads are busy waiting and pinned to cores).
perf points indicate that EVERY thing gets slower - which basically mean that the processors are slowing down when not under continuous load - but nothing else suggests that ( 17z/turbostat) or I am reading them incorrectly.
Does someone has color on what might be happening?
In my experience, this would be the nasty doings of Nagle's devilish device. Sounds extremeliy plausible to me that more data in the buffer fills up TCP packet which than get's sent. Without much data, tcp packet is waiting for the ack from the other side, which is, as we all know, delayed.
Solution - learn to make sure Nagle's program killer is disabled the first thing after you create any send-capable TCP socket.
I want to measure UDP latency and drop rate between two machines on Linux. Preferably (but not crucial) to perform measurement between multiple machines at the same time.
As a result I want to get a histogram, e.g. RTT times of each individual packet at every moment during measurement. Expected frequency is about 10 packets per second.
Do you know of any tool that I can use for this purpose?
What I tried so far is:
ping - uses icmp instead of UDP
iperf - measures only jitter but not latency.
D-ITG - measures per flow statistics, no histograms
tshark - uses TCP for pings instead UDP
I have also created a simple C++ socket program where I have Client and Server on each side, and I send UDP packets with counter and timestamp. My program seems to work ok, although since I am not a network programmer I am not 100% sure that I handled buffers correctly (specifically in the case of partial packets etc). So I would prefer to use some proven software for this task.
Can you recommend something?
Thanks
It depends. If all you want is a trace with timestamps, Wireshark is your friend: https://www.wireshark.org/
I would like to remind you that UDP is a message based protocol and packets have definite boundaries. There cannot be reception of partial packets. That is, you will either get the complete message or you will not get it. So, you need not worry about partial packets in UDP.
The method of calculating packet drop using counter & calculating latency using time delta appears fine for UDP. However the important point to be taken in to consideration is ensuring the synchronization of the system time of client and server.
I have a device which sends data every 20 milliseconds over TCP. I have an application which connects to this device, starts the socket communication. My Application listens on a seperate thread and reads the data as fast as data is ready, puts data aside, and some other thread processes it. Device is directly connected to the computer via ethernet cable.
I see a strange problem and I am trying to understand the reason why, Almost once in every minute, it takes approximately 50 milliseconds to receive a packet from the device. I do a blocking read which will try reading for a second, and will finish as fast as data is ready, normally it takes approximately 20 ms as I would expect, but like I said before there are times it takes 50 ms even though it is very rare(1 in 3000). What I noticed is the packets after late packet arrives immediately, so it makes me think that there's some delay on the network layer. I also examined the timestamps of the packets(which is given by the device), they are consistenly increasing by 20 ms's.
Is it normal to see delays like that when the device is directly connected to the computer, Since it is TCP there might be lots of effort under the hood(CRC checks, out of order packages, retransmissions, etc). I still want to find an alternative way to prevent this delay than accepting the fact that it might happen.
Any insights will be greatly appreciated.
It's probably result of Nagle's algorithm which is turned on by default in TCP/IP socket.
Use setsockopt() to set the TCP_NODELAY flag on socket that sends data to turn it off.
I currently have an application that receives real time messages at a very high rate and my application needs to display those messages instantly. I read about nagles algorithm and I understand that it combines small messages into one big message and then transmits it (It is designed to reduce the number of acknowledgement packets by delaying the ACK for a short time).My question is will disabling Nagles algorithm help my application ? all my messages are to be displayed in realtime as soon as they are received.Any suggestions on this issue would be appreciated.
Update:
Also I only have control of the receiver , will disabling nagles algo. on the receiver have any affect or does it only have affect when its disabled on the sender ?
Nagle is a sender side algorithm only, so if you can only affect the receiver, you cannot disable it.
Even if you could affect the sender, disabling Nagle is not very effective when doing one directional communication. In bidirectional communication, disabling Nagle can improve throughput because the benefits of removing delays can accumulate as each node can send its responses slightly sooner, letting the other side respond even sooner than that. However, in the one direction case, disabling Nagle can decrease latency by one round trip, but those benefits cannot accumulate because the fact that you are not delaying packets does not slow down the generation of new packets. You never get ahead by more than one round trip. Over the internet, that's ~20-30ms. Over a LAN, that's usually ~1ms
If your system is sufficiently hard real time that a single round-trip latency matters, then TCP is a poor protocol, and you should be using UDP instead. Nagle is a TCP only algorithm, so it would not affect UDP.
Just for fun: Pinging a local computer on my LAN is <1ms. This means Nagle can only delay my messages by something under 1ms. Quantums for desktop computer schedulers can 20-60ms [1], and even longer for servers, so I would expect removing the nagle algorithm to have no visible effect on my LAN, dwarfed by the effect of other threads on my computer consuming the CPUs
[1] http://recoverymonkey.org/2007/08/17/processor-scheduling-and-quanta-in-windows-and-a-bit-about-unixlinux/
I am working on a C++ application that can be qualified as a router. This application receives UDP packets on a given port (nearly 37 bytes each second) and must multicast them to another destinations within a 10 ms period. However, sometimes after packet reception, the retransmission exceeds the 10 ms limit and can reach the 100 ms. these off-limits delays are random.
The application receives on the same Ethernet interface but on a different port other kind of packets (up to 200 packets of nearly 100 bytes each second). I am not sure that this later flow is disrupting the other one because these delay peaks are too scarce (2 packets among 10000 packets)
What can be the causes of these sporadic delays? And how to solve them?
P.S. My application is running on a Linux 2.6.18-238.el5PAE. Delays are measured between the reception of the packet and after the success of the transmission!
An image to be more clear :
10ms is a tough deadline for a non-realtime OS.
Assign your process to one of the realtime scheduling policies, e.g. SCHED_RR or SCHED_FIFO (some reading). It can be done in the code via sched_setscheduler() or from command line via chrt. Adjust the priority as well, while you're at it.
Make sure your code doesn't consume CPU more than it has to, or it will affect entire system performance.
You may also need RT_PREEMPT patch.
Overall, the task of generating Ethernet traffic to schedule on Linux is not an easy one. E.g. see BRUTE, a high-performance traffic generator; maybe you'll find something useful in its code or in the research paper.