We are trying to use send scheduling on a Connectx-6 LX. If we set no timestamps on the packet buffers and manually send each packet at approximately the right time everything works. However if we set timestamps in the buffers then the first 25 packets are sent and the packets are received at the expected times but all subsequent calls to rte_eth_tx_burst return 0. If its relevant we are sending a single packet in each burst with timestamps 125 us apart.
We've tried setting the timestamps to low values and the packets are transmitted correctly and as expected the tx_pp_timestamp_past_errors value is incremented. We also set high values and this worked too with tx_pp_timestamp_future_errors incrementing.
Any ideas where to start debugging this? I couldn't see any API which would give an error code for rte_eth_tx_burst failing.
We're using DPDK 21.08 with the 5.4.3.1 Mellanox driver on Ubuntu 20.04.
It looks like this was caused by not having enough TX descriptors, we were only specifying 64, increasing to 1024 fixes the problem.
Related
Here's the summary, I send a packet from a server to a client run on the same computer. For some reason the packet sent is not the same as the packet received.
Here's the details:
The packet was sent using RakNet with the calling function:
rakPeer->Send(&bitStream, MEDIUM_PRIORITY, RELIABLE_ORDERED, 0, UNASSIGNED_RAKNET_GUID, true);
Here are the first 10 bytes of the packet sent by the server:
27,50,39,133,202,135,0,0,0,99 ... 1180 more bytes
Here are the first 10 bytes of the packet as seen by the receiving client (Note: 50% of the time it is right, the other half it is this):
27,50,43,40,247,134,255,255,255,99 ... 1180 more bytes
The first byte is ID_TIMESTAMP. Bytes 2-5 contain the time stamp, which I presume RakNet messes with somehow. Byte 6 is the packed ID which is clearly changed, as well as the following 3 bytes.
My suspicion is that the error is some how caused by the length of the packet, as smaller packets seem to send without any detectable errors, however I understand RakNet automatically handles packet corruption and internally splits the packet if it is too large.
Any help is appreciated.
Well for anyone who has the same issue, here is the solution.
RakNet time stamps are 32 bit or 64 bit depending on your build configuration. In this case I was sending 32 bit timestamps using a 64 bit build. That is a no-no since RakNet will change the bits it thinks are the timestamp to account for the relative time between computers.
this is my first "question", I hope I do it right :)
I am experimenting with network programming and in particular I want to broadcast data from one machine to some other >10 devices using UDP, over a wireless network. The data comes in packets of about 300 bytes, and at about 30 frames per second, i.e., one every ~33ms.
My implementation is based on the qt example: http://qt-project.org/doc/qt-4.8/network-broadcastreceiver.html
I am testing the application with just one client and experiencing quite a few dropped frames, not really sure why. All works fine if I used ethernet cables. I hope someone here can help me find a reason.
I can spot dropped frames because the packets contain a timestamp: After I receive one datagram, I can check for the difference between its timestamp and the last one received, if this is greater than e.g. 50ms, it means that I lost one packet on the way.
This happens quite often, even though I have a dedicated wi-fi network (not connected to the internet and with just 3 machines connected to a router I just bought). Most of the times I drop one or two packets, which would not be a problem, but sometimes the difference between the timestamps suggests that some >30 packets are lost, which is not good for what I am trying to achieve.
When I ping from one machine to the other, I get these values:
50 packets transmitted, 50 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.244/91.405/508.959/119.074 ms
pretty bad for a new router, in a dedicated network with just 3 clients, isn't it? The router is advertised as a very fast Wi-Fi router, with three times faster performance than 802.11n routers.
Compare it with the values I get from an older router, sitting in the same room, with some 10 machines connected to it, during office hour:
39 packets transmitted, 39 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.458/47.297/142.201/37.186 ms
Perhaps the router is defective?
One thing I cannot explain is that, if I ping while running my UDP client/server application, the statistics improve:
55 packets transmitted, 55 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1.164/6.174/197.962/26.181 ms
I was wondering if anyone had tips on what to test, hints on how to achieve a "reliable" UDP connection between these machines over wi-fi. By reliable I mean that I would be ok dropping 2 consecutive packets, but not more.
Thanks.
Edit
It seems that the router (?) sends the packets in bursts. I am measuring the time it passes between receiving two datagrams on the client and this value is about 3 ms for a sequence of ~10 packets, and then, around 300 ms for the next packet. I think my issues at the client is more related to this inconsistency in the intervals between frames, rather than the dropped frames. I probably just need to have a queue and a delay of >300ms wrt to the server.
The first and easiest way to tackle any problem related to network is to capture them on wireshark.
And also check if packets are really being sent out from broadcasting machine.
And also, based on your description if packets being transmitted fine with etherne cables and not with UDP then
it could be issue with UDP port too.
I have a java app on linux which opens UDP socket and waits for messages.
After couple of hours under heavy load, there is a packet loss, i.e. the packets are received by kernel but not by my app (we see the lost packets in sniffer, we see UDP packets lost in netstat, we don't see those packets in our app logs).
We tried enlarging socket buffers but this didnt help - we started losing packets later then before, but that's it.
For debugging, I want to know how full the OS udp buffer is, at any given moment. Googled, but didn't find anything. Can you help me?
P.S. Guys, I'm aware that UDP is unreliable. However - my computer receives all UDP messages, while my app is unable to consume some of them. I want to optimize my app to the max, that's the reason for the question. Thanks.
UDP is a perfectly viable protocol. It is the same old case of the right tool for the right job!
If you have a program that waits for UDP datagrams, and then goes off to process them before returning to wait for another, then your elapsed processing time needs to always be faster than the worst case arrival rate of datagrams. If it is not, then the UDP socket receive queue will begin to fill.
This can be tolerated for short bursts. The queue does exactly what it is supposed to do – queue datagrams until you are ready. But if the average arrival rate regularly causes a backlog in the queue, it is time to redesign your program. There are two main choices here: reduce the elapsed processing time via crafty programming techniques, and/or multi-thread your program. Load balancing across multiple instances of your program may also be employed.
As mentioned, on Linux you can examine the proc filesystem to get status about what UDP is up to. For example, if I cat the /proc/net/udp node, I get something like this:
$ cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops
40: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 3466 2 ffff88013abc8340 0
67: 00000000:231D 00000000:0000 07 00000000:0001E4C8 00:00000000 00000000 1006 0 16940862 2 ffff88013abc9040 2237
122: 00000000:30D4 00000000:0000 07 00000000:00000000 00:00000000 00000000 1006 0 912865 2 ffff88013abc8d00 0
From this, I can see that a socket owned by user id 1006, is listening on port 0x231D (8989) and that the receive queue is at about 128KB. As 128KB is the max size on my system, this tells me my program is woefully weak at keeping up with the arriving datagrams. There have been 2237 drops so far, meaning the UDP layer cannot put any more datagrams into the socket queue, and must drop them.
You could watch your program's behaviour over time e.g. using:
watch -d 'cat /proc/net/udp|grep 00000000:231D'
Note also that the netstat command does about the same thing: netstat -c --udp -an
My solution for my weenie program, will be to multi-thread.
Cheers!
Linux provides the files /proc/net/udp and /proc/net/udp6, which lists all open UDP sockets (for IPv4 and IPv6, respectively). In both of them, the columns tx_queue and rx_queue show the outgoing and incoming queues in bytes.
If everything is working as expected, you usually will not see any value different than zero in those two columns: as soon as your application generates packets they are sent through the network, and as soon those packets arrive from the network your application will wake up and receive them (the recv call immediately returns). You may see the rx_queue go up if your application has the socket open but is not invoking recv to receive the data, or if it is not processing such data fast enough.
rx_queue will tell you the queue length at any given instant, but it will not tell you how full the queue has been, i.e. the highwater mark. There is no way to constantly monitor this value, and no way to get it programmatically (see How do I get amount of queued data for UDP socket?).
The only way I can imagine monitoring the queue length is to move the queue into your own program. In other words, start two threads -- one is reading the socket as fast as it can and dumping the datagrams into your queue; and the other one is your program pulling from this queue and processing the packets. This of course assumes that you can assure each thread is on a separate CPU. Now you can monitor the length of your own queue and keep track of the highwater mark.
The process is simple:
If desired, pause the application process.
Open the UDP socket. You can snag it from the running process using /proc/<PID>/fd if necessary. Or you can add this code to the application itself and send it a signal -- it will already have the socket open, of course.
Call recvmsg in a tight loop as quickly as possible.
Count how many packets/bytes you got.
This will discard any datagrams currently buffered, but if that breaks your application, your application was already broken.
I have a device which sends data every 20 milliseconds over TCP. I have an application which connects to this device, starts the socket communication. My Application listens on a seperate thread and reads the data as fast as data is ready, puts data aside, and some other thread processes it. Device is directly connected to the computer via ethernet cable.
I see a strange problem and I am trying to understand the reason why, Almost once in every minute, it takes approximately 50 milliseconds to receive a packet from the device. I do a blocking read which will try reading for a second, and will finish as fast as data is ready, normally it takes approximately 20 ms as I would expect, but like I said before there are times it takes 50 ms even though it is very rare(1 in 3000). What I noticed is the packets after late packet arrives immediately, so it makes me think that there's some delay on the network layer. I also examined the timestamps of the packets(which is given by the device), they are consistenly increasing by 20 ms's.
Is it normal to see delays like that when the device is directly connected to the computer, Since it is TCP there might be lots of effort under the hood(CRC checks, out of order packages, retransmissions, etc). I still want to find an alternative way to prevent this delay than accepting the fact that it might happen.
Any insights will be greatly appreciated.
It's probably result of Nagle's algorithm which is turned on by default in TCP/IP socket.
Use setsockopt() to set the TCP_NODELAY flag on socket that sends data to turn it off.
Using 2 PC's with Windows XP, 64kB Tcp Window size, connected with a crossover cable
Using Qt 4.5.3, QTcpServer and QTcpSocket
Sending 2000 messages of 40kB takes 2 seconds (40MB/s)
Sending 1 message of 80MB takes 80 seconds (1MB/s)
Anyone has an explanation for this? I would expect the larger message to go faster, since the lower layers can then fill the Tcp packets more efficiently.
This is hard to comment on without seeing your code.
How are you timing this on the sending side? When do you know you're done?
How does the client read the data, does it read into fixed sized buffers and throw the data away or does it somehow know (from the framing) that the "message" is 80MB and try and build up the "message" into a single data buffer to pass up to the application layer?
It's unlikely to be the underlying Windows sockets code that's making this work poorly.
TCP, from the application side, is stream-based which means there are no packets, just a sequence of bytes. The kernel may collect multiple writes to the connection before sending it out and the receiving side may make any amount of the received data available to each "read" call.
TCP, on the IP side, is packets. Since standard Ethernet has an MTU (maximum transfer unit) of 1500 bytes and both TCP and IP have 20-byte headers, each packet transferred over Ethernet will pass 1460 bytes (or less) of the TCP stream to the other side. 40KB or 80MB writes from the application will make no difference here.
How long it appears to take data to transfer will depend on how and where you measure it. Writing 40KB will likely return immediately since that amount of data will simply get dropped in TCP's "send window" inside the kernel. An 80MB write will block waiting for it all to get transferred (well, all but the last 64KB which will fit, pending, in the window).
TCP transfer speed is also affected by the receiver. It has a "receive window" that contains everything received from the peer but not fetched by the application. The amount of space available in this window is passed to the sender with every return ACK so if it's not being emptied quickly enough by the receiving application, the sender will eventually pause. WireShark may provide some insight here.
In the end, both methods should transfer in the same amount of time since an application can easily fill the outgoing window faster than TCP can transfer it no matter how that data is chunked.
I can't speak for the operation of QT, however.
Bug in Qt 4.5.3
..................................