Realtime receiving of UDP packets with QNX RTOS - c++

Question

I have a source which sends UDP packets at a rate of 819.2 Hz (~1.2ms) to my QNX Neutrino machine. I want to receive and process those messages with as little delay and jitter as possible.
My first code was basically:
SetupUDPSocket();
while (true) {
recv(socket, buffer, BufferSize, MSG_WAITALL); // blocks until whole packet is received
processPacket(buffer);
}
The problem is that recv() only checks at each timer tick of the system if there is a new packet available. The timer tick is usually 1ms. So, if I use this I will get a huge jitter, because I process a packet every 1ms or every 2ms. I could reset the size of the timer ticks, but that would affect the whole system (and other timers of other processes, etc). And I still would have a jitter, because I certainly would never exactly match the 819.2 Hz.
So, I tried to use the interrupt line of the network card (5). But it seems as there are also other things which causes the interrupt to rise. I used to following code:
ThreadCtl(_NTO_TCTL_IO, 0);
SIGEV_INTR_INIT(&event);
iID = InterruptAttachEvent(IRQ5, &event, _NTO_INTR_FLAGS_TRK_MSK);
while(true) {
if (InterruptWait(0, NULL) == -1) {
std::cerr << "errno: " << errno << std::endl;
}
length = recv(socket, buffer, bufferSize, 0); // non-blocking this time
LogTimeAndLength();
InterruptUnmask(IRQ5, iID;
}
This results in a single succesful read in the beginning, followed by reads with 0 byte length after 0 time passing. It seems, that after do the InterruptUnmask(), the InterruptWait() does not wait at all, so there must already be a new interrupt (or the same?!).
Is it possible to do something like that with the interrupt line of the network card? Are there any other possibilties to receive the packets at a rate of 819.2 Hz?
Some information about the network card:
'pci -vvv' outputs:
Class = Network (Ethernet)
Vendor ID = 8086h, Intel Corporation
Device ID = 107ch, 82541PI Gigabit Ethernet Controller
PCI index = 0h
Class Codes = 020000h
Revision ID = 5h
Bus number = 4
Device number = 15
Function num = 0
Status Reg = 230h
Command Reg = 17h
I/O space access enabled
Memory space access enabled
Bus Master enabled
Special Cycle operations ignored
Memory Write and Invalidate enabled
Palette Snooping disabled
Parity Error Response disabled
Data/Address stepping disabled
SERR# driver disabled
Fast back-to-back transactions to different agents disabled
Header type = 0h Single-function
BIST = 0h Build-in-self-test not supported
Latency Timer = 40h
Cache Line Size= 8h un-cacheable
PCI Mem Address = febc0000h 32bit length 131072 enabled
PCI Mem Address = feba0000h 32bit length 131072 enabled
PCI IO Address = ec00h length 64 enabled
Subsystem Vendor ID = 8086h
Subsystem ID = 1376h
PCI Expansion ROM = feb80000h length 131072 disabled
Max Lat = 0ns
Min Gnt = 255ns
PCI Int Pin = INT A
Interrupt line = 5
CPU Interrupt = 5h
Capabilities Pointer = dch
Capability ID = 1h - Power Management
Capabilities = c822h - 28002000h
Capability ID = 7h - PCI-X
Capabilities = 2h - 400000h
Device Dependent Registers:
0x040: 0000 0000 0000 0000 0000 0000 0000 0000
...
0x0d0: 0000 0000 0000 0000 0000 0000 01e4 22c8
0x0e0: 0020 0028 0700 0200 0000 4000 0000 0000
0x0f0: 0500 8000 0000 0000 0000 0000 0000 0000
and 'nicinfo' outputs:
wm1:
INTEL 82544 Gigabit (Copper) Ethernet Controller
Physical Node ID ........................... 000E0C C5F6DD
Current Physical Node ID ................... 000E0C C5F6DD
Current Operation Rate ..................... 100.00 Mb/s full-duplex
Active Interface Type ...................... MII
Active PHY address ....................... 0
Maximum Transmittable data Unit ............ 1500
Maximum Receivable data Unit ............... 0
Hardware Interrupt ......................... 0x5
Memory Aperture ............................ 0xfebc0000 - 0xfebdffff
Promiscuous Mode ........................... Off
Multicast Support .......................... Enabled
Thanks for reading!

Answer 1

I am quite not sure why the statement "The problem is that recv() only checks at each timer tick of the system if there is a new packet available. The timer tick is usually 1ms." would be true for preemptive OS. There must be something in the system configuration or the network protocol stack implementation has some issues.
Years ago when I was working on some IPTV STB project for Yahoo BB Japan, i got an issue in RTP receiving. The issues is not delay or jitter, but the overall system performance in the STB after we add some NDS algorithm. We are using vxWorks, and vxWorks support ethernet hook interface, which will be called each time a ethernet packet is received by the driver.
I hook an API into it and just parse the UDP with specified port from the ethernet packets directly. Of course we have some assumption that there is no fragmentation, which is guaranteed by the network setup for performance issues. Maybe you can also check to see if you can get the same hook in the QNX ethernet driver. At lease you found out if the jitter comes from driver or not.

Answer 2

How big are your UDP packets ? If the packet size is small you will gain greater efficiency by packing more data into single packet and decreasing transmission rate.

Answer 3

I suspect the interrupt service routing (ISR) is not masking the interrupt. Perhaps it is designed for edge-sensitivity and the interrupt is level-sensitive.

Answer 4

sorry I'm a bit late to the party, but I came across your question and saw that it was similar to a situation I encountered. Instead of hardware interrupts, you could try a software interrupt using signals. QNX has some documentation here: http://www.qnx.com/developers/docs/qnx_4.25_docs/qnx4/sysarch/microkernel.html#IPCSIGNALS . I was using CentOS at the time but the theory is the same. According to http://www.qnx.com/developers/docs/6.3.0SP3/neutrino/lib_ref/s/socket.html you can use ioctl() to set up a receive group for the SIGIO signal for a given file descriptor...in your case a UDP socket. When the socket has data that is ready for reading, a SIGIO signal is sent to the process indicated by ioctl(). Use sigaction() to tell the OS what signal handling function to use. In your case, the signal handler can read the data off the socket and store it in a buffer for processing. Use pause() to suspend the process until it handles the SIGIO signal. When the signal handler returns, the process will wake up and you can process the data in the buffer.
That should allow you to process your data as it comes in without having to deal with timers or hardware interrupts. One thing to be aware of is that your system can process those signals as fast as the UDP traffic is coming in.