Hello members of stackoverflow
I am developing a project with DPDK but have encountered a silly issue, that is not obvious to me.
I want to find out the right approach to tackle my current problem.
I am sending and receiving 4kb rte_mbuf between remote and local node,
that alone works fine,
however when I combine the implementation with 3rd party library, the DPDK stops receiving the data after approximately receiving 8000 packets.
I have debugged all the possible program side and to my astonishment. there is no error found and all the packets that are within 8000 are received correctly.
I have no idea the approach to find out the problem of this situation. but the situation that I have mentioned can be replicated. It always stops at approximately 8000 packets received.
and there are absolutely no bugs found in dpdk (user interface). The only problem is the rte_rx_queue stops returning the packets after 8000 packets.
Would there be a good approach to identify the problem of this case?
Would there be a good approach to identify the problem of this case?
The best approach would be to start with the stats. Have a look at rte_eth_stats_get()
We need to check if any counter is increasing after the DPDK app has stopped. I bet the rx_nombuf counter is still increasing, which might mean your mempool has exhausted.
If we pass the mbuf to an external lib, we have to make sure that each mbuf is freed after return from the lib.
Related
Here is the main problem.
I have 10-gigabit ethernet interface and current flow is 6-7 Gbit/sec
I need to implement a firewall Then I need to capture raw packets to filter some packets.
Simply I started to implement as a raw socket necessary codes are at below. Socket bound to a specific interface.
socketfd=socket(AF_PACKET,SOCK_RAW,htons(ETH_P_ALL));
strncpy(ifopts.ifr_name,interfaceName,IFNAMSIZ-1);
ioctl(socketfd,SIOCGIFINDEX,&ifopts);
sll.sll_family=AF_INET;
sll.sll_ifindex=ifopts.ifr_ifindex;
sll.sll_protocol=htons(ETH_PALL);
bind(socketfd,&sll,sizeof(sll));
Here is how I read and mtu size is 9000
while(true)
recvfrom(socketfd,buffer,9000,0,0,0);
Without any process on a packet I got ~150Mbit/sec.
This is the problem I need to solve. I realize that nload or ip -s link shows the actual rate; but I cannot reach these numbers around 6-7Gbit/sec.
~150Mbit/sec is so ridiculous rate for me. I need to increase performance as much as I can do using one CPU. I will try to use PF_INET, if you want I can share the result of it.
Here is the answer.
First of all capturing speed is not only depend on the size of bytes on the interface, but the number of packets is also important. So socket programming is also limited by the number of packets. I measured as 200k packets per second (pps).
Using better network driver is the one way of the increasing pps. PF_RING is the possible library and driver. You may use the trial version to test. I simply test it on my network and result is 14M pps. Then this rate is almost 10Gbit/sec. That's all I experienced.
Thanks all.
Multiple clients are connected to a single ZMQ_PUSH socket. When a client is powered off unexpectedly, server does not get an alert and keep sending messages to it. Despite of using ZMQ_OBLOCK and setting ZMQ_HWM to 5 (queue only 5 messages at max), my server doesn't get an error until unless client is reconnected and all the messages in queue are received at once.
I recently ran into a similar problem when using ZMQ. We would cut power to interconnected systems, and the subscriber would be unable to reconnect automatically. It turns out the there has recently (past year or so) been implemented a heartbeat mechanism over ZMTP, the underlying protocol used by ZMQ sockets.
If you are using ZMQ version 4.2.0 or greater, look into setting the ZMQ_HEARTBEAT_IVL and ZMQ_HEARTBEAT_TIMEOUT socket options (http://api.zeromq.org/4-2:zmq-setsockopt). These will set the interval between heartbeats (ZMQ_HEARTBEAT_IVL) and how long to wait for the reply until closing the connection (ZMQ_HEARTBEAT_TIMEOUT).
EDIT: You must set these socket options before connecting.
There is nothing in zmq explicitly to detect the unexpected termination of a program at the other end of a socket, or the gratuitous and unexpected failure of a network connection.
There has been historical talk of adding some kind of underlying ping-pong are-you-still-alive internal messaging to zmq, but last time I looked (quite some time ago) it had been decided not to do this.
This does mean that crashes, network failures, etc aren't necessarily handled very cleanly, and your application will not necessarily know what is going on or whether messages have been successfully sent. It is Actor model after all. As you're finding your program may eventually determine something had previously gone wrong. Timeouts in zmtp will spot the failure, and eventually the consequences bubble back up to your program.
To do anything better you'd have to layer something like a ping-pong on top yourself (eg have a separate socket just for that so that you can track the reachability of clients) but that then starts making it very hard to use the nice parts of ZMQ such as push / pull. Which is probably why the (excellent) zmq authors decided not to put it in themselves.
When faced with a similar problem I ended up writing my own transport library. I couldn't find one off the shelf that gave nice behaviour in the face of network failures, crashes, etc. It implemented CSP, not actor model, wasn't terribly fast (an inevitability), didn't do patterns in the zmq sense, but did mean that programs knew exactly where messages were at all times, and knew that clients were alive or unreachable at all times. The CSPness also meant message transfers were an execution rendezvous, so programs know what each other is doing too.
I've written a socket client server program in c++. The program works perfectly and what I want to do now is to check periodically whether the connected clients are available. I know that it can be done using a while(true) loop in the server program. But it will use more cpu resources. Is there any other efficient way to check the availability of the clients? I've heard something called heartbeat. Does that help me? Is there any other way to do this?
By definition, is a client is "connected" it is "available" otherwise it will not be "connected".
If you need a persistent connection just use a transport protocol that provides for persistence (like TCP) and let TCP to do its job. Its own keep-alive and retransmission timers are already set up to satisfy network characteristics (and if they are not, this is a system problem, not something a specific application should manage alone).
Just send and receive data and manage the connection errors that eventually arise if one of the party goes away or becomes unreachable.
In any case:
Don't do dumb infinite loops sending/receiving empy messages just to test: there are billions on node in the internet: if anyone will behave as such, the entire internet will be fullfilled by "ping pong-s". Bendwidth has a cost. Much more than CPU and memory. Don't waste it.
Don't attempt to timeout yourself o recover a missing packet. There are hundredth of good reason you can even never imagine why a network can decide to reroute or discard a packet. Let TCP and IP to play their role consistently. All application trying to fix themselves netwotking issues do nothing more to adding mess to network managers.
In very short terms: if a socket is opened and is not in error, than the computer is connected. If it is in error, just close it and attempt to reopen it (and wait for the transport protocol timeouts. It ca be up to minutes, but don't try to escape from networking protocol rules: you are mot alone in the network, and it's not you who can make those rules).
Why not firing the main event in a timely manner instead of an endless loop ?
I heard from someone that modern MB architecture and memory designs can eliminate the need of allocating and reallocating mem-chunks by applying caching methods with some specialties to implement re-usability which helps also to avoid creating too many fragments both externally and internally. I don't mind checking this out again as I am listening to my online professor's online interview right now...:D. Good luck searching!
[EDIT] I read up on the other reply by ongard using ping method timely, which is very nice but please also consider that clients can block pings from other computers as well.
I want to get number of SYN and ESTABLISHED connection to my server with C/C++. But I don't want to call popen to run netstat, or any other Linux command. I've managed to scan /proc/net/ip_conntrack and get the numbers. But I realize that scanning ip_conntrack requires great resources, each time my application invoke that method. Is there any other simple way?
Scanning /proc/net/ip_conntrack is not reliable because it only works if netfilter/connection tracking is enabled. And it doesn't only count connections to your server but also through your server (if it's acting as a router).
Better would be to get the information in the same places as netstat does: /proc/net/tcp, /proc/net/tcp6 (and similar files for UDP and other protocols if you care about those). That amount more or less to reimplementing netstat inside your application though. You have to wonder if it's worth it. Also, it's portable (more or less) to call netstat whereas reading those files directly is Linux-specific.
I know you are concerned about the resources requires to scan the full table every time, but I don't think there's a say to "subscribe" and get notifications when new connections are established or torn down. The closest thing I can think of to something like that would be to sniff the network interface (using libpcap) and keeping track of connection setups and teardowns yourself.
I need some help understanding a peculiar issue I'm having when using asio.
I've a client -server app, with a c++ client(using boost asio) sending 2 byte hearbeat (say, every second) to a server(written in java) (and receiving lots of data as well).
for a quite a few minutes the server correctly receives the 2 byte HeartBeat, but after that the server's 'read' complains abt a 0 byte read, and closes the connection (which I guess is correct for a blocking read). The client however always prints out that it's been transferring the correct amount.
I've experimented with almost all variants of the 'write' family of functions. are all of them implemented in terms of 'write_some' and does that mean that this behavior is expected?
I must be making some mistake in my usage, basically I'm looking for something within asio that guarantees a write ( at least a byte) . please help me figure out where I'm going wrong(and if any further info is reqd.)...
any advice, most appreciated!
thanks!
If it's sockets, you can't "guarantee a write"; what if the network is down, the cable yanked out, the switch is on fire, or the power is out worldwide and your computer happens to be the only one running on batteries?
That said, it sounds as if you have some kind of buffering/emptying issue perhaps, check over your read code to make sure it really consumes all data that appears.
A 0-byte read is not an error, look over that code again, check for any error status flags on the socket(s) and so on. A read can fail with a "AGAIN"-status, which really means you should try again.
strace the applications at both ends. It will show any error codes that are returned by read(), write() etc. Use strace -f if the application is multithreaded.
The advantage of this approach is that all applications - java, c++, python appear the same in an strace, so it's easy to spot bad behaviour.
In this case, it would probably show that the tcp connection ended (gracefully).