I am writing a gtk application (in C++) to communicate with motor controller through serial port. I am working with Linux Ubuntu and termios lib.
I need advice on the best solution to do it.
So here are the constraints I have:
1- when i send a request, the controler send me back a message
2- the controler can send me error notification at any time if an error occurs
3- request is ANSII characters string finished with [CR]
4- controller answer is ANSII characters string finished with [CR][NL]
Because of (3) and (4), I thought it was appropriate to configure serial port in CANONICAL mode.
Because of the GUI + (2), I thought about multi-threading: a main thread who write user request on serial port and an other infinite thread to read controller answer. Do you think it is a good idea?
Second question: if I am using multi-threading I want to be able to write data when I need it so I have to find a way to stop/sleep the reading thread during writing maybe with pthread_cond_wait. Am I right? I've seen poll and select functions but I don't really understand them and I am not sure they are compatible with canonical mode.
I am getting started with multi-threading and serial port. I read lots of things on google, forum...but the large amount of information is a little overwhelming for a beginner.
Thank you for your help.
The main thing to consider here for separating the GUI from the serial port is going to be your delays. Are you ever going to be performing any actions that will cause you to need to poll the port for a specific amount of time that would be noticeable to the user? If you are just doing request/reply and the latency of those is really low your user probably wouldn't notice any of those delays. Additionally receiving those asynchronous error messages would also not cause any sort of noticeable delay I would imagine. Unless you know for a fact that there could be numerous seconds of delay after an Init message or something like that gets sent to the controller it will probably make your life much simpler to keep the application single threaded.
On the other hand if there will be those large latencies or you just want to mess around with multi thread I would just start with 1 thread that does all the GUI work and another thread that handles all the serial IO. Use message passing or event notification between those two threads for coordinating your activities and it should be pretty straight forward.
I've written a socket client server program in c++. The program works perfectly and what I want to do now is to check periodically whether the connected clients are available. I know that it can be done using a while(true) loop in the server program. But it will use more cpu resources. Is there any other efficient way to check the availability of the clients? I've heard something called heartbeat. Does that help me? Is there any other way to do this?
By definition, is a client is "connected" it is "available" otherwise it will not be "connected".
If you need a persistent connection just use a transport protocol that provides for persistence (like TCP) and let TCP to do its job. Its own keep-alive and retransmission timers are already set up to satisfy network characteristics (and if they are not, this is a system problem, not something a specific application should manage alone).
Just send and receive data and manage the connection errors that eventually arise if one of the party goes away or becomes unreachable.
In any case:
Don't do dumb infinite loops sending/receiving empy messages just to test: there are billions on node in the internet: if anyone will behave as such, the entire internet will be fullfilled by "ping pong-s". Bendwidth has a cost. Much more than CPU and memory. Don't waste it.
Don't attempt to timeout yourself o recover a missing packet. There are hundredth of good reason you can even never imagine why a network can decide to reroute or discard a packet. Let TCP and IP to play their role consistently. All application trying to fix themselves netwotking issues do nothing more to adding mess to network managers.
In very short terms: if a socket is opened and is not in error, than the computer is connected. If it is in error, just close it and attempt to reopen it (and wait for the transport protocol timeouts. It ca be up to minutes, but don't try to escape from networking protocol rules: you are mot alone in the network, and it's not you who can make those rules).
Why not firing the main event in a timely manner instead of an endless loop ?
I heard from someone that modern MB architecture and memory designs can eliminate the need of allocating and reallocating mem-chunks by applying caching methods with some specialties to implement re-usability which helps also to avoid creating too many fragments both externally and internally. I don't mind checking this out again as I am listening to my online professor's online interview right now...:D. Good luck searching!
[EDIT] I read up on the other reply by ongard using ping method timely, which is very nice but please also consider that clients can block pings from other computers as well.
I'm looking for a way to add network emulation to a socket.
The basic solution would be some way to add bandwidth limitation to a connection.
The ideal solution for me would:
Support advanced network properties (latency, packet-loss)
Open-source
Have a similar API as standard sockets (or wraps around them)
Work on both Windows and Linux
Support IPv4 and IPv6
I saw a few options that work on the system level, or even as proxy (Dummynet, WANem, neten, etc.), but that won't work for me, because I want to be able to emulate each socket manually (for example, open one socket with modem emulation and one with 3G emulation. Basically I want to know how these tools do it.
EDIT: I need to embed this functionality in my own product, therefore using an extra box or a third-party tool that needs manual configuration is not acceptable. I want to write code that does the same thing as those tools do, and my question is how to do it.
Epilogue: In hindsight, my question was a bit misleading. Apparently, there is no way to do what I wanted directly on the socket. There are two options:
Add delays to send/receive operation (Based on #PaulCoccoli's answer):
by adding a delay before sending and receiving, you can get a very crude network simulation (constant delay for latency, delay sending, as to not send more than X bytes per second, for bandwidth).
Paul's answer and comment were great inspiration for me, so I award him the bounty.
Add the network simulation logic as a proxy (Based on #m0she and others answer):
Either send the request through the proxy, or use the proxy to intercept the requests, then add the desired simulation. However, it makes more sense to use a ready solution instead of writing your own proxy implementation - from what I've seen Dummynet is probably the best choice (this is what webpagetest.org does). Other options are in the answers below, I'll also add DonsProxy
This is the better way to do it, so I'm accepting this answer.
You can compile a proxy into your software that would do that.
It can be some implementation of full fledged socks proxy (like this) or probably better, something simpler that would only serve your purpose (and doesn't require prefixing your communication with the destination and other socks overhead).
That code could run as a separate process or a thread within your process.
Adding throttling to a proxy shouldn't be too hard. You can:
delay forwarding of data if it passes some bandwidth limit
add latency by adding timer before read/write operations on buffers.
If you're working with connection based protocol (like TCP), it would be senseless to drop packets, but with a datagram based protocol (UDP) it would also be simple to implement.
The connection creation API would be a bit different from normal posix/winsock (unless you do some macro or other magic), but everything else (send/recv/select/close/etc..) is the same.
If you're building this into your product, then you should implement a layer of abstraction over the sockets API so you can select your own implementation at run time. Alternatively, you can implement wrappers of each socket function and select whether to call your own version or the system's version.
As for adding latency, you could have your implementation of the sockets API spin off a thread. In that thread, have a priority queue ordered by time (i.e. this background thread does a very basic discrete event simulation). Each "packet" you send or receive could be enqueued along with a delivery time. Each delivery time should have some amount of delay added. I would use some kind of random number generator with a Gaussian distribution.
The background thread would also have to simulate the other side of the connection, though it sounds like you may have already implemented that part?
I know only Network Link Conditioner for Mac OS X Lion. You should be mac developer to download it, so i cannot put download link there. Only description from 9to5mac.com: http://9to5mac.com/2011/08/10/new-in-os-x-lion-network-link-conditioner-utility-lets-you-simulate-internet-and-bandwidth-conditions/
This answer might be a partial solution for you when using linux:
Simulate delayed and dropped packets on Linux. It refers to a kernel module called netem, which can simulate all kinds of network problems.
If you want to work with TCP connections, having "packet loss" could be problematic since a lot of error-handling (like recovering lost packages) is done in the kernel. Simulating this in a cross-platform way could be hard.
you usually add a network device to your network that throttles the bandwidth or latency, on a port by port basis, you can then achieve what you want just by connecting to the port allocated to the particular type of crappy network you want to test, with no code changes or modifications required.
The easiest ways to do this is just add iptables rules to a Linux server acting as a proxy.
If you want it to work without the separate device, try trickle that is a software package that throttles your network on your client PC. (or for Windows)
You may would like to check WANem http://wanem.sourceforge.net/ . WANEM is Open Source and licensed under the GNU General Public License.
WANem allows the application development team to setup a transparent application gateway which can be used to simulate WAN characteristics like Network delay, Packet loss, Packet corruption, Disconnections, Packet re-ordering, Jitter, etc.
I think you could use a tool like Network Simulator. It's free, for Windows.
The only thing to do is to setup your program to use the right ports (and the settings for the network, of course).
If you want a software only solution that you control, you will have to implement it yourself. I know of no such existing package.
While a wrapper layer over a socket may give you the ability to introduce delay, it won't be sufficient to introduce loss or out of order delivery. In order to simulate those activities, you actually need intercept the data in transit between the two TCP stacks.
The approach I would recommend is to use a tunneling device (say tunX). Routes should be set so the client believes the way to the server is through tunX. Additional code (perhaps running in a different thread) would promiscuously intercept traffic on tunX, and perform your augmented behavior, before forwarding packets over the true physical interface that will get the traffic to your server. The reverse would happen for packets arriving from the server on the physical interface. Those packets would be intercepted by the client code, behavior augmented, before forwarding through tunX.
However, since you are testing client software, I am unclear as to why you would want to embed this code in your released software, unless the software itself is a WAN simulating client.
I want to get number of SYN and ESTABLISHED connection to my server with C/C++. But I don't want to call popen to run netstat, or any other Linux command. I've managed to scan /proc/net/ip_conntrack and get the numbers. But I realize that scanning ip_conntrack requires great resources, each time my application invoke that method. Is there any other simple way?
Scanning /proc/net/ip_conntrack is not reliable because it only works if netfilter/connection tracking is enabled. And it doesn't only count connections to your server but also through your server (if it's acting as a router).
Better would be to get the information in the same places as netstat does: /proc/net/tcp, /proc/net/tcp6 (and similar files for UDP and other protocols if you care about those). That amount more or less to reimplementing netstat inside your application though. You have to wonder if it's worth it. Also, it's portable (more or less) to call netstat whereas reading those files directly is Linux-specific.
I know you are concerned about the resources requires to scan the full table every time, but I don't think there's a say to "subscribe" and get notifications when new connections are established or torn down. The closest thing I can think of to something like that would be to sniff the network interface (using libpcap) and keeping track of connection setups and teardowns yourself.
Given: multithreaded (~20 threads) C++ application under RHEL 5.3.
When testing under load, top shows that CPU usage jumps in range 10-40% every second.
The design mostly pretty simple - most of the threads implement active object design pattern: thread has a thread-safe queue, requests from other queues are pushed to the queue, while the thread only polling on the queue and process incomming requests. Processed request causes to a new request to be pushed to next processing thread.
The process has several TCP/UDP connection over each a data is received/sent in a high load.
I know I did not provided sufficiant data. This is pretty big application, and I'n not familiar well with all it's parts. It's now ported from Windows on Linux over ACE library (used for networking part).
Suppusing the problem is in the application and not external one, what are the techicues/tools/approaches can be used to discover the problem. For example I suspect that this maybe caused by some mutex contention.
I have faced similar problem some time back and here are the steps that helped me.
1) Start with using strace to see where the application is spending the time executing system calls.
2) Use OProfile to profile both the application and the kernel.
3) If you are using an SMP system , look at the numa settings,
In my case that caused a havoc .
/proc/appPID/numa_maps will give a quick look at how the access to the memory is happening.
numa misses can cause the jumps.
4) You have mentioned about TCP connections in your app.
Look at the MTU size and see its set to right value and
Depending upon the type of Data getting transferred use the Nagles Delay appropriately.
Nagles Delay