I'm working on an instant messenger client in C++ (Win32) and I'm experimenting with different asynchronous socket models. So far I've been using WSAAsyncSelect for receiving notifications via my main window. However, I've been experiencing some unexpected results with Winsock spawning additionally 5-6 threads (in addition to the initial thread created when calling WSAAsyncSelect) for one single socket.
I have plans to revamp the client to support additional protocols via DLL:s, and I'm afraid that my current solution won't be suitable based on my experiences with WSAAsyncSelect in addition to me being negative towards mixing network with UI code (in the message loop).
I'm looking for advice on what a suitable asynchronous socket model could be for a multi-protocol IM client which needs to be able to handle roughly 10-20+ connections (depending on amount of protocols and protocol design etc.), while not using an excessive amount of threads -- I am very interested in performance and keeping the resource usage down.
I've been looking on IO Completion Ports, but from what I've gathered, it seems overkill. I'd very much appreciate some input on what a suitable socket solution could be!
Thanks in advance! :-)
There are four basic ways to handle multiple concurrent sockets.
Multiplexing, that is using select() to poll the sockets.
AsyncSelect which is basically what you're doing with WSAAsyncSelect.
Worker Threads, creating a single thread for each connection.
IO Completion Ports, or IOCP. dp mentions them above, but basically they are an OS specific way to handle asynchronous I/O, which has very good performance, but it is a little more confusing.
Which you choose often depends on where you plan to go. If you plan to port the application to other platforms, you may want to choose #1 or #3, since select is not terribly different from other models used on other OS's, and most other OS's also have the concept of threads (though they may operate differently). IOCP is typically windows specific (although Linux now has some async I/O functions as well).
If your app is Windows only, then you basically want to choose the best model for what you're doing. This would likely be either #3 or #4. #4 is the most efficient, as it calls back into your application (similar, but with better peformance and fewer issues to WSAsyncSelect).
The big thing you have to deal with when using threads (either IOCP or WorkerThreads) is marshaling the data back to a thread that can update the UI, since you can't call UI functions on worker threads. Ultimately, this will involve some messaging back and forth in most cases.
If you were developing this in Managed code, i'd tell you to look at Jeffrey Richter's AysncEnumerator, but you've chose C++ which has it's pros and cons. Lots of people have written various network libraries for C++, maybe you should spend some time researching some of them.
consider to use the ASIO library you can find in boost (www.boost.org).
Just use synchronous models. Modern operating systems handle multiple threads quite well. Async IO is really needed in rare situations, mostly on servers.
In some ways IO Completion Ports (IOCP) are overkill but to be honest I find the model for asynchronous sockets easier to use than the alternatives (select, non-blocking sockets, Overlapped IO, etc.).
The IOCP API could be clearer but once you get past it it's actually easier to use I think. Back when, the biggest obstacle was platform support (it needed an NT based OS -- i.e., Windows 9x did not support IOCP). With that restriction long gone, I'd consider it.
If you do decide to use IOCP (which, IMHO, is the best option if you're writing for Windows) then I've got some free code available which takes away a lot of the work that you need to do.
Latest version of the code and links to the original articles are available from here.
And my views on how my framework compares to Boost::ASIO can be found here: http://www.lenholgate.com/blog/2008/09/how-does-the-socket-server-framework-compare-to-boostasio.html.
Related
I'm working on an multi-threaded server application for learning puposes. My problem right now is receiving the data.The first time I wrote the application I used WSAAsyncSelect, but I didn't like how it was working ( the window dependency was stupid in my opinion, even if you hid the window.). So I re-wrote it and now I have a thread that goes through the connected clients and checks if there is any data to be be received and pass it to the worker threads. This works for a small ammount of clients, but I think that for a larger amount it might delay other clients too much. A solution I've read about is having a thread for each client, but there are thread limitations. Another solution would be IOCP ( Windows ), but I need to find a good documentation, since the examples I found were a bit too ambigous ( I might be the problem here )
The language I'm using C/C++ under Microsoft Visual Studio 2013 ( WinSock, but I would like to write it in a multiplatform way )
If one thread is enough to handle all your clients, consider having N threads and distribute the clients (e.g. through hashing or in order of reception) among the available threads.
The key is that the overall number of threads (first in your process: polling+worker... but also in the whole system) must remain constant and not exceeding the number of processing resources (CPUs/cores)
This distinction between polling thread(s) and worker threads is the right way to go. Decoupling using queue structures, freeing the polling threads to do their job (i.e. polling) unhindered from the (arbitrary) logic running on the worker threads.
On Windows, IOCP is the standard model for scalable async IO. It solves all the problems you mention. It has a sane programming model (with a few API design errors). I believe there are self-contained samples. To learn IOCP I'd create a very simple chat server using this technology and try to get it absolutely right.
The select has the disadvantage that it does not scale (as you said).
If you want portability, look into async IO libraries like Boost asio. They use an IOCP-like model on all modern platforms. They are callback-driven.
To refresh and expand my C++ knowledge I am trying to implement a fcgi application and implement the fcgi interface myself. However, I have no expierience using sockets. Research has lead me to Boost.asio, but with my lack of socket knowledge I find it hard to use the library by the tutorials, which are sometimes only code examples, and browsing an apireference when you do not know what you are looking for is difficult.
Questions:
Which Tutorial(Other than the one coming wiht boost.asio)/Book would you recommend to read up on sockets/asynchronus IO?
For fcgi I find it hard to understand the benefits of AsyncIO, the header has to be decode first and then the data can be received, how does one get benefits by asynchronus I/O?
Is there some heuristic to know which parameters (number of threads, sockets per thread, socket multiplexing(yes(how many connections?)/no, async IO, buffersize) yield thebest performance for an fcgi interface?
I would recommend (for free) BeeJs networking guide or (for pay) W Richard Stevens book on sockets.
AIO (in this case) is a method of multiplexing multiple connections. If you don't have multiple connections you won't see any benefit. It's also the fastest though hardest method for doing this task. The idea is that any individual read or write to a connection may take time just waiting for resources to be available. AIO allows your application to move on to another connection and service that whilst it waits for those resources. In short, your app spends more time working and less time waiting, over multiple connections.
No. It depends on your architecture and your expected load. Really clever web servers will adjust all of that based on configuration and also on measured usage.
I have read that working with more than 64 sockets in a thread is dangerous(?). But -at least for me- Non-blocking sockets are used for avoiding complicated thread things. Since there is only one listener socket, how am i supposed to split sockets into threads and use them with select() ? Should i create fd_sets for each thread or what ? And how am i supposed to assign a client to a thread, since I can only pass values in the beginning with CreateThread() ?
No no no, you got a few things wrong there.
First, the ideal way to handle many sockets is to have a thread pool which will do the work in front of the sockets (clients).
Another thread, or two (actually in the amount of CPUs as far as I know), do the connection accepting.
Now, when a an event occurs, such as a new connection, it is being dispatched to the thread pool to be processed.
Second, it depends on the actual implementation and environment.
For example, in Windows there's something called IOCP.
If you ask me - do not bother with the lower implementation but instead use a framework such as BOOST::ASIO or ACE.
I personally like ASIO. The best thing about those frameworks is that they are usually cross-platform (nix, Windows etc').
So, my answer is a bit broad but I think it's to the best that you take these facts into consideration before diving into code/manuals/implementation.
Good luck!
Well, what you have read is wrong. Many powerful single-threaded applications have been written with non-blocking sockets and high-performance I/O demultiplexers like epoll(4) and kqueue(2). Their advantage is that you setup your wait events upfront, so the kernel does not have to copy ton of file descriptors and [re-]setup lots of stuff on each poll.
Then there are advantages to threading if your primary goal is throughput, and not latency.
Check out this great overview of available techniques: The C10K problem.
The "ideal way to handle many sockets" is not always - as Poni seems to believe - to "have a thread pool."
What does "ideal" pertain to? Is it ease of programming? Best performance?
Since he recommends not bothering "with the lower implementation" and "use a framework such as BOOST::ASIO or ACE" I guess he means ease of programming.
Had he had a performance angle on Windows he would have recommended "something called IOCPs." IOCPs are "IO Control Ports" which will allow implementation of super-fast IO-applications using just a handful of threads (one per available core is recommended). IOCP applications run circles around any thread-pool equivalent which he would have known if he'd ever written code using them. IOCPs are not used alongside thread pools but instead of them.
There is no IOCP equivalent in Linux.
Using a framework on Windows may result in a faster "time to market" product but the performance will be far from what it might have been had a pure IOCP implementation been chosen.
The performance difference is such that OS-specific code implementations should be considered. If a generic solution is chosen anyway, at least performance would "not have been given away accidentally."
For some time now I have been googling a lot to get to know about the various ways to acheive asynchronous programming/behavior on nix machines and ( as known earlier to me ) got confirmed on the fact that there is still no TRULY async pattern (concurrency using single thread) for Linux as available for Windows(IOCP).
Below are the few alternatives present for linux:
select/poll/epoll :: Cannot be done using single thread as epoll is still blocking call. Also the monitored file descriptors must be opened in non-blocking mode.
libaio:: What I have come to know about is that its implementation sucks and its still notification based instead of being completion based as with windows I/O completion ports.
Boost ASIO :: It uses epoll under linux and thus not a true async pattern as it spawns thread which are completely abstracted from user code to acheive the proactor design pattern
libevent :: Any reason to go for it if I prefer ASIO?
Now Here comes the questions :)
What would be the best design pattern for writing fast scalable network server using epoll (ofcourse, will have to use threads here :( )
I had read somewhere that "only sockets can be opened in non-blocking mode" hence epoll supports only sockets and hence cannot be used for disk I/O.
How true is the above statement and why async programming cannot be done on disk I/O using epoll ?
Boost ASIO uses one big lock around epoll call. I didnt actually understand what can be its implications and how to overcome it using asio itself. Similar question
How can I modify ASIO pattern to work with disk files? Is there any recommended design pattern ?
Hope somebody will able to answer all the questions with nice explanations also. Any link to source where the implementation details of epoll and AIO design patterns are exaplained is also appreciated.
Boost ASIO :: It uses epoll under linux and thus not a true async
pattern as it spawns thread which are completely abstracted from user
code to acheive the proactor design pattern
This is not correct. The Asio library uses epoll() by default on most recent Linux kernel versions. however, threads invoking io_service::run() will invoke callback handlers as needed. There is only one place in the Asio library that a thread is used to emulate an asynchronous interface, it is well described in the documentation:
An additional thread per io_service is used to emulate asynchronous
host resolution. This thread is created on the first call to either
ip::tcp::resolver::async_resolve() or
ip::udp::resolver::async_resolve().
This does not make the library "not a true async pattern" as you claim, in fact its name would disagree with you by definition.
1) What would be the best design pattern for writing fast scalable network server using epoll (of course, will have to use threads here :(
)
I suggest using Boost Asio, it uses the proactor design pattern.
3) Boost ASIO uses one big lock around epoll call. I didnt actually
understand what can be its implications and how to overcome it using
asio itself
The epoll reactor uses a mutex to dispatch handlers, though in practice this is not a big concern for most applications. There are application specific ways to mitigate this behavior, such as an io_service per CPU to exploit data locality. See my answer to a similar question on this topic. It is also discussed on the Asio mailing list frequently.
4) How can I modify ASIO pattern to work with disk files? Is there any
recommended design pattern?
The Asio library does not natively support file I/O as you noted. There have been several attempts to add it to the library, I'd suggest discussing on the mailing list.
First of all:
got confirmed on the fact that there is still no TRULY async pattern (concurrency using single thread) for Linux as available for Windows(IOCP).
You probably has a small misconception, asynchronous can be build on top of "polling" api.
More then that "reactor" (epoll-like) API is more powerful then "proactor" API (IOCP) as
the second can be implemented in terms of the first one (but not the other way around).
Also some operations that are "truly" asynchronous for example like disk I/O, some some other tools can be with combination of signals and Linux specific signalfd can provide full coverage of some other cases.
Bottom line. epoll is truly asynchronous I/O
I'm programming a network protocol over UDP, using C/C++ in Linux. The protocol must provide reliability, so I'm going to simulate something like TCP retransmission over UDP.
This can be done using pthreads or fork, but I believe that's an overkill and consumes a lot of system resources. A better approach is to exploit a scheduler.
I probably can't use Linux internal scheduler, since I'm programming in user space. Are there standard C/C++ libraries to accomplish this? How about 3rd party libraries?
Edit: Some people asked why I'm doing this. Why not use the TCP instead?
The answer is, since I'm implementing a tunneling protocol. If someone tunnels TCP over TCP, the efficiency will drop considerably. Here's more info Why TCP Over TCP Is A Bad Idea.
The "scheduler" you're after is called "select", and it's a user-space call available in linux. Type "man 2 select" to read the help page for how to use it.
If you need a timeout, just call select() with a timeout value. The select call will return either when new data has arrived, or a timeout has expired. You can then do retransmissions if there was a timeout.
Here's a sample of how to accomplish asynchronous coroutines with Boost. Boost manages the overhead of creating a thread to run the coroutine in this case so that you don't need to. If you would like the kernel to manage your interrupts, you can use alarm & setitimer, but they're very limited in what they can do.
Any solution will include threads, forks, or some variant of them at some level, unless you synchronously manage the transmission in the main thread using something like select().
Not clear what exactly you are trying to schedule. You can use libevent for efficient and somewhat portable interface. This is basically similar to Matthew's suggestion of using select, but using the most efficient interface (which select is not) on FreeBSD, Linux and MacOS X (actually their page now claims Windows support as well but I'm not too familiar with that).
This will give ability to do non-blocking event-driven network calls. It will not solve the scheduling part. DOing it in a separate thread is not going to hurt your performance. I think running a pthread per connection is not the best approach, but having a single scheduling thread and some worker threads dealing with the network events and maybe some non-trivial processing usually works well.