In our application we use Boost libraries (and ASIO for network communications).
Recently, we discovered that if we're sending our data from different threads via same socket, our client application is receiving garbaged data.
Small test to highlight the issue:
#include <stdio.h>
#include <boost/thread.hpp>
#include <boost/asio.hpp>
void send_routine(boost::shared_ptr<boost::asio::ip::tcp::socket> s, char c)
{
std::vector<char> data(15000, c);
data.push_back('\n');
for (int i=0; i<1000; i++)
boost::asio::write(*s, boost::asio::buffer(&data[0], data.size()));
}
int main()
{
using namespace boost::asio;
using namespace boost::asio::ip;
try {
io_service io_service;
io_service::work work(io_service);
const char* host = "localhost";
const char* service_name = "18000";
tcp::resolver resolver(io_service);
tcp::resolver::query query(tcp::v4(), host, service_name);
tcp::resolver::iterator iterator = resolver.resolve(query);
auto socket = boost::shared_ptr<tcp::socket>(new tcp::socket(io_service));
socket->connect(*iterator);
boost::thread t1(send_routine, socket, 'A');
boost::thread t2(send_routine, socket, 'B');
boost::thread t3(send_routine, socket, 'C');
t1.join();
t2.join();
t3.join();
}
catch (std::exception& e) {
printf("FAIL: %s\n", e.what());
}
return 0;
}
So, we create socket here, connect to localhost:18000 and start 3 threads which will write to the socket.
In different terminal window, I run nc -l -p 18000 | tee out.txt | sort | uniq | wc -l. I expect 3 as output, but it returns more then 100 "different strings" in the network stream (so, data is corrupted). But it works with small buffer sizes (if we'll change 15000 to 80, for example).
So, the question is: is it a correct behavior of ASIO library? And another: how to fix it? Should I use mutex inside my send_routine function (or there is another solution)?
write and async_write are not thread safe in the manner you are using them. The canonical way to approach this is to queue your messages, then write them out one at a time.
Yes there is another solution !
Strands: Use Threads Without Explicit Locking. Be care that strands only provides "atomic" access to socket for the "event handlers", of course you need to use asio "event handlers" which is not the case of your code. In other words you need to use boost::asio::async_write instead of boost::asio::write.
Well according to the documentation tcp::socket is not thread safe when shared between multiple threads.
So you either do a synchronisation like you suggested with boost::mutex or you use async write. The io_service the work for you.
You might have two problems, the threading issue could be solve for example by having one thread dedicated to writing and a queue where all threads post there response. You can also change your design to an asynchronous one and use the write_some() function and let the threading be done by the io_service::run(), which can be run by more than one thread.
Second, you might have a protocol problem, if a client expects the answers to it's questions in the same order.
hth
Torsten
Related
I am trying to create a concurrent c++ TCP server using threads. In particular I was wondering if I could use std::async to accept connections and serve each one in its own thread.
So far I have created a rough mockup but can't really tell if I am on the correct path.
void networking::TCP_Server::acceptConnection() {
std::string stringToSend{"This is a test string to be replied to"};
int new_fd = accept(listeningFD, nullptr, nullptr);
send(new_fd, stringToSend.c_str(), stringToSend.size(), 0);
sleep(3);
std::cout << ("End of thread");
}
///LISTEN FOR CONNECTIONS ON listeningFD
///CREATE A LIST OF FILE DESCRIPTORS FOR POLL fds[]
(fds[i].fd == listeningFD) {
do {
std::cout << ("New incoming connection - %d\n", new_fd);
std::async(std::launch::async, acceptConnection)
} while (new_fd != -1);
} /* End of existing connection is readable */
} /* End of loop through pollable descriptors */
I am connecting at the same time to the server with two clients and would expect for the loop to run through both new connections and create a thread for each one. As of now it is as it runs in deferred mode, one gets accepted, the other waits until the first finishes.
Any ideas?
(Pardon any mistakes in the code)
std::async returns a std::future which the code doesn't save into a variable, hence its destructor is called immediately. std::future::~future() blocks the calling thread until the future becomes ready.
You may like to use (detached) std::thread instead of std::async.
There are more scalable strategies to handle many clients. I highly recommend reading old but instructive The C10K problem.
You may also like to get familar with Asio C++ Library.
I am writing a server that accepts data from a device and processes it. Everything works fine unless there is an interruption in the network (i.e., if I unplug the Ethernet cable, then reconnect it). I'm using read_until() because the protocol that the device uses terminates the packet with a specific sequence of bytes. When the data stream is interrupted, read_until() blocks, as expected. However when the stream starts up again, it remains blocked. If I look at the data stream with Wireshark, the device continues transmitting and each packet is being ACK'ed by the network stack. But if I look at bytes_readable it is always 0. How can I detect the interruption and how to re-establish a connection to the data stream? Below is a code snippet and thanks in advance for any help you can offer. [Go easy on me, this is my first Stack Overflow question....and yes I did try to search for an answer.]
using boost::asio::ip::tcp;
boost::asio::io_service IOservice;
tcp::acceptor acceptor(IOservice, tcp::endpoint(tcp::v4(), listenPort));
tcp::socket socket(IOservice);
acceptor.accept(socket);
for (;;)
{
len = boost::asio::read_until(socket, sbuf, end);
// Process sbuf
// etc.
}
Remember, the client initiates a connection, so the only thing you need to achieve is to re-create the socket and start accepting again. I will keep the format of your snippet but I hope your real code is properly encapsulated.
using SocketType = boost::asio::ip::tcp::socket;
std::unique_ptr<SocketType> CreateSocketAndAccept(
boost::asio::io_service& io_service,
boost::asio::ip::tcp::acceptor& acceptor) {
auto socket = std::make_unique<boost::asio::ip::tcp::socket>(io_service);
boost::system::error_code ec;
acceptor.accept(*socket.get(), ec);
if (ec) {
//TODO: Add handler.
}
return socket;
}
...
auto socket = CreateSocketAndAccept(IOservice, acceptor);
for (;;) {
boost::system::error_code ec;
auto len = boost::asio::read_until(*socket.get(), sbuf, end, ec);
if (ec) // you could be more picky here of course,
// e.g. check against connection_reset, connection_aborted
socket = CreateSocketAndAccept(IOservice, acceptor);
...
}
Footnote: Should go without saying, socket needs to stay in scope.
Edit: Based on the comments bellow.
The listening socket itself does not know whether a client is silent or whether it got cut off. All operations, especially synchronous, should impose a time limit on completion. Consider setting SO_RCVTIMEO or SO_KEEPALIVE (per socket, or system wide, for more info How to use SO_KEEPALIVE option properly to detect that the client at the other end is down?).
Another option is to go async and implement a full fledged "shared" socket server (BOOST example page is a great start).
Either way, you might run into data consistency issues and be forced to deal with it, e.g. when the client detects an interrupted connection, it would resend the data. (or something more complex using higher level protocols)
If you want to stay synchronous, the way I've seen things handled is to destroy the socket when you detect an interruption. The blocking call should throw an exception that you can catch and then start accepting connections again.
for (;;)
{
try {
len = boost::asio::read_until(socket, sbuf, end);
// Process sbuf
// etc.
}
catch (const boost::system::system_error& e) {
// clean up. Start accepting new connections.
}
}
As Tom mentions in his answer, there is no difference between inactivity and ungraceful disconnection so you need an external mechanism to detect this.
If you're expecting continuous data transfer, maybe a timeout per connection on the server side is enough. A simple ping could also work. After accepting a connection, ping your client every X seconds and declare the connection dead if he doesn't answer.
What Im trying to achieve is the following:
Run radio in the receive mode and check for incoming packages. Once a second Id like a timer interrupt to execute and switch the radio into transmit mode to send a package and then return back to the receive mode.
Right now I have the following code:
int main(int argc, char** argv)
{
boost::asio::io_service io;
boost::asio::deadline_timer t(io, boost::posix_time::seconds(1));
t.async_wait(boost::bind(sendSyncToDevices, boost::asio::placeholders::error, &t));
io.run();
}
void sendSyncToDevices(const boost::system::error_code& ,
boost::asio::deadline_timer* t)
{
DoSomething();
t->expires_at(t->expires_at() + boost::posix_time::seconds(1));
t->async_wait(boost::bind(sendSyncToDevices, boost::asio::placeholders::error, t));
}
however io.run() is a blocking call. Can somebody advice how I could add additional while(1) loop that would be checking for incoming messages on the radio?
Thanks!
You may get your answer in similar post
How do I perform a nonblocking read using asio?
They have discussed a way of having the read/write call non-blocking.
Alternatively,
Doing while(1) on the app code may not be the best way to wait for an event or I/O. You can also try polling style (more of a C -style programming) using select() on those file descriptors. Select() is inherently a poll() function which serves much better than having while(1) and reading the file again and again.
Hope this helps.
Ive currently been messing around with boost trying it out. When i try to make a simple multi threaded echo server it exits when receiving with error code 3. I have looked over the documentation many times and still no luck. I know it is probably something very simple i'm overlooking. I have decent experience with winsock but i would like to learn the boost library.
here is the code thats failing i took out the
typedef boost::shared_ptr<tcp::socket> socket_ptr;
boost::asio::io_service io;
boost::array<char, 512> buf;
void startserver ( std::string host, std::string port )
{
tcp::acceptor a (io, tcp::endpoint(tcp::v4(), atoi(port.c_str())));
for(;;)
{
socket_ptr sock (new tcp::socket(io));
a.accept(*sock);
std::cout << sock->remote_endpoint() << std::endl;
boost::thread t (boost::bind(session, sock));
}
}
void session ( socket_ptr sock )
{
sock->send(boost::asio::buffer("welcome"),0,er);
size_t len;
for(;;)
{
len = sock->receive(boost::asio::buffer(buf));
sock->send(boost::asio::buffer(buf,len),0,er);
}
}
I can connect to it fine with netcat and it receives the welcome message but right when it goes to receive it crashs. Ive tried catching an error using boost::system::error_code on each one but nothing was returned
There are too many issues. Check asio documentation for correct examples. Some of issues:
Creating boost::thread object t and then immediately exit scope. This deattaches thread and it not controllable now; or, as mentioned Joachim Pileborg it can terminate (im not very familiar with boost::threads, so correct me if i wrong).
Right after this you starting new acceptor. You should hold only 1 acceptor per listening port.
No point to create thread for this at all, it is ASIO, use async ;)
receive does not wait data, it just fetch packet data ASIO already had (what is not true in this case)
Check examples at boost site, i think your case is blocking tcp echo server
It's most likely because the thread goes out of scope. From the manual page of the boost::thread destructor:
If the thread is joinable calls to std::terminate. Destroys *this.
This means that when the thread is started it might run for a little while before the thread in startserver gets control again and the thread object is destructed and your thread is terminated.
I wanted to create a multi-threaded socket server using C++11 and standard linux C-Librarys.
The easiest way doing this would be opening a new thread for each incoming connection, but there must be an other way, because Apache isn't doing this. As far as I know Apache handles more than one connection in a Thread. How to realise such a system?
I thought of creating one thread always listening for new clients and assigning this new client to a thread. But if all threads are excecuting an "select()" currently, having an infinite timeout and none of the already assigned client is doing anything, this could take a while for the client to be useable.
So the "select()" needs a timeout. Setting the timeout to 0.5ms would be nice, but I guess the workload could rise too much, couldn't it?
Can someone of you tell me how you would realise such a system, handling more than one client for each thread?
PS: Hope my English is well enough for you to understand what I mean ;)
The standard method to multiplex multiple requests onto a single thread is to use the Reactor pattern. A central object (typically called a SelectServer, SocketServer, or IOService), monitors all the sockets from running requests and issues callbacks when the sockets are ready to continue reading or writing.
As others have stated, rolling your own is probably a bad idea. Handling timeouts, errors, and cross platform compatibility (e.g. epoll for linux, kqueue for bsd, iocp for windows) is tricky. Use boost::asio or libevent for production systems.
Here is a skeleton SelectServer (compiles but not tested) to give you an idea:
#include <sys/select.h>
#include <functional>
#include <map>
class SelectServer {
public:
enum ReadyType {
READABLE = 0,
WRITABLE = 1
};
void CallWhenReady(ReadyType type, int fd, std::function<void()> closure) {
SocketHolder holder;
holder.fd = fd;
holder.type = type;
holder.closure = closure;
socket_map_[fd] = holder;
}
void Run() {
fd_set read_fds;
fd_set write_fds;
while (1) {
if (socket_map_.empty()) break;
int max_fd = -1;
FD_ZERO(&read_fds);
FD_ZERO(&write_fds);
for (const auto& pr : socket_map_) {
if (pr.second.type == READABLE) {
FD_SET(pr.second.fd, &read_fds);
} else {
FD_SET(pr.second.fd, &write_fds);
}
if (pr.second.fd > max_fd) max_fd = pr.second.fd;
}
int ret_val = select(max_fd + 1, &read_fds, &write_fds, 0, 0);
if (ret_val <= 0) {
// TODO: Handle error.
break;
} else {
for (auto it = socket_map_.begin(); it != socket_map_.end(); ) {
if (FD_ISSET(it->first, &read_fds) ||
FD_ISSET(it->first, &write_fds)) {
it->second.closure();
socket_map_.erase(it++);
} else {
++it;
}
}
}
}
}
private:
struct SocketHolder {
int fd;
ReadyType type;
std::function<void()> closure;
};
std::map<int, SocketHolder> socket_map_;
};
First off, have a look at using poll() instead of select(): it works better when you have large number of file descriptors used from different threads.
To get threads currently waiting in I/O out of waiting I'm aware of two methods:
You can send a suitable signal to the thread using pthread_kill(). The call to poll() fails and errno is set to EINTR.
Some systems allow a file descriptor to be obtained from a thread control device. poll()ing the corresponding file descriptor for input succeeds when the thread control device is signalled. See, e.g., Can we obtain a file descriptor for a semaphore or condition variable?.
This is not a trivial task.
In order to achieve that, you need to maintain a list of all opened sockets (the server socket and the sockets to current clients). You then use the select() function to which you can give a list of sockets (file descriptors). With correct parameters, select() will wait until any event happen on one of the sockets.
You then must find the socket(s) which caused select() to exit and process the event(s). For the server socket, it can be a new client. For client sockets, it can be requests, termination notification, etc.
Regarding what you say in your question, I think you are not understanding the select() API very well. It is OK to have concurrent select() calls in different threads, as long as they are not waiting on the same sockets. Then if the clients are not doing anything, it doesn't prevent the server select() from working and accepting new clients.
You only need to give select() a timeout if you want to be able to do things even if clients are not doing anything. For example, you may have a timer to send periodic infos to the clients. You then give select a timeout corresponding to you first timer to expire, and process the expired timer when select() returns (along with any other concurrent events).
I suggest you have a long read of the select manpage.