I have this piece of code as part of a socks5 proxy server implementation. This is the part from where once the server established communication sockets with proxy client (in code - socket_) and destination server (in code clientSock_) it takes data send on a socket and exchanges it with data sent on the other socket.
I specify that this exchange happens already in a thread spawn by the server for a proxy client.
std::size_t readable = 0;
boost::asio::socket_base::bytes_readable command1(true);
boost::asio::socket_base::bytes_readable command2(true);
try
{
while (1)
{
socket_->io_control(command1);
clientSock_->io_control(command2);
if ((readable = command1.get()) > 0)
{
transf = ba::read(*socket_, ba::buffer(data_,readable));
ba::write(*clientSock_, ba::buffer(data_,transf));
boost::this_thread::sleep(boost::posix_time::milliseconds(500));
}
if ((readable = command2.get()) > 0)
{
transf = ba::read(*clientSock_, ba::buffer(data_,readable));
ba::write(*socket_, ba::buffer(data_,transf));
boost::this_thread::sleep(boost::posix_time::milliseconds(500));
}
}
}
catch (std::exception& ex)
{
std::cerr << "Exception in thread while exchanging: " << ex.what() << "\n";
return;
}
The problem here is that I have very high CPU in the loop. Also I am not sure if here the way to know if one of the parts has closed the socket is to catch boost socket exception -> and end the data exchange.
The problem ca be solved by using asynchronous write/read functions. Basically use async_read_some() or async_write() - or other async functions in these categories. Also in order for async processing to work one must call io_service.run() after at least one async function was called - that will dispatch completion handler for async processing.
Related
I am writing a plugin that interfaces with a desktop application through a ZeroMQ REQ/REP request-reply communication archetype. I can currently receive a request, but the application seemingly crashes if a reply is not sent quick enough.
I receive the request on a spawned thread and put it in a queue. This queue is processed in another thread, in which the processing function is invoked by the application periodically.
The message is correctly being received and processed, but the response cannot be sent until the next iteration of the function, as I cannot get the data from the application until then.
When this function is conditioned to send the response on the next iteration, the application will crash. However, if I send fake data as the response soon after receiving the request, in the first iteration, the application will not crash.
Constructing the socket
zmq::socket_t socket(m_context, ZMQ_REP);
socket.bind("tcp://*:" + std::to_string(port));
Receiving the message in the spawned thread
void ZMQReceiverV2::receiveRequests() {
nInfo(*m_logger) << "Preparing to receive requests";
while (m_isReceiving) {
zmq::message_t zmq_msg;
bool ok = m_respSocket.recv(&zmq_msg, ZMQ_NOBLOCK);
if (ok) {
// msg_str will be a binary string
std::string msg_str;
msg_str.assign(static_cast<char *>(zmq_msg.data()), zmq_msg.size());
nInfo(*m_logger) << "Received the message: " << msg_str;
std::pair<std::string, std::string> pair("", msg_str);
// adding to message queue
m_mutex.lock();
m_messages.push(pair);
m_mutex.unlock();
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
nInfo(*m_logger) << "Done receiving requests";
}
Processing function on seperate thread
void ZMQReceiverV2::exportFrameAvailable()
// checking messages
// if the queue is not empty
m_mutex.lock();
if (!m_messages.empty()) {
nInfo(*m_logger) << "Reading message in queue";
smart_target::SMARTTargetCreateRequest id_msg;
std::pair<std::string, std::string> pair = m_messages.front();
std::string topic = pair.first;
std::string msg_str = pair.second;
processMsg(msg_str);
// removing just read message
m_messages.pop();
//m_respSocket.send(zmq::message_t()); wont crash if I reply here in this invocation
}
m_mutex.unlock();
// sending back the ID that has just been made, for it to be mapped
if (timeToSendReply()) {
sendReply(); // will crash, if I wait for this to be exectued on next invocation
}
}
My research shows that there is no time limit for the response to be sent, so this, seeming to be, timing issue, is strange.
Is there something that I am missing that will let me send the response on the second iteration of the processing function?
Revision 1:
I have edited my code, so that the responding socket only ever exists on one thread. Since I need to get information from the processing function to send, I created another queue, which is checked in the revised the function running on its own thread.
void ZMQReceiverV2::receiveRequests() {
zmq::socket_t socket = setupBindSocket(ZMQ_REP, 5557, "responder");
nInfo(*m_logger) << "Preparing to receive requests";
while (m_isReceiving) {
zmq::message_t zmq_msg;
bool ok = socket.recv(&zmq_msg, ZMQ_NOBLOCK);
if (ok) {
// does not crash if I call send helper here
// msg_str will be a binary string
std::string msg_str;
msg_str.assign(static_cast<char *>(zmq_msg.data()), zmq_msg.size());
NLogger::nInfo(*m_logger) << "Received the message: " << msg_str;
std::pair<std::string, std::string> pair("", msg_str);
// adding to message queue
m_mutex.lock();
m_messages.push(pair);
m_mutex.unlock();
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
if (!sendQueue.empty()) {
sendEntityCreationMessage(socket, sendQueue.front());
sendQueue.pop();
}
}
nInfo(*m_logger) << "Done receiving requests";
socket.close();
}
The function sendEntityCreationMessage() is a helper function that ultimately calls socket.send().
void ZMQReceiverV2::sendEntityCreationMessage(zmq::socket_t &socket, NUniqueID id) {
socket.send(zmq::message_t());
}
This code seems to be following the thread safety guidelines for sockets. Any suggestions?
Q : "Is there something that I am missing"
Yes,the ZeroMQ evangelisation, called a Zen-of-Zero, since ever promotes never try to share a Socket-instance, never try to block and never expect the world to act as one wishes.
This said, avoid touching the same Socket-instance from any non-local thread, except the one that has instantiated and owns the socket.
Last, but not least, the REQ/REP-Scalable Formal Communication Pattern Archetype is prone to fall into a deadlock, as a mandatory two-step dance must be obeyed - where one must keep the alternating sequence of calling .send()-.recv()-.send()-.recv()-.send()-...-methods, otherwise the principally distributed-system tandem of Finite State Automata (FSA) will unsalvageably end up in a mutual self-deadlock state of the dFSA.
In case one is planning to professionally build on ZeroMQ, the best next step is to re-read the fabulous Pieter HINTJENS' book "Code Connected: Volume 1". A piece of a hard read, yet definitely worth one's time, sweat, tears & efforts put in.
I have a TCP-server with multiple clients/sessions. Each session has its own thread for receiving data from the client, but there is only one thread ("writeThread") to respond to all clients.
Now there is the problem, if a client closes the connection during the "writeThread" is writing to this socket it takes multiple seconds until the write operation notices that the connection is closed remotely. Sometimes its not notices at all, just when I send a signal for an installed sighandler manually, the application will detect it and break the write operation.
The time is measuared between Logger::trace("start write"); and Logger::trace("remote term, closed socket ");
Despite the fact that this may not the best design, is there a possibility to detect the closed connection immediately, or do I really have to redesign?
bool myWrite(UINT8 *pu8_buffer, UINT32 u32_size)
{
bool b_success = false;
try
{
Logger::trace("start write");
b_success = (u32_size == boost::asio::write(_x_socket, boost::asio::buffer(pu8_buffer, u32_size)))
}
catch (boost::system::system_error &er)
{
if (er.code() == boost::asio::error::eof ||
er.code() == boost::asio::error::connection_reset)
{
boost::system::error_code x_er;
_x_socket.close(x_er);
if (!x_er)
{
Logger::trace("remote term, closed socket ");
}
else
{
Logger::err("remote term, closed socket failed");
}
}
}
catch(std::exception &ex)
{
Logger::err("write exception\n\t",ex.what());
}
catch(...)
{
Logger::err("write unknown exception",(uint32_t)this);
}
return b_success;
}
If you use asychronous write operations, it is possible to multiplex writes on the same thread without one blocking the other. You can even do the same for reads.
Just to close this question I will summarize "Richard Critten"s comments.
The issue was that there was no graceful disconnect from the client connected to my server. If there is a proper disconnect the write function will break immediately. To avoid long term or infinite blocking of the write operation its possible to configure a timeout for how long a write operation can take before reporting an error. This timeout can be set with the SO_SNDTIMEO socket option. http://man7.org/linux/man-pages/man7/socket.7.html
I have a server method that waits for new incoming TCP connections, for each connection I'm creating two threads (detached) for handling various tasks.
void MyClass::startServer(boost::asio::io_service& io_service, unsigned short port) {
tcp::acceptor TCPAcceptor(io_service, tcp::endpoint(tcp::v4(), port));
bool UARTToWiFiGatewayStarted = false;
for (;;) {
auto socket(std::shared_ptr<tcp::socket>(new tcp::socket(io_service)));
/*!
* Accept a new connected WiFi client.
*/
TCPAcceptor.accept(*socket);
socket->set_option( tcp::no_delay( true ) );
MyClass::enableCommunicationSession();
// start one worker thread.
std::thread(WiFiToUARTWorkerSession, socket, this->LINport, this->LINbaud).detach();
// only if this is the first connected client:
if(false == UARTToWiFiGatewayStarted) {
std::thread(UARTToWifiWorkerSession, socket, this->UARTport, this->UARTbaud).detach();
UARTToWiFiGatewayStarted = true;
}
}
}
This works fine for starting the communication, but the problem appears when the client disconnects and connects again (or at least tries to connect again).
When the current client disconnects, I stop the communication (by stopping the internal infinite loops from both functions, then they'll return).
void Gateway::WiFiToUARTWorkerSession(std::shared_ptr<tcp::socket> socket, ...) {
/*!
* various code here...
*/
try {
while(true == MyClass::communicationSessionStatus) {
/*!
* Buffer used for storing the UART-incoming data.
*/
unsigned char WiFiDataBuffer[max_incoming_wifi_data_length];
boost::system::error_code error;
/*!
* Read the WiFi-available data.
*/
size_t length = socket->read_some(boost::asio::buffer(WiFiDataBuffer), error);
/*!
* Handle possible read errors.
*/
if (error == boost::asio::error::eof) {
break; // Connection closed cleanly by peer.
}
else if (error) {
// this will cause the infinite loops from the both worker functions to stop, and when they stop the functions will return.
MyClass::disableCommunicationSession();
sleep(1);
throw boost::system::system_error(error); // Some other error.
}
uart->write(WiFiDataBuffer, length);
}
}
catch (std::exception &exception) {
std::cerr << "[APP::exception] Exception in thread: " << exception.what() << std::endl;
}
}
I expect that when I reconnect the communication should work again (the MyClass::startServer(...) will create and detach again two worker threads that will do the same things.
The problem is that when I connect the second time I get:
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'
what(): write: Broken pipe
From what I found about this error it seems that the server (this application) sends something via TCP to a client that was disconnected.
What I'm doing wrong?
How can I solve this problem?
A read of length 0 with no error is also an indication of eof. The boost::asio::error::eof error code is normally more useful when you're checking the result of a composed operation.
When this error condition is missed, the code as presented will call write on a socket which has now been shutdown. You have used the form of write which does not take a reference to an error_code. This form will throw if there is an error. There will be an error. The read has failed.
Im building a TCP client using Boost::asio Libs. My program has a write() thread that sends a command to the server
write(*_socket,boost::asio::buffer("sspi l1\n\n",sizeof("sspi l1\n\n")));
Then a read thread is started that reads from the buffer all the time, as there can be messages broadcasted from the server due to any other client
void TCP_IP_Connection::readTCP()
{
size_t l=0;
this->len=0;
boost::system::error_code error;
try
{//loop reading all values from router
while(1)
{
//wait for reply??
l=_socket->read_some(boost::asio::buffer(this->reply,sizeof(this->reply)),error);
if(error)
throw boost::system::system_error(error);
if(l>0)
{
this->dataProcess(l);
}
else
boost::this_thread::sleep(boost::posix_time::milliseconds(5000));
_io.run();
if(error==boost::asio::error::eof) //connection closed by router
std::cout<<"connection closed by router";
_io.reset();
}
}
catch (std::exception& e)
{
std::cerr << e.what() << std::endl;
}
}
This thread runs al time in a while(1) loop and is supposed to sleep when the received data length is less than zero. It reads all the data and calls the data parser function. After that the write thread is used to send another command, with read thread running. But instead of the required response the server sends back
? ""
ERROR: Unknown command
I tried using the wireshark. I can see the command being send properly. What can be mistake I'm doing here?
sizeof("sspi l1\n\n") returns 10, but I can only count 9 characters in that string.
Try this instead:
const std::string cmd("sspi l1\n\n");
write(*_socket,boost::asio::buffer(cmd, cmd.length()));
Or when you have it as a string it is enough to do
const std::string cmd("sspi l1\n\n");
write(*_socket,boost::asio::buffer(cmd));
The second argument specifies a maximum length of the string to use. But since it is a constant string, the second argument is not strictly necessary.
I use boost asio to handle a session per thread like this:
Server::Server(ba::io_service& ioService, int port): ioService_(ioService), port_(port)
{
ba::ip::tcp::acceptor acceptor(ioService_, ba::ip::tcp::endpoint(ba::ip::tcp::v4(), port_));
for (;;)
{
socket_ptr sock(new ba::ip::tcp::socket(ioService_));
acceptor.accept(*sock);
boost::thread thread(boost::bind(&Server::Session, this, sock));
}
}
void Server::Session(socket_ptr sock)
{
const int max_length = 1024;
try
{
char buffer[256] = "";
// HandleRequest() function performs async operations
if (HandleHandshake(sock, buffer))
HandleRequest(sock, buffer);
ioService_.run();
}
catch (std::exception& e)
{
std::cerr << "Exception in thread: " << e.what() << "\n";
}
std::cout << "Session thread ended \r\n"; // THIS LINE IS NEVER REACHED
}
In Server::Session() I do at some point async io using async_read_some() and async_write() functions.
All works well and in order for this to work I have to have a call to ioService_.run() inside my spawn thread otherwise Server::Session() function exits and it does not process the required io work.
The problem is that ioService_.run() called from my thread will lead for the thread not to exit at all because in the meantime other requests come to my listening server socket.
What I end up with is threads starting and processing for now sessions but never releasing resources (ending). Is it possible to use only one boost::asio::io_service when using this approach ?
I believe you are looking for run_one() or poll_one() this will allow you to have the thread either execute a ready handler (poll) or wait for a handler (run). By only handling one, you can pick how many to execute before exiting your thread. As opposed to run() which executes all the handlers until the io_service is stopped. Where as poll() would stop after it handled all the ones that are currently ready.
The way I structured handling connection here was bad.
There is quite a good video presentation about how to design your asio server bellow(made by asio creator)
Thinking Asynchronously: Designing Applications with Boost Asio