Connect with an already created named pipe with boost - c++

I'm looking in to using third party libraries for IPC communication using named pipes on Windows. I've been using the Win32 API for this for the last few years. I was interested in replacing my implementation with a tried and true open source library.
I noticed that boost::process has an implementation of an async_pipe which would allow me to use it with boost::asio which would be really helpful for my application.
What I'm trying to do is create the named pipe on the server, which is a C# application.
Once the pipe has been created, connect to it with a client using the boost::process::async_pipe.
The problem I'm having is I don't see an API in boost::process that would allow me to connect with an already created named pipe. The constructors for async_pipe create the pipe instead of connecting to an already created pipe.
Below is the code I'm currently using in the client which is erroneously creating the pipe
boost::asio::io_context ctx;
std::vector<char> buffer( 8196, 0 );
boost::process::async_pipe pipe{ ctx, R"(\\.\pipe\TestPipe)" };
boost::asio::async_read( pipe, boost::asio::buffer( buffer ),
[ &buffer ]( const boost::system::error_code& ec, std::size_t size )
{
if ( ec )
std::cout << "Error: " << ec.message( ) << '\n';
else
{
std::string message{ std::begin( buffer ), std::begin( buffer ) + size };
std::cout << "Received message: " << message << '\n';
}
} );
ctx.run( );
I'm unsure if I can use boost::process to achieve what I want. I'm wondering if there is a way I could use CreateFileW to connect with the named pipe and then pass the HANDLE to async_pipe but I haven't found any documentation regarding that.
Question
How can I connect with an already created named pipe using boost

OK, so I was going about it the wrong way.
After reading this issue on Github Link I realized I needed to use a stream_handle instead. Note, the pipe must be opened in OVERLAPPED mode for this to work.
Creating the stream_handle
static boost::asio::windows::stream_handle OpenPipe( io_context& context )
{
constexpr const wchar_t* pipeName{ LR"(\\.\pipe\TestPipe)" };
return { context, CreateFileW( pipeName,
GENERIC_READ | GENERIC_WRITE,
0, nullptr,
OPEN_EXISTING,
FILE_FLAG_OVERLAPPED,
nullptr ) };
}
Once the stream_handle is created you can use the async functions it provides for communication.
Using the stream_handle
std::vector<char> buffer( 8196, 0 );
pipe.async_read_some( boost::asio::buffer( buffer ),
[ &buffer ]( auto ec, auto size ) { } );

Related

Adding protocol layers over socketpair() to get SSL-WebSocket (boost::beast?)

I am experimenting with boost::beast and WebSockets for a project. I would like to layer WebSockets (and ideally SSL) onto a socketpair (boost::asio::local::stream_protocol::socket). This seems like it should be easy, but I'm lost in g++ template errors.
Is there a way to get WebSockets running over an already-connected generic::stream-protocol::socket?
int
main( int argc, char **argv )
{
io_context ioctx;
local::stream_protocol::socket socket1( ioctx );
local::stream_protocol::socket socket2( ioctx );
local::connect_pair( socket1, socket2 );
boost::beast::websocket::??? ws1( socket1 );
boost::beast::websocket::??? ws2( socket2 );
cout << "Here we go..." << endl;
// queue an initial read or write on ws1 or ws2...
ioctx.run( );
// all I/O operations completed
}

ZeroMQ with NORM - address already in use error was thrown on 2nd .bind() - why?

I'm using ZeroMQ with NACK-Oriented Reliable Multicast ( NORM ) norm:// protocol. The documentation contains only a Python code, so here is my C++ code:
PUB Sender :
string sendHost = "norm://2,127.0.0.1:5556";// <NormNodeId>,<addr:port>
string tag = "MyTag";
string sentMessage = "HelloWorld";
string fullMessage = tag + sentMessage;
zmq::context_t *context = new zmq::context_t( 20 );
zmq::socket_t publisher( *context, ZMQ_PUB );
zmq_connect( publisher, sendHost.c_str() );
zmq_send( publisher,
fullMessage.c_str(),
fullMessage.size(),
0
);
SUB Receiver :
char message[256];
string receiveHost = "norm://1,127.0.0.1:5556";// <NormNodeId>,<addr:port>
string tag = "MyTag";
zmq::context_t *context = new zmq::context_t( 20 );
zmq::socket_t subscriber( *context, ZMQ_SUB );
zmq_bind( subscriber, receiveHost.c_str() );
zmq_setsockopt( subscriber, ZMQ_SUBSCRIBE, tag.c_str(), tag.size() );
zmq_recv( subscriber,
message,
256,
0
);
cout << bytesReceived << endl;
cout << message << endl;
The problem I'm facing is that according to the documentation both .bind() and .connect() are interchangeable.
In my case they both do a .bind(), which causes ZeroMQ to throw an error saying the second bind fails, due to address already in use error.
... they both do a bind, which causes ZeroMQ to throw an error saying the second bind fails
Yes, this is a correct state to fail.
The first .bind() "takes ownership" of the port and this is an exclusive role.
The interchangeability of { .bind() | .connect() } is to be understood so that it does not matter which side .bind()-s and which one .connect()-s.
Until this moment, I saw no one interpreting this property in such a manner, that both sides would try to .connect() ( a non-existent .bind()-(not)-exposed Access Point ), the less to try to .bind() an already "occupied" port ( in case of residing on the same localhost ), or to remain in a nox-et-solitudo state, for the cases that either of the .bind()-s establishes such a .connect()-ready state on both ports on different localhost-s, which both after that remain in a silent solitude ( forever ), as there is ( and will be ) no attempt to make any .connect()-ion going live and operational.
No, you need just 1 .bind(), that may since that moment handle 0+ future .connect()-requests, arriving to establish a live-channel PUB/SUB, for any respective <transport-class> protocol, including the newly added norm://.
Anyways, welcome norm:// to the Family of ZeroMQ protocols.
Confused ?
May enjoy a further 5-seconds read
about the main conceptual differences in [ ZeroMQ hierarchy in less than a five seconds ] or other posts and discussions here.

ZeroMQ socket.recv() raised a STACK_OVERFLOW exception

if use this code in .dll, a call to a socket.recv() raised an exception STACK_OVERFLOW, but when this code compiled as .exe it works.
Why?
I run a .dll-test by "C:\windows\system32\rundll32.exe myDll.dll StartUp"
void StartUp()
{
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("tcp://127.0.0.1:3456");
zmq::message_t msgIN, msgOUT("test", 4);
while (true){
socket.recv(&msgIN);
socket.send(msgOUT);
};
}
callstack :
libzmq-v120-mt-gd-4_2_2.dll!zmq::mailbox_t::recv(zmq::command_t * cmd_=0x0231f700, int timeout_=0x00000000)
libzmq-v120-mt-gd-4_2_2.dll!zmq::io_thread_t::in_event()
libzmq-v120-mt-gd-4_2_2.dll!zmq::select_t::loop()
libzmq-v120-mt-gd-4_2_2.dll!zmq::select_t::worker_routine(void * arg_=0x002f1778)
libzmq-v120-mt-gd-4_2_2.dll!thread_routine(void * arg_=0x002f17c0)
main thread callstack:
libzmq-v120-mt-gd-4_2_2.dll!zmq::signaler_t::wait(int timeout_=0xffffffff)
libzmq-v120-mt-gd-4_2_2.dll!zmq::mailbox_t::recv(zmq::command_t * cmd_=0x0019f3c0, int timeout_=0xffffffff)
libzmq-v120-mt-gd-4_2_2.dll!zmq::socket_base_t::process_commands(int timeout_, bool throttle_)
libzmq-v120-mt-gd-4_2_2.dll!zmq::socket_base_t::recv(zmq::msg_t * msg_=0x0019f628, int flags_=0x00000000)
libzmq-v120-mt-gd-4_2_2.dll!s_recvmsg(zmq::socket_base_t * s_=0x006f6c70, zmq_msg_t * msg_=0x0019f628, int flags_=0x00000000)
libzmq-v120-mt-gd-4_2_2.dll!zmq_msg_recv(zmq_msg_t * msg_=0x0019f628, void * s_=0x006f6c70, int flags_=0x00000000)
mydll.dll!zmq::socket_t::recv(zmq::message_t * msg_=0x0019f628, int flags_=0x00000000)
mydll.dll!StartUp()
Update:
this example, also crashed with the same reason. Does someone know any reasons for exception stack overflow?
zmq::context_t context(1);
zmq::socket_t socket(context, ZMQ_REP);
socket.bind("tcp://*:7712");
while (1){
Sleep(10);
}
A reverse problem-isolation MCVE:
And how did this myDll.dll-test work,
if run by C:\windows\system32\rundll32.exe myDll.dll StartUp? Post the screen outputs.
void StartUp()
{
std::cout << "INF:: ENTRY POINT ( C:\windows\system32\rundll32.exe myDll.dll StartUp )" << std::endl;
std::cout << "INF:: WILL SLEEP ( C:\windows\system32\rundll32.exe myDll.dll StartUp )" << std::endl;
Sleep( 10 );
std::cout << "INF:: SLEPT WELL ( C:\windows\system32\rundll32.exe myDll.dll StartUp )" << std::endl;
std::cout << "INF:: WILL RETURN ( C:\windows\system32\rundll32.exe myDll.dll StartUp )" << std::endl;
}
The reason of crash is SizeOfStackCommit value in OPTIONAL_HEADER rundll32 file.
It too small (0xC000), i change it to 0x100000. Now all works.
ZeroMQ objects require certain respect to work with:
there are many features under the radar, that may go wreck havoc, as you have already seen on your screen.
Best read with due care both the ZeroMQ C++ binding reference documentation plus the original ZeroMQ API ( which is often mentioned in the C++ binding either ).
Both do emphasise to never handle zmq::message_t instances directly, but via using "service"-functions ( often re-wrapped as instance methods in C++ ).
zmq::message_t messageIN,
messageOUT;
bool successFlag;
while (true){
successFlag = socket.recv( &messageIN );
assert( successFlag && "EXC: .recv( &messageIN )" );
/* The zmq_recv() function shall receive a message
from the socket referenced by the socket argument
and store it in the message referenced by the msg
argument.
Any content previously stored in msg shall be
properly deallocated.
If there are no messages available on the specified
socket the zmq_recv() function shall block
until the request can be satisfied.
*/
messageOUT.copy( messageIN );
successFlag = socket.send( messageOUT );
assert( successFlag && "EXC: .send( messageOUT )" );
/* The zmq_send() function shall queue the message
referenced by the msg argument to be sent to
the socket referenced by the socket argument.
The flags argument is a combination of the flags
defined { ZMQ_NOBLOCK, ZMQ_SNDMORE }
The zmq_msg_t structure passed to zmq_send()
is nullified during the call.
If you want to send the same message to multiple
sockets you have to copy it using (e.g.
using zmq_msg_copy() ).
A successful invocation of zmq_send()
does not indicate that the message
has been transmitted to the network,
only that it has been queued on the socket
and ØMQ has assumed responsibility for the message.
*/
};
My suspect is a reference counting, adding more and more instances, produced by a zmq::message_t message; constructor in an infinite while( true ){...}-loop, none of which has ever met it's own fair destructor. The STACK, having a physically-limited capacity and none STACK-management care inside DLL, will fail sooner or later.
zmq::message_t instances are quite an expensive toy, so a good resources-management practices ( pre-allocation, reuse, controlled destructions ) are always welcome for professional code.
Q.E.D.
Tail remarks for clarity purposes:
A bit paraphrasing Dijkstra's view on error hunting and software testing: "If I see no Error, that does not mean, there is none in the piece of code ( the less if any external functions are linked in addition to it )."
No stack allocations?
Yes, no visible ones.
ZeroMQ API puts more light into it:
"The zmq_msg_init_size() function shall allocate any resources required to store a message size bytes long and initialise the message object referenced by msg to represent the newly allocated message.
The implementation shall choose whether to store message content on the stack (small messages) or on the heap (large messages). For performance reasons zmq_msg_init_size() shall not clear the message data."
Many years, so far spent on using cross-platform distributed systems, based on ZeroMQ API since v.2.1+, has taught me lot on being careful on explicit resources control. The more once you did not develop your own language binding for the native API.
After all unsupported criticism, let's add one more citation from ZeroMQ:
This adds a view, how a proper indirect manipulation of the message_t content is done by the library C++ bindings itself, wrapped into trivial helper functions:
from zhelpers.hpp:
// Receive 0MQ string from socket and convert into string
static std::string
s_recv (zmq::socket_t & socket) {
zmq::message_t message;
socket.recv(&message);
return std::string(static_cast<char*>(message.data()), message.size());
}
// Convert string to 0MQ string and send to socket
static bool
s_send (zmq::socket_t & socket, const std::string & string) {
zmq::message_t message(string.size());
memcpy (message.data(), string.data(), string.size());
bool rc = socket.send (message);
return (rc);
}
// Sends string as 0MQ string, as multipart non-terminal
static bool
s_sendmore (zmq::socket_t & socket, const std::string & string) {
zmq::message_t message(string.size());
memcpy (message.data(), string.data(), string.size());
bool rc = socket.send (message, ZMQ_SNDMORE);
return (rc);
}

boost::asio::async_write and buffers over 65536 bytes

I have a very simple method, with the purpose of responding to an incoming message, and then closing the connection:
void respond ( const std::string message )
{
std::string str = "<?xml version=\"1.0\"?>";
Controller & controller = Controller::Singleton();
if ( auto m = handleNewMessage( _message ) )
{
auto reply = controller.FIFO( m );
str.append( reply );
}
else
str.append ( "<Error/>" );
std::size_t bytes = str.size() * sizeof( std::string::value_type );
std::cout << "Reply bytesize " << bytes << std::endl;
boost::asio::async_write(
socket_,
boost::asio::buffer( str ),
boost::bind(
&TCPConnection::handle_write,
shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
));
}
void handle_write ( const boost::system::error_code & error, size_t bytes_transferred )
{
if ( error )
{
std::cerr << "handle_write Error: " << error.message() << std::endl;
std::cerr << "handle_write Bytes sent: " << bytes_transferred << std::endl;
}
else
{
std::cerr << "handle_write Bytes sent: " << bytes_transferred << std::endl;
socket_.close();
}
}
I know the problem is that boost::asio::async_write does not complete the writing operation, because the output from the above operations is:
Reply bytesize: 354275
handle_write Bytes sent: 65536
Implying that the maximum buffer size (65536) was not enough to write the data?
Searching around Stack Overflow, I discovered that my problem is that the buffer created by the method:
boost::asio::buffer( str )
goes out of scope before the operation has a chance to finish sending all the data.
It seems like I can't use a boost::asio::mutable_buffer, but only a boost::asio::streambuf
Furthermore and more importantly, a second error complains about the actual boost::asio::async_write being passed a boost::asio::const_buffer OR boost::asio::mutable_buffer:
/usr/include/boost/asio/detail/consuming_buffers.hpp:164:5: error: no type named ‘const_iterator’ in ‘class boost::asio::mutable_buffer’
const_iterator;
^
/usr/include/boost/asio/detail/consuming_buffers.hpp:261:36: error: no type named ‘const_iterator’ in ‘class boost::asio::mutable_buffer’
typename Buffers::const_iterator begin_remainder_;
So I am left with only one choice: To use a boost::asio::streambuf
I've tried using:
boost::asio::streambuf _out_buffer;
As a class member, and then made method respond:
std::ostream os( &_out_buffer );
os << str;
boost::asio::async_write(
socket_,
_out_buffer,
boost::asio::transfer_exactly( bytes ),
boost::bind(
&TCPConnection::handle_write,
shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred
));
However, although I get no errors, not the entire data is sent!
So, I am guessing, not the entire string is written into the streambuf?
Alternatively, I would love to know what is the most elegant way to write using boost::asio::async_write, data that is larger than 65536 bytes!
Alex, you understand asio async operations wrong. Your problem is all about lifetime of buffer and socket.
The buffer has to be alive and socket opened during the whole transmition time (from asio::async_write call to handle_write callback is to be called by Asio io_service dispatcher.
To better understand how it works, consider that every time you doing some boost::asio::async_{operation} you are posting the pointer to data for operation and pointer to callback function to the job queue. And it's Asio decision when to execute your job (but of course it tries to do it as faster as possible =)). When the whole (possible big) I/O operation completes the Asio informs you using specified callback. And you can release resources then freely.
So, to get your code work you have to ensure that std::string str is still exist and _socket not closed until the handle_write callback. You can replace the stack allocated std::string str variable by some member variable in the class that agregates _socket. And move the socket_.close(); line from respond function to handle_write.
Hope, I helped you.
P.S. When you do boost::asio::buffer( str ), you don't copy content of the string but just create thin wpapper above data of string.
The code:
_out_buffer( static_cast<void*>( &str.front() ), bytes );
Is only valid when initializing _out_buffer, i.e. Before the body of your class's constructor begins.
That code equivalent to
_out_buffer.operator()( static_cast<void*>(&str.front()), bytes )
Of course there is no such operator in class mutable_buffer, and that's what the compiler is complaining about.
I think the simplest thing to do (but not the best), is to change that line to:
_out_buffer = boost::asio::mutable_buffer(
static_cast<void*>( &str.front() ),
bytes
);

Boost::asio::async_write doesn't seem to free memory

I have a cluster program using boost asio to make the network part.
I'm using async_write function to write the message from the server to the client :
boost::asio::async_write( *m_Socket,
boost::asio::buffer( iData, iSize ),
boost::bind(
&MyObject::handle_write, this,
boost::asio::placeholders::error ) );
My handle_write method :
void
MyObject::handle_write( const boost::system::error_code& error )
{
std::cout << "handle_write" << std::endl;
if (error)
{
std::cout << "Write error !" << std::endl;
m_Server->RemoveSession(this);
}
}
It seems to work well. When I use memory leak detector program, there is no leak at all.
But, my program is supposed to run many days without interuption and during test, it appears that I don't have anough memory... After some inspection, I found that my program was allocating around 0.3Mo by seconds. And with a memory validor I found that it was into boost::asio::async_write...
I checked the documentation and I think I use it in the correct way... Am I missing something ?
EDIT 1:
That is how I call the function who call async_write itself :
NetworkMessage* msg = new NetworkMessage;
sprintf(msg->Body(), "%s", iData );
m_BytesCount += msg->Length();
uint32 nbSessions = m_Sessions.size();
// Send to all clients
for( uint32 i=0; i < nbSessions; i++)
{
m_Sessions[i]->Write( msg->Data(), msg->Length() );
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
}
delete msg;
msg->Data is the data passed to async_write.