I am trying to use this Windows IOCP sample code as a starting point in my own IOCP server development.
There is a structure _PER_IO_CONTEXT in IocpServer.h.
//
// data to be associated for every I/O operation on a socket
//
typedef struct _PER_IO_CONTEXT {
WSAOVERLAPPED Overlapped;
char Buffer[MAX_BUFF_SIZE];
WSABUF wsabuf;
int nTotalBytes;
int nSentBytes;
IO_OPERATION IOOperation;
SOCKET SocketAccept;
struct _PER_IO_CONTEXT *pIOContextForward;
} PER_IO_CONTEXT, *PPER_IO_CONTEXT;
It's used in another structure _PER_SOCKET_CONTEXT.
//
// data to be associated with every socket added to the IOCP
//
typedef struct _PER_SOCKET_CONTEXT {
SOCKET Socket;
LPFN_ACCEPTEX fnAcceptEx;
//
//linked list for all outstanding i/o on the socket
//
PPER_IO_CONTEXT pIOContext;
struct _PER_SOCKET_CONTEXT *pCtxtBack;
struct _PER_SOCKET_CONTEXT *pCtxtForward;
} PER_SOCKET_CONTEXT, *PPER_SOCKET_CONTEXT;
From comments we can guess that pIOContext could be used as linked list, pIOContextForward member serves for this purpose.
And it is even used during resources cleanup in IocpServer.Cpp:
//
// Free all i/o context structures per socket
//
pTempIO = (PPER_IO_CONTEXT)(lpPerSocketContext->pIOContext);
do {
pNextIO = (PPER_IO_CONTEXT)(pTempIO->pIOContextForward);
if( pTempIO ) {
//
//The overlapped structure is safe to free when only the posted i/o has
//completed. Here we only need to test those posted but not yet received
//by PQCS in the shutdown process.
//
if( g_bEndServer )
while( !HasOverlappedIoCompleted((LPOVERLAPPED)pTempIO) ) Sleep(0);
xfree(pTempIO);
pTempIO = NULL;
}
pTempIO = pNextIO;
} while( pNextIO );
But pIOContextForward member is never set to anything except NULL.
May be pIOContextForward implicitly set during operations on overlapped structures?
May be this member was assumed to be used, but code is not complete?
I want to understand how this code will handle multiple asynchronous tasks on one socket and it seams that pIOContextForward should be used to implement such functionality.
So my question is how pIOContextForward member is assigned with it's corresponding value?
And if this code is not complete, how can I elaborate it?
Related
Below I add sockets to epoll and set an application-context index within epoll_event.data.u32.
When receiving packets, recv() requires the socket file descriptor. In all the examples events[i].data.fd is used.
However, events[i].data.fd and events[i].data.u32 are in a union, so how do I also access my user-context index events[i].data.u32? It looks like it is overwritten with the socket file descriptor?
// Initially
int epollFd = epoll_create1(0);
// Adding each socket, along with a user-context index for callbacks
struct epoll_event event;
event.events = EPOLLIN;
event.data.u32 = callbackIndex; // Here is the user-defined index
int sock = createSocket(port, address);
assert(epoll_ctl(epollFd, EPOLL_CTL_ADD, sock, &event));
// Later when receiving packets
struct epoll_event events[MAX_EVENTS];
while (true)
{
int event_count = epoll_wait(epollFd, events, MAX_EVENTS, 30000);
for (i = 0; i < event_count; i++)
{
int n = recv(events[i].data.fd, &buffer[0], sizeof(buffer), flags);
// How do I access the user-context index I set when adding the socket to epoll?
}
}
You tell epoll_ctl() which socket descriptor you want to listen for events for, and provide an epoll_event struct to associate with that listen operation.
Whenever epoll_wait() detects a registered event on a socket, it gives you back only the epoll_event struct that you had provided for that event, exactly as you had provided it. It does not tell you which socket triggered the event.
So, if you want to discover the socket, you have to either:
store the socket descriptor itself in the epoll_event, but then you can't use any other user-defined data.
store the socket descriptor somewhere else (ie, in an array, an object pool, etc) and then put identifying information needed to get back to the socket descriptor as user-defined data in the epoll_event (ie, array index, object pointer, etc).
Whatever you put in the epoll_event when calling epoll_ctl() is what you will get back from epoll_wait(). No more, no less.
Design of epoll is beautifully simple. The role of epoll_data_t is to provide lightweight mapping rather than storage. Notice that it has a void* ptr member, which allows you to map from fd (passed to epoll_ctl) to anything.
In your particular case, you could allocate a struct Context { int fd; uint32_t index; /*...*/ }; on the heap and point to that structure on EPOLL_CTL_ADD. You would have to also deallocate it after calling EPOLL_CTL_DEL by some object (e.g. container) which owns that context.
Since you are using C++, you could store a pointer to an abstract EventListener base class, reinterpret_cast from void* after epoll_wait to that class and dispatch event to an arbitrary derived handler.
I am implementing a connection multiplexer - class, which wraps a single connection in order to provide an ability to create so-called Stream-s over it. There can be dozens of such streams over one physical connection.
Messages sent over that connection are defined by a protocol and can be service ones (congestion control, etc), which are never seen by the clients, and data ones - they contain some data for the streams, for which one - defined in the header of the corresponding message.
I have encountered a problem when implementing a method read for a Stream. It must be blocking, but asynchronous, so that it returns some value - data read or error happened - but the request itself must be is some kind of async queue.
To implement asynchronous network IO we have used Boost's async_read-s, async_write-s, etc with a completion token, taken from another library. So, a call to MyConnection::underlying_connection::read(size_t) is asynchronous already in the terms I described before.
One solution I have implemented is function MyConnection::processFrame(), which is reading from the connection, processing message and, if it is a data message, puts the data into the corresponding stream's buffer. The function is to be called in a while loop by the stream's read. But, in that case there can be more than one simulteneous calls to async_read, which is UB. Also, this would mean that even service messages are to wait until some stream wants to read the data, which is not appropriate as well.
Another solution I came up is using future-s, but as I checked, their methods wait/get would block the whole thread (even with defered policy or paired promise), which must be avoided too.
Below is a simplified example with only methods, which are needed to understand the question. This is current implementation, which contains bugs.
struct LowLevelConnection {
/// completion token of 3-rd part library - ufibers
yield_t yield;
/// boost::asio socket
TcpSocket socket_;
/// completely async (in one thread) method
std::vector<uint8_t> read(size_t bytes) {
std::vector<uint8_t> res;
res.reserve(bytes);
boost::asio::async_read(socket_, res, yield);
return res;
}
}
struct MyConnection {
/// header is always of that length
constexpr uint32_t kHeaderSize = 12;
/// underlying connection
LowLevelConnection connection_;
/// is running all the time the connection is up
void readLoop() {
while (connection_.isActive()) {
auto msg = connection_.read(kHeaderSize);
if (msg.type == SERVICE) { handleService(msg); return; }
// this is data message; read another part of it
auto data = connection_.read(msg.data_size);
// put the data into the stream's buffer
streams_.find(data.stream_id).buffer.put(data);
}
}
}
struct Stream {
Buffer buffer;
// also async blocking method
std::vector<uint8_t> read(uint32_t bytes) {
// in perfect scenario, this should look like this
async_wait([]() { return buffer.size() >= bytes; });
// return the subbuffer of 'bytes' size and remove them
return subbufer...
}
}
Thanks for future answers!
I am working with C++ based Networking project using winsock library.
The code is:
class NetworkCom
{
private:
SOCKADDR_IN ClientAddress;
int ClientAddressSize;
SOCKET SenderSocket;
public:
NetworkCom(SOCKET& sock)
{
ClientAddressSize = sizeof(ClientAddress);
getpeername(sock,(SOCKADDR*)&ClientAddress,&ClientAddressSize);
sock = socket(AF_INET,SOCK_STREAM,0);
SenderSocket = socket(AF_INET,SOCK_STREAM,0);
}
int SendData(char message[])
{
int val;
val = send(sock,message,sizeof(message),0); // if val <= 0 then error
return val;
}
string RecieveData()
{
string message;
char msg[100];
connect(SenderSocket,(SOCKADDR*)&ClientAddress,&ClientAddressSize);
recv(SenderSocket,msg,100,0);
message = msg;
return message;
}
~NetworkCom()
{
cout<<"Closing Connection..";
closesocket(SenderSocket);
closesocket(sock);
}
};
I am going to create a new NetworkCom Object for each new client in a new thread.
So when i do so do i need to use mutex to lock the member function of the class evertime a object in a thread is using the member function SendData and RecieveData.
If i have to use mutex....
Is there any way this can be done without mutex
As most socket APIs, Winsock does not like concurrent access to the same socket:
send should not be called on the same stream-oriented socket
concurrently from different threads, because some Winsock providers
may split a large send request into multiple transmissions, and this
may lead to unintended data interleaving from multiple concurrent send
requests on the same stream-oriented socket.
From MSDN, recv has similar constraints.
So if you are sharing the same socket between multiple threads, all access to that socket need to be synchronized. A mutex is one way of achieving this synchronisation. If you are using different sockets for the different threads though, you do not need to synchronize, as long as you are not sharing any other (mutable) data between threads.
I'm trying to write a handler class that subscribes to a message published via zeromq and buffers the last received message.
I tried doing this as follows. The method ReceivedMessage() is to be called by a wrapper application in a cyclic called function. Once it returns true, I tried to access the message using GetReceivedMessageData(). Unfortunately, it seems that the data is not saved properly in the member zmq_receivedMessage_.
I guess this is because of zmq_receivedMessage_ being initialized with fixed size and the call zmq_subscriber_.recv(&zmq_receivedMessage_) does not automatically resize it?
What would be the easiest and most robust way to this? The only way I can think of is using realloc() and memcpy() every time a new message is received. Or is there a simpler way?
#include <cstdint>
#include "zeromq_cpp/zmq.hpp"
class HandlerClass
{
public:
/// #brief Initializes a AirSimToRos class instance.
HandlerClass(std::string const& addr);
// #brief Gets the message data received via ZeroMq as pointer.
void* GetReceivedMessageData();
// #brief Gets the message size received via ZeroMq as size_t.
std::size_t GetReceivedMessageSize();
// #brief Returns true if a new, full message was received via ZeroMq, false otherwise
bool ReceivedMessage();
private:
/// #brief A ZeroMq context object encapsulating functionality dealing with the initialisation and termination.
zmq::context_t zmq_context_;
/// #brief A ZeroMq socket for subscribing to incoming messages.
zmq::socket_t zmq_subscriber_;
/// #brief A ZeroMq message that was received last. Might be empty if ReceivedMessage() never was true.
zmq::message_t zmq_receivedMessage_;
};
HandlerClass::HandlerClass(std::string const& addr)
: zmq_context_(1)
, zmq_subscriber_(zmq_context_, ZMQ_SUB)
{
zmq_subscriber_.setsockopt(ZMQ_IDENTITY, "HandlerSubscriber", 5);
zmq_subscriber_.setsockopt(ZMQ_SUBSCRIBE, "", 0);
zmq_subscriber_.setsockopt(ZMQ_RCVTIMEO, 5000);
zmq_subscriber_.connect(addr);
}
void* HandlerClass::GetReceivedMessageData()
{
return zmq_receivedMessage_.data();
}
std::size_t HandlerClass::GetReceivedMessageSize()
{
return zmq_receivedMessage_.size();
}
bool HandlerClass::ReceivedMessage()
{
int received_bytes = zmq_subscriber_.recv(&zmq_receivedMessage_);
return received_bytes > 0;
}
One way would be a redesign w/ Poller-instance + ZMQ_CONFLATE
Having zero context of the intended class use-cases, the original design seems to be a rather "mechanical" wrapper of a data-mover, not any MVP-slim-design, that can squeeze maximum of the benefits the ZeroMQ Scalable Formal Communication Archetypes Signalling/Messaging framework has already built-in.
The much smarter ( and also a ZMQ_RCV_HWM-safer ( goes beyond the scope of this topic ) ) would be not to just always mechanically read each message from the ZeroMQ Context-domain of control, unless in a real need to re-transmit such data from the HandlerClass somewhere further down the line.
Add a private instance of the Poller that would allow to redesign data-flow mechanics -- using non-destructive query, using a .poll()-method for testing a new message arrival ( having also a Real-Time / Event-Handling Loop stability control tools, not to wait longer than an ad-hoc set .poll()-method timeout ), while yet able to defer any actual data-move as late as possible, until a data indeed need to flow outside of the HandlerClass-instance, not anywhere earlier.
HandlerClass::HandlerClass(std::string const& addr)
: zmq_context_(1)
, zmq_subscriber_(zmq_context_, ZMQ_SUB)
{
zmq_subscriber_.setsockopt( ZMQ_IDENTITY, "HandlerSubscriber", 5 );
zmq_subscriber_.connect( addr );
zmq_subscriber_.setsockopt( ZMQ_SUBSCRIBE, "", 0 );
zmq_subscriber_.setsockopt( ZMQ_LINGER, 0 ); // ALWAYS, READY 4 .term()
zmq_subscriber_.setsockopt( ZMQ_CONFLATE, 1 ); // SMART
zmq_subscriber_.setsockopt( ZMQ_TOS, T ); // WORTH DEPLOY & MANAGE
zmq_subscriber_.setsockopt( ZMQ_RCVTIMEO, 5000 );
// ------------------------------------------------- // ADD Poller-instance
...
// ------------------------------------------------- // RTO
}
Nota Bene: In case the exgress flow is also made on ZeroMQ infrastructure, there are time-saving API tools for Zero-Copy message re-marshalling into another ZeroMQ socket-transport -- ( almost ) for free -- cool, isn't it?
I am currently writing a winsock server-side socket in managed C++. After creating the LPWSAOVERLAPPED object and passing it to the WSASend function, I do not see where to delete it when the operation completes nonblocking (WSASend returns SOCKET_ERROR and WSAGetLastError() returns WSA_IO_PENDING). My current solution was to create a System::Threading::WaitHandle, get the unsafe pointer to the wait handle and pass that onto hEvent under the LPWSAOVERLAPPED object. However, this is causing unnessecary object creation since I do not really care about when the send operation is completed. On the other hand, I need a LPWSAOVERLAPPED object in order to make the operation complete non-blocking. Does anyone have any better solution to solve this? Here is my current code:
void Connectivity::ConnectionInformation::SendData(unsigned char data[], const int length)
{
if (isClosed || sendError)
return;
Monitor::Enter(this->sendSyncRoot);
try
{
LPWSAOVERLAPPED overlapped = OverlappedObjectPool::GetOverlapped();
WaitHandle ^ handle = gcnew ManualResetEvent(false);
IntPtr handlePointer = handle->SafeWaitHandle->DangerousGetHandle();
sendInfo->buf = (char*)data;
sendInfo->len = length;
overlapped->Internal = 0;
overlapped->InternalHigh = 0;
overlapped->Offset = 0;
overlapped->OffsetHigh = 0;
overlapped->Pointer = 0;
overlapped->hEvent = (void*)handlePointer; //Set pointer
if (WSASend(connection, sendInfo, 1, NULL, 0, overlapped, NULL) == SOCKET_ERROR)
{
if (WSAGetLastError() == WSA_IO_PENDING)
{
ThreadPool::UnsafeRegisterWaitForSingleObject(handle, sentCallback, (IntPtr)((void*)overlapped), -1, true);
}
else
{
this->sendError = true;
//The send error bool makes sure that the close function doesn't get called
//during packet processing which could lead to a lot of null reffernce exceptions.
OverlappedObjectPool::GiveObject(overlapped);
}
}
else
{
handle->Close();
sentData((IntPtr)((void*)overlapped), false);
}
}
finally
{
Monitor::Exit(this->sendSyncRoot);
}
}
For async I/O, completion is notified either by the calling of a completion routine or by the queueing of an IOCP completion message to an IOCP completion queue. In both cases, it should be noted that the OVL struct should have the lifetime of at least the entire async operation, but can be longer if convenient:)
In the case of a completion routine, the unused hEvent parameter in the OVL can be used to transfer a pointer to an 'IOrequest' class instance that contains the data buffer/s, WSABUF array and the OVL struct as members, (and surely a pointer to the socket object for which the I/O has been issued). The OVL pointer is supplied as a parameter to the completion routine and so the hEvent can be retrieved and cast to the class type, so retrieving the complete class instance - OVL, data buffer etc. When the data has been processed, (or immediately in the completion routine the case of WSASend), and this IOrequest is eventually destroyed, (or repooled), the OVL will go with it. This sounds a bit incestuous, but works fine and does not need any nasty macro or other tricks.
A similar approach can be used with full IOCP or, alternatively, the OVL passed as the lpCompletionKey 'spare' parameter.
Oh - and you do care if the operation is completed - you need to at least check for errors.