I'm writing a packet encryption for a friends gameserver.
Client is using ws2_32 recv/send but server uses WSARecv/WSASend.
I've managed to encrypt/decrypt send/recv/WSASend, but WSARecv seems impossible.
I'm using the same method as on recv, but it doesn't seem to work.
int WINAPI MyWSARecv(SOCKET socket, LPWSABUF lpBuffers, DWORD dwBufferCount, LPDWORD lpNumberOfBytesRecvd, LPDWORD lpFlags,LPWSAOVERLAPPED lpOverlapped, LPWSAOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine)
{
LPWSABUF buffers = lpBuffers;
int ret = pWSARecv(socket, buffers, dwBufferCount, lpNumberOfBytesRecvd, lpFlags, lpOverlapped, lpCompletionRoutine);
cryptPacket(buffers->buf, buffers->len);
lpBuffers = buffers;
return ret;
}
Any ideas would be appreciated.
A handful of things to consider.
You are not actually checking the return value from pWASRecv before calling your cryptPacket function. You need to start with that fix first before any other assumptions can be made about how the detour hooks work with the socket code. If the call indicates an error, it's probably not well defined what buffers->len is going to be or what is going to be in those buffers.
Also, you are likely assuming that the socket was initialized to be synchronous. If the socket was initialized for overlapped I/O, then the lpOverlapped and lpCompletionRoutine parameters become very relevant. You may need to hook the completion routine or WSAGetOverlappedResult to actually intercept the socket data.
Finally, may I suggest another approach. Rather than trying to "detour" the socket API calls, run a "proxy socket". That is, when the server socket is created (via a call to "socket"), you change it to listen on a different port. Then you create a separate listen socket that listens on the original port. When an incoming connection comes into your socket, you make a separate "proxy" connection to the actual port the game server is listening on. You can have a dedicated thread that just calls send and recv on the client socket that encrypts/decrypts data as needed.
Related
In the MSDN doc for the lpOverlapped parameter of GetQueuedCompletionStatus it is said that the application can prevent completion port notification by setting the low-order bit of the hEvent member of the OVERLAPPED structure. But is it possible to resume the notifications after they were stopped?
I need to use this for monitoring network folders for changes:
When GetQueuedCompletionStatus returns FALSE and GetLastError() returns ERROR_NETNAME_DELETED, I do this (works):
di->Overlapped.hEvent = CreateEvent( NULL, FALSE, FALSE, di->lpszDirName );
reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) |= 0x1;
And when the network problem was resolved, I tried to do the reverse operation - but it DID NOT work:
reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) &= ~(0x1);
(It will be good if the solution be compatible with Windows 7)
first of all completion port notification can not be "suspended" or "resumed"
Even if you have passed the function a file handle associated with a
completion port and a valid OVERLAPPED structure, an application can
prevent completion port notification. This is done by specifying a
valid event handle for the hEvent member of the OVERLAPPED structure,
and setting its low-order bit. A valid event handle whose low-order
bit is set keeps I/O completion from being queued to the completion
port.
this mean the next - when we call some win32 I/O api (api which take pointer to OVERLAPPED as in/out parameter, such as ReadFile, ReadDirectoryChangesW, LockFileEx etc) and file handle (passed to this api) associated with a completion port - despite this we can prevent completion port notification for this call by event handle with low-order bit. this is for only concrete api call and not affect any another api calls. and all this unrelated to GetQueuedCompletionStatus
(strictly said we can simply pass 1 in place hEvent too. but in this case question - how we get notify about I/O complete, if api return pending status ? yes possible wait and on file handle only, call GetOverlappedResult. but this will be correct only in no any another I/O call on this file in concurent)
in any case need understand how this is internally work. all native I/O api have the next signature:
NTSTATUS NTAPI SomeIoApi(
_In_ HANDLE FileHandle,
_In_opt_ HANDLE Event,
_In_opt_ PIO_APC_ROUTINE ApcRoutine,
_In_opt_ PVOID ApcContext,
_Out_ PIO_STATUS_BLOCK IoStatusBlock,
...
);
all have this common 5 parameters at begin. for queue I/O completion as result of this call several conditions must be met. of course FileHandle must be associated with some completion port (to this port and can be packet sent). but else one mandatory condition - ApcContext must be not zero (ApcContext != 0). if this 2 condition met and device return not error status (if FILE_SKIP_COMPLETION_PORT_ON_SUCCESS set on file - must be pending status only) - when I/O complete - ApcContext pointer will be pushed to port. and then it can be removed by
NTSTATUS
NTAPI
NtRemoveIoCompletion(
_In_ HANDLE IoCompletionHandle,
_Out_ PVOID *KeyContext,
_Out_ PVOID *ApcContext,
_Out_ PIO_STATUS_BLOCK IoStatusBlock,
_In_opt_ PLARGE_INTEGER Timeout
);
or by it win32 shell GetQueuedCompletionStatus.
so solution for not sent packet to port (even is file handle associated with completion port) - set ApcContext = 0. win32 layer do this in next way (pseudo - code):
BOOL WINAPI SomeWin32Api(
HANDLE FileHandle,
LPOVERLAPPED lpOverlapped,
LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
)
{
HANDLE hEvent = lpOverlapped->hEvent;
PVOID ApcContext = lpOverlapped;
if ((ULONG_PTR)hEvent & 1)
{
reinterpret_cast<uintptr_t&>(hEvent) &= ~1;
ApcContext = 0;
}
NTSTATUS status = SomeIoApi(
FileHandle,
hEvent,
lpCompletionRoutine, // not exactly, but by sense
ApcContext,
(PIO_STATUS_BLOCK)lpOverlapped,...);
}
it check low-order bit of hEvent in OVERLAPPED - if it set - pass 0 inplace ApcContext otherwise pass lpOverlapped (pointer to OVERLAPPED) as context ( ApcContext = lpOverlapped;)
note that nt layer let pass any void* pointer as ApcContext. but win32 layer always pass here pointer to OVERLAPPED structure or 0. because this and GetQueuedCompletionStatus return this pointer back as _Out_ LPOVERLAPPED *lpOverlapped (compare with NtRemoveIoCompletion - return as _Out_ PVOID *ApcContext)
anyway this trick affect only concrete single win32 I/O call, and if you late reset low-order bit in hEvent from overlapped ( reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) &= ~(0x1);) this already can not have any effect - the 0 in place ApcContext already passed.
also from general view this is rarely when we associate file handle with a completion port, but want not use it in some call. usually this is another api call. for example we can create asynchronous file handle, associate it with a completion port. and use port notifications in call WriteFile, but before begin write we can set/remove compression on file via FSCTL_SET_COMPRESSION. because file is asynchronous, the FSCTL_SET_COMPRESSION also can complete asynchronous, but we can want prevent completion port notification for this ioctl, instead wait inplace (on event) for it complete. for such situation and can be used this trick.
and in most case applications (if this not server with huge count of i/o requests) can instead manual call GetQueuedCompletionStatus, bind callback to file via BindIoCompletionCallback or CreateThreadpoolIo. as result system for you create iocp, thread pool which will be listen on this iocp (via GetQueuedCompletionStatus or NtRemoveIoCompletion) and then call your callback. this is very simplify your src code and logic
findings:
i almost sure (despite not view your code) that you not need at all
use trick with event low-order bit
if you use this trick in some I/O request (say ReadDirectoryChangesW)
this affect only this particular request
you can not change the behaviour by reset low-order bit in event
handle after request is sent, or by any another way
you in general not need use GetQueuedCompletionStatus and self thread
pool at all. instead simply call BindIoCompletionCallback for file
I'm trying to write a server which can support many clients connections simultaneously so I'm trying to do it with IOCP. So let me brief about my code flow and then I can explain my problem. First of all, server is opening a port for listening and waiting on an "accept" call for new incoming connections. For reference I have used same code as mentioned here So it accepts every new incoming connection and returns a new socket descriptor (sd), and then it marks as nonblocking with:
arg = 1;
ioctlsocket(sd, FIONBIO, &arg);
and then enable TCP_NODELAY:
level = IPPROTO_TCP;
optName = TCP_NODELAY;
value = 1;
setsockopt(sd, level, optName, (const char*)&value, sizeof(value));
thereafter associating with an IOCP port as:
CreateIoCompletionPort((HANDLE)sd, iocp_port, (DWORD)completion_key, 4);
completion_key is a class object which is nothing but a container, it contains data buffer, overlapped-buffer, query-type recv/send etc.
and in last issuing a read call:
WSARecv(sd, wsabuf, 1, &bytes, &flags, overlapped, NULL);
wsabuf and overlapped are part of completion_key object.
In 90% cases it works fine i.e. when there is some incoming data available on this socket "GetQueuedCompletionStatus" gets unblocked and it has valid data. But sometimes WSARecv call returns with an error and GetLastError() returns 6 which is "invalid handle" error. I'm bit bewildered why it's happening so.
The way I'm creating an iocp port:
iocp_port = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0);
and there are threads which are waiting on "GetQueuedCompletionStatus".
I monitored all system calls which were happening in background. WSARecv internally calls NtDeviceIoControlFile and there is an argument "Event" which is same as what is passed in lpOverlapped structure of WSARecv as hEvent. I wasn't setting hEvent to NULL, so it was taking some garbage value, when it was NULL then NtDeviceIoControlFile returned successfully and for other cases it returned "INVALID_HANDLE" error. Unfortunately, it was NULL most of the time.
I am currently trying some new libraries (IOCP) for socket programming. And I've stumbled upon the AcceptEx functionality to enable async connections.
As the documentation says:
The AcceptEx function uses overlapped I/O, unlike the accept function. If your application uses AcceptEx, it can service a large number of clients with a relatively small number of threads. As with all overlapped Windows functions, either Windows events or completion ports can be used as a completion notification mechanism.
But I am not receving any completion when a client connects. I do however get a completion when the client sends data..
This is my code:
DWORD dwBytes;
GUID GuidAcceptEx = WSAID_ACCEPTEX;
int iResult = WSAIoctl(m_hSocket, SIO_GET_EXTENSION_FUNCTION_POINTER,
&GuidAcceptEx, sizeof (GuidAcceptEx),
&m_lpfnAcceptEx, sizeof (m_lpfnAcceptEx),
&dwBytes, NULL, NULL);
if (iResult == SOCKET_ERROR)
{
CloseSocket();
}
And then:
WSAOVERLAPPED olOverlap;
memset(&olOverlap, 0, sizeof (olOverlap));
char lpOutputBuf[1024];
int outBufLen = 1024;
DWORD dwBytes;
BOOL bRet = m_lpfnAcceptEx( m_hSocket, hSocket, lpOutputBuf,
outBufLen - ((sizeof (sockaddr_in) + 16) * 2),
sizeof (sockaddr_in) + 16, sizeof (sockaddr_in) + 16,
&dwBytes, &olOverlap);
if ( bRet == FALSE )
{
DWORD dwRet = WSAGetLastError();
if( dwRet != WSA_IO_PENDING )
{
return dwRet;
}
}
Any suggestion of what to do to receive completions?
EDIT:
I bind the hSocket to the completionport after m_lpfnAcceptEx()
Firstly the WSAOVERLAPPED and data buffer that you're declaring on the stack above your call to AcceptEx() will not be in existence when a completion occurs (unless you are calling GetQueuedCompletionStatus() in the same function, which would be a trifle odd). You need to dynamically allocate them or pool them.
Secondly you state that you associate the socket to the completion port after you call AcceptEx(). That's wrong. You need to do these things before you call AcceptEx().
Create a socket with WSA_FLAG_OVERLAPPED set.
Bind it to the address you want to listen on.
Call listen on it with your desired backlog.
Load AcceptEx() dynamically using the listening socket and a call to WSAIoctl (not strictly necessary and the code you show should work but this way you can be sure you get your listening socket from the same underlying winsock provider and that it supports AcceptEx().
Load GetAcceptExSockaddrs() in the same way as you load AcceptEx() - you'll need it once the accept completes.
Associate the listening socket to your IOCP.
Now you can post a number of AcceptEx() calls using the listening socket and new 'accept' socket which you create like this:
Create a socket with WSA_FLAG_OVERLAPPED set.
Associate the socket to your IOCP.
As stated above you need to ensure that the buffer and the OVERLAPPED are unique per call and last until the completion occurs.
When the completion occurs you have to do the following....
Call setsockopt() with SO_UPDATE_ACCEPT_CONTEXT on the accepted socket using the listening socket as the data...
Deblock your addresses using GetAcceptExSockaddrs().
Process any data (if you allocated enough space in the buffer for data).
Note that by design AcceptEx() can be used to accept a new connection and return the initial data from that connection in one operation (this leads to slightly better performance in situations where you know you will always want some data before you can start doing things but is horribly complex to manage if you want to defend aginst the denial of service attack that can be launched simply by connecting and NOT sending data - I wrote about this here).
If you do not want AcceptEx() to wait for data to arrive then simply provide a data buffer that is ONLY big enough for the addresses to be returned and pass 0 as the 'buffer size'. This will cause the AcceptEx() to operate like an overlaped accept() and return as soon as the connection is established.
Note that Martin James' initial comment to your question is in fact the answer you're looking for. Don't pass outBufLen - ((sizeof (sockaddr_in) + 16) * 2), pass 0.
I am hooking WSASend, and WSARecv in C++ using the same method I've used to hook the client's WSASend and WSARecv functions. In the client I am able to get the IP, Port, and Socket from the SOCKET structure passed by WSASend/WSARecv; however, for the server when I try to use getpeername or getsockname() they both return the error 10057 (Socket not connected)...
I'm fairly sure that the hook is correct on the server, since it prints the bytes successfully, and I'm also sure the socket SHOULD be valid seeing how client and server establish a successful connection.
Is there a way to resolve this problem by any other alternative methods? I've been looking around the internet to find a solution, but I haven't seen anyone with the same problem.
I've tried this:
sockaddr *address = new sockaddr;
int peer_len;
getpeername(s, address, &peer_len);
int err = WSAGetLastError();
if(err==0)
{
char *Str = inet_ntoa(((sockaddr_in*)address)->sin_addr);
printf("[%s", Str);
printf(":%d]",ntohs(((sockaddr_in*)address)->sin_port));
}
else
{
printf("Error %i\n",err);
}
(Using both getpeername and getsockname)Both result in the same socket not connected error.
I'm planning on using the packets the C++ dll gets and forward the information to the C# dll since it'll be easier to manage on that (for me anyways), but I'd need to distinguish each packet with it's socket id.
You can only do that on the connected socket, i.e. the one returned from the accept() call, not on the listening "server" socket.
I'm trying to write an IOCP server. Basically, I have it accepting new connections. For the purpose of my testing, I'm running and connecting to 127.0.0.1.
I create the pseudo socket prior to calling AcceptEx(). Once a connection is accepted, the new pseudo socket is used for communication. This new socket is associated with an io completion port [CreateIoCompletionPort], I then assign it a few options, [SO_EXCLUSIVEADDRUSE] and [SO_CONDITIONAL_ACCEPT], and then I call WSARecv() to accept incoming data.
The problem is that once my remote connection connects to the server, it sends data, but that data is never received. I'm wondering if someone could offer some ideas as to why it's not receiving data? Perhaps my logic is flawed? I stepped through my code several times. no errors are recorded.
EDIT: Fixed the wording. I create the socket before AcceptEx() call.
Basic logic in my code:
// Create socket, associate with IOCP
WSASocket(af, type, proto, lpProtoInfo, g, dwFlags);
HANDLE hIOCP = GetPool()->GetQueueHandle();
CreateIoCompletionPort(hSource, hIOCP, 0, 0) != NULL;
// Server bind and listen
bind(m_shSocket, pAddr, nAddrLen);
listen(m_shSocket, nBacklog);
// Creation of the pseudo socket
SOCKET s = ::WSASocket(m_iSocketAf, m_iSocketType, m_iSocketProto, m_pWpi, m_SocketGroup, m_dwSocketFlags);
DWORD dwBytes;
BOOL bRet = m_fnAcceptEx(m_shSocket, s, chOutput, 0, sizeof(SOCKADDR_STORAGE) + 16, sizeof(SOCKADDR_STORAGE) + 16, &dwBytes, m_pcbAccept);
// ... New Connection comes in, it's accepted ...
// Associate new pseudo socket with IOCP
HANDLE hNewIOCP = GetPool()->GetQueueHandle();
CreateIoCompletionPort((HANDLE) s, hNewIOCP , 0, 0) != NULL;
// ... Remote socket sends ...
// ... Remote socket and Pseudo socket call WSARecv ...
// ... Pseudo socket does not receive ...
NOTE: I tried sending from the pseudo socket to the remote socket, same problem as sending data in the reverse way.
You need to post some code but your description doesn't make sense. That's NOT how AcceptEx() based servers operate.
With an AcceptEx() based server you create your accepted socket before you post the AcceptEx(). You then post the AcceptEx() with the listening socket and the new socket and a buffer which allows you to receive the remote address and, optionally, data.
So if you are describing your code in your original question then your code is wrong or you're not using AcceptEx(). I'm currently ignoring the 'few options' that you throw into the mix as they simply further confuse things at present without any code to analyse.
You might be interested in downloading my free IOCP based server framework, which includes working AcceptEx() and traditional Accept() based server code. You can get it from here: http://www.serverframework.com/products---the-free-framework.html
Are you calling GetQueuedCompletionStatus to get the data?
In case you are not doing this just to learn for yourself, I would also recommend that you use boost::asio - an excellent library that allows you to let someone else do the tedious code for handling the io completion ports.
I figured it out. I'm an idiot. I was sending zero bytes.