I'm not very good in C++, you if you see something in the code fragment which could be better, please educate me!
I'm implementing winhttp in an asynchronous fashion. But im having trouble retrieving the response. I cant figure it out. Because you should be able to parsethe whole response at once. Since multiple concurent request can occur, buffering the response (headers+body) in a global variable is not the way to go.
How can I retrieve the response of the http get request? Or else, is it an good practice to execute winhttp synchronous on a new thread (so the main loop doesn;t get blocked and then calls a function when done?):
void __stdcall cb(HINTERNET h, DWORD_PTR d, DWORD dwInternetStatus, LPVOID lpvStatusInformation, DWORD dwStatusInformationLength){
char* s=new char[1];
DWORD dwSize = 0;
if (dwInternetStatus==WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE){
MessageBoxA(0,s,"",0);
WinHttpQueryDataAvailable( h, &dwSize);
.....
}
}
And the call in the main:
...winhttpopen...
WinHttpSetStatusCallback(request, (WINHTTP_STATUS_CALLBACK)whCallback,WINHTTP_CALLBACK_FLAG_ALL_NOTIFICATIONS,0);
...winhttpsend....
Check this sample code on MSDN - Asynchronous Completion in WinHTTP.
The call to WinHttpQueryDataAvailable in QueryData generates a status
callback with a WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE completion in
the dwInternetStatus parameter. By checking the value pointed to by
the lpvStatusInformation parameter, the callback can determine how
much data is left to be read, and if there is no remaining data, can
proceed to display all the data that has been read.
This shows you that your callback is called with buffer pointer and length of data in it.
Related
I've been working with easy_perform till now and it worked as expected. But due to timeouts and single threaded application, there's latency in running multiple operations. I'm looking at optimizing these calls by converting them into asynchronous with multi_perform interface, Though I am having hard time understanding correct way to make use of it.
From my understanding, Flow looks something like following :
Create a easy_handle for call
Add this standard easy handle to the multi stack using curl_multi_add_handle
curl_multi_perform : This is where it gets tricky.
As I understand it, This call is happening in a loop.
My application is calling this API to read/write whatever there is to read or write right now etc.
If running_handles is changed from the previous call, there is data to read which we should retrieve using curl_multi_info_read
Clean up when easy handle is processed
curl_multi_remove_handle
curl_easy_cleanup
curl_multi_cleanup
Q:
Does that mean, My application needs to do periodic polling to check if there's data to read?
Is there a way to handle this with callbacks? and the callback method should trigger action in my application in asynchronous way.
Refs I've already reviewed :
Looking at http://www.godpatterns.com/2011/09/asynchronous-non-blocking-curl-multi.html , It says the same thing. Correct me if I'm wrong.
stackoverflow thread and other related : How to do curl_multi_perform() asynchronously in C++?
The prerequisite knowledge needed to understand curl_multi API is Async Sockets.
curl_multi_perform is not a blocking API. As explained in documentation:
When an application has found out there's data available for the multi_handle or a timeout has elapsed, the application should call this function to read/write whatever there is to read or write right now etc. curl_multi_perform returns as soon as the reads/writes are done.
It just needs to be called periodically.
Does that mean, My application needs to do periodic polling to check if there's data to read?
Yes. curl_multi_fdset conveniently extracts the related file descriptors so that you can select on them (select = wait), but you're free to add other descriptors to the same select call, thus interleaving curl work with your own work. Here's an example of how to do it.
Is there a way to handle this with callbacks?
Yes. The transferred data is passed during the curl_multi_perform call into a CURLOPT_WRITEFUNCTION callback. Note: curl_multi_info_read is not for reading data, it's for reading information about the transfer.
for (/* each transfer */) {
curl_easy_setopt(eh, CURLOPT_WRITEFUNCTION, write_cb);
curl_easy_setopt(eh, CURLOPT_WRITEDATA, /* pass userp value */);
curl_multi_add_handle(multi_handle, easy_handle);
}
int still_running;
do {
if (curl_multi_perform(cm, &still_running)) { // will call write_cb() when data is read
/* handle error */ break;
}
if (curl_multi_wait(cm, NULL, 0, 1000, NULL)) {
/* handle error */ break;
}
} while(still_running);
Here's a complete example of using a data callback with multi-transfer: 10-at-a-time.
Note: curl_multi_wait used in this example is a convenience wrapper around a select call.
In the MSDN doc for the lpOverlapped parameter of GetQueuedCompletionStatus it is said that the application can prevent completion port notification by setting the low-order bit of the hEvent member of the OVERLAPPED structure. But is it possible to resume the notifications after they were stopped?
I need to use this for monitoring network folders for changes:
When GetQueuedCompletionStatus returns FALSE and GetLastError() returns ERROR_NETNAME_DELETED, I do this (works):
di->Overlapped.hEvent = CreateEvent( NULL, FALSE, FALSE, di->lpszDirName );
reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) |= 0x1;
And when the network problem was resolved, I tried to do the reverse operation - but it DID NOT work:
reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) &= ~(0x1);
(It will be good if the solution be compatible with Windows 7)
first of all completion port notification can not be "suspended" or "resumed"
Even if you have passed the function a file handle associated with a
completion port and a valid OVERLAPPED structure, an application can
prevent completion port notification. This is done by specifying a
valid event handle for the hEvent member of the OVERLAPPED structure,
and setting its low-order bit. A valid event handle whose low-order
bit is set keeps I/O completion from being queued to the completion
port.
this mean the next - when we call some win32 I/O api (api which take pointer to OVERLAPPED as in/out parameter, such as ReadFile, ReadDirectoryChangesW, LockFileEx etc) and file handle (passed to this api) associated with a completion port - despite this we can prevent completion port notification for this call by event handle with low-order bit. this is for only concrete api call and not affect any another api calls. and all this unrelated to GetQueuedCompletionStatus
(strictly said we can simply pass 1 in place hEvent too. but in this case question - how we get notify about I/O complete, if api return pending status ? yes possible wait and on file handle only, call GetOverlappedResult. but this will be correct only in no any another I/O call on this file in concurent)
in any case need understand how this is internally work. all native I/O api have the next signature:
NTSTATUS NTAPI SomeIoApi(
_In_ HANDLE FileHandle,
_In_opt_ HANDLE Event,
_In_opt_ PIO_APC_ROUTINE ApcRoutine,
_In_opt_ PVOID ApcContext,
_Out_ PIO_STATUS_BLOCK IoStatusBlock,
...
);
all have this common 5 parameters at begin. for queue I/O completion as result of this call several conditions must be met. of course FileHandle must be associated with some completion port (to this port and can be packet sent). but else one mandatory condition - ApcContext must be not zero (ApcContext != 0). if this 2 condition met and device return not error status (if FILE_SKIP_COMPLETION_PORT_ON_SUCCESS set on file - must be pending status only) - when I/O complete - ApcContext pointer will be pushed to port. and then it can be removed by
NTSTATUS
NTAPI
NtRemoveIoCompletion(
_In_ HANDLE IoCompletionHandle,
_Out_ PVOID *KeyContext,
_Out_ PVOID *ApcContext,
_Out_ PIO_STATUS_BLOCK IoStatusBlock,
_In_opt_ PLARGE_INTEGER Timeout
);
or by it win32 shell GetQueuedCompletionStatus.
so solution for not sent packet to port (even is file handle associated with completion port) - set ApcContext = 0. win32 layer do this in next way (pseudo - code):
BOOL WINAPI SomeWin32Api(
HANDLE FileHandle,
LPOVERLAPPED lpOverlapped,
LPOVERLAPPED_COMPLETION_ROUTINE lpCompletionRoutine
)
{
HANDLE hEvent = lpOverlapped->hEvent;
PVOID ApcContext = lpOverlapped;
if ((ULONG_PTR)hEvent & 1)
{
reinterpret_cast<uintptr_t&>(hEvent) &= ~1;
ApcContext = 0;
}
NTSTATUS status = SomeIoApi(
FileHandle,
hEvent,
lpCompletionRoutine, // not exactly, but by sense
ApcContext,
(PIO_STATUS_BLOCK)lpOverlapped,...);
}
it check low-order bit of hEvent in OVERLAPPED - if it set - pass 0 inplace ApcContext otherwise pass lpOverlapped (pointer to OVERLAPPED) as context ( ApcContext = lpOverlapped;)
note that nt layer let pass any void* pointer as ApcContext. but win32 layer always pass here pointer to OVERLAPPED structure or 0. because this and GetQueuedCompletionStatus return this pointer back as _Out_ LPOVERLAPPED *lpOverlapped (compare with NtRemoveIoCompletion - return as _Out_ PVOID *ApcContext)
anyway this trick affect only concrete single win32 I/O call, and if you late reset low-order bit in hEvent from overlapped ( reinterpret_cast<uintptr_t &>(di->Overlapped.hEvent) &= ~(0x1);) this already can not have any effect - the 0 in place ApcContext already passed.
also from general view this is rarely when we associate file handle with a completion port, but want not use it in some call. usually this is another api call. for example we can create asynchronous file handle, associate it with a completion port. and use port notifications in call WriteFile, but before begin write we can set/remove compression on file via FSCTL_SET_COMPRESSION. because file is asynchronous, the FSCTL_SET_COMPRESSION also can complete asynchronous, but we can want prevent completion port notification for this ioctl, instead wait inplace (on event) for it complete. for such situation and can be used this trick.
and in most case applications (if this not server with huge count of i/o requests) can instead manual call GetQueuedCompletionStatus, bind callback to file via BindIoCompletionCallback or CreateThreadpoolIo. as result system for you create iocp, thread pool which will be listen on this iocp (via GetQueuedCompletionStatus or NtRemoveIoCompletion) and then call your callback. this is very simplify your src code and logic
findings:
i almost sure (despite not view your code) that you not need at all
use trick with event low-order bit
if you use this trick in some I/O request (say ReadDirectoryChangesW)
this affect only this particular request
you can not change the behaviour by reset low-order bit in event
handle after request is sent, or by any another way
you in general not need use GetQueuedCompletionStatus and self thread
pool at all. instead simply call BindIoCompletionCallback for file
I am currently working with the simulation engine VBS2 and am attempting to write a TCP socket plugin. I have a client application which I want to connect to the plugin and send a single message. Perhaps this will make more sense if I post the existing plugin code:
#include <windows.h>
#include "VBSPlugin.h"
// Command function declaration
typedef int (WINAPI * ExecuteCommandType)(const char *command, char *result, int resultLength);
// Command function definition
ExecuteCommandType ExecuteCommand = NULL;
// Function that will register the ExecuteCommand function of the engine
VBSPLUGIN_EXPORT void WINAPI RegisterCommandFnc(void *executeCommandFnc)
{
ExecuteCommand = (ExecuteCommandType)executeCommandFnc;
}
// This function will be executed every simulation step (every frame) and took a part in the simulation procedure.
// We can be sure in this function the ExecuteCommand registering was already done.
// deltaT is time in seconds since the last simulation step
VBSPLUGIN_EXPORT void WINAPI OnSimulationStep(float deltaT)
{
//{ Sample code:
ExecuteCommand("0 setOvercast 1", NULL, 0);
//!}
}
// This function will be executed every time the script in the engine calls the script function "pluginFunction"
// We can be sure in this function the ExecuteCommand registering was already done.
// Note that the plugin takes responsibility for allocating and deleting the returned string
VBSPLUGIN_EXPORT const char* WINAPI PluginFunction(const char *input)
{
//{ Sample code:
static const char result[]="[1.0, 3.75]";
return result;
//!}
}
// DllMain
BOOL WINAPI DllMain(HINSTANCE hDll, DWORD fdwReason, LPVOID lpvReserved)
{
switch(fdwReason)
{
case DLL_PROCESS_ATTACH:
OutputDebugString("Called DllMain with DLL_PROCESS_ATTACH\n");
break;
case DLL_PROCESS_DETACH:
OutputDebugString("Called DllMain with DLL_PROCESS_DETACH\n");
break;
case DLL_THREAD_ATTACH:
OutputDebugString("Called DllMain with DLL_THREAD_ATTACH\n");
break;
case DLL_THREAD_DETACH:
OutputDebugString("Called DllMain with DLL_THREAD_DETACH\n");
break;
}
return TRUE;
}
The message sent to the plugin will be used in the OnSimulationStep() function by being passed as an argument to ExecuteCommand(). However, I've also got to be careful about blocking here as the OnSimulationStep() function must be allowed to run every simulation step.
I've been staring at this for a few days now and have tried looking at the winsock tutorials, but I'm not a C++ programmer and am feeling rather stuck. Please would anyone be kind enough to give me a few pointers in the right direction?
Thanks in advance, all advice is greatly appreciated.
I would personally go with boost::asio to save myself all the hassle in dealing with asynchronous IO.
It is relatively straightforward to use, and it works well in the plugin environment - I've done something similar (also in VBS2).
When your plugin must handle data in a short time and you fear that the winsock send function could block, you need to queue the data or write a mechanism that only the important data is written, if this is an option.
One option is a queue in your plugin and a worker thread that pumps the data from the queue to the socket. With the additional thread you can just use a potential blocking call. If the blocking calll is a problem for you, you can set the socket to non-blocking mode and and wait with WSAAsyncSelect for an event indicating that you can write to the peer again.
You could implement an TCP-server which stores incoming messages in an ordered list. In each OnSimulationStep you then query the TCP-server for received messages and apply them to VBS2 via ExecuteCommand.
Remember to use ExecuteCommand always in the thread which calls OnSimulationStep. This means you're not able to execute incoming messages directly in the TCP-server.
I am learning how to network program using c/c++ and I have created a server(TCP) that is suppose to respond in specific ways to messages from a client in order to do this I created a class that the server class passes the message to and returns a string to report back to the client.
Here is my problem sometimes it reports the correct string back other times if just repeats what I sent to the message handler. Which no where in the code do I have it return what was passed in. So I am wondering am I handling getting the message correctly?
Secondly, I am unsure of how to keep a connection open in a while loop to continually pass messages back and forth. You can see how I did it in the code below but I am pretty sure this is incorrect, any help on this would be great. Thanks!
if (!fork())
{ // this is the child process
close(sockfd); // child doesn't need the listener
while ((numbytes = recv(new_fd, buf, MAXDATASIZE-1, 0)) > 0)
{
//numbytes = recv(new_fd, buf, MAXDATASIZE-1, 0);
buf[numbytes-1] = '\0';
const char* temp = ash.handleMessage(buf).c_str();
int size_of_temp = ash.handleMessage(buf).length();
send(new_fd, temp, size_of_temp, 0);
//send(new_fd, temp, size_of_temp+1, 0);
}
}//end if
Please excuse my ghetto code
Handles the message
Class Method handler uses
If your learning about sockets you should also know that you can't assume that what you send() as a "complete message", will be delivered as a complete message.
If you send() some big data from your client you might need to use multiple recv() on the server (or vice versa) to read it all. This is a big difference of how files usually work...
If you'r designing your own protocol you can opt to also send the messages length, like [LEN][message]. An easy example would be if the strings you send are limited to 256 bytes you can start with send()ing a short representing the strings length,
Or easier, decide that you use line-feeds (newline - \n) to terminate messages. The the protocol would look like
"msg1\nmsg2\n"
then you would have to recv(), and append the data, until you get a newline. This is all I can muster right now, there are a lot of great examples on the internet, but I would recommend getting the source of some "real" program and look at how it handles its network.
You are calling handleMessage twice. You didn't post the code, but it looks like you're returning a string. It might be better to do:
string temp = ash.handleMessage(buf);
int size_of_temp = temp.length();
This would avoid repeating any action that takes place in handleMessage.
I am using the KbFilter example in the WDK, trying to send an IOCTL in a function that is called by KbFilter_ServiceCallback and therefore is executed at DISPATCH_LEVEL. The function just has to send an IOCTL and return, am not waiting for an output buffer to be filled so it can be asynchronous, fire and forget.
I am currently using the WDF functions WdfIoTargetFormatRequestForIoctl and WdfRequestSend to try and send at DISPATCH_LEVEL and getting nothing. The call to WdfRequestSend is succeeding but the IOCTL doesn't appear to be received.
Using either of WdfIoTargetSendIoctlSynchronously or the WDM pattern IoBuildDeviceIoControlRequest() and IoCallDriver() requires PASSIVE_LEVEL and the only way I know to call these at PASSIVE_LEVEL is to create a separate thread that runs at PASSIVE_LEVEL and pass it instructions via a buffer or a queue, synchronized with a spinlock and semaphore.
Can someone tell me if there is an easier way to pass IOCTLs to the drivers below my filter, or is the thread/queue approach the normal pattern when you need to do things at a higher IRQL? Under what circumstances can I use KeRaiseIrql and is this what I should use? Thanks.
Use IoAllocateIrp and IoCallDriver. They can be run at IRQL <= DISPATCH_LEVEL.
You cannot lower your IRQL (unless it is you who raised it). KeRaiseIrql is used only to raise IRQL. A call to KeRaiseIrql is valid if the caller specifies NewIrql >= CurrentIrql.
Be careful: Is your IOCTL expected at DISPATCH_LEVEL?
Here is a code snippet:
PIRP Irp = IoAllocateIrp(DeviceObject->StackSize, FALSE);
Irp->Tail.Overlay.Thread = PsGetCurrentThread();
Irp->RequestorMode = KernelMode;
Irp->IoStatus.Status = STATUS_NOT_SUPPORTED;
Irp->IoStatus.Information = 0;
PIO_STACK_LOCATION stack = IoGetNextIrpStackLocation(Irp);
stack->MajorFunction = IRP_MJ_DEVICE_CONTROL;
stack->Parameters.DeviceIoControl.IoControlCode = ...