Synchronous HTTP Requests in C++ - c++

I have set up a multiplayer dedicated server that runs using PHP scripts and MySQL databases. I am trying to access the server via HTTP to send/receive game data, starting with something as simple as getting the server status.
I have been able to contact the server successfully using the Unreal Doc's IHttpRequest:
void ANetwork::getContentsOfURL(FString URL)
{
serverResponse = NULL;
TSharedRef<IHttpRequest> HttpRequest = FHttpModule::Get().CreateRequest();
HttpRequest->SetHeader(TEXT("Content-Type"), TEXT("application/json"));
HttpRequest->SetURL(URL);
HttpRequest->SetVerb(TEXT("POST"));
//Creating JSON Object
FString json = "{\"auth\":\"" + authenticator = "\"";
json += "}";
HttpRequest->SetContentAsString(json);
HttpRequest->OnProcessRequestComplete().BindUObject(this, &ANetwork::OnResponseReceived);
HttpRequest->ProcessRequest();
}
void ANetwork::OnResponseReceived(FHttpRequestPtr Request, FHttpResponsePtr Response, bool bWasSuccessful)
{
GEngine->AddOnScreenDebugMessage(-1, 15.0f, FColor::Red, Response->GetContentAsString());
if (!Response.IsValid())
{
serverResponse = "FAIL";
}
else
{
serverResponse = Response->GetContentAsString();
}
}
And this echoes the proper codes to the debugger, so I know the server is working and the code is in fact getting what it needs to get. However, I need to be able to get the HTTP response as an FString and return it to the caller so that I can use it in-game. Right now this method is asynchronous, which prevents me from returning the response.
How can I make a synchronous HTTP Request so that I can return the response as a string to the caller?
i.e.
FString ANetwork::getContentsOfURL(FString URL)

Reset (unsignal) an event at the bottom of getContentsOfUrl.
Await for it to become signaled.
Signal the event from OnResponseReceived.
CreateEvent https://msdn.microsoft.com/en-us/library/windows/desktop/ms682396(v=vs.85).aspx ResetEvent https://msdn.microsoft.com/en-us/library/windows/desktop/ms685081(v=vs.85).aspx WaitForSingleObject https://msdn.microsoft.com/en-us/library/windows/desktop/ms687032(v=vs.85).aspx SetEvent (signals it) https://msdn.microsoft.com/en-us/library/windows/desktop/ms686211(v=vs.85).aspx
HANDLE hEvent = CreateEvent(NULL, FALSE, FALSE, NULL);
// bottom getCongentsOfUrl:
ResetEvent(hEvent); // optional because inital state is unsignaled
WaitForSingleObject(hEvent);
// OnResponseReceived
SetEvent(hEvent)

Related

How to use grpc c++ ClientAsyncReader<Message> for server side streams

I am using a very simple proto where the Message contains only 1 string field. Like so:
service LongLivedConnection {
// Starts a grpc connection
rpc Connect(Connection) returns (stream Message) {}
}
message Connection{
string userId = 1;
}
message Message{
string serverMessage = 1;
}
The use case is that the client should connect to the server, and the server will use this grpc for push messages.
Now, for the client code, assuming that I am already in a worker thread, how do I properly set it up so that I can continuously receive messages that come from server at random times?
void StartConnection(const std::string& user) {
Connection request;
request.set_userId(user);
Message message;
ClientContext context;
stub_->Connect(&context, request, &reply);
// What should I do from now on?
// notify(serverMessage);
}
void notify(std::string message) {
// generate message events and pass to main event loop
}
I figured out how to used the api. Looks like it is pretty flexible, but still a little bit weird given that I typically just expect the async api to receive some kind of lambda callback.
The code below is blocking, you'll have to run this in a different thread so it doesn't block your application.
I believe you can have multiple thread accessing the CompletionQueue, but in my case I just had one single thread handling this grpc connection.
GrpcConnection.h file:
public:
void StartGrpcConnection();
private:
std::shared_ptr<grpc::Channel> m_channel;
std::unique_ptr<grpc::ClientReader<push_notifications::Message>> m_reader;
std::unique_ptr<push_notifications::PushNotificationService::Stub> m_stub;
GrpcConnection.cpp files:
...
void GrpcConnectionService::StartGrpcConnection()
{
m_channel = grpc::CreateChannel("localhost:50051",grpc::InsecureChannelCredentials());
LongLiveConnection::Connect request;
request.set_user_id(12345);
m_stub = LongLiveConnection::LongLiveConnectionService::NewStub(m_channel);
grpc::ClientContext context;
grpc::CompletionQueue cq;
std::unique_ptr<grpc::ClientAsyncReader<LongLiveConnection::Message>> reader =
m_stub->PrepareAsyncConnect(&context, request, &cq);
void* got_tag;
bool ok = false;
LongLiveConnection::Message reply;
reader->StartCall((void*)1);
cq.Next(&got_tag, &ok);
if (ok && got_tag == (void*)1)
{
// startCall() is successful if ok is true, and got_tag is void*1
// start the first read message with a different hardcoded tag
reader->Read(&reply, (void*)2);
while (true)
{
ok = false;
cq.Next(&got_tag, &ok);
if (got_tag == (void*)2)
{
// this is the message from server
std::string body = reply.server_message();
// do whatever you want with body, in my case i push it to my applications' event stream to be processed by other components
// lastly, initialize another read
reader->Read(&reply, (void*)2);
}
else if (got_tag == (void*)3)
{
// if you do something else, such as listening to GRPC channel state change, in your call, you can pass a different hardcoded tag, then, in here, you will be notified when the result is received from that call.
}
}
}
}

ERROR_WINHTTP_INCORRECT_HANDLE_STATE from WinHttpQueryHeaders

I'm trying to build a small browser using the WinHTTP API, and I'm trying to handle credentials & authentication requests. I'm using this diagram for better understanding:
When I'm trying to GET a URL that requires authentication, I'm getting an error from this function (ERROR_WINHTTP_INCORRECT_HANDLE_STATE) from the call to WinHttpQueryHeaders() and I'm not sure why:
void WinHttpCheck::init(LPCWSTR method, const char* cUrl, LPSTR data, DWORD dataLen, LPCWSTR additional headers)
{
isAborted = false;
bool postFlow = false;
// Create & Initialize the URL_COMPONENTS structure.
URL_COMPONENTS urlComp;
ZeroMemory(&urlComp, sizeof(urlComp));
urlComp.dwStructSize = sizeof(urlComp);
// Set required component lengths to non-zero
// so that they are cracked.
urlComp.dwSchemeLength = (DWORD)-1;
urlComp.dwHostNameLength = (DWORD)-1;
urlComp.dwUrlPathLength = (DWORD)-1;
urlComp.dwExtraInfoLength = (DWORD)-1;
if (lstrcmpW(method, M_POST) == 0) {
printf("********** Init Post flow! **********\n");
postFlow = true;
}
LPCWSTR lpcUrl = cStringT_convert(cUrl); // convert char* to LPCWSTR
if (!WinHttpCrackUrl(lpcUrl, (DWORD)wcslen(lpcUrl), 0/*flags*/, &urlComp)) {
reportErrorAndExit("WinHttpCrackUrl");
} else {
printUrlComponent(urlComp);
}
// return just the host name with no path or "http://" prefix
LPCWSTR validateAndConvertOutput = validateAndConvertUrl(cUrl);
// impl down
st.request = createHttpRequestHandle(validateAndConvertOutput, method, urlComp);
if (st.request) {
printf("********** open request succeed! **********\n");
// handle WinHttpSendRequest params here in case of credentials request
LPCWSTR headers = WINHTTP_NO_ADDITIONAL_HEADERS;
LPVOID optionalData = WINHTTP_NO_REQUEST_DATA;
DWORD HeadersLen = 0,
optionalLen = 0,
totalLen = 0;
if (postFlow) { // post/put data request
optionalData = (LPVOID)data;
optionalLen = dataLen;
totalLen = dataLen;
if (additionalHeaders) {
HeadersLen = (ULONG)-1L;
headers = additionalHeaders;
}
}
if (!WinHttpSendRequest(st.request, headers, HeadersLen, optionalData, optionalLen, totalLen, 0)) {
st.bResults = WinHttpQueryHeaders(st.request,
WINHTTP_QUERY_STATUS_CODE |
WINHTTP_QUERY_FLAG_NUMBER,
NULL,
&st.dwStatusCode,
&st.dwSize,
NULL);
if (st.bResults)
handleStatusCode(authForm);
else
reportErrorAndExit("WinHttpQueryHeaders - NEW"); // HERE: Error code 12019 has occurred from function WinHttpQueryHeaders - NEW
...
}
...
}
...
}
createHttpRequestHandle:
HINTERNET WinHttpCheck::createHttpRequestHandle(LPCWSTR& url, const LPCWSTR& method, URL_COMPONENTS urlComp)
{
// obtain a session handle.
st.session = WinHttpOpen(L"PTC_EDGE_INTEGRATED_BROWSER",
WINHTTP_ACCESS_TYPE_NO_PROXY,
WINHTTP_NO_PROXY_NAME,
WINHTTP_NO_PROXY_BYPASS,
0);
// Specify an HTTP server.
if (st.session) {
printf("********** open succeed! **********\n");
printf("trying to connect to url: %S\n", url);
st.connect = WinHttpConnect(st.session, url,
urlComp.nPort, 0/*reserved - must be 0*/);
}
else
reportErrorAndExit("WinHttpOpen");
// Create an HTTP request handle.
if (st.connect) {
printf("********** connect succeed! **********\n");
st.request = WinHttpOpenRequest(st.connect, method,
L"/debug.txt", NULL/*default version - HTTP/1.1*/,
WINHTTP_NO_REFERER,
WINHTTP_DEFAULT_ACCEPT_TYPES,
WINHTTP_FLAG_SECURE);
}
else {
reportErrorAndExit("WinHttpConnect");
}
return st.request;
}
The error code is 12019 = ERROR_WINHTTP_INCORRECT_HANDLE_STATE.
The URL_COMPONENT content looks like this:
I looked for this in Microsoft's docs but all it says is:
The requested operation cannot be carried out because the handle supplied is not in the correct state.
Can I have some help for a better understanding of "handle's correct state"?
The only place I see you calling WinHttpQueryHeaders() is after WinHttpSendRequest() fails, which is not a valid time to call it.
As is clearly stated in the WinHTTPQueryHeaders documentation:
[in] hRequest
HINTERNET request handle returned by WinHttpOpenRequest. WinHttpReceiveResponse must have been called for this handle and have completed before WinHttpQueryHeaders is called.
Also, as the WinHttpReceiveResponse() documentation says:
The WinHttpReceiveResponse function waits to receive the response to an HTTP request initiated by WinHttpSendRequest. When WinHttpReceiveResponse completes successfully, the status code and response headers have been received and are available for the application to inspect using WinHttpQueryHeaders. An application must call WinHttpReceiveResponse before it can use WinHttpQueryDataAvailable and WinHttpReadData to access the response entity body (if any).
Even the WinHTTPSendRequest() documentation says:
Even when WinHTTP is used in asynchronous mode, that is, when WINHTTP_FLAG_ASYNC has been set in WinHttpOpen, this function can operate either synchronously or asynchronously. In either case, if the request is sent successfully, the application is called back with the completion status set to WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE. The WINHTTP_CALLBACK_STATUS_REQUEST_ERROR completion indicates that the operation completed asynchronously, but failed. Upon receiving the WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE status callback, the application can start to receive a response from the server with WinHttpReceiveResponse. Before then, no other asynchronous functions can be called, otherwise, ERROR_WINHTTP_INCORRECT_HANDLE_STATE is returned.
You can even see in the diagram you posted that WinHttpReceiveResponse() precedes WinHttpQueryHeaders().
So, it makes perfect sense why you are getting an error about the HTTP session being in an invalid state when you try retrieve the response headers. The request failed to send, and no response was received which can be queried.

gRPC: What are the best practices for long-running streaming?

We've implemented a Java gRPC service that runs in the cloud, with an unidirectional (client to server) streaming RPC which looks like:
rpc PushUpdates(stream Update) returns (Ack);
A C++ client (a mobile device) calls this rpc as soon as it boots up, to continuously send an update every 30 or so seconds, perpetually as long as the device is up and running.
ChannelArguments chan_args;
// this will be secure channel eventually
auto channel_p = CreateCustomChannel(remote_addr, InsecureChannelCredentials(), chan_args);
auto stub_p = DialTcc::NewStub(channel_p);
// ...
Ack ack;
auto strm_ctxt_p = make_unique<ClientContext>();
auto strm_p = stub_p->PushUpdates(strm_ctxt_p.get(), &ack);
// ...
While(true) {
// wait until we are ready to send a new update
Update updt;
// populate updt;
if(!strm_p->Write(updt)) {
// stream is not kosher, create a new one and restart
break;
}
}
Now different kinds of network interruptions happen while this is happening:
the gRPC service running in the cloud may go down (for maintenance) or may simply become unreachable.
the device's own ip address keeps changing as it is a mobile device.
We've seen that on such events, neither the channel, nor the Write() API is able to detect network disconnection reliably. At times the client keep calling Write() (which doesn't return false) but the server doesn't receive any data (wireshark doesn't show any activity at the outgoing port of the client device).
What are the best practices to recover in such cases, so that the server starts receiving the updates within X seconds from the time when such an event occurs? It is understandable that there would loss of X seconds worth data whenever such an event happens, but we want to recover reliably within X seconds.
gRPC version: 1.30.2, Client: C++-14/Linux, Sever: Java/Linux
Here's how we've hacked this. I want to check if this can be made any better or anyone from gRPC can guide me about a better solution.
The protobuf for our service looks like this. It has an RPC for pinging the service, which is used frequently to test connectivity.
// Message used in IsAlive RPC
message Empty {}
// Acknowledgement sent by the service for updates received
message UpdateAck {}
// Messages streamed to the service by the client
message Update {
...
...
}
service GrpcService {
// for checking if we're able to connect
rpc Ping(Empty) returns (Empty);
// streaming RPC for pushing updates by client
rpc PushUpdate(stream Update) returns (UpdateAck);
}
Here is how the c++ client looks, which does the following:
Connect():
Create the stub for calling the RPCs, if the stub is nullptr.
Call Ping() in regular intervals until it is successful.
On success call PushUpdate(...) RPC to create a new stream.
On failure reset the stream to nullptr.
Stream(): Do the following a while(true) loop:
Get the update to be pushed.
Call Write(...) on the stream with the update to be pushed.
If Write(...) fails for any reason break and the control goes back to Connect().
Once in every 30 minutes (or some regular interval), reset everything (stub, channel, stream) to nullptr to start afresh. This is required because at times Write(...) does not fail even if there is no connection between the client and the service. Write(...) calls are successful but the outgoing port on the client does not show any activity on wireshark!
Here is the code:
constexpr GRPC_TIMEOUT_S = 10;
constexpr RESTART_INTERVAL_M = 15;
constexpr GRPC_KEEPALIVE_TIME_MS = 10000;
string root_ca, tls_key, tls_cert; // for SSL
string remote_addr = "https://remote.com:5445";
...
...
void ResetStreaming() {
if (stub_p) {
if (strm_p) { // graceful restart/stop, this pair of API are called together, in this order
if (!strm_p->WritesDone()) {
// Log a message
}
strm_p->Finish(); // Log if return value of this is NOT grpc::OK
}
strm_p = nullptr;
strm_ctxt_p = nullptr;
stub_p = nullptr;
channel_p = nullptr;
}
}
void CreateStub() {
if (!stub_p) {
ChannelArguments chan_args;
chan_args.SetInt(GRPC_ARG_KEEPALIVE_TIME_MS, GRPC_KEEPALIVE_TIME_MS);
channel_p = CreateCustomChannel(
remote_addr,
SslCredentials(SslCredentialsOptions{root_ca, tls_key, tls_cert}),
chan_args);
stub_p = GrpcService::NewStub(m_channel_p);
}
}
void Stream() {
const auto restart_time = steady_clock::now() + minutes(RESTART_INTERVAL_M);
while (!stop) {
// restart every RESTART_INTERVAL_M (15m) even if ALL IS WELL!!
if (steady_clock::now() > restart_time) {
break;
}
Update updt = GetUpdate(); // get the update to be sent
if (!stop) {
if (channel_p->GetState(true) == GRPC_CHANNEL_SHUTDOWN ||
!strm_p->Write(updt)) {
// could not write!!
return; // we will Connect() again
}
}
}
// stopped due to stop = true or interval to create new stream has expired
ResetStreaming(); // channel, stub, stream are recreated once in every 15m
}
bool PingRemote() {
ClientContext ctxt;
ctxt.set_deadline(system_clock::now() + seconds(GRPC_TIMEOUT_S));
Empty req, resp;
CreateStub();
if (stub_p->Ping(&ctxt, req, &resp).ok()) {
static UpdateAck ack;
strm_ctxt_p = make_unique<ClientContext>(); // need new context
strm_p = stub_p->PushUpdate(strm_ctxt_p.get(), &ack);
return true;
}
if (strm_p) {
strm_p = nullptr;
strm_ctxt_p = nullptr;
}
return false;
}
void Connect() {
while (!stop) {
if (PingRemote() || stop) {
break;
}
sleep_for(seconds(5)); // wait before retrying
}
}
// set to true from another thread when we want to stop
atomic<bool> stop = false;
void StreamUntilStopped() {
if (stop) {
return;
}
strm_thread_p = make_unique<thread>([&] {
while (!stop) {
Connect();
Stream();
}
});
}
// called by the thread that sets stop = true
void Finish() {
strm_thread_p->join();
}
With this we are seeing that the streaming recovers within 15 minutes (or RESTART_INTERVAL_M) whenever there is a disruption for any reason. This code runs in a fast path, so I am curious to know if this can be made any better.

FastCGI c++ store requests?

I need to make any applciation using fastcgi and c++. A thread different from the one that has the FCGX_Accept_r(...) loop is the one doing the processing, so I was hoping I could store the requests as they come in a queue and have the second thread send the response on the output stream in each requests.
This is what I got so far.
void Listen(){
FCGX_Request request;
FCGX_InitRequest(&request, 0, 0);
while(FCGX_Accept_r(&request)>= 0)
{
AddRequest(request); //just adds it to a std::queue<FCGX_Request> queue
request = *NewRequest();
}
}
FCGX_Request* NewRequest()
{
FCGX_Request* request;
request = new FCGX_Request();
FCGX_InitRequest(request, 0, 0);
return request;
}
The second thread is just checking if something is on the queue (they both share it). If there's a request, it'll do whatever and it's supposed to print the response and call FCGX_Finish_r(&request). When I get to printing the output, the app doesn't crash, but nginx puts the error "upstream prematurely closed FastCGI stdout while reading response header from upstream" in the error file. I'm not really sure what's going on. Maybe it has something to do with the fact that the output stream in FCGX_Request is a pointer.
If I uncomment out the line
request = *NewRequest();
it's able to print the response, but if I can't reinitialize the request to a new one each time, then I can't store them in the queue.
What can I do to proberly store these FCGX_Requests in the queue for another thread to process? Thanks
Do something like this:
void Listen(){
FCGX_Request *request = NewRequest();
while(FCGX_Accept_r(request)>= 0)
{
// Make your queue hold pointers
AddRequest(request); //just adds it to a std::queue<FCGX_Request*> queue
request = NewRequest();
}
}
FCGX_Request* NewRequest()
{
FCGX_Request* request = new FCGX_Request();
FCGX_InitRequest(request, 0, 0);
return request;
}
Don't forget to delete the requests when you are done with them.

Wait for response from server on client

I'm trying to validate a user's login, so I send a username and password to the server, the server checks that data against the database, and will send a yes/no if the validation was a success or failure. The client receives this and the readyRead() signal is emitted, and I handle that with a slot.
I have this login function:
bool Client::login(QString username, QString password){
//some code
client.write(clientSendBuf); //send the data to the server
//wait for response
//if response is good, return true
//else return false
}
I want to wait for a response from the server before I return a true or false with login. I know how to accept a response from the server just fine, but I basically want the data to be sent, and the client program to stop until either we get a response or some time has passed and we get a time out.
How do I do this in Qt?
http://qt-project.org/doc/qt-4.8/qiodevice.html#waitForReadyRead
QTcpSocket client;
if(client.waitForReadyRead(15000)){
//do something if signal is emitted
}
else{
//timeout
}
I didn't look through the docs properly. I found my answer.
You really do not want to write code like that. Remember that all waitFor... and exec methods can reenter your code and thus are a source of hard to find bugs. No, they will reenter at the most inopportune moment. Perhaps when you're demoing to a client, or perhaps when you've shipped your first system to Elbonia :)
The client should emit a signal when the login succeeds. There's a request to login, and a response to such a request. You can use the QStateMachine to direct the overall application's logic through such responses.
The example below presumes that the network protocol supports more than one request "on the wire" at any time. It'd be simple to get rid of the handler queue and allow just one handler.
class Client : public QObject {
Q_OBJECT
typedef bool (Client::*Handler)(); // returns true when the request is finished
QTcpSocket m_client;
QByteArray m_buffer;
QQueue<Handler> m_responses; // always has the idle response on the bottom
...
Q_SLOT void hasData() {
Q_ASSERT(! m_responses.isEmpty());
m_buffer += m_client.readAll();
while (! m_buffer.isEmpty()) {
if (m_reponses.head()()) m_responses.dequeue();
}
}
bool processIdleRsp() {
// Signal an error condition, we got data but expect none!
return false; // Must never return true, since this response mustn't be dequeued.
}
bool processLoginRsp() {
const int loginRspSize = ...;
if (m_buffer.size() < loginRspSize) return false;
bool success = false;
... // process the response
emit loginRsp(success);
m_buffer = m_buffer.mid(loginRspSize); // remove our response from the buffer
return true;
}
public:
Client(QObject * parent = 0) : QObject(parent), m_state(Idle) {
connect(&m_client, SIGNAL(readyRead()), SLOT(hasData());
m_responses.enqueue(&Client::processIdleRsp);
}
Q_SLOT void loginReq(const QString & username, const QString & password) {
QByteArray request;
QDataStream req(&request, QIODevice::WriteOnly);
...
m_client.write(request);
m_responses.enqueue(&Client::processLoginRsp);
}
Q_SIGNAL void loginRsp(bool success);
};
You could use a circular queue for the buffer to speed things up if you're transmitting lots of data. As-is, the remaining data is shoved to the front of the buffer after each response is processed.