How can I stop QNetworkRequest from buffering the entire contents of a QIODevice during a put/post to an HTTPS connection? It works fine when posting to HTTP but HTTPS causes the entire file to be read into memory before the post starts.
This isn't supported using the Qt classes. The reason is that Qt needs to know the total data length for the SSL headers. Chunked encoding is not supported from a send perspective. You can however roll your own - you'll need to create your own SSL header, then create your own chunks of SSL-encoded data.
I suggest you wrap this all up in your own class, so it's nicely re-usable (why not post it online?).
BTW, most of this information was taken from a recent thread on the Qt-interest mailing list - a thread on the 30th September 2009 discussed this exact problem.
You may probably have more success with Qt 4.6. It has some bugfixes regarding that.
Related
I have cases when there is a chunked response which is too big for beast, and I want to stop before I get to beast's body_limit, and continue handling the message from that point using plain boost::asio. Mind that this (obviously) means I already received the header and a large part of the body.
I'm using it for a reverse proxy, so basically what I want to do is somehow send the incomplete response to the http client, while continuing relaying the remaining response data using boost::asio.
I'm guessing I'll need to somehow serialize the incomplete response, maybe using operator<< to std::stringstream, send that to the client using boost::asio, and continue the communication from there.
Will this work? Is this the correct way of doing that, or is there a better way, maybe even using beast api? Is there another way to handle chunked messages that are about to exceed body_limit in beast's api?
Thanks in advance,
David.
UPDATE
I finally abandoned the idea of falling back to boost asio, and am now trying to receive the http message (chunked or regular) in chunks with a fixed size buffer, so that I don't reach body limit. I'm just done skimming over Receive/parse the message body one chunk at a time · Issue #154 · boostorg/beast, and it seems that it's exactly what I need. I'm trying to implement a reverse proxy as well.. I tried to use Incremental Read 💡 - 1.70.0 but get a Reference to non-static member function must be called error when trying to compile this line:
ctx->response.get().body().data = response_buffer;
Maybe the incremental read example page is not updated with the latest syntax? Do you have an example relevant for the reverse proxy I'm trying to write?
Thanks in advance,
David
Maybe the incremental read example page is not updated with the latest syntax? Do you have an example relevant for the reverse proxy I'm trying to write?
The examples in the docs are compiled, so they can't possibly be out of date. Perhaps you are mixing different versions of the example and Beast? Are you using http::buffer_body? What does the declaration of your message look like?
By default, Beast's parser limits the size of the body to 1MB for requests and 8MB for responses. This is to prevent trivial resource exhaustion attacks. You can always increase the limit, or eliminate it entirely (by setting it to the largest uint64_t) by calling parser::body_limit :
https://www.boost.org/doc/libs/1_71_0/libs/beast/doc/html/beast/ref/boost__beast__http__parser/body_limit.html
I am trying to use CURL to implement Microsoft’s EWS Streaming Notifications i.e. HTTP Streaming where the request is sent once and the server responds with a header with "Transfer Encoding: chunked". The server will send multiple keepalive or notification chunks before the final packet. The chunks are terminated with cr lf.
If I create a standard CURL client then curl_easy_perform will not return until the final chunk is received whereas I need curl_easy_perform to return upon receipt of each chunk whereupon the application will process the received chunk and call curl_easy_perform to wait for the next chunk.
I realize that I could process the chunk in the CURLOPT_WRITEFUNCTION callback but the architecture of the application doesn’t allow for that (this is a GSOAP plugin)
Any suggestions other than switching to CURLOPT_CONNECT_ONLY and handling the write all subsequent reads with curl_easy_send and curl_easy_recv? Which seems a shame as I will have to duplicate CURL’s formatting and parsing.
Alan
curl_easy_perform is completely synchronous and will return only once the entire transfer is done. There's really no way around that (you already mentioned CURLOPT_CONNECT_ONLY and I wouldn't recommend that either) with this API.
If you want control back in the same thread before the entire transfer is done, which your question suggests, you probably rather want to use the multi interface.
Using that interface, curl_multi_perform will only do as much as it can right now without blocking and return control back to your function. It does however put the responsibility over to your code to wait for socket activity and call libcurl again when there is.
(Sorry, but I don't know what restrictions a "GSOAP plugin" has and you didn't state them here, so maybe this is all crap)
So I'm wondering what is a good way of getting the progress of a download when using SFML's HTTP Class / using HTTP in general. The only way I've thought of being able to do it is using tons of ranged GET requests in a separate thread, but that ofc makes the download take much longer with all the requests and all.
Any ideas?
You can't. If you want progress information, you should either implement it yourself (not recommended) or use another library for networking.
From the documentation of sf::Http::sendRequest:
Warning: this function waits for the server's response and may not return instantly; use a thread if you don't want to block your application, or use a timeout to limit the time to wait.
In other words, it's a blocking method that return only on timeout or completion (success or error).
Maybe have a look at libcurl, cpp-netlib or maybe some other libraries.
Good day!
I am having a wierd problem: QNAM sends my SECOND request twice. I am not sure how to reproduce the problem (what are actual prerequisites), but I have the code that will demonstrate it 100% of the time. I can also provide screenshot of HTTP sniffer catching those requests. You can see that the request I send at code line 42 effectively gets duped, but the one before and one after (not shown) are sent fine (once)...
My problem is similar to the one shown in this question.
However...
Main differences between that question and mine are:
I am using QNAM in an asynchronous manner with the help of Qt's Signals and slots
I am sending post requests instead of get.
What could be the problem? Is there any way to make it work as it's supposed to?
Thanks in advance! If you need additional info just ask :)
This can happen if the server doesn't return any content in the response. If you control the server, you can add a workaround to return some dummy content.
I just had this issue and after sending down one character in the response, I no longer see the request sent twice. This was with a BlackBerry 10 (QT) hitting an iOS device running Mongoose as the embedded server.
Firstly I think I need to say that I'm still learning C++ so apologies if this is blindingly obvious/simple.
I'm trying to use the libevent library (by trying I've looked through code in the sample folder and tested some) in my C++ program to consume an http stream. I'm wondering if anyone can provide me with an example of how I'd go about connecting to a URL e.g. live.domain.com, sending the appropriate headers, read the data returned and send data back over the same connection...
I'm not sure libevent does any blocking connections but just to be explicit, I'm after non-blocking samples.
Why am I trying to do this?
I'm using an API which requires you to open a connection and it keeps it alive unless there's an error. It'll periodically send status texts to the connected client until it receives a string with an ID over the same connection. At which point it starts sending data back about the ID given... I'm not entirely sure sending data back over the same connection after the initial request is strictly compliant but that's what the server expects so it'll work...if I knew how
Thanks in advance
Yuck. Given that this isn't really HTTP, I don't think you're going to be happy using a HTTP library - even if you get it to work today after a lot of frustration, it could easily be broken tomorrow. This is too rare to be a supported feature.
But...it sounds like it's also simple enough that you could just open a raw TCP connection with libevent, manually send something that looks kind of like an HTTP request, and handle it with raw sockets from there. You don't want the extra stuff a HTTP library gets you anyway (additional transfer/content encodings, proxy support, SSL, compatibility with other protocol versions, ...)
As far as examples go, look at the libevent book. In particular, the a "Trivial HTTP v0 client" that seems very close to what you want. Good luck!