I'm using libCURL to perform an HTTP GET request toward a device that responds with a continuous flow of data in a multipart HTTP response.
I'd like to handle the unfortunate but possible case where the device is disconnected/shutdown or is not reachable anymore on the network.
By default libCURL does not have a few seconds timeout as I need, so I tried:
setting the CURLOPT_CONNECTTIMEOUT options,
but this only works at connection stage, not while already receiving data.
setting the CURLOPT_TIMEOUT option,
but this seems to always force a timeout even when data is still received.
My question is: how can I properly handle a timeout with libCURL, in the case described above?
For your scenario instead of
curl_easy_setopt(curl, CURLOPT_TIMEOUT, <your timeout in seconds>);
use
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_easy_setopt(curl, CURLOPT_LOW_SPEED_TIME, <your timeout in seconds>);
The above two lines make sure that if the average speed drops below 1 byte per second, in a time frame of X seconds, then the operation is aborted (timeout).
See reference here.
Related
When I invoke my lambda function it takes between 1 and 15 seconds to execute. If I invoke the function via the C++ SKD, I get timeouts. These timeouts seem to occur after a few seconds (this is human-judgment only, I did not actually time it).
Question: How do I tell the SDK to wait for slow lambdas to return and not to timeout?
Things that did not work:
In the JS SDK you can change this in the HTTP settings. This is is no such option in the C++ SDK HTTPOptions.
It does not help to give the lambda client a config with a larger connectionTimeoutMS (socket timeout). Also, the httpRequestTimeoutMs of the client is set to 0 by default, meaning it will wait forever.
I am using synchronous requests, which do not seem to have an extra option for timeouts.
Additional information:
I am using a single client to run multiple requests in parallel.
Error also happens if I am using async requests.
Related:
How do I troubleshoot retry and timeout issues when invoking a Lambda function using an AWS SDK?
Same thing gave me hard time once. You may have got the solution but for others, here is what I did. There is client configuration which can edit default connection time. Default connection time for sending request is 1 sec and for receiving it is 3 sec, if you get the request done in this time period than it is good otherwise a retry according to lambda setting will be invoked. The behavior of these two is well explained in their respective header file.
you can also play with memory size of lambda higher the memory of lambda lower the response time for the same.
Aws::Client::ClientConfiguration m_ClientConfig;
m_ClientConfig.requestTimeoutMs = 300000; // i.e. 300 seconds
m_ClientConfig.connectTimeoutMs = 300000;
/**
* Socket read timeouts for HTTP clients on Windows. Default 3000 ms. This should be more than adequate for most services. However, if you are transfering large amounts of data
* or are worried about higher latencies, you should set to something that makes more sense for your use case.
* For Curl, it's the low speed time, which contains the time in number milliseconds that transfer speed should be below "lowSpeedLimit" for the library to consider it too slow and abort.
*/
long requestTimeoutMs;
/**
* Socket connect timeout. Default 1000 ms. Unless you are very far away from your the data center you are talking to. 1000ms is more than sufficient.
*/
long connectTimeoutMs;
After a lookup both on SO and other places, I've noticed there is a lot of conflicting information about the cURL options CONNECTTIMEOUT vs TIMEOUT.
CONNECTTIMEOUT is definitely the timeout just for the connection phase,
TIMEOUT is stated as being timeout for the entire cURL process (including CONNECTTIMEOUT) or the timeout after the connection phase has finished, depending on who you ask.
Furthermore, the official libcurl docs explain CONNECTTIMEOUT as
set maximum time the request is allowed to take
which is quite ambiguous language as it could be referring to e.g a HTTP request or speaking about the entire process as a request
CONNECTTIMEOUT is the time curl waits for during the connection. after that curl abandons the effort to connect. on the other hand, TIMEOUT is the total duration of receiving a response for a given request for which curl will wait, including the time it takes to connect and the time that the server takes to reply. here is the official link for both:
https://curl.haxx.se/libcurl/c/CURLOPT_CONNECTTIMEOUT.html
https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT.html
I have initialized curlHandle using curl_easy_init() and have set some options eg. url, type of request, timeout, ssl verification etc.
I read that re-using same curlHandle increases performance and did little sample code writting as well, it happened to be true.
So I initialized handle once in the constructor of my class and then resuing the same handle every time just changing header or request, but if I do not use that curlHandle for 10 minutes and then if I try to reuse then it throws an exception that could not resolve host.
Is there the time limit for which you can use curlHandle?
Because I did not see any such mention in any SO question or curl documentation.
Or is it like after certain timeout you have to do curl_easy_init() again?
[EDIT]
I am using wolfSSL for SSL communication with cURL.
By default session-id caching is disabled in wolfSSL and enabled in curl
because of CURLOPT_SSL_SESSIONID_CACHE defaulting to 1.
Session-id times out after inactivity of 500 seconds( Approx. 8 minutes) whereas cURL tries to reuse the same session-id.
This is causing the failure in SSL_set_session of wolfSSL and this causes curl to fail after 8-10 minutes of inactivity.
Curl version used 7.49.1
No, there's no time limit for a curl handle and no, it is not supposed to cause any particular errors after having been idle for ten or more minutes. If you can reproduce this with a recent version of libcurl, it might be a bug...
I have a while loop, and inside that loop I send a PUT request into google firebase REST api. It works very well, but if I want to fasten things up (the while loop waits for the curl response every round of the loop which is very slow sometimes, over 200ms), I'm trying to add the CURLOPT_TIMEOUT_MS and set it to a low 1 millisecond.
TLDR;
after adding line
curl_easy_setopt(curl, CURLOPT_TIMEOUT_MS, 1L);
My curl does not send anything to the server anymore. Or does the server somehow force the client to receive the returning value from the request?
You tell curl to fail the operation if it isn't completed within 1 millisecond. Not many requests are completed that quickly, especially not if you're using DNS or just use connections over the Internet.
So yes, most transfers will then just return CURLE_OPERATION_TIMEDOUT (28) with no content.
This is a bug of CURL.
If your timeout setting is less than 1s, it will directly return an error.
Solution is:
curl_easy_setopt(conn, CURLOPT_NOSIGNAL, 1);
conn is the pointer of CURL, e.g.:
CURL *conn = NULL;
curl_easy_setopt(conn, CURLOPT_NOSIGNAL, 1);
curll_handle = Curl init()
curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 20 )
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 30);
Request1:(curll_handle)
Curl_easy_perform(curll_handle)
For connection = 20 secs
Chal response request = 30 secs
request2:same handle used as above
curl_slist_append(headers, "Connection: Close");
Curl_easy_perform(curll_handle)
Questions:
Request 1:
will it use both timeouts(connection and response timeouts) for curl perform?
what is the total time will take? as per my understand 30 secs(CURLOPT_CONNECTTIMEOUT + CURLOPT_TIMEOUT)
Request 2:
if we use same request1 curl handle for request2. is it(req2) curl_easy_perform() will d0 both connection and response sequence requestS?
what the timeout value use for sencond request if use same req1 curl handle?
Request 1:
CURLOPT_TIMEOUT is the maximum time for the entire operation. It will not spend more time. You know it will be done within 30 seconds, successful or not.
CURLOPT_CONNECTTIMEOUT is the longest time you allow the "connection phase" to take. If the connection to the remote server is not done within 20 seconds, the transfer will return failure.
The times are set totally independent of each other. So if the connection phase takes 19 seconds, there is 11 seconds left for the transfer to complete or the maximum timeout will trigger.
Request 2:
Uses the same options and options are sticky and will be the same until changed, so it will have the exact same timeout values as request 1.
If the connection is kept and re-used from request 1, the CURLOPT_CONNECTTIMEOUT won't trigger since it is already connected.