Problem with libcurl cookie engine - libcurl

Question

[Cross-posted from lib-curl mailing list]
I have a single threaded app (MSVC C++ 2005) build against a static
LIBCURL 7.19.4
A test application connects to an in house server & performs a bespoke
authentication process that includes posting a couple of forms, and
when this succeeds creates a new resource (POST) and then updates the
resource (PUT) using If-Match.
I only use a single connection to libcurl (i.e. only one CURL*)
The cookie engine is enabled from the start using
curl_easy_setopt(CURLOPT_COOKIEFILE, "")
The cookie cache is cleared at the end of the authentication process
using curl_easy_setopt(CURLOPT_COOKIELIST, "SESS"). This is required
by the authentication process.
The next call, which completes a successful authentication, results in
a couple of security cookies being returned from the server - they
have no expiry date set.
The server (and I) expect the security cookies to then be sent with
all subsequent requests to the server. The problem is that sometimes
they are sent and sometimes they aren't.
I'm not a CURL expert, so I'm probably doing something wrong, but I
can't figure out what. Running the test app in a loop results shows a
random distribution of correct cookie handling.
As a workaround I've disabled the cookie engine and am doing basic
manual cookie handling. Like this it works as expected, but I'd prefer
to use the library if possible.
Does anyone have any ideas?
Thanks
Seb

Answer 1

We've experienced issues with libcurl losing "session" when the headers are a particular size.
The two known cases we've seen are 1425 and 2885.
When the sent headers are this specific size the server doesn't appear to receive the proper cookies. We haven't actually tested against a controlled server to see what is actually received by the server.
The work-around we came up with was to alter the User-Agent slightly by adding a space at the end to change the header size.
Here is some code to predict the header size before the request is sent
size_t PredictHeaderOutSize(CURL *curl, bool doPost, const char* request, char* userAgent, const char* host, const char* form)
{
size_t predictedHeaderOutSize = 0;
// Note, so far predicting 1 byte per newline, fix the hard coded #'s below if that turns out to be wrong
// POST/GET line
predictedHeaderOutSize += (doPost ? 4 : 3); // POST vs GET
predictedHeaderOutSize += strlen(request);
predictedHeaderOutSize += 11; // Extra characters in 'POST <request> HTTP/1.1' not accounted for above
// User-Agent line
predictedHeaderOutSize += strlen(userAgent);
predictedHeaderOutSize += 13;
// Host: header
predictedHeaderOutSize += strlen(host);
predictedHeaderOutSize += 7;
// Accept: */*
predictedHeaderOutSize += 12;
// Cookie:
struct curl_slist *cookies=NULL;
struct curl_slist *next_cookie;
int num_cookies = 0;
CURLcode res = curl_easy_getinfo(curl, CURLINFO_COOKIELIST, &cookies);
if (res == CURLE_OK)
{
if (cookies != NULL)
{
// At least 1 cookie so add the extra space taken on cookie line
predictedHeaderOutSize += 7;
next_cookie = cookies;
num_cookies = 1;
while (next_cookie)
{
std::vector<std::string> cookie = QueueHelper::Split("\t", next_cookie->data, 7);
if (cookie.size() != 7)
{
// wtf?
}
else
{
// For each cookie we add length of key + value + 3 (for the = ; and extra space)
predictedHeaderOutSize += cookie[5].length() + cookie[6].length() + 3;
}
next_cookie = next_cookie->next;
num_cookies++;
}
curl_slist_free_all(cookies);
}
}
else
{
printf("curl_easy_getinfo failed: %s\n", curl_easy_strerror(res));
}
if (doPost)
{
// Content-Length:
size_t formLength = strlen(form);
if (formLength < 10)
predictedHeaderOutSize += 1;
if (formLength >= 10 && formLength < 100)
predictedHeaderOutSize += 2;
if (formLength >= 100 && formLength < 1000)
predictedHeaderOutSize += 3;
if (formLength >= 1000 && formLength < 10000)
predictedHeaderOutSize += 4;
predictedHeaderOutSize += 17;
// Content-Type: application/x-www-form-urlencoded
predictedHeaderOutSize += 48;
}
predictedHeaderOutSize += 2; // 2 newlines at the end? something else? not sure
return predictedHeaderOutSize;
}