JSON response split up over multiple cURL callbacks - c++

Messing around with the League of Legends API.
I've had an issue for a couple of days now so I've simplified what's going on. I'm sending off a URL via cURL which should return a block of Json. The URL opens fine in my browser and displays the expected data. However for some strange reason, cURL (or the API?) is sending data to my callback function multiple times.
A few snippets of what returns:
Starts with - {"20278403":[{"name":"Pop...
Ends with - {"name":"Karthus's Overlords","ti
Literally cuts out with "ti. A new callback then begins, continuing on with the old data:
Starts with - er":"PLATINUM","que...
Ends with - "isInactive":false}]}]}
As you may notice, the correct termination for Json is present with the second callback's output. I know the suggestion will be 'why not just shove it all into one string and parse it after?' - the problem is that I need to send off several requests as you can only request X many players data at a time. So it's difficult to tell where one request's Json begins and the other ends!
Most importantly - does anyone know why this is happening? It seems extremely bizarre to return data across multiple callbacks.
If it helps.. Just a generic cURL call:
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://euw...")
curl_easy_setopt(m_pCurl, CURLOPT_WRITEFUNCTION, &DataSuccessCB);
curl_easy_perform(m_pCurl);
size_t CAPIReader::DataSuccessCB(char* cBuffer, size_t iSize, size_t nmemb, void* userData)
{
string sBuffer = string(cBuffer);
vStrVec.push_back(sBuffer); // vector holding all the returned json strings - intended to have a whole block of json in each one!
return (iSize * nmemb);
}
Thanks.

This is the normal behaviour of libcurl, you could see this in the getinmemory.c sample. I suppose that cURL callback the function when data are available from the socket. So if the TCP message is fragmented, the callback is called several times.
A possible solution to concatenate the message is to transmit a pointer to the string to fill :
size_t CAPIReader::DataSuccessCB(char* cBuffer, size_t iSize, size_t nmemb, void* userData)
{
std::string & buffer = *(std::string*)userData;
buffer.append((char*)contents,nmemb*iSize);
return (iSize * nmemb);
}
std::string data;
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://euw...")
curl_easy_setopt(m_pCurl, CURLOPT_WRITEFUNCTION, &DataSuccessCB);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&data);
if(curl_easy_perform(m_pCurl) == CURLE_OK)
{
// Parse the JSON data in data string
}

Related

C++ libcurl: using non-blocking loop to detect HTTP status code change

Scenario:
Before updating at a scheduled time, a web page has a HTTP status code of 503. When new data is added to the page after the scheduled time, the HTTP status code changes to 200.
Goal:
Using a non-blocking loop, to detect this change in the HTTP status code from 503 to 200 as fast as possible. With the current code seen further below, a WHILE loop successfully listens for the change in HTTP status code and prints out a success statement. Once 200 is detected, a break statement stops the loop.
However, it seems that the program must wait for a response every time a HTTP request is made before moving to the next WHILE loop iteration, behaving in a blocking manner.
Question:
Using libcurl C++, how can the below program be modified to transmit requests (to a single URL) to detect a HTTP status code change without having to wait for the response before sending another request?
Please note: I am aware that excessive requests may be deemed as unfriendly (this is an experiment for my own URL).
Before posting this question, the following SO questions and resources have been consulted:
How to do curl_multi_perform() asynchronously in C++?
Is curl_easy_perform() synchronous or asynchronous?
http://www.godpatterns.com/2011/09/asynchronous-non-blocking-curl-multi.html
https://curl.se/libcurl/c/multi-single.html
https://curl.se/libcurl/c/multi-poll.html
What's been tried so far:
Using multi-threading with a FOR loop in C to repeatedly call function to detect HTTP code change, which had a slight latency advantage. See code here: https://pastebin.com/73dBwkq3
Utilised OpenMP, again when using a FOR loop instead of the original WHILE loop. Latency advantage wasn't substantial.
Using the libcurl documentation C tutorials to try to replicate a program that listens to just one URL for changes, using the asynchronous multi-interface with difficulty.
Current attempt using curl_easy_opt:
#include <iostream>
#include <iomanip>
#include <vector>
#include <string>
#include <curl/curl.h>
// Function for writing callback
size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata) {
std::vector<char> *response = reinterpret_cast<std::vector<char> *>(userdata);
response->insert(response->end(), ptr, ptr+nmemb);
return nmemb;
}
long request(CURL *curl, const std::string &url) {
std::vector<char> response;
long response_code;
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response_code);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
auto res = curl_easy_perform(curl);
if (response_code == 200) {
std::cout << "SUCCESS" << std::endl;
}
return response_code;
}
int main() {
curl_global_init(CURL_GLOBAL_ALL);
CURL *curl = curl_easy_init();
while (true) {
long response_code = request(curl, "www.example.com");
if (response_code == 200) {
break; // Page updated
}
}
curl_easy_cleanup(curl);
curl_global_cleanup();
return 0;
}
Summary:
Using C++ and libcurl, does anyone know how a WHILE loop can be used to repeatedly send a request to one URL only, without having to wait for the response in between sending requests? The aim of this is to detect the change as quickly as possible.
I understand that there is ample libcurl documentation, but have had difficulties grasping the multi-interface aspects to help apply them to this issue.
/* get us the resource without a body - use HEAD! */
curl_easy_setopt(curl, CURLOPT_NOBODY, 1L);
If HEAD does not work for you, the server may reject HEAD, another solution:
size_t header_callback(char *buffer, size_t size, size_t nitems, void *userdata) {
long response_code = 0;
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response_code);
if (response_code != 200)
return 0; // Aborts the request.
return nitems;
}
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_callback);
The second solution will consume network traffic, the HEAD is much better, once you receive 200, you can request GET.

Curl easy perform using multiple threads C++ Programming

So far, I've been successful in pulling information from a service provider. However, I need to invoke this over parallel process with multiple threads for millions of requests.
Following is the piece of code
size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
int main()
{
CURL *curl = curl_easy_init();
std::string readBuffer;
if(curl) {
CURLcode res;
curl_easy_setopt(curl, CURLOPT_URL, "service-url");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
}
Here are my two options
a) One is thread pool (Visual studio C++ 2010 - Thus no access to C++ 11)
b) Using curl_multi_perform
When I use thread pool -> Does invoking curl become a worker thread. How do I make user that the WriteCallback is specific to the thread so that no two threads overwrite the contents.
If I use curl_multi_perform, what do I need to do, to make sure that WriteCallback gives me the output for that particular handle?
You can use the multi interface to send X simultaneously requests and handle each request when response given.
Have a look at that C Example.

C++ curl_easy_perform() inserts newline into response

I'm fetching 43182 chars long JSON from remote REST API using following code snippet:
string result_;
curl_easy_setopt(curlGET_, CURLOPT_TIMEOUT, CURL_TIMEOUT);
curl_easy_setopt(curlGET_, CURLOPT_URL, url.c_str());
curl_easy_setopt(curlGET_, CURLOPT_VERBOSE, CURL_DEBUG_VERBOSE);
curl_easy_setopt(curlGET_, CURLOPT_WRITEDATA, &result_);
curl_easy_setopt(curlGET_, CURLOPT_WRITEFUNCTION, WriteCallback);
static size_t WriteCallback(void *response,
size_t size,
size_t nmemb,
void *userp) noexcept
{
(static_cast<string*>(userp))->append(static_cast<char*>(response));
return size * nmemb;
};
curlStatusCode_ = curl_easy_perform(curlGET_);
What I get in result_ is complete JSON but with newline character after character 31954:
If I fetch same JSON in browser or command-line curl there is no newline character. How to fix this problem for "arbitrarily" long JSON or other generic response?
From CURL document of the write callback:
The data passed to this function will not be zero terminated!
The response in WriteCallback is not necessarily null terminated. So calling append by just casting it to char* will invoke undefined behavior. You have to pass the amount of data too in append.
(static_cast<string*>(userp))->append(static_cast<char*>(response), size * nmemb)

Posting with libcurl in C++, write callback params are garbage

Developing on Win64 with Visual Studio 2013 Community, deploying to both Win64 and Linux with cross platform wxWidgets. I am trying to emulate the following curl.exe command line with C++ using libcurl:
curl.exe -X POST -g "single-url-string"
This is for an IoT feature of an app, where an end-user supplies the single-url-string to control their device. The reason this logic is not just executing curl.exe as an external process is because this logic runs in its own thread, and wxWidgets does not support launching external executables when outside the main thread.
Normally when performing a POST with curl.exe, the post data is supplied as an option. This tells curl.exe the operation is a POST to the supplied url, and here is the data for that POST. As you can see, what I'm trying to do is a GET style url (with the parameters embedded in the url) but then changing the operation to a POST. It's done this way because research shows asking end-users to supply two separate url and data strings is simply too complex for them. So we came up with this easier single string end-users must supply, which is usually just copying a string from their device manual without having to interpret the string, much less break it into separate meaningful strings.
So, the issue at hand is: I have my simple C++ libcurl POST routine in two versions, but in both versions the parameters received by the write callback are bad. The two versions are a POST with a single url string, and a POST with the post data provided as a separate option to the url string.
The problems are 1) using the single string version does not execute a POST, and it's write callback params are bad; and 2) using the two string version does execute a POST, but the write callback params are bad, in a different way.
The data pointer parameter in the write callback points to memory address 1 in both versions, the size parameter appears good in both versions, but the nmemb parameter is either a huge random value (single string version) or zero (two string POST version).
Here's my code, and yes I call curl_global_init() at app start.
size_t CX_IOT_THREAD::curl_write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
// storage for transferred data:
const int dataStoreSize = CURL_MAX_WRITE_SIZE + 1;
char dataStore[dataStoreSize];
memset(dataStore, 0, dataStoreSize); // zeroed out
size_t dataSize = size * nmemb; // bytes sent
if (dataSize)
{
memcpy(dataStore, ptr, dataSize); // copy into buffer sized so we'll have a terminating NULL char
wxString msg = wxString::Format(wxT("%s"), dataStore); // send as event, eventually to the log
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg);
// must return byte count processed for libcurl to be happy:
return dataSize; /**/
}
return size; // should be dataSize, but because nmemb is bad, I’m using size; it works.
}
cx_int CX_IOT_THREAD::Post(std::string& url)
{
if (url.length() == 0)
return -1;
char errBuf[CURL_ERROR_SIZE];
errBuf[0] = '\0';
static const char *postthis = "name=Bloke&age=67";
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, url.c_str());
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, postthis);
curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
curl_easy_setopt(hnd, CURLOPT_ERRORBUFFER, errBuf);
curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, &CX_IOT_THREAD::curl_write_callback);
curl_easy_setopt(hnd, CURLOPT_WRITEDATA, NULL);
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.49.1");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
// curl_easy_setopt(hnd, CURLOPT_CUSTOMREQUEST, "POST");
ret = curl_easy_perform(hnd);
curl_easy_cleanup(hnd);
if (ret != CURLE_OK)
{
wxString msg = wxString::Format(wxT("Attempted POST failed, libcurl return code '%d'."), (cx_int)ret);
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg, (cx_int)ret);
cx_int len = strlen(errBuf);
if (len > 0)
msg = wxString::Format("%s%s", errBuf, ((errBuf[len - 1] != '\n') ? "\n" : ""));
else msg = wxString::Format("%s\n", curl_easy_strerror(ret));
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg, (cx_int)ret);
}
return (cx_int)ret;
}
Any ideas why the write callback parameters are bad? Any idea why the single string version does not even do a post? (The single string version is the above with the 2 POSTFIELDS options commented out and the CUSTOMREQUEST one enabled.)
As Igor Tandetnik points out, the callback must be static.

how to use CURL to login site

I am very new to cURL, so hopefully could get some help. Currently, I am in a Window environment, and using Visual Studio.
I am trying to use cURL to access a DLink IP camera through the DLink website (https://mydlink.com/login). And grab the video stream by the IP camera to do some processing. But to do this, I have to first login. But I am not sure how to do it.
Below is my code.
int main()
{
CURL *curl;
CURLcode result;
char *url_1 = "https://mydlink.com/login";
char *postdata = "email=xyz#gmail.com&password=123456";
char *cookiefile = "tempcookie";
curl = curl_easy_init();
if( curl )
{
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiefile);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dummy);
curl_easy_setopt(curl, CURLOPT_URL, url_1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postdata);
// Connect to target (login)
result = curl_easy_perform(curl);
if( result )
cout << "Cannot connect to site, check your url!\n";
else
{
//...
}
}
return 0;
}
Could someone please enlighten me, or provide some piece of code for it?
Thank you
1) Your example code is incomplete: you use dummy function which is not in your listing.
It is important that the dummy function returns size*nmemb (see manual for CURLOPT_WRITEFUNCTION), so it is difficult to say what went wrong.
2) you don't output your error code: please use curl_easy_strerror to decode your error in the result, then you would know why it failed.
3) if I supply my own "dummy" callback, then I get an HTML page without errors, and the page itself does not complain about wrong password or anything (which is strange, but it kind of works).
Here is my dummy:
size_t dummy(char *ptr, size_t size, size_t nmemb, void *userdata)
{
printf("%.*s", size*nmemb, ptr);
return size*nmemb;
}
I looked a bit further what mydlink.com is doing and it is doing acrobatics with the email address (like deciding if it is local, tries to guess a region etc), then manipulates cookies -- it is all in javascript, thus I am afraid one has to dig that Javascript in order to emulate proper login POST, or perhaps find some proper documentation about mydlink.com services, sorry.