Developing on Win64 with Visual Studio 2013 Community, deploying to both Win64 and Linux with cross platform wxWidgets. I am trying to emulate the following curl.exe command line with C++ using libcurl:
curl.exe -X POST -g "single-url-string"
This is for an IoT feature of an app, where an end-user supplies the single-url-string to control their device. The reason this logic is not just executing curl.exe as an external process is because this logic runs in its own thread, and wxWidgets does not support launching external executables when outside the main thread.
Normally when performing a POST with curl.exe, the post data is supplied as an option. This tells curl.exe the operation is a POST to the supplied url, and here is the data for that POST. As you can see, what I'm trying to do is a GET style url (with the parameters embedded in the url) but then changing the operation to a POST. It's done this way because research shows asking end-users to supply two separate url and data strings is simply too complex for them. So we came up with this easier single string end-users must supply, which is usually just copying a string from their device manual without having to interpret the string, much less break it into separate meaningful strings.
So, the issue at hand is: I have my simple C++ libcurl POST routine in two versions, but in both versions the parameters received by the write callback are bad. The two versions are a POST with a single url string, and a POST with the post data provided as a separate option to the url string.
The problems are 1) using the single string version does not execute a POST, and it's write callback params are bad; and 2) using the two string version does execute a POST, but the write callback params are bad, in a different way.
The data pointer parameter in the write callback points to memory address 1 in both versions, the size parameter appears good in both versions, but the nmemb parameter is either a huge random value (single string version) or zero (two string POST version).
Here's my code, and yes I call curl_global_init() at app start.
size_t CX_IOT_THREAD::curl_write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
// storage for transferred data:
const int dataStoreSize = CURL_MAX_WRITE_SIZE + 1;
char dataStore[dataStoreSize];
memset(dataStore, 0, dataStoreSize); // zeroed out
size_t dataSize = size * nmemb; // bytes sent
if (dataSize)
{
memcpy(dataStore, ptr, dataSize); // copy into buffer sized so we'll have a terminating NULL char
wxString msg = wxString::Format(wxT("%s"), dataStore); // send as event, eventually to the log
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg);
// must return byte count processed for libcurl to be happy:
return dataSize; /**/
}
return size; // should be dataSize, but because nmemb is bad, I’m using size; it works.
}
cx_int CX_IOT_THREAD::Post(std::string& url)
{
if (url.length() == 0)
return -1;
char errBuf[CURL_ERROR_SIZE];
errBuf[0] = '\0';
static const char *postthis = "name=Bloke&age=67";
CURLcode ret;
CURL *hnd = curl_easy_init();
curl_easy_setopt(hnd, CURLOPT_URL, url.c_str());
curl_easy_setopt(hnd, CURLOPT_POSTFIELDS, postthis);
curl_easy_setopt(hnd, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
curl_easy_setopt(hnd, CURLOPT_ERRORBUFFER, errBuf);
curl_easy_setopt(hnd, CURLOPT_WRITEFUNCTION, &CX_IOT_THREAD::curl_write_callback);
curl_easy_setopt(hnd, CURLOPT_WRITEDATA, NULL);
curl_easy_setopt(hnd, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(hnd, CURLOPT_USERAGENT, "curl/7.49.1");
curl_easy_setopt(hnd, CURLOPT_MAXREDIRS, 50L);
// curl_easy_setopt(hnd, CURLOPT_CUSTOMREQUEST, "POST");
ret = curl_easy_perform(hnd);
curl_easy_cleanup(hnd);
if (ret != CURLE_OK)
{
wxString msg = wxString::Format(wxT("Attempted POST failed, libcurl return code '%d'."), (cx_int)ret);
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg, (cx_int)ret);
cx_int len = strlen(errBuf);
if (len > 0)
msg = wxString::Format("%s%s", errBuf, ((errBuf[len - 1] != '\n') ? "\n" : ""));
else msg = wxString::Format("%s\n", curl_easy_strerror(ret));
mp_queue->Report(CX_IOTTHR_CMD_ACCESS_JOB, msg, (cx_int)ret);
}
return (cx_int)ret;
}
Any ideas why the write callback parameters are bad? Any idea why the single string version does not even do a post? (The single string version is the above with the 2 POSTFIELDS options commented out and the CUSTOMREQUEST one enabled.)
As Igor Tandetnik points out, the callback must be static.
Related
I am using libcurl to fetch json data using GET request from a webserver.
This is my sample code:
char *DownloadedResponse;
static int writer(char *data, size_t size, size_t nmemb, char *buffer_in)
{
if (buffer_in != NULL)
{
buffer_in = new char[size*nmemb];
strcpy(buffer_in,data);
DownloadedResponse = buffer_in;
return size * nmemb;
}
return 0;
}
char * DownloadJSON(string URL)
{
CURL *curl;
CURLcode res;
struct curl_slist *headers=NULL;
curl_slist_append(headers, "Accept: application/json");
curl_slist_append( headers, "Content-Type: application/json");
curl_slist_append( headers, "charsets: utf-8");
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPGET,1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,writer);
res = curl_easy_perform(curl);
if (CURLE_OK == res)
{
char *ct;
res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);
if((CURLE_OK == res) && ct)
{
cout<<"\nresponse received: "<<DownloadedResponse;
}
else
{
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
curl = NULL;
return NULL;
}
}
}
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
curl = NULL;
}
Here I am able to get json data in DownloadedResponse in callback "writer" of CURLOPT_WRITEFUNCTION.
But if I print using custom pointer of CURLOPT_WRITEDATA,
char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);
cout<<dataPointer;
Output of dataPointer is empty.
What is the issue here since i able to print json data in callback of CURLOPT_WRITEFUNCTION but not in the pointer of CURLOPT_WRITEDATA
You write a function that takes data read from the network, and writes it to where you want it.
static int writer(char *data, size_t size, size_t nmemb, char *buffer_in){
if (buffer_in != NULL) {
// very bad code which is never executed
}
return 0;
}
In order for that function to write the data, it has to know where to write it, so you tell it to write to NULL
char *dataPointer = NULL;
CURLcode curl_easy_setopt(curl, CURLOPT_WRITEDATA, dataPointer);
What value do you tell it to use as buffer_in? You pass it dataPointer, which is NULL, so you just told it buffer_in = NULL. I think instead you meant to say "the address of dataPointer", which would be &dataPointer.
Technically, I have answered your question now. You passed it NULL for the buffer, so the write function exited immediately. But there's more. Now you get to execute that really bad code in writer().
if (buffer_in != NULL)
{
// if buffer_in already has allocated memory then leak it immediately
// create a new buffer of memory to leak later
buffer_in = new char[size*nmemb];
// store the data in buffer_in
// assume it is null terminated (it is not)
// rather than using the length we already know
strcpy(buffer_in,data);
// remember buffer_in? We don't use it so assign that data pointer to a global variable.
DownloadedResponse = buffer_in;
// return size of this particular chunk of data
return size * nmemb;
}
This function MUST use the length of the data, and not assume data is null terminated (see https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html).
This function MUST be able to handle the data in multiple small pieces by adding them to what it has already read. You can't call new and then discard the new memory. And you can't do that anyway because you just leaked that memory -- every new must be matched with exactly one delete. In fact, you would be very well advised not to use new or delete at all, now that we have the standard library.
This function should use the buffer_in argument you give it rather than a global variable, but you can use a global variable if you want, it's just error prone. It's not literally an error like the other stuff.
The whole point of buffer_in is to give you a persistent data structure where you can accumulate the answers. It probably should be in local scope around the curl_easy_execute, so you can then just return the content from that data structure if you got CURLE_OK. I strongly recommend you write the data to std::vector, so you don't have to keep track of memory allocation. You have trouble with it, but you don't need to do it at all. Modern style says everybody has trouble with it, so just let the standard library handle it.
You claim to follow the example in the docs, which links to https://curl.haxx.se/libcurl/c/getinmemory.html If you look again, you will see what they are doing, and how your code doesn't match. In particular, they pass &chunk (the address of chunk) and then write data into chunk so they keep what was there before.
struct MemoryStruct {
char *memory;
size_t size;
};
static size_t
WriteMemoryCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
// here is where they get access to the buffer
struct MemoryStruct *mem = (struct MemoryStruct *)userp;
In the call to curl, you will find the struct locally defined, then the remote call:
struct MemoryStruct chunk;
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, WriteMemoryCallback);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, (void *)&chunk);
res = curl_easy_perform(curl_handle);
if (stuff)
printf("%lu bytes retrieved\n", (long)chunk.size);
Messing around with the League of Legends API.
I've had an issue for a couple of days now so I've simplified what's going on. I'm sending off a URL via cURL which should return a block of Json. The URL opens fine in my browser and displays the expected data. However for some strange reason, cURL (or the API?) is sending data to my callback function multiple times.
A few snippets of what returns:
Starts with - {"20278403":[{"name":"Pop...
Ends with - {"name":"Karthus's Overlords","ti
Literally cuts out with "ti. A new callback then begins, continuing on with the old data:
Starts with - er":"PLATINUM","que...
Ends with - "isInactive":false}]}]}
As you may notice, the correct termination for Json is present with the second callback's output. I know the suggestion will be 'why not just shove it all into one string and parse it after?' - the problem is that I need to send off several requests as you can only request X many players data at a time. So it's difficult to tell where one request's Json begins and the other ends!
Most importantly - does anyone know why this is happening? It seems extremely bizarre to return data across multiple callbacks.
If it helps.. Just a generic cURL call:
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://euw...")
curl_easy_setopt(m_pCurl, CURLOPT_WRITEFUNCTION, &DataSuccessCB);
curl_easy_perform(m_pCurl);
size_t CAPIReader::DataSuccessCB(char* cBuffer, size_t iSize, size_t nmemb, void* userData)
{
string sBuffer = string(cBuffer);
vStrVec.push_back(sBuffer); // vector holding all the returned json strings - intended to have a whole block of json in each one!
return (iSize * nmemb);
}
Thanks.
This is the normal behaviour of libcurl, you could see this in the getinmemory.c sample. I suppose that cURL callback the function when data are available from the socket. So if the TCP message is fragmented, the callback is called several times.
A possible solution to concatenate the message is to transmit a pointer to the string to fill :
size_t CAPIReader::DataSuccessCB(char* cBuffer, size_t iSize, size_t nmemb, void* userData)
{
std::string & buffer = *(std::string*)userData;
buffer.append((char*)contents,nmemb*iSize);
return (iSize * nmemb);
}
std::string data;
curl_easy_setopt(m_pCurl, CURLOPT_URL, "https://euw...")
curl_easy_setopt(m_pCurl, CURLOPT_WRITEFUNCTION, &DataSuccessCB);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *)&data);
if(curl_easy_perform(m_pCurl) == CURLE_OK)
{
// Parse the JSON data in data string
}
I am very new to cURL, so hopefully could get some help. Currently, I am in a Window environment, and using Visual Studio.
I am trying to use cURL to access a DLink IP camera through the DLink website (https://mydlink.com/login). And grab the video stream by the IP camera to do some processing. But to do this, I have to first login. But I am not sure how to do it.
Below is my code.
int main()
{
CURL *curl;
CURLcode result;
char *url_1 = "https://mydlink.com/login";
char *postdata = "email=xyz#gmail.com&password=123456";
char *cookiefile = "tempcookie";
curl = curl_easy_init();
if( curl )
{
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiefile);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, dummy);
curl_easy_setopt(curl, CURLOPT_URL, url_1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postdata);
// Connect to target (login)
result = curl_easy_perform(curl);
if( result )
cout << "Cannot connect to site, check your url!\n";
else
{
//...
}
}
return 0;
}
Could someone please enlighten me, or provide some piece of code for it?
Thank you
1) Your example code is incomplete: you use dummy function which is not in your listing.
It is important that the dummy function returns size*nmemb (see manual for CURLOPT_WRITEFUNCTION), so it is difficult to say what went wrong.
2) you don't output your error code: please use curl_easy_strerror to decode your error in the result, then you would know why it failed.
3) if I supply my own "dummy" callback, then I get an HTML page without errors, and the page itself does not complain about wrong password or anything (which is strange, but it kind of works).
Here is my dummy:
size_t dummy(char *ptr, size_t size, size_t nmemb, void *userdata)
{
printf("%.*s", size*nmemb, ptr);
return size*nmemb;
}
I looked a bit further what mydlink.com is doing and it is doing acrobatics with the email address (like deciding if it is local, tries to guess a region etc), then manipulates cookies -- it is all in javascript, thus I am afraid one has to dig that Javascript in order to emulate proper login POST, or perhaps find some proper documentation about mydlink.com services, sorry.
I am trying to read the content of a PHP / HTML file on a remote web server using C++, but haven't found a way to do it. I want to pass GET statements to it, so http://example.com/login.php?user=abc&password=def.
How would I do it?
Your best bet is to use an external library. libcurl is popular and fairly easy to use.
Here's a simple example, you need to add error checking though:
string data;
CURL *curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, url_.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &data);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curlWrite);
curl_easy_perform(curl);
Your callback would look something like this:
size_t curlWrite(void *ptr, size_t size, size_t nmemb, void *usrPtr)
{
size_t bytes = size * nmemb;
string *data = static_cast<string *>(usrPtr);
data->append(static_cast<const char *>(ptr), bytes);
return bytes;
}
You can add your GET parameters on the end of the URL.
I would like to convert this code (using a scripting language with a Sockets extension) over to C++ using LibCurl. I have only used LibCurl once previously, so I am at a bit of a loss as to what else I am going to need. My main point of confusing is wether I should be able to just use curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers); to send this, or if I am going to need to extract the socket, then send it over that.
Here is the relevant snippet from the script...
public OnSocketConnected(Handle:socket, any:friendId)
{
decl String:CommunityId[32];
FriendIDToCommunityId(friendId, CommunityId, sizeof(CommunityId));
decl String:query[2048];
decl String:cookieString[100];
decl String:inviterString[32];
decl String:groupString[32];
GetConVarString(cookie, cookieString, sizeof(cookieString));
GetConVarString(inviter, inviterString, sizeof(inviterString));
GetConVarString(group, groupString, sizeof(groupString));
Format(query, sizeof(query), "GET /actions/GroupInvite?type=groupInvite&inviter=%s&invitee=%s&group=%s HTTP/1.1\r\nHost: steamcommunity.com\r\nConnection: close\r\nCookie: steamLogin=%s\r\n\r\n", inviterString, CommunityId, groupString, cookieString);
SocketSend(socket, query);
LogMessage("%s", query);
}
and here is what I have in C++ so far. It is looking like I am going to need to extract the socket first, but I am not proficient with network coding, so I am not exactly sure where I need to go from here.
void InviteToGroup(const char *szUserSteamID, const char *szInviterSteamID, const char *steamUser, const char *steamPass)
{
CURL *curl;
CURLcode res, result;
//int sockfd; /* socket */
char errorBuffer[CURL_ERROR_SIZE];
const char *szUserID = GetCommunityID(szUserSteamID); // User's Steam Community ID
const char *szInviterID = GetCommunityID(szInviterSteamID); // Inviter's Steam Community ID
char *szGroupID = "";
GetGroupCommunityID(1254745, &szGroupID); // Group Steam Community ID
const char *szCookie = "76561198018111441%7C%7CC7D70E74A3F592F3E130CCF4CAACD4A7B9CAD993"; // Steam Community Login Cookie
char *buffer = new char[512];
// Create the GET request
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "GET /actions/GroupInvite?type=groupInvite&inviter=");
snprintf(buffer, sizeof(buffer), "%s&invitee=%s&group=%s ", szInviterID, szUserID, szGroupID);
headers = curl_slist_append(headers, buffer);
headers = curl_slist_append(headers, "HTTP/1.1\r\nHost: steamcommunity.com\r\nConnection: close\r\nCookie: ");
snprintf(buffer, sizeof(buffer), "steamLogin=%s\r\n\r\n", szCookie);
headers = curl_slist_append(headers, buffer);
delete buffer;
// Init CURL
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "https://www.steamcommunity.com");
curl_easy_setopt(curl, CURLOPT_PORT, 443); // Check this before using
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorBuffer);
//curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1); // No transfer, just extract the socket
// Find out if we need to use Proxy stuff here
char *userpass = new char[64];
snprintf(userpass, sizeof(userpass), "%s:%s", steamUser, steamPass);
curl_easy_setopt(curl, CURLOPT_USERPWD, userpass);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
// Attempt to Connect the Steam Community Server
res = curl_easy_perform(curl);
if (res == CURLE_OK)
Msg("Connected Successfully!\n");
else
Msg("Connection Failed! Error: %s\n", errorBuffer);
// Close the connection
curl_easy_cleanup(curl);
}
}
So corrections here as your C++ is incorrect which will mean your code won't work.
sizeof( char * ) is the size of a pointer, not the length of the string. So your snprintf will print up to 4 characters on a 32-bit system which will include the null terminator so you will actually get only a string of length 3.
If you are going to use a "should be big enough" buffer do not allocate it with new. You could use std::vector if you want to use snprintf, then check the return value of snprintf which is the length of the actual string that would be written and then increase the buffer size if necessary (to the return value + 1).
vector<char> buffer;
size_t required = 511;
do
{
buffer.resize( required + 1 );
required = snprintf( &buffer[0], buffer.size(), formatstr, ... );
}
while( required >= buffer.size() );
There is no need to "free" the buffer at the end (which incidentally you did incorrectly) as vector will do that automatically.
You now pass &buffer[0] to the function, it will point to a null-terminated string and will be writable if the API requires a char* rather than a const char*. (libcurl though takes ... as the 3rd parameter to curl_easy_setopt so you can pass in "anything". You therefore do have to be careful as you won't get warned by the compiler if you accidentally pass in a std::string or std::vector).