So basically what I want to do is use libcurl to fetch slightly different urls, e.g.:
http://foo.com/foo.asp?name=*NAMEHERE*
What I would like to do is iterate through a vector of names and get each one, e.g.:
http://foo.com/foo.asp?name=James
Then
http://foo.com/foo.asp?name=Andrew
And so on.
However, when I try doing this:
int foo (){
CURL *curl;
CURLcode success;
char errbuf[CURL_ERROR_SIZE];
int m_timeout = 15;
if ((curl = curl_easy_init()) == NULL) {
perror("curl_easy_init");
return 1;
}
std::vector<std::string> names;
names.push_back("James");
names.push_back("Andrew");
for (std::vector<std::string>::const_iterator i = names.begin(); i != names.end(); ++i){
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, long(m_timeout));
curl_easy_setopt(curl, CURLOPT_URL, "http://foo.com/foo.asp?name=" + *i);
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, 1L);
}
if ((success = curl_easy_perform(curl)) != 0) {
fprintf(stderr, "%s: %s\n", "curl_easy_perform", errbuf);
return 1;
}
curl_easy_cleanup(curl);
return 0;
}
It gives me an error:
Cannot pass object of non-trivial type 'std::__1::basic_string<char>' through variadic function; call will abort at runtime
On this line:
curl_easy_setopt(curl, CURLOPT_URL, "http://foo.com/foo.asp?name=" + *i);
because of the + *i.
Is what I want to do possible? Is there a solution?
EDIT: Thanks for the answer, but for some reason, when I run this it only gets the website with the last string in the vector, and it ignores the other ones. In my case, it skips James and goes straight to Andrew. Why does that happen?
The parameter passed to curl_easy_setopt for CURLOPT_URL needs to be a char * instead of a std::string. You can get a const char * from a std::string by calling its c_str member function:
std::string url = "http://foo.com/foo.asp?name=" + *i;
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
For anyone who ends up here in the future, take a look at the curl manual if you haven't already. There's also an online book.
The documentation for curl_easy_setopt says you need to read about a specific option to know what parameter type to use.
All options are set with an option followed by a parameter. That parameter can be a long, a function pointer, an object pointer or a curl_off_t, depending on what the specific option expects. Read this manual carefully as bad input values may cause libcurl to behave badly!
The documentation for CURLOPT_URL says exactly what the parameter type needs to be.
Pass in a pointer to the URL to work with. The parameter should be a char * to a zero terminated string which must be URL-encoded in the following format:
scheme://host:port/path
for some reason, when I run this it only gets the website with the last string in the vector, and it ignores the other ones. In my case, it skips James and goes straight to Andrew. Why does that happen?
You are looping through all of the names configuring the curl object with a different name on each loop iteration, THEN you are calling curl_easy_perform() only after the loop has finished, so it will retrieve only the last name that was configured. You need to move your call to curl_easy_perform() inside of the loop instead:
for (std::vector<std::string>::const_iterator i = names.begin(); i != names.end(); ++i)
{
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, long(m_timeout));
string url = "http://foo.com/foo.asp?name=" + *i;
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_AUTOREFERER, 1L);
success = curl_easy_perform(curl);
if (success != 0) {
fprintf(stderr, "%s: %s\n", "curl_easy_perform", errbuf);
// handle the error as needed...
continue;
}
// use the downloaded content as needed...
}
Related
I'm developing a library that communicates with a REST API. The method I wrote for PUT calls has worked up until this point.
void Command::put(const std::string url, const std::string body)
{
CURLcode ret;
struct curl_slist *slist1;
slist1 = NULL;
// slist1 = curl_slist_append(slist1, "Content-Type: multipart/form-data;");
slist1 = curl_slist_append(slist1, "Content-Type: application/json;");
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 102400L);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)12);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, slist1);
curl_easy_setopt(curl, CURLOPT_USERAGENT, "curl/7.79.1");
curl_easy_setopt(curl, CURLOPT_MAXREDIRS, 50L);
curl_easy_setopt(curl, CURLOPT_SSH_KNOWNHOSTS, "/Users/$USER/.ssh/known_hosts");
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_setopt(curl, CURLOPT_FTP_SKIP_PASV_IP, 1L);
curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, 1L);
ret = curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl = NULL;
curl_slist_free_all(slist1);
slist1 = NULL;
}
curl_easy_cleanup(curl);
curl = nullptr;
curl_slist_free_all(slist1);
slist1 = NULL;
}
However, it seems to not work with arrays. Below are three debug put calls and their output. The first two work. The third does not.
// *** DEBUG ***
commandWrapper->put(
"http://192.168.1.7/api/x1CAh16Nr8NcKwgkUMVbyZX3YKFQrEaMAj5pFz0Z/lights/6/state", "{\"on\":true}");
commandWrapper->put(
"http://192.168.1.7/api/x1CAh16Nr8NcKwgkUMVbyZX3YKFQrEaMAj5pFz0Z/lights/6/state", "{\"bri\":123}");
commandWrapper->put(
"http://192.168.1.7/api/x1CAh16Nr8NcKwgkUMVbyZX3YKFQrEaMAj5pFz0Z/lights/6/state", "{\"xy\":[0.04,0.98]}");
[{"success":{"/lights/6/state/on":true}}]
[{"success":{"/lights/6/state/bri":123}}]
[{"error":{"type":2,"address":"/lights/6/state","description":"body contains invalid json"}}]2022-04-02T02:56:47.220Z - Living Room 1 - Status: 0
I'm a little lost on how to proceed. I verified that this command does work on the command line.
You need to change this line:
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)12);
To this instead:
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)body.size());
Or, simply omit CURLOPT_POSTFIELDSIZE_LARGE completely, since body.c_str() is a pointer to a null-terminated string in this situation. Curl can determinant the size for you.
The JSON {xy:[0.04,0.98]} is 16 characters, longer than the 12 characters you are claiming, so you are truncating the JSON during transmission, which is why the server thinks it is invalid. Both {"on":true} and {"bri":123} are 11 characters (12 if you count the null terminator), so they are not truncated.
When running my code (pertinent sections pasted below), I periodically get the following error:
program(34010,0x70000e58b000) malloc: *** error for object
0x7fc43d93fcf0: pointer being freed was not allocated set a breakpoint
in malloc_error_break to debug Signal: SIGABRT (signal SIGABRT)
I am running multi-threaded C++ code on a Macbook (OS-10.13) wherein different threads make use of the code in question simultaneously. To my knowledge, libcurl is indeed thread safe as long as I do not utilize the same "curl handle" (which I understand to be an instance of "CURL" aka "CURL *curl = curl_easy_init();") in two different threads at the same time. In my case, since each thread calls the function separately and initializes a new instance of a CURL object, I should be "safe", right? Hopefully there is something obvious that I'm missing that is causing me (or lib curl in this case) to attempt to free memory that has already been freed. If there is any more information I should have included (below) please don't hesitate to let me know.
The function that seg faults is
string http_lib::make_get_request(string url)
on the line that reads
curl_easy_cleanup(curl);
and sometimes (less often) on the line that reads
res = curl_easy_perform(curl);
Below is what I think would be the pertinent sections of my code:
size_t http_lib::CurlWrite_CallbackFunc_StdString(void *contents, size_t size, size_t nmemb, std::string *s)
{
size_t newLength = size*nmemb;
size_t oldLength = s->size();
try
{
s->resize(oldLength + newLength);
}
catch(std::bad_alloc &e)
{
//handle memory problem
return 0;
}
std::copy((char*)contents,(char*)contents+newLength,s->begin()+oldLength);
return size*nmemb;
}
string http_lib::make_post_request(string url, vector<string> headers, string post_params) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if(curl)
{
struct curl_slist *chunk = NULL;
for(int i=0; i<headers.size(); i++){
/* Add a custom header */
chunk = curl_slist_append(chunk, headers[i].c_str());
}
/* set our custom set of headers */
res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_params.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
{
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
/* always cleanup */
curl_easy_cleanup(curl);
}
// Debug output
if (networking_debug){
cout<<"Response: " << s <<endl;
}
return s;
}
string http_lib::make_get_request(string url) {
//SslCurlWrapper sslObject;
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
//tell libcurl to follow redirection
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
if (networking_debug){
cout << "Response: " << s << endl;
}
return s;
}
In main() I have
int main(int argc, char *argv[]){
// Initialize http_lib (curl)
curl_global_init(CURL_GLOBAL_DEFAULT);
... spin up 10 or so threads that make get/post requests to https site (some requests utilize the make_post_request() function and others utilize make_get_requet() function).
}
CMAKE doesn't/didn't seem to want to use anything other than CURL_ROOT_DIR of "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include" for libcurl (aka curl).
Thus it was using the curl lib that mac (and/or Xcode) ships with. I haven't figured out what version that is, but I can say that not using it and instead using CURL version 7.57 is what fixed my issue.
I used "brew" package manager to
brew install curl
Doing so created /usr/local/Cellar/curl/7.57.0 directory and put all libs/includes in there.
Then I added
-I/usr/local/Cellar/curl/7.57.0/include -L/usr/local/Cellar/curl/7.57.0/lib
to my CMAKE CMAKE_CXX_FLAGS.
TLDR; Solution was to ensure I was using the newest version of the curl lib. Now that I am, no problem.
I'm trying tu use curl in c++ using function.
Example:
#define ...
...
...
/* CURL parameters */
CURL *curl = curl_easy_init();
CURLcode res;
string readBuffer;
void setHeader(){
if(curl) {
/* Headers */
struct curl_slist *chunk = NULL;
chunk = NULL;
chunk = curl_slist_append(chunk, "Connection:keep-alive");
..
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
}
}
void myFunction1(){
setHeader();
curl_easy_setopt(curl, CURLOPT_URL, "....");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
res = curl_easy_perform(curl);
cout << readBuffer;
curl_easy_cleanup(curl);
}
void myFunction2(){
setHeader();
curl_easy_setopt(curl, CURLOPT_URL, "....");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
res = curl_easy_perform(curl);
cout << readBuffer;
curl_easy_cleanup(curl);
}
In the main function i call myFunction1 or myFunction2;
I don't know if is right to use libcurl in this way, but I'm having a problem.
I use this to login in my site, so I save the cookie whit this code:
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
int main(){
curl_global_init(CURL_GLOBAL_ALL);
myFunction1();
myFunction2();
curl_global_cleanup();
}
Suppose i login and save the cookie in myFunction1.
When I'm trying to set in the myFunction2 it crash on the CURLOPT_COOKIEFILE row.
I don't know why the cookie is saved in the file, but it can't be used beacuse when I run the program it crash on that row.
Sorry for my English
Thank's
The reason of my problem is that when I use the curl_easy_cleanup(curl) and then curl_easy_init I change the sessionID, so the cookie that I've saved are no longer good. So I use at the first curl_easy_init, then I make all my request in my functions, and at the end of my program I use curl_easy_cleanup. To clean the CURL OPTION you can use: curl_easy_reset(). I've update this post for people that can have my same problem.
I used Curl 7.2.9 and checked connection this way:
Here's example:
curl = curl_easy_init();
bool result = false;
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
if(res != CURLE_OK)
{
}
else
{
// connection is available
}
Now I switched to curl-7.33.0 and got *CURLE_WRITE_ERROR* error,
and to make it work I must code it like
std::string output;
char* encodedUrl = curl_easy_escape(curl, m_checkConnectionUrl, 0);
curl_easy_setopt(curl, CURLOPT_POST, 0);
curl_easy_setopt(curl, CURLOPT_URL, encodedUrl);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeMemoryCurlCallbackStub);
CURLcode res = curl_easy_perform(curl);
But I don't need to write anything. Any ideas?
Manily the Curl option *CURLOPT_WRITEFUNCTION* is used to have a certain amount of data periodically(at the callback functoin) to handle a large file download. I don't see any reason to use this with your curl purpose, regardless the version.
Remove the *CURLOPT_POST*(by default its 0) and *CURLOPT_WRITEFUNCTION* from the code and it should work. If it doesn't, then you are doing something wrong at other places in your code!
Also, if you are checking whether the URL is ok or not, then using CURL is ok. But to only check for connection, you can only check whether the port 80 of the domain is on or not.
You need to write a writecallback as well
size_t CurlWriteCallback(char* buf, size_t size, size_t nmemb, void* up)
{
TRACE("CURL - Response received:\n%s", buf);
TRACE("CURL - Response handled %d bytes:\n%s", size*nmemb);
// tell curl how many bytes we handled
return size*nmemb;
}
if(curl)
{
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlWriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
Old question, but I have just encountered a similar problem. After some more googling this is the solution:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1L);
curl_easy_perform(curl);
// OK, now we are connected (if nothing bad happened),
// but it would be nice to communicate with the server:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 0L);
//now we can do the actual communication
I used this to separate authentication from the actual emails sending.
I am trying to make a c++ application that uses Dropbox. When I am trying to create a new folder I take the following error: {"error": "Not Found"} and "HTTP/1.1 404 Not Found".
I have followed the instruction of Dropbox's REST Api, so I have used the POST method. (I am not sure if I can use PUT, instead).
Here is my code:
void createDirectory(const char *new_Directory_Path)
{
string create_Directory_Url = "https://api.dropbox.com/1/fileops/create_folder/sandbox/test_folder";
CURL* curl = curl_easy_init();
if(!curl)
{
CURLcode res;
struct curl_slist *headers = NULL;
string my_header = "Authorization: OAuth oauth_version=\"1.0\", oauth_signature_method=\"PLAINTEXT\",
oauth_consumer_key=\""
+ m_consumer_key + "\", oauth_token=\"" + m_accessKey + "\", oauth_signature=\""
+ m_consumer_secret + "&" + m_accessSecret + "\"";
headers = curl_slist_append(headers, my_header.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_POST, 1L);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);
curl_easy_setopt(curl, CURLOPT_URL, create_Directory_Url.c_str());
res = curl_easy_perform(curl);
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
}
}
Please, does anyone have an idea about what am I doing wrong?
Looking at the documentation I belive you want to leave sandbox/test_folder off of the url and use root=sandbox & path=test_folder as form values.
Here is an example of submitting form parameters: http://curl.haxx.se/libcurl/c/postit2.html
EDIT:
I found another example, here: https://github.com/geersch/DropboxRESTApi/blob/master/src/part-3/README.md which seems to show the parameters being added to the url's query string rather than as form values. The doc page isn't clear. It just shows them as parameters but doesn't specify which kind.