CURL finish executing and timeout - c++

I'm performing a server request with curl in C++ which return responses in pieces and those pieces's size may also vary.
At the time of arrival of each piece, the callback function is being called. The problem is I can't detect when the connection finished in order to perform an another callback to my parent class.
And by the way, I want to know if we can set and detect timeout for a curl?
Here is my code in short:
CURL *curl = curl_easy_init();
curl_global_init(CURL_GLOBAL_ALL);
curl_easy_setopt(curl, CURLOPT_URL, "My URL");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "My Postfields");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeCallback);
curl_easy_perform(curl);
curl_easy_cleanup(curl);
curl_global_cleanup();
The default callback:
size_t writeCallback(char* buf, size_t size, size_t nmemb, void* up)
{
//do something
//But how can I detect the last callback when connection finished
//in order to call an another one?
return size*nmemb;
}

The data you want can be saved off during the callback, then used once curl_easy_perform returns. Example:
CURL *curl = curl_easy_init();
curl_global_init(CURL_GLOBAL_ALL);
// NOTE: added to accumulate data.
std::string result;
curl_easy_setopt(curl, CURLOPT_URL, "My URL");
curl_easy_setopt(curl, CURLOPT_POST, 1);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "My Postfields");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &result); // NOTE: added
curl_easy_perform(curl);
// TODO: do something with your data stored in result
curl_easy_cleanup(curl);
curl_global_cleanup();
And in your write callback:
size_t writeCallback(char* buf, size_t size, size_t nmemb, void* up)
{
std::string* pstr = static_cast<std::string*>(up);
std::copy(buf, buf+size*nmemb, std::back_inserter(*pstr));
return size*nmemb;
}
or something along those lines. I leave all the error checking to you (and sorry for any typos; I don't have a compiler to validate this on immediately available to me).
Regarding timeout length, there are a multitude of timeout options available to a easy-mode curl request. Too many to mention here, in fact. See the documentation for curl_easy_setopt, in particular the connection options approximately 2/3rd of the way down the page.
Best of luck.

Related

libcurl downloads no data to buffer

I am using following code to download data from an url to memory (stream). Around 2% chance, the size of the stream is zero. I can download proper data from the same failing url if I try it another time. I am not sure if this is a network issue, CPU usage issue, or it's just the code not covering some corner cases. Please advice. Thanks!
static size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata)
{
std::vector<uchar> *stream = (std::vector<uchar>*)userdata;
size_t count = size * nmemb;
stream->insert(stream->end(), ptr, ptr + count);
return count;
}
static void CurlUrl(const char* img_url, std::vector<uchar>* stream) {
CURL *curl = curl_easy_init(); // curl_global_init is called eleswhere.
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(curl, CURLOPT_URL, img_url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, stream);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
CURLcode res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
If it didn't deliver any download data into the buffer via the callback, it means that the transfer either failed or that there was exactly zero bytes to transfer.
Check the return code from curl_easy_perform() as it might actually tell you exactly what happened.
Use CURLOPT_VERBOSE to see what's going on if (1) is not enough.
Use CURLOPT_ERRORBUFFER to get a better error description if it fails if (2) is not enough.

How to login on a page and read url content from it?

I have a problem.
I wanted to login e.g. in ogame and read the url source code from it.
But everytime I start my code, my file returns the login page and not the second page after login.
#include <stdio.h>
#include <curl/curl.h>
FILE *fptr;
static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
fptr = fopen("ogametest.html", "w");
fputs((const char*)contents, (FILE*)fptr);
fclose (fptr);
//printf("%s", (char *) contents);
return size * nmemb;
}
int main(int argc, char **argv){
CURL *curl;
CURLcode res;
char strlist[16000]="";
char* str= &strlist[0];
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "https://de.ogame.gameforge.com/");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "usernameLogin=myLogin&passwordLogin=myPassw&serverLogin=s146-de.ogame.gameforge.com");
res = curl_easy_perform(curl);
//curl_easy_reset(curl);
curl_easy_setopt(curl, CURLOPT_URL, "https://s146-de.ogame.gameforge.com/game/index.php?page=overview&relogin=1");
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, str);
res = curl_easy_perform(curl);
fptr = fopen("ogametest.html", "r");
fgets(str, 16000, fptr);
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return 0;
}
It would be fine if you can help me.
Have a good day
There are several issues with your code.
The url is wrong for login. The right url is "https://de.ogame.gameforge.com/main/login"
The post message is not the right format.
You need to handle the cookies. The cookies are used to save the login session. Without the cookies you'll be redirected to main page everytime you make any request at ogame.
Here is some curl code that works to login ogame
CURL* curl;
//Local initializing of curl object
curl = curl_easy_init();
//Defining parameters
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //print html header request/responses in console
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, TRUE);//Follow redirects
curl_easy_setopt(curl, CURLOPT_URL, "https://de.ogame.gameforge.com/main/login");//Url
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookie.txt");//Save cookies here
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookie.txt");//Load cookies here
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "kid=&login=my%40email.here&pass=mypassword&uni=s148-de.ogame.gameforge.com");//Post message
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &writeCallback);//html code response function
curl_easy_setopt(curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30");//Setting agent. Just do this
//Executing curl object
curl_easy_perform(curl);
where
string htmlcode;
size_t writeCallback(char* buf, size_t size, size_t nmemb, void* up){
for (int c = 0; c<size*nmemb; c++)
{
htmldata.push_back(buf[c]);
}
return size*nmemb;}
is globally defined. So in this code the html code will be store in the string htmlcode.
Now the "my%40email.here" should be changed to your email. The %40 is #. "mypassword" should be changed to your password. The "s148-de.ogame.gameforge.com" depends the universe you play on. 148 is Virgo.
I don't know what you're trying to do, but I suggest you use a program to intercept the html requests/responses. In chrome you can go to DevTools Overview (shortcut f12). Under network you can track the exchange between the browser and server, which you can then try to mimic.

Using CURL in function in c++

I'm trying tu use curl in c++ using function.
Example:
#define ...
...
...
/* CURL parameters */
CURL *curl = curl_easy_init();
CURLcode res;
string readBuffer;
void setHeader(){
if(curl) {
/* Headers */
struct curl_slist *chunk = NULL;
chunk = NULL;
chunk = curl_slist_append(chunk, "Connection:keep-alive");
..
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
}
}
void myFunction1(){
setHeader();
curl_easy_setopt(curl, CURLOPT_URL, "....");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
res = curl_easy_perform(curl);
cout << readBuffer;
curl_easy_cleanup(curl);
}
void myFunction2(){
setHeader();
curl_easy_setopt(curl, CURLOPT_URL, "....");
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
res = curl_easy_perform(curl);
cout << readBuffer;
curl_easy_cleanup(curl);
}
In the main function i call myFunction1 or myFunction2;
I don't know if is right to use libcurl in this way, but I'm having a problem.
I use this to login in my site, so I save the cookie whit this code:
curl_easy_setopt(curl, CURLOPT_COOKIEFILE, cookiePath);
curl_easy_setopt(curl, CURLOPT_COOKIEJAR, cookiePath);
int main(){
curl_global_init(CURL_GLOBAL_ALL);
myFunction1();
myFunction2();
curl_global_cleanup();
}
Suppose i login and save the cookie in myFunction1.
When I'm trying to set in the myFunction2 it crash on the CURLOPT_COOKIEFILE row.
I don't know why the cookie is saved in the file, but it can't be used beacuse when I run the program it crash on that row.
Sorry for my English
Thank's
The reason of my problem is that when I use the curl_easy_cleanup(curl) and then curl_easy_init I change the sessionID, so the cookie that I've saved are no longer good. So I use at the first curl_easy_init, then I make all my request in my functions, and at the end of my program I use curl_easy_cleanup. To clean the CURL OPTION you can use: curl_easy_reset(). I've update this post for people that can have my same problem.

Extract specific data from webpage

Basically this is my code :
int main()
{
CURL *curl;
FILE *fp;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
char outfilename[FILENAME_MAX] = "C:\\Users\\admin\\desktop\\test.txt";
if(curl) {
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "user=123&pass=123");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
Sleep(1000);
curl_easy_cleanup(curl);
fclose(fp);
}
return EXIT_SUCCESS;
}
The output is successfully saved in the text file.
My concern is how to extract specific content in between specific tags.
For example i want only the content between < bla> .............. < /bla> .
Whats the easiest way and thank you.
In your Example, you are dumping the response from the website to a file, libcURL writes the data returned by the webpage that you hit as it is, it does not take efforts for restructuring the returned data.
You can obtain the data in a memory, by defining the write_data function, which needs the following format only:
size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata);
Once you get the data in a memory, you can parse it and restructure it as required.
See Example Here for using write_data function.
For XML Parsing you may use This sample code

Curl - checking connection

I used Curl 7.2.9 and checked connection this way:
Here's example:
curl = curl_easy_init();
bool result = false;
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
if(res != CURLE_OK)
{
}
else
{
// connection is available
}
Now I switched to curl-7.33.0 and got *CURLE_WRITE_ERROR* error,
and to make it work I must code it like
std::string output;
char* encodedUrl = curl_easy_escape(curl, m_checkConnectionUrl, 0);
curl_easy_setopt(curl, CURLOPT_POST, 0);
curl_easy_setopt(curl, CURLOPT_URL, encodedUrl);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeMemoryCurlCallbackStub);
CURLcode res = curl_easy_perform(curl);
But I don't need to write anything. Any ideas?
Manily the Curl option *CURLOPT_WRITEFUNCTION* is used to have a certain amount of data periodically(at the callback functoin) to handle a large file download. I don't see any reason to use this with your curl purpose, regardless the version.
Remove the *CURLOPT_POST*(by default its 0) and *CURLOPT_WRITEFUNCTION* from the code and it should work. If it doesn't, then you are doing something wrong at other places in your code!
Also, if you are checking whether the URL is ok or not, then using CURL is ok. But to only check for connection, you can only check whether the port 80 of the domain is on or not.
You need to write a writecallback as well
size_t CurlWriteCallback(char* buf, size_t size, size_t nmemb, void* up)
{
TRACE("CURL - Response received:\n%s", buf);
TRACE("CURL - Response handled %d bytes:\n%s", size*nmemb);
// tell curl how many bytes we handled
return size*nmemb;
}
if(curl)
{
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlWriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, m_checkConnectionUrl);
CURLcode res = curl_easy_perform(curl);
}
Old question, but I have just encountered a similar problem. After some more googling this is the solution:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1L);
curl_easy_perform(curl);
// OK, now we are connected (if nothing bad happened),
// but it would be nice to communicate with the server:
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 0L);
//now we can do the actual communication
I used this to separate authentication from the actual emails sending.