uploading file with libcurl - c++

Take a look at the following code
static size_t reader(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t retcode = fread(ptr, size, nmemb, stream);
cout << "*** We read " << retcode << " bytes from file" << endl;
return retcode;
}
void upload() { //upload() is called from ouside
FILE *pFile;
pFile = fopen("map.txt" , "r");
struct stat file_info;
stat("map.txt", &file_info);
size_t size = (size_t)file_info.st_size;
uploadFile(pFile, size);
}
bool uploadFile(void* data, size_t datasize) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if (curl) {
char *post_params = ...;
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_params);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long) strlen(post_params));
curl_easy_setopt(curl, CURLOPT_READFUNCTION, reader);
curl_easy_setopt(curl, CURLOPT_READDATA, data);
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t) datasize);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
return true;
}
When the code is executed, the following is outputed
*** We read 490 bytes from file
*** We read 0 bytes from file
after that the app does nothing (even not exiting).
Can someone point out at what's wrong here?
Will be grateful for any help!!!

There's some serious confusions shown in this code. Let me try to explain:
CURLOPT_UPLOAD - this will ask libcurl to PUT the file when the protocol of choice is HTTP
CURLOPT_POSTFIELDS - tells libcurl to POST the data that is provided in the additional argument (which has the size set with CURLOPT_POSTFIELDSIZE)
CURLOPT_READFUNCTION - provides libcurl an alternative way to get data than CURLOPT_POSTFIELDS to allow a POST that reads the data from a file. When using CURLOPT_UPLOAD this is the only way to provide data.
So in the end the questions left for you are:
Do you want PUT or POST?
Do you want to provide the data as a string or do you want it provided with a callback?

Related

libcurl downloads no data to buffer

I am using following code to download data from an url to memory (stream). Around 2% chance, the size of the stream is zero. I can download proper data from the same failing url if I try it another time. I am not sure if this is a network issue, CPU usage issue, or it's just the code not covering some corner cases. Please advice. Thanks!
static size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata)
{
std::vector<uchar> *stream = (std::vector<uchar>*)userdata;
size_t count = size * nmemb;
stream->insert(stream->end(), ptr, ptr + count);
return count;
}
static void CurlUrl(const char* img_url, std::vector<uchar>* stream) {
CURL *curl = curl_easy_init(); // curl_global_init is called eleswhere.
curl_easy_setopt(curl, CURLOPT_NOSIGNAL, 1);
curl_easy_setopt(curl, CURLOPT_URL, img_url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, stream);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 10);
CURLcode res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
If it didn't deliver any download data into the buffer via the callback, it means that the transfer either failed or that there was exactly zero bytes to transfer.
Check the return code from curl_easy_perform() as it might actually tell you exactly what happened.
Use CURLOPT_VERBOSE to see what's going on if (1) is not enough.
Use CURLOPT_ERRORBUFFER to get a better error description if it fails if (2) is not enough.

C++ libcurl how to send FTP file usign STOR and REST

I have an XML that I want to incrementally upload to the ftp server.
The problem is that it does not work for me with an APPEND, because I have to eliminate the final labels. For example.
File on the server
< xml>< root>< a>ssss< /a>< b>kkkkk< /b>< /root>
File in local
< xml>< root>< a>ssss< /a>< b>kkkkk< /b>< c>hhhhh< /c>< /root>
I need to upload from before < /root>.
< xml>< root>< a>ssss< /a>< b>kkkkk< /b>< /root>
----------------------------------------^
My code is the following
size_t read_callback(void *ptr, size_t size, size_t nmemb, void *stream) {
curl_off_t nread;
size_t retcode = fread(ptr, size, nmemb, (FILE*) stream);
nread = (curl_off_t) retcode;
fprintf(stderr, "*** We read %" CURL_FORMAT_CURL_OFF_T " bytes from file\n", nread);
return retcode;
}
bool uploadFile(std::string urlFtp, std::string rutaFtp, std::string archivoOrigen, std::string userPassword, long posicion) {
bool resultado = false;
try {
CURL *curl;
CURLcode res;
FILE *hd_src;
struct stat file_info;
curl_off_t fsize;
/* get the file size of the local file */
if (stat(archivoOrigen.c_str(), &file_info)) {
LOGF_ERROR("Couldn't open '%s': %s\n", archivoOrigen.c_str(), strerror(errno));
return false;
}
fsize = (curl_off_t) file_info.st_size;
LOGF_INFO("Local file size: %" CURL_FORMAT_CURL_OFF_T " bytes.\n", fsize);
hd_src = fopen(archivoOrigen.c_str(), "rb"); // get a FILE * of the same file
curl_global_init(CURL_GLOBAL_ALL); // In windows, this will init the winsock stuff
curl = curl_easy_init(); // get a curl handle
if (curl) {
// build a list of commands to pass to libcurl
struct curl_slist *headerlist = NULL;
std::string cadena(rutaFtp);
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback); // we want to use our own read function
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L); // enable uploading
cadena = urlFtp;
cadena.append(rutaFtp);
curl_easy_setopt(curl, CURLOPT_URL, cadena.c_str()); // specify target
curl_easy_setopt(curl, CURLOPT_USERPWD, userPassword.c_str());
if (posicion > 0) {
curl_easy_setopt(curl, CURLOPT_RESUME_FROM, posicion);
}
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src); // now specify which file to upload
curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t) fsize);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, 15L);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl); // Now run off and do what you've been told!
if (res != CURLE_OK)
LOGF_ERROR("curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
else
resultado = true;
curl_slist_free_all(headerlist); // clean up the FTP commands list
curl_easy_cleanup(curl);
}
fclose(hd_src); // close the local file
curl_global_cleanup();
} catch (std::exception &e) {
LOGF_ERROR(e.what());
}
return resultado;
}
I have not been able to solve the problem with CURL. In order to solve it I created a class in C ++ that makes the upload. I noticed the documentation of RFC rfc3659 page 13.
5.5. REST Example
Assume that the transfer of a largish file has previously been
interrupted after 802816 octets had been received, that the previous
transfer was with TYPE=I, and that it has been verified that the file
on the server has not since changed.
C> TYPE I
S> 200 Type set to I.
C> PORT 127,0,0,1,15,107
S> 200 PORT command successful.
C> REST 802816
S> 350 Restarting at 802816. Send STORE or RETRIEVE
C> RETR cap60.pl198.tar
S> 150 Opening BINARY mode data connection
[...]
S> 226 Transfer complete.
Instead of putting the RETR I have put a STOR and it works correctly.

C++ CURL post file and then get response error

I'm trying to POST a file to an API and then check the response for various things. But whenever I try to POST a file that is larger than 0 bytes i get an error:
First-chance exception at 0x77AADF63 (ntdll.dll) in Test.exe:
0xC0000005: Access violation writing location 0x00000014.
The relevant code is:
std::string Network::Post(std::string url, std::map<std::string, std::string> requestBody, std::string file, int port) {
std::ostringstream response;
std::stringstream bodyStream;
struct stat file_info;
for (std::map<std::string, std::string>::iterator iterator = requestBody.begin(); iterator != requestBody.end(); iterator++) {
bodyStream << iterator->first << "=" << iterator->second << "&";
}
//Get file
FILE* file_handle = fopen(file.c_str(), "rb");
int t = fstat(_fileno(file_handle), &file_info);
CURL *ch;
curl_global_init(CURL_GLOBAL_ALL);
ch = curl_easy_init();
struct curl_slist *headers = NULL;
curl_slist_append(headers, "Content-Type: application/x-www-form-urlencoded");
std::string bodyStreamString = bodyStream.str();
const char* bodyStreamCharPtr = bodyStreamString.c_str();
curl_easy_setopt(ch, CURLOPT_URL, url.c_str());
curl_easy_setopt(ch, CURLOPT_PORT, port);
curl_easy_setopt(ch, CURLOPT_POSTFIELDS, bodyStreamCharPtr);
curl_easy_setopt(ch, CURLOPT_POSTFIELDSIZE, bodyStream.str().length());
curl_easy_setopt(ch, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(ch, CURLOPT_AUTOREFERER, 1);
curl_easy_setopt(ch, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(ch, CURLOPT_COOKIEFILE, "cookies.dat");
curl_easy_setopt(ch, CURLOPT_COOKIEJAR, "cookies.dat");
curl_easy_setopt(ch, CURLOPT_WRITEFUNCTION, &ResponseToString);
curl_easy_setopt(ch, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(ch, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(ch, CURLOPT_READDATA, file_handle);
curl_easy_setopt(ch, CURLOPT_INFILESIZE_LARGE, (curl_off_t)file_info.st_size);
curl_easy_setopt(ch, CURLOPT_WRITEDATA, &response);
int res = curl_easy_perform(ch);
curl_easy_cleanup(ch);
return response.str();
}
size_t Network::ResponseToString(char *ptr, size_t size, size_t nmemb, void *stream) {
std::ostringstream *s = (std::ostringstream*)stream;
size_t count = size * nmemb;
s->write(ptr, count);
return count;
}
Any idea what's happenning? Been stuck on it for a day or so, sigh :(
More details:
It just breaks at " int res = curl_easy_perform(ch);" , can't step into it.
If I remove the WRITEFUNCTION and WRITEDATA options , it works, but then I can't get the response.
The problem doesn't seem to be in the ResponseToString method either.
The problem is your WRITEFUNC.
I'm quite sure Network::ResponseToString is not static, and this cause the problem, since a non static function will pass "this" as first argument and then screw your function parameters, I suggest to avoid using a class member at all and use this (to be placed in the same file as Network::Post and before Post method definition):
static size_t ResponseToString(char *ptr, size_t size, size_t nmemb, std::ostringstream *stream) {
size_t count = size * nmemb;
stream->write(ptr, count);
return count;
}
Remember also to change:
curl_easy_setopt(ch, CURLOPT_WRITEFUNCTION, &ResponseToString);
in:
curl_easy_setopt(ch, CURLOPT_WRITEFUNCTION, ResponseToString);
Also, a lot of the headers you specify in curl opt declaration are not needed if your scope is a simple "post":
curl_easy_setopt(ch, CURLOPT_POSTFIELDSIZE, bodyStream.str().length());
curl_easy_setopt(ch, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(ch, CURLOPT_READDATA, file_handle);
curl_easy_setopt(ch, CURLOPT_INFILESIZE_LARGE, (curl_off_t)file_info.st_size);
Decided to pipe all the requests from my app through Fiddler and I discovered that nothing was being sent and back came an error (sent 0 bytes out of n). So i poked around a bit and discovered the issue.
Aaand, the problem was my lack of a READFUNCTION. Solved it like this:
curl_easy_setopt(ch, CURLOPT_READDATA, file_handle);
curl_easy_setopt(ch, CURLOPT_READFUNCTION, &RequestToFile);
And this is the read function. Apparently it's required on windows.
size_t EasyCloud::Network::RequestToFile(void *ptr, size_t size, size_t nmemb, FILE *stream) {
curl_off_t nread;
size_t retcode = fread(ptr, size, nmemb, stream);
nread = (curl_off_t)retcode;
return retcode;
}

Extract specific data from webpage

Basically this is my code :
int main()
{
CURL *curl;
FILE *fp;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
char outfilename[FILENAME_MAX] = "C:\\Users\\admin\\desktop\\test.txt";
if(curl) {
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, "http://www.example.com");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "user=123&pass=123");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
Sleep(1000);
curl_easy_cleanup(curl);
fclose(fp);
}
return EXIT_SUCCESS;
}
The output is successfully saved in the text file.
My concern is how to extract specific content in between specific tags.
For example i want only the content between < bla> .............. < /bla> .
Whats the easiest way and thank you.
In your Example, you are dumping the response from the website to a file, libcURL writes the data returned by the webpage that you hit as it is, it does not take efforts for restructuring the returned data.
You can obtain the data in a memory, by defining the write_data function, which needs the following format only:
size_t write_data(char *ptr, size_t size, size_t nmemb, void *userdata);
Once you get the data in a memory, you can parse it and restructure it as required.
See Example Here for using write_data function.
For XML Parsing you may use This sample code

"Failed writing body" CURLOPT_WRITEDATA

The below code is to get response from a server using wsdl, here the problem is curl returns response but am unable to print it.
Error:
Failed writing body
Failed writing data
#include<stdio.h>
#include<string.h>
#include"../include/curl.h"
size_t write_data(void *ptr, size_t size, size_t count, void *stream)
{
/* ptr - your string variable.
stream - data chuck you received */
printf("%.*s", size, (char*)stream);
}
int main()
{
int res=0,i=0;
char buffer[4098]="",buff[128]="",buf[256]="",buf7[30]="",buf6[30]="",buf5[30]="";
char machineid[]="SUBANI";
char filename1[50]="";
int refno=0,paymode=0,taxtype=0;
FILE *fbc;
memset(filename1,0,sizeof(filename1));
sprintf(filename1,"/mnt/jffs2/Response_Details1.xml");
lk_dispclr();
lk_disptext(1,0,(unsigned char *)"Sending Request",0);
lk_disptext(2,0,(unsigned char *)"Please Wait",0);
memset(buffer,0,sizeof(buffer));
sprintf(buffer,"<?xml version=\"1.0\" encoding=\"utf-8\"?>\
<soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:log=\"http://wsdlclassess.application.sims.test.com\">\
<soap:Header>\
</soap:Header>\
<soap:Body>\
<log:loginMethod>\
<log:loginid>%s</log:loginid>\
<log:password>%s</log:password>\
</log:loginMethod>\
</soap:Body>\
</soap:Envelope>","raja","test");
res=GET_FILE1(buffer,filename1);
return 0;
}
int GET_FILE1(char *buffer,char *filename)
{
CURL *curl;
CURLcode res;
struct curl_slist *headers = NULL;
FILE *out_fd = (FILE *) 0;
char errorbuf[300] = "",tmpbuff[128]="";
char errmsg[256];
int Timeout=120; //Default timeout is = 2 mins
int buffer_size = 0;
char urlbuff[256]="";
char mstr[10240];
memset(urlbuff,0,sizeof(urlbuff));
memset(tmpbuff,0,sizeof(tmpbuff));
buffer_size = strlen(buffer);
strcpy(tmpbuff,"http://10.10.1.111:8081/test_server/services/application?wsdl");
tmpbuff[strlen(tmpbuff)]='\0';
curl = curl_easy_init();
if(curl)
{
out_fd = fopen (filename, "w");
curl_easy_setopt(curl, CURLOPT_FILE, out_fd);
printf("%s:Sign-In Request\n", __func__);
headers = curl_slist_append(headers, "Content-type:application/soap+xml; charset=utf-8; action=\"http://wsdlclassess.application.sims.test.com/loginMethod\"");
curl_easy_setopt(curl, CURLOPT_URL, tmpbuff);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, mstr);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, buffer_size);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, buffer);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
curl_easy_setopt(curl, CURLOPT_TIMEOUT, Timeout);
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER,errmsg);
printf("The Server%s:Performing Transaction.....\n",__func__);
res = curl_easy_perform(curl);
printf("res=after culreasey perform%d\n",res);
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
printf("\nerrorbuf:%s\n",errmsg);
fclose(out_fd);
if(CURLE_OK != res)
{
puts("error occured is\n" );
//ppp_close();
return -1;
}
}
return 0;
}
The error is that you don't return the correct value from the function, in fact you don't return anything.
Also, the data provided to the function is actually the first ptr argument.
I agree that the documentation is not very clear, but it says:
The size of the data pointed to by ptr is size multiplied with nmemb, it will not be zero terminated.
The above line (emphasis mine) tells you that the data is in ptr which is the first argument in the function declaration provided in the documentation.
The documentation also states:
Return the number of bytes actually taken care of. If that amount differs from the amount passed to your function, it'll signal an error to the library. This will abort the transfer and return CURLE_WRITE_ERROR.
You don't return a value from the function, and so you have undefined behavior with a seemingly random value being returned causing the whole operation to fail. To fix this you should return size * count.
You also uses size to print the string, which is the size of the underlying type used (probably 1), your count variable is the number of characters read by CURL. To be fully working, without invoking more undefined behavior (since the data is not terminated) you should call printf like:
printf("%*.*s", size * count, size * count, ptr);