I have started working with libcurl, and simply tried running the basic code to get a file from an url. When I get this file using curl.exe, compiled with the same library, I get detect no random traffic on my localhost. However, when I run with my own executable, I get around 19 packets sent between two localhost ports.
I make sure to call curl_global_init(CURL_GLOBAL_WIN32) and curl_global_cleanup() after the method call.
What could be the cause of this traffic, and how could I make it go away?
int CurlFileDownloader::downloadSingleFile(const std::string& url, const std::string& destination) {
CURLcode res = CURLE_READ_ERROR;
mHandle = curl_easy_init();
if(mHandle) {
mData.destinationFolder = destination;
// Get the file name from the url
auto lastPos = url.find_last_of("/");
mData.fileName = url.substr(lastPos + 1);
curl_easy_setopt(mHandle, CURLOPT_URL, url.c_str());
/* Define our callback to get called when there's data to be written */
curl_easy_setopt(mHandle, CURLOPT_WRITEFUNCTION, &CurlFileDownloader::writeFileContent);
/* Set a pointer to our struct to pass to the callback */
curl_easy_setopt(mHandle, CURLOPT_WRITEDATA, &mData);
/* Switch on full protocol/debug output */
curl_easy_setopt(mHandle, CURLOPT_VERBOSE, 1L);
mLastError = curl_easy_perform(mHandle);
/* always cleanup */
curl_easy_cleanup(mHandle);
if (mData.fileStream.is_open()) {
mData.fileStream.close();
}
if(CURLE_OK != mLastError) {
std::cerr << "Curl error " << mLastError << std::endl;
}
}
return mLastError;
}
size_t CurlFileDownloader::writeFileContent(char *buffer, size_t size, size_t nmemb, void *cb_data) {
struct CurlCallbackData *data = (CurlCallbackData*)cb_data;
size_t written = 0;
if (data->fileStream.is_open()) {
data->fileStream.write(buffer, nmemb);
}
else {
/* listing output */
if (data->destinationFolder != "") {
data->fileStream.open(data->destinationFolder + "\\" + data->fileName, std::ios::out | std::ios::binary);
}
else {
data->fileStream.open(data->fileName, std::ios::out | std::ios::binary);
}
data->fileStream.write(buffer, nmemb);
}
return nmemb;
}
Here is a sample of what RawCap.exe is capturing.
The source of the localhost communication was the use of a socket pair for IPV4 Loopback. When removing the #USE_SOCKETPAIR from libCURL's socketpair.h, the issue went away.
Related
How do you add a prefix for the output of command execution with c++
localhost is a Flask web application
std::string exec(const char* cmd) {
std::array<char, 128> buffer;
std::string result;
std::unique_ptr<FILE, decltype(&_pclose)> pipe(_popen(cmd, "r"), _pclose);
if (!pipe) {
throw std::runtime_error("popen() failed!");
}
while (fgets(buffer.data(), buffer.size(), pipe.get()) != nullptr) {
result += buffer.data();
//std::cout << typeid(result).name() << std::endl;
// read form pipe and add to the output string
std::string output = "output=";
output += buffer.data()
std::cout << output << std::endl;
// call report_ to send a post request to the server
report_(output);
}
char* c = const_cast<char*>(result.c_str());
return result;
}
As far as I understand this is a c++ function that returns a string value of the output from the command prompt
int report_(std::string report )
{
CURL* curl;
CURLcode res;
/* In windows, this will init the winsock stuff */
curl_global_init(CURL_GLOBAL_ALL);
/* get a curl handle */
curl = curl_easy_init();
if (curl) {
/* First set the URL that is about to receive our POST. This URL can
just as well be a https:// URL if that is what should receive the
data. */
curl_easy_setopt(curl, CURLOPT_URL, "http://localhost/api/00000000000000000000/report");
/* Now specify the POST data */
// report starts with "output="
std::cout << report << std::endl;
// this is where we add the post data
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, output );
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return 0;
}
This function reports the output of the exec() function but before you do that you have to add the prefix output= to the output of exec() which takes a string as an argument
The server returns
400 Bad Request: The browser (or proxy) sent a request that this server could not understand. KeyError: 'output'
If you change curl_easy_setopt(curl, CURLOPT_POSTFIELDS, output ); to curl_easy_setopt(curl, CURLOPT_POSTFIELDS, "output=hello world" ); then the server receives the output
This link explains how to add post data to post fields you have to pass a pointer to the data you want to send so using const char*
// this line refers to the pointer of the string needed to be send over
// just replace output with and std::string value and you can send it as post data
// do not use std::string as post data
const char* c = const_cast<char*>(output.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, c );
My application uses libcurl to upload files to an SFTP server, and sometimes I need to close the application before the current upload is complete. I would like to abort the upload, and resume it later. However, when the upload is aborted there is corrupted data at the end of the partially uploaded file on the server. The amount various, up to about 10 bytes. This makes resuming the upload by appending data impossible.
See some sample code below. Press Ctrl-C to abort the upload, then manually download the partial file and compare it with the original file. The partial file will usually contain incorrect data at the end.
#include <stdio.h>
#include <signal.h>
#include "curl.h"
// Handle Ctrl-C to abort.
static bool s_bAborted = false;
static void __cdecl ctrlCHandler(int)
{
printf("Aborting...\n");
s_bAborted = true;
}
// File reading function.
static size_t readFunction(char * ptr, size_t size, size_t nmemb, void * stream)
{
FILE * pFile = (FILE *)stream;
if (ferror(pFile))
{
return CURL_READFUNC_ABORT;
}
return fread(ptr, size, nmemb, pFile);
}
// Progress function so transfers can be aborted.
static int progressFunction(void * clientp, curl_off_t dltotal, curl_off_t dlnow, curl_off_t ultotal, curl_off_t ulnow)
{
if (s_bAborted)
{
return 1;
}
return CURL_PROGRESSFUNC_CONTINUE;
}
int main()
{
// Add your local file and server URL.
// Note: This problem was observed using an SFTP server.
const char * szLocalFile = "Test.bin";
const char * szRemoteUrl = "sftp://user:password#host/Test.bin";
curl_global_init(CURL_GLOBAL_ALL);
CURL * curl = curl_easy_init();
FILE * pFile = fopen(szLocalFile, "rb");
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_URL, szRemoteUrl);
curl_easy_setopt(curl, CURLOPT_READFUNCTION, readFunction);
curl_easy_setopt(curl, CURLOPT_READDATA, pFile);
curl_easy_setopt(curl, CURLOPT_APPEND, 0L);
curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 0L);
curl_easy_setopt(curl, CURLOPT_XFERINFOFUNCTION, progressFunction);
printf("Uploading... Press Ctrl-C to abort\n");
signal(SIGINT, ctrlCHandler);
CURLcode eResult = curl_easy_perform(curl);
fclose(pFile);
if (eResult == CURLE_OK)
{
printf("Uploaded OK\n");
}
else if (eResult == CURLE_ABORTED_BY_CALLBACK)
{
printf("Upload aborted\n");
}
else
{
printf("Error: Upload failed: %d\n", eResult);
}
curl_easy_cleanup(curl);
curl_global_cleanup();
// After aborting the upload, use something else to download the partial file.
// The last few bytes of the partial file are usually incorrect,
// i.e. different to the original file in that position.
// This makes resuming the upload impossible, since it will append after incorrect bytes.
}
Please can anyone help? Thank you.
Following the advice of rustyx above, I modified it to abort using the read function, and removed the progress callback. This appears to fix the problem (after testing about 15 times).
Note: This method means the partial file is always a multiple of 64 kB. When aborting from the progress callback it was any random size. Perhaps this has something to do with it.
See updated code below:
#include <stdio.h>
#include <signal.h>
#include "curl.h"
// Handle Ctrl-C to abort.
static bool s_bAborted = false;
static void __cdecl ctrlCHandler(int)
{
s_bAborted = true;
}
// File reading function.
static size_t readFunction(char * ptr, size_t size, size_t nmemb, void * stream)
{
FILE * pFile = (FILE *)stream;
if (s_bAborted || ferror(pFile))
{
return CURL_READFUNC_ABORT;
}
return fread(ptr, size, nmemb, pFile);
}
int main()
{
// Add your local file and server URL.
// Note: This problem was observed using an SFTP server.
const char * szLocalFile = "Test.bin";
const char * szRemoteUrl = "sftp://user:password#host/Test.bin";
curl_global_init(CURL_GLOBAL_ALL);
CURL * curl = curl_easy_init();
FILE * pFile = fopen(szLocalFile, "rb");
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
curl_easy_setopt(curl, CURLOPT_URL, szRemoteUrl);
curl_easy_setopt(curl, CURLOPT_READFUNCTION, readFunction);
curl_easy_setopt(curl, CURLOPT_READDATA, pFile);
curl_easy_setopt(curl, CURLOPT_APPEND, 0L);
printf("Uploading... Press Ctrl-C to abort\n");
signal(SIGINT, ctrlCHandler);
CURLcode eResult = curl_easy_perform(curl);
fclose(pFile);
if (eResult == CURLE_OK)
{
printf("Uploaded OK\n");
}
else if (eResult == CURLE_ABORTED_BY_CALLBACK)
{
printf("Upload aborted\n");
}
else
{
printf("Error: Upload failed: %d\n", eResult);
}
curl_easy_cleanup(curl);
curl_global_cleanup();
}
I'm trying to play with libcurl SMTP and everything works fine with this example:
http://curl.haxx.se/libcurl/c/smtp-mail.html
#include <stdio.h>
#include <string.h>
#include <curl/curl.h>
/* This is a simple example showing how to send mail using libcurl's SMTP
* capabilities. For an exmaple of using the multi interface please see
* smtp-multi.c.
*
* Note that this example requires libcurl 7.20.0 or above.
*/
#define FROM "<sender#example.org>"
#define TO "<addressee#example.net>"
#define CC "<info#example.org>"
static const char *payload_text[] = {
"Date: Mon, 29 Nov 2010 21:54:29 +1100\r\n",
"To: " TO "\r\n",
"From: " FROM "(Example User)\r\n",
"Cc: " CC "(Another example User)\r\n",
"Message-ID: <dcd7cb36-11db-487a-9f3a-e652a9458efd#rfcpedant.example.org>\r\n",
"Subject: SMTP example message\r\n",
"\r\n", /* empty line to divide headers from body, see RFC5322 */
"The body of the message starts here.\r\n",
"\r\n",
"It could be a lot of lines, could be MIME encoded, whatever.\r\n",
"Check RFC5322.\r\n",
NULL
};
struct upload_status {
int lines_read;
};
static size_t payload_source(void *ptr, size_t size, size_t nmemb, void *userp)
{
struct upload_status *upload_ctx = (struct upload_status *)userp;
const char *data;
if((size == 0) || (nmemb == 0) || ((size*nmemb) < 1)) {
return 0;
}
data = payload_text[upload_ctx->lines_read];
if(data) {
size_t len = strlen(data);
memcpy(ptr, data, len);
upload_ctx->lines_read++;
return len;
}
return 0;
}
int main(void)
{
CURL *curl;
CURLcode res = CURLE_OK;
struct curl_slist *recipients = NULL;
struct upload_status upload_ctx;
upload_ctx.lines_read = 0;
curl = curl_easy_init();
if(curl) {
/* This is the URL for your mailserver */
curl_easy_setopt(curl, CURLOPT_URL, "smtp://mail.example.com");
/* Note that this option isn't strictly required, omitting it will result in
* libcurl sending the MAIL FROM command with empty sender data. All
* autoresponses should have an empty reverse-path, and should be directed
* to the address in the reverse-path which triggered them. Otherwise, they
* could cause an endless loop. See RFC 5321 Section 4.5.5 for more details.
*/
curl_easy_setopt(curl, CURLOPT_MAIL_FROM, FROM);
/* Add two recipients, in this particular case they correspond to the
* To: and Cc: addressees in the header, but they could be any kind of
* recipient. */
recipients = curl_slist_append(recipients, TO);
recipients = curl_slist_append(recipients, CC);
curl_easy_setopt(curl, CURLOPT_MAIL_RCPT, recipients);
/* We're using a callback function to specify the payload (the headers and
* body of the message). You could just use the CURLOPT_READDATA option to
* specify a FILE pointer to read from. */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, payload_source);
curl_easy_setopt(curl, CURLOPT_READDATA, &upload_ctx);
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* Send the message */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
/* Free the list of recipients */
curl_slist_free_all(recipients);
/* curl won't send the QUIT command until you call cleanup, so you should be
* able to re-use this connection for additional messages (setting
* CURLOPT_MAIL_FROM and CURLOPT_MAIL_RCPT as required, and calling
* curl_easy_perform() again. It may not be a good idea to keep the
* connection open for a very long time though (more than a few minutes may
* result in the server timing out the connection), and you do want to clean
* up in the end.
*/
curl_easy_cleanup(curl);
}
return (int)res;
}
but if I greatly increases a payload function, I get an error:
read function returned funny value
Segmentation fault
How could I solve this problem?
The following code is to send a plain text message
rather than a email message. but the basic ideas are the same.
did you set the curl write function option to point to a callback function?
curl_easy_setopt(curlHandle, CURLOPT_WRITEFUNCTION, &CurlWriteFunction);
//////////////////
where the CurlWriteFunction() would look similar to the following:
//////////////////
size_t CurlWriteFunction(void *ptr, size_t size, size_t nmemb, void *stream)
{
memset(CurlMsgStr, 0, MAX_MSG_STR);
phpCurlError = FALSE;
if(((char *)ptr)[0] == '1' || ((char *)ptr)[1] == '1')
{
phpCurlError = FALSE;
sprintf(CurlMsgStr,"Record sent to URL OK...[%c%c]\n",
((char *)ptr)[0],((char *)ptr)[1]);
}
else if(((char *)ptr)[0] == '0' || ((char *)ptr)[1] == '0')
{
phpCurlError = TRUE;
sprintf(CurlMsgStr,"Record sent to URL FAILED...[%c%c]\n",
((char *)ptr)[0],((char *)ptr)[1]);
}
else
{
phpCurlError = TRUE;
sprintf(CurlMsgStr,"UNKNOWN URL RETURN VALUE...[%c%c]\n",
((char *)ptr)[0],((char *)ptr)[1]);
}
return (nmemb);
} // CurlWriteFunction()
////////////////////
Then send the curl message with something like this:
////////////////////
void sendViaCurl(char *str)
{
#if 0 // eliminate heap thrashing of 8k malloc/free
char* postData = NULL;
#else
static char* postData = NULL;
#endif
CURLcode curlErr;
long curlRespCode = 0;
char recNameStr[MAX_STRING_LENGTH];
if(curlHandle)
{
// Specify the POST data
if(postData == NULL) // then first time through
{
postData = malloc(MAX_RESPONSE_LENGTH * 2);
}
memset(postData,0,MAX_RESPONSE_LENGTH * 2);
sprintf(postData, "%s", str);
curl_easy_setopt(curlHandle, CURLOPT_POSTFIELDS, postData);
// Execute the POST, response goes to STDOUT
curlErr = curl_easy_perform(curlHandle);
// Report any CURL errors
if(curlErr)
{
log_error("CURL ERROR -- %s\n", curl_easy_strerror(curlErr));
}
// Get info on the POST data transfer to URL
curl_easy_getinfo(curlHandle, CURLINFO_RESPONSE_CODE, &curlRespCode);
// log_error( "CURL RESPONSE CODE -- %ld\n", curlRespCode);
if(phpCurlError)
{
log_error( " %s rec rejected by DB -- %s\n", recNameStr, str);
curlRecRejected = TRUE;
}
else
{
log_error( "%s [%s] rec accepted by DB...\n",recNameStr, str);
curlRecRejected = FALSE;
}
}
else
{
log_error( "NOT ABLE TO GET CURL HANDLE...RECORD NOT SENT\n");
curlRecRejected = TRUE;
}
#if 0 // eliminate thrashing of heap with 8k malloc/free
if(postData)
{
free(postData);
postData = NULL;
}
#endif
return;
} // sendViaCurl()
In windows, this code works file, but now, I want to convert it into ubuntu:
// callback read function to upload file from local to ftp server
size_t read_callback (void* ptr, size_t size, size_t nmemb, FILE *stream){
//return fread(ptr,size,nmemb, (FILE*) stream);
return fread(ptr,size,nmemb,stream);
}
// get file name from a path
string FTPClientConnector::getFileName(string path){
int length = path.size();
for(int i = length - 1; i >= 0; i--){
if(path[i] == '/' || path[i] == '\\'){
return path.substr(i+1, length-i-1);
}
}
}
//function to upload a file to FTP server
int FTPClientConnector::uploadFile(string filePath, string serverPath ){
CURL *curl;
CURLcode res;
FILE *hd_src;
struct stat file_info;
curl_off_t fsize;
char* local_file = new char[filePath.size()+1];
std::copy(filePath.begin(), filePath.end(), local_file);
local_file[filePath.size()] = '\0';
// stat the local file
if(stat(local_file, &file_info)){
printf("couldn't open file\n");
delete local_file;
return -1;
}
// convert URL and username and password to connect to remote server
string urlPath = this->hostName + serverPath;
urlPath += getFileName(filePath);
char *url = new char[urlPath.size() + 1];
std::copy(urlPath.begin(), urlPath.end(), url);
url[urlPath.size()] = '\0';
string userAndPassString = this->userName + ":" + this->password;
char* usernameAndPassword = new char[userAndPassString.size() +1];
std::copy(userAndPassString.begin(), userAndPassString.end(), usernameAndPassword);
usernameAndPassword[userAndPassString.size()] = '\0';
// get the file to open
hd_src = fopen(local_file, "rb");
curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
if(curl){
/* specify target */
curl_easy_setopt(curl,CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_PORT, this->port);
curl_easy_setopt(curl, CURLOPT_USERPWD, usernameAndPassword);
/* we want to use our own read function */
curl_easy_setopt(curl, CURLOPT_READFUNCTION, read_callback);
/* enable uploading */
curl_easy_setopt(curl, CURLOPT_UPLOAD, 1L);
/* now specify which file to upload */
curl_easy_setopt(curl, CURLOPT_READDATA, hd_src);
/* Now run off and do what you've been told! */
res = curl_easy_perform(curl);
if(res != CURLE_OK){
printf("Upload file failed!\n");
delete local_file;
delete url;
delete usernameAndPassword;
return -1;
}
curl_easy_cleanup(curl);
}
fclose(hd_src);
delete local_file;
delete url;
delete usernameAndPassword;
return 0;
}
This is what I call in main.cpp:
FTPClientConnector connector(host,user,password,port);
connector.uploadFile("xml/kingfisher.xml", "/xml_test_upload");
The code above doesn't work in Ubuntu with errors:
220 ProFTPD 1.3.4a Server (Debian) [::ffff:10.244.31.244]
500 PUT not understood
500 AUTHORIZATION: not understood
500 HOST: not understood
550 */*: Forbidden command argument
500 TRANSFER-ENCODING: not understood
500 EXPECT: not understood
500 Invalid command: try being more creative
500 2A2 not understood
Edit: This is my Makefile:
uploader:
g++ -o uploader FTPClientConnector.cpp main.cpp -lcurl
The output seems to indicate that you speak HTTP to a FTP server. Make sure your URL properly uses a FTP:// prefix for FTP, as without a protocol prefix libcurl guesses which protocol you want and it defaults to HTTP...
It appears from your comments that you need to use IPv4. Add this to your list of setopt calls:
curl_easy_setopt(curl, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
I have a problem when using CURL to get MJPEG Stream from IP Camera Panasonic.
Here is my code.
int _tmain(int argc, _TCHAR* argv[])
{
CURL *curl;
CURLcode res;
/* Minimalistic http request */
char request[1000];
strcpy(request, "GET /nphMotionJpeg?Resolution=640x480&Quality=Standard HTTP/1.1\r\n\r\n") ;
curl_socket_t sockfd; /* socket */
long sockextr;
size_t iolen;
curl = curl_easy_init();
if(curl)
{
curl_easy_setopt(curl, CURLOPT_URL, "http://192.168.1.253");
curl_easy_setopt(curl, CURLOPT_USERPWD, "my_usr:my_pass");
/* Do not do the transfer - only connect to host */
curl_easy_setopt(curl, CURLOPT_CONNECT_ONLY, 1L);
res = curl_easy_perform(curl);
if(CURLE_OK != res)
{
printf("Error: %s\n", strerror(res));
return 1;
}
res = curl_easy_getinfo(curl, CURLINFO_LASTSOCKET, &sockextr);
if(CURLE_OK != res)
{
printf("Error: %s\n", curl_easy_strerror(res));
return 1;
}
sockfd = sockextr;
/* wait for the socket to become ready for sending */
if(wait_on_socket(sockfd, false, 6000L) < 0)
{
printf("Error: timeout.\n");
return 1;
}
iolen = 0;
res = curl_easy_send(curl, request, strlen(request), &iolen);
if(CURLE_OK != res)
{
printf("Error: %s\n", curl_easy_strerror(res));
return 1;
}
//puts("Reading response.");
/* read the response */
for(;;)
{
char* buf = new char[1024*100];
iolen = 0;
wait_on_socket(sockfd, true, 60000L);
res = curl_easy_recv(curl, buf, 1024*100, &iolen);
if(CURLE_OK != res)
break;
fstream f;
f.open("out.txt", ios::out|ios::binary|ios::app);
if(!f.fail())
{
f.write(buf,iolen);
f.close();
}
delete []buf;
}
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}
The result is content in buf after the first loop as description of Panasonic document.
But it only have 3 frames jpeg in data responding. and total size only is 3056 bytes. That mean is about 1KB/Jpeg image.It is wrong.
And in the second loop, the curl_easy_recv() always return CURLE_UNSUPPORTED_PROTOCOL.
I also change the request string same as description of Panasonic document:
"GET http://xxx.xxx.xxx.xxx:yy/nphMotionJpeg?Resolution=640x480&Quality=Standard HTTP/1.0\r\n"
OR
"GET http://usr:pass#xxx.xxx.xxx.xxx:yy/nphMotionJpeg?Resolution=640x480&Quality=Standard HTTP/1.0\r\n"
--> it will response "BAD REQUEST".
The model of my camera is Panasonic BL-C111CE.
Here's a much better example source code you could start from:
http://curl.haxx.se/libcurl/c/url2file.html
You really SHOULD avoid CURLOPT_CONNECT_ONLY, curl_easy_recv() and curl_easy_send() unless you know PERFECTLY well what you're doing and why the "normal" API isn't good enough. More often than otherwise they are the wrong answer to your problems. libcurl does HTTP perfectly fine on its own already!
Nitpick: at one point you call strerror() on a return code from libcurl, which won't show the correct error string...
I have found out the problem.
The reason with command "GET /nphMotionJpeg?Resolution=640x480&Quality=Standard HTTP/1.1\r\n\r\n" is not permitted to access video.
I solve it by changing the setting in my camera that can permit Guest user access to video.
Thanks all!