I'm having trouble getting an image from a URL using curl. It works if I pass the URL in as the constructor of an ImageMagick Image object. But using curl I'm not having much luck and I need to use curl.
Right now I'm doing...
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &curlCallback);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_perform(curl);
And then
size_t curlCallback(char* buf, size_t size, size_t nmemb, void* up)
{
ofstream out;
out.open("/home/name/Desktop/img.png");
out.write(buf, nmemb * size);
return size * nmemb;
}
It does seem to get the start of a PNG, but not the whole thing. It only returns 251 bytes (header info or something maybe??). An image viewer will open it as a png and know its resolution, but the image itself is blank. If I print the buffer to console, I see ?PNG and then the binary data symbol.
I know its not a problem with the remote host because if I use ImageMagick:
Image image = Image(url);
Then I get the image in its entirety and can save it and it's just fine.
The function set with CURLOPT_WRITEFUNCTION (curlCallback in your case) can be called multiple times during the download (see the docs).
Using CURLOPT_WRITEDATA passing in a FILE* might be easier.
Related
My Question is closed so I have to update this.
1- My purpose is to send my compressed file google cloud storage URL.
2- To do that I have generated a postman request. I have stored my file to my google cloud storage by using the postman tool and the tool has generated the following code.
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "PUT");
curl_easy_setopt(curl, CURLOPT_URL, "my URL");
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Content-Type:
application/octet-stream");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_POSTFIELDS,"<file contents here>");
res = curl_easy_perform(curl);
}
curl_easy_cleanup(curl);
3- Then I have copied the code above into my c++ project to send my compressed file to the URL.
4- To create CURLOPT_POSTFIELDS content I did implement the following code;
std::ifstream ifs;
ifs.open ("./compressed.gz", std::ios::binary |
std::ios::ate);
PRINT("Size ->", ifs.tellg());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &ifs);
And when I compile and run my code, the request returns me the 200 success response.
But when I checked the google storage dashboard, it just contains 6 bytes of data. Actually, the size of my ifstream data is 1090.
So my problem is that why my request uploads all bytes of the compressed file to cloud storage? Whats wrong in my code ?
How to read compressed files c++
Compressed files are generally binary formats, they are not null terminated text. You cannot use strlen to get their length because that requires a null terminated text as input.
You can use any UnformattedInputFunction to read binary data. Don't forget to open any stream in binary mode.
I need to put the compressed file in a char pointer to put it to the server by using libcurl.
You don't need to read the file in order to do that. You can let libcurl take care of reading the file by passing a FILE* to CURLOPT_READDATA.
This way it won't be necessary to store the entire file in memory. Reading the entire file could be a problem if the file is very large.
You can do something like that when you want to read binary data:
std::ifstream file("./compressed.gz", std::ios::binary);
if (!file.is_open())
ERROR();
std::vector<unsigned char> buffer(std::istreambuf_iterator<char>(file), {});
Then you will have it in the vector buffer. Access the size via buffer.size() and to get raw data use buffer.data() Note that since buffer is an std::vector if it goes outside of scope it will be destructed so the data will be deleted.
I am writing an simple file downloader with a help of libcurl. Here's the code for downloading the file from HTTP server:
static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp) {
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
std::wstring result; //result with polish letters (ą, ę etc.)
CURL *curl;
CURLcode res;
std::string readBuffer;
curl = curl_easy_init();
ERROR_HANDLE(curl, L"CURL could not been inited.", MOD_INTERNET);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_easy_setopt(curl, CURLOPT_USERPWD, (login + ":" + password).c_str()); //e.g.: "login:password"
curl_easy_setopt(curl, CURLOPT_POST, true);
//curl_easy_setopt(curl, CURLOPT_ENCODING, "UTF-8"); //does not change anything
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
result = C::toWString(readBuffer);
return res == 0; //0 = OK
It works fine when the file I want to download is encoded as ANSI (according to e.g. Notepad++). But when I try to download the UTF-8 file (UTF-8 without BOM), I get an error with some characters (e.g. polish letters) due to encoding problem.
For example, I run the code for two files with the same text ("to jest teść to") and saved it to std::wstring. The result is from ANSI file and result2 (problematic) from UTF-8 version:
Both files opened on server with e.g. Notepad++ displays the right text.
So, how can I get the UTF-8 file content with libcurl and save it to std::wstring with the proper encoding (so the debugger of Visual Studio will show it as to jest teść to)?
This is not a libcurl issue. You are storing the raw data in a std::string and then converting that to a std::wstring after the download is finished. You have to look at the charset reported in the HTTP response and decode the data to std::wstring accordingly. C::toWString() has no concept of charsets, so you should use something else, like ICONV or ICU. Or, if you know the data is always UTF-8, do the conversion manually (UTF conversions are easy to code by hand), or use C++11's built in UTF conversions using the std::wstring_convert class.
libcurl won't convert or translate the contents for you. It will deliver the exact bytes to your application that the server sent out.
You can use HTTP Accept headers etc to affect what the server responds, but then you need to check the received charset and convert accordingly by yourself if you're not satisfied with what you get.
I'll preface this by saying I'm still a new C/C++ programmer, so please excuse me for what may be a redundant question.
I'm writing a program in C/C++ to interact with this website: http://www.youtube-mp3.org/.
From what I understand, to get my program to download a link for me I'll have to send a POST request to the server containing the URL I want to convert, then find a way of getting it to follow the URL that is generated allowing me to download the file. I also understand that libcurl is a good way of doing this sort of thing in C/C++.
I've tried using the POST examples on the libcurl website (http://curl.haxx.se/libcurl/c/simplepost.html and one other) but neither seems to work. In addition, I'm not sure how to then get my program to follow the link that appears saying 'Download' . I've tried sending a POST request, then telling my program to get the html source of the page and store this in a file, but that file doesn't seem to contain any download link. When this is done through a browser, the page source definitely includes a working download link.
Would really appreciate some help, as I'm not sure whether I've got completely the wrong idea!
EDIT: My question wasn't very clear at all. Here is the relevant code I'm using for the POST request:
static const char *postthis="http://www.youtube.com/watch?v=KMU0tzLwhbE";
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "http://www.youtube-mp3.org/");
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postthis);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(postthis));
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
And for writing the html source to file:
static size_t write_data(void *ptr, size_t size, size_t nmemb, void *stream)
{
int written = fwrite(ptr, size, nmemb, (FILE *)stream);
return written;
}
{
static const char *filename = "head.txt";
FILE *htmlfile;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
// open the file
htmlfile = fopen(filename,"w");
if (htmlfile == NULL) {
curl_easy_cleanup(curl);
return -1;
}
curl_easy_setopt(curl,CURLOPT_WRITEDATA, htmlfile);
curl_easy_perform(curl);
/* close the header file */
fclose(htmlfile);
/* always clean up */
curl_easy_cleanup(curl);
}
Your code does not work because you are assuming the wrong logic to begin with.
http://www.youtube-mp3.org does NOT use POST, in fact its download form doesn't even submit to a server-side URL at all. When you click on the "Convert Video" button, a client-side JavaScript is invoked to process the input URL, download the relevant information from YouTube, and modify the calling page's HTML to display the actual download link and video preview image. This is why you don't see the download link when you simply retrieve the HTML - you are not invoking the JavaScript that performs the actual work of preparing the download link. And you will not be able to do that from an application (without a LOT of extra work), it has to be done inside of a web browser that has a real JavaScript engine and a real DOM for the script to manipulate.
I have installed cURL, and I was able to download an image from website, and it works fine.
Here is the code:
#define CURL_STATICLIB
#include <stdio.h>
#include <stdlib.h>
#include </usr/include/curl/curl.h>
#include </usr/include/curl/stdcheaders.h>
#include </usr/include/curl/easy.h>
size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written = fwrite(ptr, size, nmemb, stream);
return written;
}
int main(void) {
CURL *curl;
FILE *fp;
CURLcode res;
char *url = "http://www.example.com/test_img.png";
char outfilename[FILENAME_MAX] = "/home/c++_proj/output/web_req_img.png";
curl = curl_easy_init();
if (curl) {
fp = fopen(outfilename,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
/* always cleanup */
curl_easy_cleanup(curl);
fclose(fp);
}
return 0;
}
I also have a dlink DCS-930L camera. I can easily connect my camera a static IP address, and I was able to view live video on the camera, by logging into the camera (e.g. http://192.168.1.5).
I don't need any special software or anything to start watch video.
Now, I would like to use cURL to download images from camera. But I am not sure how to do it.
Could someone please tell me, or provide some piece of code for it?
All I want to do is to capture (sample) few of the images that are being streamed.
How do I know when to make a request, and when would be the boundary between the images.
I would truly appreciate some advise and piece of code that could get me going.
T
According to the manual for this camera [1], you need use a Java or ActiveX plugin to receive and watch the video:
Please make sure that you have the latest version of Java application
installed on your computer to ensure proper operation when viewing the
video in Java mode. The Java application can be downloaded at no cost
from Sun’s web site (http://www.java.com).
When you connect to the home page of your camera, you will be prompted
to download ActiveX. If you want to use ActiveX to view your video
images instead of Java, then you must download ActiveX.
This suggests that grabbing the image is going to be more difficult than simply making an HTTP request.
[1] http://www.dlink.com/us/en/support/product/-/media/Consumer_Products/DCS/DCS%20930L/Manual/DCS%20930L_Manual_EN_US.pdf
I am transferring a binary file (.exe) with FTP using libcurl, and saving it to a local file. The problem is that after the file is transferred, it is altered and is no longer a valid Win32 application, and doesn't run. Here's how I'm doing it:
CURL *curl;
curl = curl_easy_init();
FILE* f = fopen("C:\\blah.exe", "w");
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, "ftp://ftp.mysite.com");
curl_easy_setopt(curl, CURLOPT_USERPWD, "blah:blah");
curl_easy_setopt(curl, CURLOPT_FTP_FILEMETHOD, CURLFTPMETHOD_SINGLECWD);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, NULL);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &f);
} else {
fclose(f);
return CURL_EASY_INIT_FAIL;
}
fclose(f);
The file is written but is bigger than it is on the FTP server. Like I said, trying to run it results in the "%1 is not a valid Win32 application" error. Did I forget to set an option or something?
You forgot the binary flag.
This is the correct code:
FILE* f = fopen("C:\\blah.exe", "wb");
The reason is that you transfer as ASCII and not as binary. So your end of lines might get broken. Of there are CRs in the binary they might turn into CR LF or the other way around. Tune CURL to make a binary transfer.