I've been using libcurl in my Visual C++ programs for a while. I'm trying to write a new program that duplicates this command line:
curl -d 'assetclass=1&viewall=1&action=Go' https://stockmarketmba.com/stockscreener.php
This is the first time that I've had to 'click' on buttons to get the data that I want. This is my code, minus the code that checks the returns of the curl_easy_setopt() calls:
static size_t WriteCallBack( void *contents, size_t size, size_t nmemb, void *userp ) noexcept {
size_t nbytes( size * nmemb );
((std::string*)userp)->append( (char*)contents, nbytes );
return nbytes;
}
ReadWebData() {
CURLcode res;
string bufr;
CURL *handle = curl_easy_init();
curl_easy_setopt( handle, CURLOPT_URL, "https://stockmarketmba.com/stockscreener.php" );
char const *postfields = { "assetclass=1&viewall=1&action=Go" };
curl_easy_setopt( handle, CURLOPT_POSTFIELDS, postfields );
curl_easy_setopt( handle, CURLOPT_POSTFIELDSIZE, (long)strlen( postfields ) );
curl_easy_setopt( this->handle, CURLOPT_WRITEFUNCTION, WriteCallBack );
curl_easy_setopt( this->handle, CURLOPT_WRITEDATA, &bufr )
res = curl_easy_perform( handle );
if ( res == CURLE_OK ) {
// process data in bufr here
}
}
When I look in bufr, it contains the same data that I got before I added the curl_easy_setopt() calls with CURLOPT_POSTFIELDS and CURLOPT_POSTFIELDSIZE. I added curl_easy_setopt() calls with CURLOPT_DEBUGFUNCTION, CURLOPT_DEBUGDATA, and CURLOPT_VERBOSE, and don't see where the post fields are sent to the server.
What am I missing?
Related
I use libcurl to download file, url may be like this below
url = "https://***.com/**/***/abc"
url = "http://***:8000/**/test.txt?***2484d197c16b2e"
url = "http://***:8000/**/test.txt"
It is troublesome to get its file name, so is there a way to get file name using libcurl when downloading it? My download code is below.
size_t writeData(void *ptr, size_t size, size_t nmemb, FILE *stream) {
size_t written = fwrite(ptr, size, nmemb, stream);
return written;
}
int downloadFile(const char *url, const char *saveFile)
{
CURL *curl;
FILE *saveFd;
CURLcode res;
curl = curl_easy_init();
if (curl) {
saveFd = fopen(saveFile,"wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeData);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, saveFd);
std::string strurl = string(url);
if (strurl.find("https") != string::npos) {
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, false);
}
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(saveFd);
}
return 0;
}
Use CURLOPT_HEADERFUNCTION to get a callback for each of the HTTP headers. You are looking for the header called Content-Disposition.
ok so basically i want the response headers after i sendt my post request.
this is my send post code
pDataInfo->recvHeadBuff = (char*)VirtualAlloc(NULL, 4096, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
InitPost(pDataInfo);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 5L);
curl_easy_setopt(curl, CURLOPT_URL, pDataInfo->URL);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, pDataInfo->PostData.c_str());
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_callback);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, pDataInfo->recvHeadBuff);
curl_easy_perform(curl);
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &pDataInfo->repCode);
k so right here im trying to create a buffer that im going to use for my userdata*
callback func:
static size_t header_callback(char* buffer, size_t size,
size_t nitems, void* userdata)
{
strcpy((char*)userdata, buffer);
std::cout << (char*)userdata; // This prints the headers correctly
return nitems * size;
}
now i thought that my buffer would be filled with the characters from the header buffer but when i try to cout the buffer again from outside this callback function i get nothing.
basically need help with a way to use userdata and buffer to get my variable out of this func, ik its noobxd.
Your CURLOPT_HEADERFUNCTION callback is assuming the provided buffer is null-terminated, but it is not. The documentation even says so:
This function gets called by libcurl as soon as it has received header data. The header callback will be called once for each header and only complete header lines are passed on to the callback. Parsing headers is very easy using this. buffer points to the delivered data, and the size of that data is nitems; size is always 1. Do not assume that the header line is null-terminated!
You are also assuming the full header will fit inside the 4K recvHeadBuff that you are allocating, but that is not guaranteed either, so you are risking a buffer overflow.
Try something more like this instead:
std::string headers;
InitPost(pDataInfo);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 5L);
curl_easy_setopt(curl, CURLOPT_URL, pDataInfo->URL);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, pDataInfo->PostData.c_str());
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_callback);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &headers);
curl_easy_perform(curl);
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &pDataInfo->repCode);
// use headers as needed...
DWORD size = (headers.size() + 1) * sizeof(char);
pDataInfo->recvHeadBuff = (char*) VirtualAlloc(NULL, size, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
if (pDataInfo->recvHeadBuff)
CopyMemory(pDataInfo->recvHeadBuff, headers.c_str(), size);
...
static size_t header_callback(char* buffer, size_t size,
size_t nitems, void* userdata)
{
std::string *headers = (std::string*) userdata;
headers->append(buffer, nitems * size);
return nitems * size;
}
Alternatively, you should change pDataInfo->recvHeadBuff to use the same string type that pDataInfo->PostData is using (I'm assuming std::string) rather than being a char*, and then you can do this instead:
InitPost(pDataInfo);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 5L);
curl_easy_setopt(curl, CURLOPT_URL, pDataInfo->URL);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, pDataInfo->PostData.c_str());
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, header_callback);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, &(pDataInfo->recvHeaders));
curl_easy_perform(curl);
curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &pDataInfo->repCode);
...
static size_t header_callback(char* buffer, size_t size,
size_t nitems, void* userdata)
{
std::string *headers = (std::string*) userdata;
headers->append(buffer, nitems * size);
return nitems * size;
}
I have a curl script:
curl -k -H "Content-Type: application/txt" -X POST --data-binary "#name_file.txt" -g "https://ya.com/file.txt" > .\out.txt
I can use it via console ant it works good.
I implement this script in visual studio on C++:
auto curl = curl_easy_init();
if (curl)
{
FILE* fp = fopen(path_to_file.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, "https://ya.com/file.txt");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
struct curl_slist *list = NULL;
list = curl_slist_append(list, "Content-Type: application/txt");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list); // - H
std::string data_for_send = data_for_send_arr.c_str();
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, data_for_send.size());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data_for_send.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, callbackfunction);
res = curl_easy_perform(curl);
if (res != CURLE_OK) AfxMessageBox(_T("error"));
curl_easy_cleanup(curl);
curl_global_cleanup();
fclose(fp);
}
size_t callbackfunction(void *ptr, size_t size, size_t nmemb, void* userdata)
{
FILE* stream = (FILE*)userdata;
if (!stream)
{
printf("!!! No stream\n");
return 0;
}
size_t written = fwrite((FILE*)ptr, size, nmemb, stream);
return written;
}
i also i need to tell that at one moment i use async call:
async(func_curl_1, data_1);
async(func_curl_2, data_2);
each function create curl and do request for server at the same time.
but this script do not work. it work one time from seven request
server returned for me empty file or broken file.
i do not know why...
try to add in your settings CURLOPT_VERBOSE
i.e.
/* ask libcurl to show us the verbose output */
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
and investigate output
Get remote file size without content length.
size_t readHeader(char* header, size_t size, size_t nitems, void *userdata) {
// this works I get the file size when Content-length is available
if (startsWith(header, "Content-Length:")) {
std::string header_in_string = replaceAll(header, "Content-Length:", "");
long size = atol(header_in_string.c_str());
*file_size = size;
}
else { // do something else }
return size * nitems;
}
// main function
if (curl) {
fp = fopen(temp_file_name, "wb");
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_CAINFO, "./ca-bundle.crt");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, false);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, false);
curl_easy_setopt(curl, CURLOPT_HEADERDATA, this);
curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, readHeader);
// writing the downloading content to file
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
This provide the solution in C# via GET
Getting remote file size without content-length but I want to know the solution in C++ with libcurl.
I'm trying to upload an image to the imgur API. I tried with the postman REST client and it returns exactly what I want. When I use libcurl in C++, it returns this:
{"data":{"id":"NsmDwZN","title":null,"description":null,"datetime":1494462656,"type":"image\/png","animated":false,"width":28,"height":22,"size":336,"views":0,"bandwidth":0,"vote":null,"favorite":false,"nsfw":null,"section":null,"account_url":null,"account_id":0,"is_ad":false,"in_most_viral":false,"tags":[],"in_gallery":false,"deletehash":"...","name":"","link":"..."},"success":true,"status":200}
0
Where is that 0 coming from?
Here is my main POST code:
int postToImgur(std::string file)
{
curl_global_cleanup();
CURL *curl;
CURLcode res;
// Init winsock stuff
curl_global_init(CURL_GLOBAL_ALL);
// Get curl handle
curl = curl_easy_init();
struct curl_httppost *formpost = NULL;
struct curl_httppost *lastptr = NULL;
struct curl_slist *headerlist = NULL;
if (curl) {
curl_formadd(&formpost, &lastptr, CURLFORM_COPYNAME, "image", CURLFORM_FILE, file.c_str(), CURLFORM_END);
headerlist = curl_slist_append(headerlist, "Expect:");
headerlist = curl_slist_append(headerlist, "Authorization: Client-ID bea02xxxxxaf4");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt(curl, CURLOPT_URL, "https://api.imgur.com/3/image");
curl_easy_setopt(curl, CURLOPT_HTTPPOST, formpost);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_callback);
res = curl_easy_perform(curl);
if (res != CURLE_OK) {
MessageBox(NULL, _T("Could not upload image. Please check your internet connection."), _T("Error"), NULL);
}
curl_easy_cleanup(curl);
curl_formfree(formpost);
curl_slist_free_all(headerlist);
}
return 0;
}
And in my write_callback function I simply do:
size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
CStringA responseStringA(ptr);
OutputDebugStringA((LPCSTR) responseStringA);
And it outputs a "0". This is breaking my parsing and I'm having to manually delete the 0 but it seems kinda sketch to have to do that. Any suggestions on where I could be getting this 0 from?
Thanks to Dai for the help. Libcurl does not pass the ptr in the response as a null-terminated char array. That was where my error was coming from. I think I have a working solution to create another (bigger) char array, copy the previous one over, and set the last character to the null terminating symbol.
Please check my solution and let me know if I'm messing up strncpy somehow:
size_t write_callback(char *ptr, size_t size, size_t nmemb, void *userdata)
{
char * responseString = new char[nmemb + 1]();
strncpy(responseString, ptr, nmemb);
responseString[nmemb] = '\0';
OutputDebugStringA(responseString);
...
}