C++ libcurl seg fault in write callback function - c++

I'm trying to get something done quick and dirty. I saw another SO question and tried to reuse the code. I'm hitting a couple rest services (not multithreaded) that return json and when the CURLOPT_WRITEFUNCTION is called it throws a seg fault. I'm still trying to grasp all the c++ concepts so it's been pretty difficult diagnosing.
Here's what I see
static std::string *DownloadedResponse;
static size_t writer(char *data, size_t size, size_t nmemb, std::string *buffer_in)
{
cout << "In writer callback" << endl;
// Is there anything in the buffer?
if (buffer_in != NULL)
{
cout << "Buffer not null" << endl;
// Append the data to the buffer
buffer_in->append(data, size * nmemb);
cout <<" Buffer appended, seting response" << endl;
DownloadedResponse = buffer_in;
cout << "Set downloadedResponse" << endl;
return size * nmemb;
}
return 0;
}
std::string downloadJSON(std::string URL)
{
CURL *curl;
CURLcode res;
struct curl_slist *headers=NULL; // init to NULL is important
std::ostringstream oss;
curl_slist_append(headers, "Accept: application/json");
curl_slist_append( headers, "Content-Type: application/json");
curl_slist_append( headers, "charsets: utf-8");
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPGET,1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,writer); // I comment this to display response in stdout.
cout << "calling easy_perform" << endl;
res = curl_easy_perform(curl);
cout << "call made.." << endl;
if (CURLE_OK == res)
{
char *ct;
res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);
if((CURLE_OK == res) && ct)
{
cout << "returning downloaded resposne" << endl;
return *DownloadedResponse;
}
}
else
{
cout << "CURLCode: " << res << endl;
}
}
cout << "Returning null" << endl;
return NULL;
}
Output
$ ./test-rest
calling easy_perform
In writer callback
Buffer not nullSegmentation fault (core dumped)
How am I improperly using the string in the writer callback function?

You forgot to pass in a string pointer/reference with CURLOPT_WRITEDATA.

Related

http challenge response POST with curl

I have written a small c++ program to connect to my router (avm Fritzbox 7590) using a challenge response method.
For this I use openssl (SHA256), curl(GET & POST) and tinyXML (parsing the response).
I have a member function "query_challenge()" to query the FritzBox for a challenge.
This I receive as .xml, this I receive:
<?xml version="1.0" encoding="utf-8"?>
<SessionInfo>
<SID>0000000000000000</SID>
<Challenge>2$60000$*******************$6000$15a84421a7dfc4d9ac8026f776de48f3</Challenge>
<BlockTime>0</BlockTime>
<Rights/>
<Users>
<User last="1">adminxxx</User>
</Users>
</SessionInfo>
With the member function "std::string calculate_pbkdf2_response()" I evaluate the received challenge and send this "query_sessionID()" with a POST to the FritzBox to log in and get a valid SessionID.
When calling my query_sessionID() method, if everything is correct, I should get another XML via the POST in which a valid SID(SessionID) is contained.
However, I get the following response:
<?xml version="1.0" encoding="utf-8"?>
<SessionInfo>
<SID>0000000000000000</SID>
<Challenge>2$60000$diffent*******************$6000$1268d1df004ffdb2fd8074b557467a01</Challenge>
<BlockTime>128</BlockTime>
<Rights/>
<Users>
<User last="1">adminxxx</User>
</Users>
</SessionInfo>
which shows that either the calculated resopnse is incorrect or the POST may not have worked correctly.
I tested my calculate_pbkdf2_response() with the example below.
The example challenge “2$10000$5A1711$2000$5A1722” and the password
“1example!“ (utf8-encoded) results in: hash1 =
pbdkf2_hmac_sha256(“1example!”, 5A1711, 10000)
=> 0x23428e9dec39d95ac7a40514062df0f9e94f996e17c398c79898d0403b332d3b (hex) response = 5A1722$ + pbdkf2_hmac_sha256(hash1, 5A1722,
2000).hex()
=> 5A1722$1798a1672bca7c6463d6b245f82b53703b0f50813401b03e4045a5861e689adb
https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID_english_2021-05-03.pdf
This "5A1722$1798a1672bca7c6463d6b245f82b53703b0f50813401b03e4045a5861e689adb" response gets my calculate_pbkdf2_response() method also out with the respective PW and challenge.
So I suspect that my CURL POST request may not be correct.
Maybe someone has a tip who has more experience with this stuff, this is my first http attempt with CURL as well as the first use of openSSL or challenge response method.
main.cpp:
#include <iostream>
#include "../public/FritzBox_Session.hpp"
int main(int argc, char const *argv[])
{
std::string user = "adminxxx";
std::string pw = "passwort";
homeAutomat::FritzBox::FritzBox_Session TEST(user, pw);
TEST.establish_connection();
return 0;
}
FritzBox_Session.hpp:
#include <iostream>
#include <string>
#include <sstream>
#include <curl/curl.h>
#include <nlohmann/json.hpp>
#include <tinyxml2.h>
#include <openssl/evp.h>
#include <openssl/sha.h>
namespace homeAutomat {
namespace FritzBox {
class FritzBox_Session
{
private:
CURL *curl;
const std::string fritzbox_addr = "http://fritz.box/login_sid.lua?version=2";
std::string mUsername;
std::string mPassword;
std::stringstream mChallenge;
std::string mSID;
tinyxml2::XMLDocument XML_response;
bool query_challenge();
bool query_sessionID();
std::string calculate_pbkdf2_response();
public:
FritzBox_Session(std::string &username, std::string &password);
~FritzBox_Session();
bool establish_connection();
};
} // FritzBox
} // homeAutomat
FritzBox_Session.cpp:
#include "FritzBox_Session.hpp"
using namespace homeAutomat;
using namespace FritzBox;
using namespace nlohmann;
namespace {
// CURL - ReadReceivedData Callback-Fuction
size_t writeFunction(void *ptr, size_t size, size_t nmemb, void *userdata)
{
((std::string*)userdata)->append((char*)ptr, size * nmemb);
return size * nmemb;
}
}
FritzBox_Session::FritzBox_Session(std::string &username, std::string &password)
:
mUsername(username),
mPassword(password)
{
curl = curl_easy_init();
if(!curl){
std::cerr << "curl isn't initialized! " << std::endl;
}
}
FritzBox_Session::~FritzBox_Session()
{
curl_easy_cleanup(curl);
}
bool FritzBox_Session::establish_connection()
{
if(query_challenge()){
if(query_sessionID()){
}
else{
std::cerr << "error at query_sessionID() occurred" << std::endl;
return false;
}
}
else{
std::cerr << "error at query_challenge() occurred" << std::endl;
return false;
}
return true;
}
bool FritzBox_Session::query_challenge()
{
std::string readBuffer;
CURLcode retVal;
if(!curl){
std::cerr << "curl isn't initialized!'" << std::endl;
return 1;
}
curl_easy_setopt(curl, CURLOPT_URL, fritzbox_addr.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeFunction);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
retVal = curl_easy_perform(curl);
if(retVal != CURLE_OK){
std::cerr << "curl_easy_perform() failed: " << curl_easy_strerror(retVal) << std::endl;
return false;
}
int error_id = XML_response.Parse(&readBuffer[0]);
if(error_id != tinyxml2::XML_SUCCESS){
std::cout << "xml parse error" << XML_response.ErrorName() << std::endl;
XML_response.Clear();
return false;
}
auto save_xml_to_file = XML_response.SaveFile("response_query_challenge.xml");
if(save_xml_to_file != tinyxml2::XML_SUCCESS){
std::cout << "xml save to file error: " << XML_response.ErrorName() << std::endl;
}
// search for <SessionInfo>
tinyxml2::XMLElement* sessionInfo = XML_response.FirstChildElement("SessionInfo");
if(sessionInfo == nullptr){
std::cerr << "XML response has no <SessionInfo>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
// search for <SessionInfo> <Challenge> </SessionInfo>
tinyxml2::XMLElement* challenge = sessionInfo->FirstChildElement("Challenge");
if(challenge == nullptr){
std::cerr << "XML response has no <Challenge>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
// get String Value of <SessionInfo> <Challenge> </SessionInfo>
mChallenge << challenge->GetText();
if(mChallenge.str().empty()){
std::cerr << "XML response <Challenge> is empty!" << std::endl;
XML_response.Clear();
return false;
}
XML_response.Clear();
return true;
}
bool FritzBox_Session::query_sessionID()
{
CURLcode retVal;
std::string readBuffer;
std::string response = calculate_pbkdf2_response();
if (!curl) {
std::cerr << "Fehler beim Initialisieren von CURL" << std::endl;
return 1;
}
// HTTP Method 'POST'
curl_easy_setopt(curl, CURLOPT_POST, 1L);
// Set POST Data
std::string post_data = "username=" + mUsername + "&response=" + response;
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, post_data.length());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &post_data[0]);
// Set Content-Type-Header
struct curl_slist* headers = NULL;
headers = curl_slist_append(headers, "Content-Type: application/x-www-form-urlencoded");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
// Set Callback to writeFunction() received data
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeFunction);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
// Send POST data and received answer
retVal = curl_easy_perform(curl);
if(retVal != CURLE_OK){
std::cerr << "Error while sending the request: " << curl_easy_strerror(retVal) << std::endl;
return false;
}
else{
int error_id = XML_response.Parse(&readBuffer[0]);
if(error_id != tinyxml2::XML_SUCCESS){
std::cout << "xml parse error" << XML_response.ErrorName() << std::endl;
}
auto save_xml_to_file = XML_response.SaveFile("response_query_sessionID.xml");
if(save_xml_to_file != tinyxml2::XML_SUCCESS){
std::cout << "xml save to file error: " << XML_response.ErrorName() << std::endl;
}
tinyxml2::XMLElement* sessionInfo = XML_response.FirstChildElement("SessionInfo");
if(sessionInfo == nullptr){
std::cerr << "XML response has no <SessionInfo>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
tinyxml2::XMLElement* SID = sessionInfo->FirstChildElement("SID");
if(SID == nullptr){
std::cerr << "XML response has no <SID>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
mSID = SID->GetText();
if(mSID.empty()){
std::cerr << "XML response <SID> is empty!" << std::endl;
XML_response.Clear();
return false;
}
if(mSID.compare("0000000000000000") == 0){
std::cerr << "username or password wrong!" << std::endl;
}
else{
std::cout << "login was successful SID is: " << mSID << std::endl;
}
return true;
}
}
std::string FritzBox_Session::calculate_pbkdf2_response()
{
// Split the challenge into parts
std::vector<std::string> challenge_parts;
std::string item;
while (std::getline(mChallenge, item, '$'))
{
challenge_parts.push_back(item);
}
// Extract all necessary values encoded into the challenge
int iter1 = std::stoi(challenge_parts[1]);
int iter2 = std::stoi(challenge_parts[3]);
// Convert the salt strings to unsigned char arrays
unsigned char salt1[challenge_parts[2].size() / 2];
unsigned char salt2[challenge_parts[4].size() / 2];
for (int i = 0; i < challenge_parts[2].size(); i += 2) {
sscanf(challenge_parts[2].substr(i, 2).c_str(), "%02x", &salt1[i / 2]);
}
for (int i = 0; i < challenge_parts[4].size(); i += 2) {
sscanf(challenge_parts[4].substr(i, 2).c_str(), "%02x", &salt2[i / 2]);
}
// Hash twice, once with static salt...
unsigned char hash1[SHA256_DIGEST_LENGTH];
PKCS5_PBKDF2_HMAC(mPassword.c_str(), mPassword.size(), salt1, sizeof(salt1), iter1, EVP_sha256(), SHA256_DIGEST_LENGTH, hash1);
// Once with dynamic salt.
unsigned char hash2[SHA256_DIGEST_LENGTH];
PKCS5_PBKDF2_HMAC((const char*)hash1, sizeof(hash1), salt2, sizeof(salt2), iter2, EVP_sha256(), SHA256_DIGEST_LENGTH, hash2);
// Convert the hash2 array to a hexadecimal string
char hash2_hex[SHA256_DIGEST_LENGTH * 2 + 1];
for (int i = 0; i < SHA256_DIGEST_LENGTH; i++) {
sprintf(hash2_hex + i * 2, "%02x", hash2[i]);
}
return challenge_parts[4] + '$' + hash2_hex;
}

C++ libcurl multithreading

I'm trying to build a small multithreading program which takes a subdomains and test them if they are alive on http or https, I've problem that's my program doesn't produce the correct output each time I get different output and also freeze and doesn't continue execution. I followed http://www.cplusplus.com/reference/thread/thread/thread/ when implementing the multithreading.
int main(int argc, char const *argv[] )
{
if (argc < 2){
cout << "Usage httplive <path to subdomains>" << endl;
}
ifstream http(argv[1]);
string line;
vector <std::thread> thread_pool;
while (getline(http, line)){
thread_pool.push_back(thread(httpTest,line, true));
thread_pool.push_back(thread(httpTest, line, false));
}
for (auto& t : thread_pool){
t.join();
}
return 0;
}
void httpTest(string line, bool Flag){
CURL *curl = curl_easy_init();
CURLcode res;
if (curl) {
line = Flag ? "https://" + line : "http://"+ line;
curl_easy_setopt(curl, CURLOPT_URL, const_cast<char*>(line.c_str()));
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
// curl_easy_setopt(curl, CURLOPT_TIMEOUT, 1L);
res = curl_easy_perform(curl);
// cout << res << endl;
if (res == CURLE_OK ) cout << line << endl;
}
curl_easy_cleanup(curl);
}

Why does file upload stop after sending a couple of chunks of data (using multipart/form-data)?

I'm using libcurl to upload a firmware file to a hardware device. I'm using multipart/form-data and it looks like the file upload starts okay but it doesn't load the entire file.
The file I'm uploading is 144,855,725 bytes but it appears that only two 64k chunks are being sent.
In the posted code I am using a read callback function. I had also tried just passing the file name to the curl_mime_filedata function and the results were the same.
One interesting note is that when I run the program I'll get a successful response from the curl_easy_perform about half of the time. The other half of the time I'll get error 56 "Failure when receiving data from the peer".
Another interesting note is that there is a slight discrepancy in the size of the file (144,855,725) and the size that curl perceives as the download size (144,856,042). I assume this is because it is accounting for all of the bytes in the body (not just the file). Is that correct?
Here is some (pared down) output from when I run the program.
File size: 144855725
TOTAL TIME: 0.000092
UP: 0 of 0 DOWN: 0 of 0
UP: 0 of 144856042 DOWN: 0 of 0
ReadCallback: Size=1, Nmemb=65267
We read 65267 bytes from the file
UP: 65536 of 144856042 DOWN: 0 of 0
ReadCallback: Size=1, Nmemb=65536
We read 65536 bytes from the file
UP: 131072 of 144856042 DOWN: 0 of 0
curl result ERROR = <56: Failure when receiving data from the peer>
Failed to upload firmware file
size_t ReadCallback(char *BufferOut, size_t Size, size_t Nmemb, void *StreamIn)
{
curl_off_t nread;
size_t retcode = fread(BufferOut, Size, Nmemb, (FILE *)StreamIn);
nread = (curl_off_t)retcode;
cout << "ReadCallback: Size=" << Size << ", Nmemb=" << Nmemb << endl;
cout << "We read " << nread << " bytes from the file" << endl;
return retcode;
}
int main(void)
{
CURL *pCurl;
CURLcode res;
std::stringstream ss;
struct curl_slist *headerList = NULL;
string accessToken;
struct TransferProgress transProgress;
string filePath;
FILE * pFile;
long lSize;
curl_global_init(CURL_GLOBAL_ALL);
pCurl = curl_easy_init();
if (pCurl)
{
EC520UutComms comms;
curl_mime *multipart;
curl_mimepart *part;
accessToken = comms.GetAccessToken(pCurl);
SetOptionsToDefault(pCurl);
// Specify the target URL
std::string str(comms.BaseURL() + kAPI_Upgrade);
cout << "URL <" + str + ">" << endl;
curl_easy_setopt(pCurl, CURLOPT_URL, str.c_str());
multipart = curl_mime_init(pCurl);
// Add the Content-Disposition
part = curl_mime_addpart(multipart);
curl_mime_name(part, "Content-Disposition");
curl_mime_data(part, "form-data; name=\"upgrade_file\"; filename=\"\"", CURL_ZERO_TERMINATED);
// Add the file
part = curl_mime_addpart(multipart);
curl_mime_type(part, "application/octet-stream");
filePath = "C:\\Temp\\TestFile.tst";
// curl_mime_filedata(part, filePath.c_str());
fopen_s(&pFile, filePath.c_str(), "rb");
// obtain file size:
fseek(pFile, 0, SEEK_END);
lSize = ftell(pFile);
rewind(pFile);
cout << "File size: " << lSize << endl;
curl_mime_data_cb(part, lSize, ReadCallback, NULL, NULL, pFile);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
// This is a long upload, disable the timeout
curl_easy_setopt(pCurl, CURLOPT_TIMEOUT, 0L);
// Create headers
ss.str("");
ss << "Authorization: Bearer " << accessToken;
headerList = curl_slist_append(headerList, ss.str().c_str());
// Accept
headerList = curl_slist_append(headerList,
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
curl_easy_setopt(pCurl, CURLOPT_HTTPHEADER, headerList);
curl_easy_setopt(pCurl, CURLOPT_XFERINFOFUNCTION, TransferInfo);
transProgress.curl = pCurl;
curl_easy_setopt(pCurl, CURLOPT_XFERINFODATA, &transProgress);
curl_easy_setopt(pCurl, CURLOPT_NOPROGRESS, 0L);
// Now send the message
res = curl_easy_perform(pCurl);
curl_slist_free_all(headerList);
curl_mime_free(multipart);
if (res == CURLE_OK)
{
cout << "Firmware file successfully uploaded" << endl;
}
else
{
cout << "curl result ERROR = <" + to_string(res) + ": " + curl_easy_strerror(res) + ">" << endl;
cout << "Failed to upload firmware file" << endl;
}
}
curl_easy_cleanup(pCurl);
}
I expect the entire file to be uploaded rather than just a couple of chunks of the file.
You are not populating the curl_mime structure correctly. The code you have shown does not match the MIME format you described in your previous question:
--1a2fc07a-d882-4470-a1da-79716d34cd9b
Content-Disposition: form-data; name="upgrade_file"; filename=""
Content-Type: application/octet-stream
// File data goes here //
--1a2fc07a-d882-4470-a1da-79716d34cd9b
Content-Disposition: form-data; name="submit"
Install OS
--1a2fc07a-d882-4470-a1da-79716d34cd9b--
Try this instead:
size_t ReadCallback(char *buffer, size_t size, size_t nitems, void *arg)
{
cout << "ReadCallback: size=" << size << ", nitems=" << nitems << endl;
size_t retcode = fread(buffer, size, nitems, (FILE *)arg);
cout << "We read " << retcode << " bytes from the file" << endl;
return retcode;
}
int SeekCallback(void *arg, curl_off_t offset, int origin)
{
if (fseek((FILE *)arg, offset, origin) == 0)
return CURL_SEEKFUNC_OK;
else
return CURL_SEEKFUNC_FAIL;
}
int main()
{
CURL *pCurl;
CURLcode res;
struct curl_slist *headerList = NULL;
string accessToken;
struct TransferProgress transProgress;
string filePath;
FILE * pFile;
long lSize;
curl_global_init(CURL_GLOBAL_ALL);
pCurl = curl_easy_init();
if (pCurl)
{
EC520UutComms comms;
curl_mime *multipart;
curl_mimepart *part;
accessToken = comms.GetAccessToken(pCurl);
SetOptionsToDefault(pCurl);
// Specify the target URL
std::string str(comms.BaseURL() + kAPI_Upgrade);
cout << "URL <" + str + ">" << endl;
curl_easy_setopt(pCurl, CURLOPT_URL, str.c_str());
multipart = curl_mime_init(pCurl);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "upgrade_file");
curl_mime_filename(part, "");
curl_mime_type(part, "application/octet-stream");
filePath = "C:\\Temp\\TestFile.tst";
// curl_mime_filedata(part, filePath.c_str());
fopen_s(&pFile, filePath.c_str(), "rb");
// obtain file size:
fseek(pFile, 0, SEEK_END);
lSize = ftell(pFile);
rewind(pFile);
cout << "File size: " << lSize << endl;
curl_mime_data_cb(part, lSize, ReadCallback, SeekCallback, NULL, pFile);
part = curl_mime_addpart(multipart);
curl_mime_name(part, "submit");
curl_mime_data(part, "Install OS", CURL_ZERO_TERMINATED);
curl_easy_setopt(pCurl, CURLOPT_MIMEPOST, multipart);
// This is a long upload, disable the timeout
curl_easy_setopt(pCurl, CURLOPT_TIMEOUT, 0L);
// Create headers
headerList = curl_slist_append(headerList, ("Authorization: Bearer " + accessToken).c_str());
headerList = curl_slist_append(headerList, "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
curl_easy_setopt(pCurl, CURLOPT_HTTPHEADER, headerList);
curl_easy_setopt(pCurl, CURLOPT_XFERINFOFUNCTION, TransferInfo);
transProgress.curl = pCurl;
curl_easy_setopt(pCurl, CURLOPT_XFERINFODATA, &transProgress);
curl_easy_setopt(pCurl, CURLOPT_NOPROGRESS, 0L);
// Now send the message
res = curl_easy_perform(pCurl);
fclose(pFile);
curl_slist_free_all(headerList);
curl_mime_free(multipart);
if (res == CURLE_OK)
{
cout << "Firmware file successfully uploaded" << endl;
}
else
{
cout << "curl result ERROR = <" + to_string(res) + ": " + curl_easy_strerror(res) + ">" << endl;
cout << "Failed to upload firmware file" << endl;
}
curl_easy_cleanup(pCurl);
}
return 0;
}

Writing Data Fails With CURLOPT_WRITEFUNCTION After Many Writeback Calls

Essentially, I have a write callback that I set for CURLOPT_WRITEFUNCTION. I also set my pointer for writing the data to in CURLOPT_WRITEDATA.
I run an infinite while loop for around 5 seconds, then CURLOPT fails to right data to the void* up pointer defined in CURLOPT_WRITEDATA, then after that one failure it starts working again. I got tons of successful writes sprinkled with failures. Is this a memory issue, and if so, is there a way to circumvent reaching this issue altogether?
// these locations will likely be different on your local.
#include "../cygwin64/usr/include/curl/curl.h"
#include "../cygwin64/usr/include/json/json.h"
size_t write_callback(char *buf, size_t size, size_t nmemb, void* up) {
size_t num_bytes = size*nmemb;
std::string* data = (std::string*) up;
for(int i = 0; i < num_bytes; i++) {
data->push_back(buf[i]);
}
return num_bytes;
}
CURL* init_curl(struct curl_slist *headers, std::string* chunk) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_callback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *) chunk);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
return curl;
}
bool curl_get(struct curl_slist *headers, const std::string& url,
Json::Value* json_res) {
// this is where my data should be written to
std::string data;
CURL* curl = init_curl(headers, &data);
CURLcode res;
bool success = true;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
res = curl_easy_perform(curl);
if(res != CURLE_OK) {
std::cout << "Could not perform get for " << url << std::endl;
std::cout << curl_easy_strerror(res) << std::endl;
success = false;
} else {
Json::Value json_data;
Json::CharReaderBuilder json_reader;
std::istringstream stream_data(data);
std::string errs;
if(Json::parseFromStream(json_reader, stream_data, &json_data, &errs)) {
std::cout << "successfully parsed JSON data for: " << url << std::endl;
*json_res = json_data;
} else {
std::cout << "failed to parse JSON data for: " << url << std::endl;
std::cout << errs << std::endl;
std::cout << json_data << std::endl;
std::cout << "finished failing" << std::endl;
success = false;
}
}
} else {
success = false;
}
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
return success;
}
int main(int argc, char** argv) {
curl_global_init(CURL_GLOBAL_ALL);
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
while (true) {
Json::Value response;
if (curl_get(headers, "https://api.robinhood.com/quotes/?symbols=AMZN", &response)) {
std::cout << response << std::endl;
} else {
// reaches here sometimes because response (my data) is null
std::cout << "failed to get last trade price" << std::endl;
}
}
curl_global_cleanup();
return 0;
}
Expect no failures, but I get intermittent failures to right the data to my WRITEDATA chunk.

curl_easy_getinfo does not return correct size after upload

I have this code as below
#include <stdio.h>
#include <curl/curl.h>
#include <iostream>
using namespace std;
size_t read_file_data(char* buffer, size_t size,
size_t nitems, void* instream) {
printf("\nUpload file binary data");
cout << "Read" << nitems << " items by " << size << " bytes" << endl;
size_t retcode = fread(buffer, size, nitems,
static_cast<FILE*>(instream));
size_t retsize = size * retcode;
cout << "Read " << retsize << "bytes from file\n";
return retsize;
}
int main(void)
{
CURL* curl = curl_easy_init();
if (!curl) {
return 1;
}
curl_easy_setopt(curl, CURLOPT_URL, "http://172.30.191.145:3000/upload");
struct curl_slist* chunk = NULL;
chunk = curl_slist_append(chunk, "Transfer-Encoding: chunked");
chunk = curl_slist_append(chunk,
"Content-type:application/octet-stream");
chunk = curl_slist_append(chunk, "Session: test");
CURLcode result = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
FILE* fp =
fopen("/home/thinh/Documents/SDL/curlupload/testfile.txt","rb");
curl_easy_setopt(curl, CURLOPT_POST, 1L);
curl_easy_setopt(curl, CURLOPT_READFUNCTION,
&read_file_data);
curl_easy_setopt(curl, CURLOPT_READDATA, fp);
result = curl_easy_perform(curl);
cout << "curl result " << result << endl;
CURLcode get_info_result;
double val = 0;
get_info_result = curl_easy_getinfo(curl, CURLINFO_SIZE_UPLOAD, &val);
if (CURLE_OK == get_info_result) {
printf("Size of uploaded data: %0.0f bytes.\n", val);
} else {
printf("ERROR");
}
curl_easy_cleanup(curl);
curl_slist_free_all(chunk);
fclose(fp);
return 0;
}
After execute, the Size of uploaded data always larger than actual file size.
But the file after uploaded in server is correct(same with local file)
I use ll command to check file size
problem in this line
get_info_result = curl_easy_getinfo(curl, CURLINFO_SIZE_UPLOAD, &val);
For example: I upload local file text.txt 1022 bytes, but in log: "Size of uploaded data: 1034 bytes.", in server it is 1022 bytes.
text.txt content:
abcd123
abcd123
abcd123
I couldn't find any document talk detail about upload file size.
After change this line:
chunk = curl_slist_append(chunk, "Transfer-Encoding: chunked");
to
chunk = curl_slist_append(chunk, "Transfer-Encoding: ");
I get correct size in log.
The chunked encoding modifies the body of a message in order to transfer it as a series of chunks, each with its own size indicator, followed by an optional trailer containing entity-header fields Ref.
But I have to use chunked. Is there any way to get correct uploaded data