Essentially, I have a write callback that I set for CURLOPT_WRITEFUNCTION. I also set my pointer for writing the data to in CURLOPT_WRITEDATA.
I run an infinite while loop for around 5 seconds, then CURLOPT fails to right data to the void* up pointer defined in CURLOPT_WRITEDATA, then after that one failure it starts working again. I got tons of successful writes sprinkled with failures. Is this a memory issue, and if so, is there a way to circumvent reaching this issue altogether?
// these locations will likely be different on your local.
#include "../cygwin64/usr/include/curl/curl.h"
#include "../cygwin64/usr/include/json/json.h"
size_t write_callback(char *buf, size_t size, size_t nmemb, void* up) {
size_t num_bytes = size*nmemb;
std::string* data = (std::string*) up;
for(int i = 0; i < num_bytes; i++) {
data->push_back(buf[i]);
}
return num_bytes;
}
CURL* init_curl(struct curl_slist *headers, std::string* chunk) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_callback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, (void *) chunk);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
return curl;
}
bool curl_get(struct curl_slist *headers, const std::string& url,
Json::Value* json_res) {
// this is where my data should be written to
std::string data;
CURL* curl = init_curl(headers, &data);
CURLcode res;
bool success = true;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
res = curl_easy_perform(curl);
if(res != CURLE_OK) {
std::cout << "Could not perform get for " << url << std::endl;
std::cout << curl_easy_strerror(res) << std::endl;
success = false;
} else {
Json::Value json_data;
Json::CharReaderBuilder json_reader;
std::istringstream stream_data(data);
std::string errs;
if(Json::parseFromStream(json_reader, stream_data, &json_data, &errs)) {
std::cout << "successfully parsed JSON data for: " << url << std::endl;
*json_res = json_data;
} else {
std::cout << "failed to parse JSON data for: " << url << std::endl;
std::cout << errs << std::endl;
std::cout << json_data << std::endl;
std::cout << "finished failing" << std::endl;
success = false;
}
}
} else {
success = false;
}
curl_slist_free_all(headers);
curl_easy_cleanup(curl);
return success;
}
int main(int argc, char** argv) {
curl_global_init(CURL_GLOBAL_ALL);
struct curl_slist *headers = NULL;
headers = curl_slist_append(headers, "Accept: application/json");
while (true) {
Json::Value response;
if (curl_get(headers, "https://api.robinhood.com/quotes/?symbols=AMZN", &response)) {
std::cout << response << std::endl;
} else {
// reaches here sometimes because response (my data) is null
std::cout << "failed to get last trade price" << std::endl;
}
}
curl_global_cleanup();
return 0;
}
Expect no failures, but I get intermittent failures to right the data to my WRITEDATA chunk.
Related
I have written a small c++ program to connect to my router (avm Fritzbox 7590) using a challenge response method.
For this I use openssl (SHA256), curl(GET & POST) and tinyXML (parsing the response).
I have a member function "query_challenge()" to query the FritzBox for a challenge.
This I receive as .xml, this I receive:
<?xml version="1.0" encoding="utf-8"?>
<SessionInfo>
<SID>0000000000000000</SID>
<Challenge>2$60000$*******************$6000$15a84421a7dfc4d9ac8026f776de48f3</Challenge>
<BlockTime>0</BlockTime>
<Rights/>
<Users>
<User last="1">adminxxx</User>
</Users>
</SessionInfo>
With the member function "std::string calculate_pbkdf2_response()" I evaluate the received challenge and send this "query_sessionID()" with a POST to the FritzBox to log in and get a valid SessionID.
When calling my query_sessionID() method, if everything is correct, I should get another XML via the POST in which a valid SID(SessionID) is contained.
However, I get the following response:
<?xml version="1.0" encoding="utf-8"?>
<SessionInfo>
<SID>0000000000000000</SID>
<Challenge>2$60000$diffent*******************$6000$1268d1df004ffdb2fd8074b557467a01</Challenge>
<BlockTime>128</BlockTime>
<Rights/>
<Users>
<User last="1">adminxxx</User>
</Users>
</SessionInfo>
which shows that either the calculated resopnse is incorrect or the POST may not have worked correctly.
I tested my calculate_pbkdf2_response() with the example below.
The example challenge “2$10000$5A1711$2000$5A1722” and the password
“1example!“ (utf8-encoded) results in: hash1 =
pbdkf2_hmac_sha256(“1example!”, 5A1711, 10000)
=> 0x23428e9dec39d95ac7a40514062df0f9e94f996e17c398c79898d0403b332d3b (hex) response = 5A1722$ + pbdkf2_hmac_sha256(hash1, 5A1722,
2000).hex()
=> 5A1722$1798a1672bca7c6463d6b245f82b53703b0f50813401b03e4045a5861e689adb
https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AVM_Technical_Note_-_Session_ID_english_2021-05-03.pdf
This "5A1722$1798a1672bca7c6463d6b245f82b53703b0f50813401b03e4045a5861e689adb" response gets my calculate_pbkdf2_response() method also out with the respective PW and challenge.
So I suspect that my CURL POST request may not be correct.
Maybe someone has a tip who has more experience with this stuff, this is my first http attempt with CURL as well as the first use of openSSL or challenge response method.
main.cpp:
#include <iostream>
#include "../public/FritzBox_Session.hpp"
int main(int argc, char const *argv[])
{
std::string user = "adminxxx";
std::string pw = "passwort";
homeAutomat::FritzBox::FritzBox_Session TEST(user, pw);
TEST.establish_connection();
return 0;
}
FritzBox_Session.hpp:
#include <iostream>
#include <string>
#include <sstream>
#include <curl/curl.h>
#include <nlohmann/json.hpp>
#include <tinyxml2.h>
#include <openssl/evp.h>
#include <openssl/sha.h>
namespace homeAutomat {
namespace FritzBox {
class FritzBox_Session
{
private:
CURL *curl;
const std::string fritzbox_addr = "http://fritz.box/login_sid.lua?version=2";
std::string mUsername;
std::string mPassword;
std::stringstream mChallenge;
std::string mSID;
tinyxml2::XMLDocument XML_response;
bool query_challenge();
bool query_sessionID();
std::string calculate_pbkdf2_response();
public:
FritzBox_Session(std::string &username, std::string &password);
~FritzBox_Session();
bool establish_connection();
};
} // FritzBox
} // homeAutomat
FritzBox_Session.cpp:
#include "FritzBox_Session.hpp"
using namespace homeAutomat;
using namespace FritzBox;
using namespace nlohmann;
namespace {
// CURL - ReadReceivedData Callback-Fuction
size_t writeFunction(void *ptr, size_t size, size_t nmemb, void *userdata)
{
((std::string*)userdata)->append((char*)ptr, size * nmemb);
return size * nmemb;
}
}
FritzBox_Session::FritzBox_Session(std::string &username, std::string &password)
:
mUsername(username),
mPassword(password)
{
curl = curl_easy_init();
if(!curl){
std::cerr << "curl isn't initialized! " << std::endl;
}
}
FritzBox_Session::~FritzBox_Session()
{
curl_easy_cleanup(curl);
}
bool FritzBox_Session::establish_connection()
{
if(query_challenge()){
if(query_sessionID()){
}
else{
std::cerr << "error at query_sessionID() occurred" << std::endl;
return false;
}
}
else{
std::cerr << "error at query_challenge() occurred" << std::endl;
return false;
}
return true;
}
bool FritzBox_Session::query_challenge()
{
std::string readBuffer;
CURLcode retVal;
if(!curl){
std::cerr << "curl isn't initialized!'" << std::endl;
return 1;
}
curl_easy_setopt(curl, CURLOPT_URL, fritzbox_addr.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeFunction);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
retVal = curl_easy_perform(curl);
if(retVal != CURLE_OK){
std::cerr << "curl_easy_perform() failed: " << curl_easy_strerror(retVal) << std::endl;
return false;
}
int error_id = XML_response.Parse(&readBuffer[0]);
if(error_id != tinyxml2::XML_SUCCESS){
std::cout << "xml parse error" << XML_response.ErrorName() << std::endl;
XML_response.Clear();
return false;
}
auto save_xml_to_file = XML_response.SaveFile("response_query_challenge.xml");
if(save_xml_to_file != tinyxml2::XML_SUCCESS){
std::cout << "xml save to file error: " << XML_response.ErrorName() << std::endl;
}
// search for <SessionInfo>
tinyxml2::XMLElement* sessionInfo = XML_response.FirstChildElement("SessionInfo");
if(sessionInfo == nullptr){
std::cerr << "XML response has no <SessionInfo>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
// search for <SessionInfo> <Challenge> </SessionInfo>
tinyxml2::XMLElement* challenge = sessionInfo->FirstChildElement("Challenge");
if(challenge == nullptr){
std::cerr << "XML response has no <Challenge>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
// get String Value of <SessionInfo> <Challenge> </SessionInfo>
mChallenge << challenge->GetText();
if(mChallenge.str().empty()){
std::cerr << "XML response <Challenge> is empty!" << std::endl;
XML_response.Clear();
return false;
}
XML_response.Clear();
return true;
}
bool FritzBox_Session::query_sessionID()
{
CURLcode retVal;
std::string readBuffer;
std::string response = calculate_pbkdf2_response();
if (!curl) {
std::cerr << "Fehler beim Initialisieren von CURL" << std::endl;
return 1;
}
// HTTP Method 'POST'
curl_easy_setopt(curl, CURLOPT_POST, 1L);
// Set POST Data
std::string post_data = "username=" + mUsername + "&response=" + response;
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, post_data.length());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, &post_data[0]);
// Set Content-Type-Header
struct curl_slist* headers = NULL;
headers = curl_slist_append(headers, "Content-Type: application/x-www-form-urlencoded");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
// Set Callback to writeFunction() received data
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writeFunction);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
// Send POST data and received answer
retVal = curl_easy_perform(curl);
if(retVal != CURLE_OK){
std::cerr << "Error while sending the request: " << curl_easy_strerror(retVal) << std::endl;
return false;
}
else{
int error_id = XML_response.Parse(&readBuffer[0]);
if(error_id != tinyxml2::XML_SUCCESS){
std::cout << "xml parse error" << XML_response.ErrorName() << std::endl;
}
auto save_xml_to_file = XML_response.SaveFile("response_query_sessionID.xml");
if(save_xml_to_file != tinyxml2::XML_SUCCESS){
std::cout << "xml save to file error: " << XML_response.ErrorName() << std::endl;
}
tinyxml2::XMLElement* sessionInfo = XML_response.FirstChildElement("SessionInfo");
if(sessionInfo == nullptr){
std::cerr << "XML response has no <SessionInfo>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
tinyxml2::XMLElement* SID = sessionInfo->FirstChildElement("SID");
if(SID == nullptr){
std::cerr << "XML response has no <SID>-Variable!" << std::endl;
XML_response.Clear();
return false;
}
mSID = SID->GetText();
if(mSID.empty()){
std::cerr << "XML response <SID> is empty!" << std::endl;
XML_response.Clear();
return false;
}
if(mSID.compare("0000000000000000") == 0){
std::cerr << "username or password wrong!" << std::endl;
}
else{
std::cout << "login was successful SID is: " << mSID << std::endl;
}
return true;
}
}
std::string FritzBox_Session::calculate_pbkdf2_response()
{
// Split the challenge into parts
std::vector<std::string> challenge_parts;
std::string item;
while (std::getline(mChallenge, item, '$'))
{
challenge_parts.push_back(item);
}
// Extract all necessary values encoded into the challenge
int iter1 = std::stoi(challenge_parts[1]);
int iter2 = std::stoi(challenge_parts[3]);
// Convert the salt strings to unsigned char arrays
unsigned char salt1[challenge_parts[2].size() / 2];
unsigned char salt2[challenge_parts[4].size() / 2];
for (int i = 0; i < challenge_parts[2].size(); i += 2) {
sscanf(challenge_parts[2].substr(i, 2).c_str(), "%02x", &salt1[i / 2]);
}
for (int i = 0; i < challenge_parts[4].size(); i += 2) {
sscanf(challenge_parts[4].substr(i, 2).c_str(), "%02x", &salt2[i / 2]);
}
// Hash twice, once with static salt...
unsigned char hash1[SHA256_DIGEST_LENGTH];
PKCS5_PBKDF2_HMAC(mPassword.c_str(), mPassword.size(), salt1, sizeof(salt1), iter1, EVP_sha256(), SHA256_DIGEST_LENGTH, hash1);
// Once with dynamic salt.
unsigned char hash2[SHA256_DIGEST_LENGTH];
PKCS5_PBKDF2_HMAC((const char*)hash1, sizeof(hash1), salt2, sizeof(salt2), iter2, EVP_sha256(), SHA256_DIGEST_LENGTH, hash2);
// Convert the hash2 array to a hexadecimal string
char hash2_hex[SHA256_DIGEST_LENGTH * 2 + 1];
for (int i = 0; i < SHA256_DIGEST_LENGTH; i++) {
sprintf(hash2_hex + i * 2, "%02x", hash2[i]);
}
return challenge_parts[4] + '$' + hash2_hex;
}
I'm trying to build a small multithreading program which takes a subdomains and test them if they are alive on http or https, I've problem that's my program doesn't produce the correct output each time I get different output and also freeze and doesn't continue execution. I followed http://www.cplusplus.com/reference/thread/thread/thread/ when implementing the multithreading.
int main(int argc, char const *argv[] )
{
if (argc < 2){
cout << "Usage httplive <path to subdomains>" << endl;
}
ifstream http(argv[1]);
string line;
vector <std::thread> thread_pool;
while (getline(http, line)){
thread_pool.push_back(thread(httpTest,line, true));
thread_pool.push_back(thread(httpTest, line, false));
}
for (auto& t : thread_pool){
t.join();
}
return 0;
}
void httpTest(string line, bool Flag){
CURL *curl = curl_easy_init();
CURLcode res;
if (curl) {
line = Flag ? "https://" + line : "http://"+ line;
curl_easy_setopt(curl, CURLOPT_URL, const_cast<char*>(line.c_str()));
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data);
// curl_easy_setopt(curl, CURLOPT_TIMEOUT, 1L);
res = curl_easy_perform(curl);
// cout << res << endl;
if (res == CURLE_OK ) cout << line << endl;
}
curl_easy_cleanup(curl);
}
I downloaded the source code of Curl and built the library (libcurl.lib). Following is the code to read from the site and dump the contents.
The code works well for http sites and fails for https. I tried downloading openssl libraries but unable to link them as more linker errors are thrown.
What is the best solution to handle this?
#include "stdafx.h"
#include <iostream>
#include <string>
#include <curl/curl.h>
static size_t WriteCallback(void *contents, size_t size, size_t nmemb, void *userp)
{
((std::string*)userp)->append((char*)contents, size * nmemb);
return size * nmemb;
}
int main(void)
{
CURL *curl;
CURLcode res;
std::string readBuffer;
std::string curl_url = "https://www.example.com/";
curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, curl_url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, WriteCallback);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &readBuffer);
res = curl_easy_perform(curl);
if (res != CURLE_OK) {
std::cout << "Error from cURL: " << curl_easy_strerror(res) << std::endl;
}
curl_easy_cleanup(curl);
std::cout << "Finished reading from the website" << std::endl;
std::cout << readBuffer << std::endl;
}
return 0;
}
i'm a beginner in c++. I want to send request to a API, for this i use libcurl, and stock the response on a string and copy the string in a file. it is my test file :
#include <iostream>
#include <string>
#include <curl/curl.h>
#include <fstream>
int MyCurlObject::curlWriter(char *data, size_t size, size_t nmemb, std::string *buffer) {
int result = 0;
if (buffer != NULL) {
buffer->append(data, size * nmemb);
result = size * nmemb;
}
return result;
}
int main (){
std::string url = "https://www.google.com/";
std::string content;
curl = curl_easy_init();
if(!curl)
{
std::cerr << "impossible d'initialiser curl." << std::endl;
}
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &MyCurlObject::curlWriter);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &content);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_USERAGENT, "libcurl-agent/1.0");
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
const CURLcode rc = curl_easy_perform(curl);
if( rc != CURLE_OK ) {
std::cout << "Error from cURL: " << curl_easy_strerror(rc) << std::endl;
}
std::ofstream file(fileName);
if(!file){
std::cerr << "can't open this file : " << fileName << std::endl;
}
file << content;
file.close();
return 0;
}
My files contain all of my string, but line in file not contain end of line symbol ( i display all of symbol with notepade++ and i just see CR in end of line ) and if i make this :
std::ifstream file(name);
if(file)
{
std::string crash;
int nbrOfLine = 0;
while(getline(file, crash))
{
std::cout << crash;
nbrOfLine++;
}
}
return 1 but my file contain 1500 lines.
thank you in advance
I'm trying to get something done quick and dirty. I saw another SO question and tried to reuse the code. I'm hitting a couple rest services (not multithreaded) that return json and when the CURLOPT_WRITEFUNCTION is called it throws a seg fault. I'm still trying to grasp all the c++ concepts so it's been pretty difficult diagnosing.
Here's what I see
static std::string *DownloadedResponse;
static size_t writer(char *data, size_t size, size_t nmemb, std::string *buffer_in)
{
cout << "In writer callback" << endl;
// Is there anything in the buffer?
if (buffer_in != NULL)
{
cout << "Buffer not null" << endl;
// Append the data to the buffer
buffer_in->append(data, size * nmemb);
cout <<" Buffer appended, seting response" << endl;
DownloadedResponse = buffer_in;
cout << "Set downloadedResponse" << endl;
return size * nmemb;
}
return 0;
}
std::string downloadJSON(std::string URL)
{
CURL *curl;
CURLcode res;
struct curl_slist *headers=NULL; // init to NULL is important
std::ostringstream oss;
curl_slist_append(headers, "Accept: application/json");
curl_slist_append( headers, "Content-Type: application/json");
curl_slist_append( headers, "charsets: utf-8");
curl = curl_easy_init();
if (curl)
{
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_URL, URL.c_str());
curl_easy_setopt(curl, CURLOPT_HTTPGET,1);
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,writer); // I comment this to display response in stdout.
cout << "calling easy_perform" << endl;
res = curl_easy_perform(curl);
cout << "call made.." << endl;
if (CURLE_OK == res)
{
char *ct;
res = curl_easy_getinfo(curl, CURLINFO_CONTENT_TYPE, &ct);
if((CURLE_OK == res) && ct)
{
cout << "returning downloaded resposne" << endl;
return *DownloadedResponse;
}
}
else
{
cout << "CURLCode: " << res << endl;
}
}
cout << "Returning null" << endl;
return NULL;
}
Output
$ ./test-rest
calling easy_perform
In writer callback
Buffer not nullSegmentation fault (core dumped)
How am I improperly using the string in the writer callback function?
You forgot to pass in a string pointer/reference with CURLOPT_WRITEDATA.