Incomplete response string from server in implementing POST-requests - c++

I’m writing the program sending POST-requests to the server and getting answers from it. (using curl library) (OS Linux, Red Hat Linux 3.2.2.-5). Sometimes I see , that response from server contains only second part of the message. ( I print _sResponse and sometimes I see full message(more often) and sometimes broken message(only last part of it)).
Class CurlSoapHandler
{
……..
static std::string _sResponse;
static std::string GetResponse() {return _sResponse;}
static size_t write_data(char *ptr, size_t size, size_t count, std::string *buffer)
{
int result = 0;
if (buffer != NULL)
{
std::string tmp_buffer(ptr, size * count);
_sResponse = tmp_buffer;
result = size * count;
}
else
{
std::cout<<"Buffer is not OK!"<<std::endl;
}
return result;
}
};
void CurlSoapHandler::DoRequest(const std::string& sRequest, std::string& sResponse)
{
CURL* _CURL;
CURLcode res;
struct curl_slist *headerlist=NULL;
headerlist = curl_slist_append(headerlist, "Content-Type:text/xml");
curl_global_init(CURL_GLOBAL_ALL);
_CURL = curl_easy_init();
if(_CURL)
{
curl_easy_setopt( _CURL, CURLOPT_URL, ttp://10.10.10.11:8083/Server/Server.asmx);
curl_easy_setopt( _CURL, CURLOPT_USERPWD, "user#password");
curl_easy_setopt( _CURL, CURLOPT_POSTFIELDS, sRequest.c_str());
curl_easy_setopt( _CURL, CURLOPT_HTTPHEADER, headerlist);
curl_easy_setopt( _CURL, CURLOPT_TIMEOUT, 20);
curl_easy_setopt( _CURL, CURLOPT_WRITEFUNCTION, CurlSoapHandler::write_data);
res = curl_easy_perform(_CURL);
sResponse = GetResponse();
if(res != CURLE_OK)
{
std::cerr<<"CURL message: "<<curl_easy_strerror(res)<<std::endl;
}
curl_easy_cleanup(_CURL);
}
else
{
std::cout<<"Curl initialization problem!"<<std::endl;
}
curl_global_cleanup();
curl_slist_free_all (headerlist);
}
I have no ideas why it occurs. Does someone have ideas? (May be it’s necessary to set some curl options, for example). How can I determine the reason for it and understand if it’s problem of my applications or not. I'd caught traffic using tcpdump utility (I run it on my Virtual Machine (VMWARe)) and saw full message in the dump. So, I think that server sends the correct response. Thanks in advance!

Just replace _sResponse = tmp_buffer; by _sResponse += tmp_buffer; shoulds work.

Related

How to read the JSON Error response returned from Server using Libcurl

I have a requirement where I have to read the error response from backend server which returns 500 Internal Server error. The error response is in JSON Format.
Below is the code snippet used in our application
INT CCurlHTTP::HTTPSPost(const CString& endPointUrl, const CString& urlparam,const CString& cookie){
CURL *curl;
CURLcode res;
struct curl_slist *headers=NULL;
char errbuf[CURL_ERROR_SIZE];
curl = curl_easy_init();
get_request req;
req.buffer =0;
req.len =0;
req.buflen =0;
if(curl)
{
//add url, headers, and parameters to the request
curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, "POST");
curl_easy_setopt(curl, CURLOPT_URL, endPointUrl);
curl_easy_setopt(curl, CURLOPT_DEFAULT_PROTOCOL, "https");
headers = curl_slist_append(headers, m_httpHeadAccept);
headers = curl_slist_append(headers, m_httpContentType);
//callback function used to save response
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_String);
req.buffer = (unsigned char*) malloc(CHUNK_SIZE);
req.buflen = CHUNK_SIZE;
req.len = 0;
curl_easy_setopt(curl,CURLOPT_WRITEDATA, (void *)&req);
if (!cookie.IsEmpty())
{
headers = curl_slist_append(headers, m_DBAuthCertficate); //What is difference between this and line no 118?
CString pCookie = "DBAuthTicket=" + cookie;
curl_easy_setopt(curl,CURLOPT_COOKIE, pCookie);
}
else
{
headers = curl_slist_append(headers, m_OAuthToken);
}
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, urlparam);
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);
errbuf[0] = 0;
curl_easy_setopt(curl, CURLOPT_BUFFERSIZE, 512000);
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
res = curl_easy_perform(curl);
if(res != CURLE_OK)
{
/* if errors have occured, tell us wath's wrong with 'result'*/
m_response.Format("%s", curl_easy_strerror(res));
return res;
}
m_response = (char*)req.buffer;
m_errDescription = errbuf;
len = req.len;
buflen = req.buflen;
curl_easy_cleanup(curl);
free(req.buffer);
}
return res;
}
/****************************************************************************
Function: CurlWrite_CallbackFunc_String
Description: Read data from the connected URL
Return: String of data and size
****************************************************************************/
size_t CCurlHTTP::CurlWrite_CallbackFunc_String(void *contents, size_t size, size_t nmemb, void *userdata)
{
size_t rLen = size*nmemb;
get_request* req = (get_request*)userdata;
while(req->buflen < req->len + rLen + 1)
{
req->buffer = (unsigned char*)realloc(req->buffer,req->buflen + CHUNK_SIZE);
req->buflen += CHUNK_SIZE;
}
memcpy(&req->buffer[req->len], contents, rLen);
req->len += rLen;
req->buffer[req->len] = 0;
return rLen;
}
The above code works fine for the Success 200 OK Requests. It reads the JSON Response just fine. However, when I get a 500 Internal Server error, it does not read the JSON Error response that comes along with it. How do I read the JSON response in this scenario?
By setting the CURLOPT_FAILONERROR option to TRUE, you are telling curl_easy_perform() to fail immediately with CURLE_HTTP_RETURNED_ERROR on any HTTP response >= 400. It will not call the CURLOPT_WRITEFUNCTION callback, as it will simply close the connection and not even attempt to read the rest of the response.
To get the response data you want, simply remove the CURLOPT_FAILONERROR option. Curl's default behavior is to deliver the response data to you regardless of the HTTP response code. In which case, curl_easy_perform() will return CURLE_OK, and you can then retrieve the response code using curl_easy_getinfo(CURLINFO_RESPONSE_CODE) to check if the HTTP request was successful or not.
On a side note, since the code shown is written in C++, I would strongly advise you NOT to use a dynamic char[] buffer for get_request::buffer. Not only because you are not handling malloc()/realloc() failures at all, but also because manual memory management should be avoided in C++ in general. Use std::string or std::vector<char> instead, in which case you can eliminate get_request in this code altogether, eg:
INT CCurlHTTP::HTTPSPost(const CString& endPointUrl, const CString& urlparam,const CString& cookie){
...
std::string resp;
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &resp);
...
//curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
res = curl_easy_perform(curl);
if (res != CURLE_OK)
{
/* if errors have occured, tell us what's wrong with 'result'*/
m_response.Format("%s", curl_easy_strerror(res));
curl_easy_cleanup(curl);
return res;
}
m_response = resp.c_str();
m_errDescription = errbuf;
len = resp.size();
buflen = resp.capacity();
curl_easy_cleanup(curl);
return res;
}
size_t CCurlHTTP::CurlWrite_CallbackFunc_String(void *contents, size_t size, size_t nmemb, void *userdata)
{
size_t rLen = size * nmemb;
static_cast<std::string*>(userdata)->append(static_cast<char*>(contents), rLen);
return rLen;
}

C++ cant use respond string from GET requests retrieved by ExpressJS API for anything else than printing

Scenario:
Im writing a c++ program which should retrieve files and strings from an expressJS API..
Downloading txt.files with curlRequests works perfectly but as soon as i try to retrieve plain strings, they can only be used for printing..
Problem: When trying to do anything with the response from my GET request (from the expressjs api), my response doesnt get treated as a string.
string myText = curlRequest.GetText("/templates/names");
string myTextB = "react.txt, scss.txt"
cout << myText << endl; // prints"react.txt, scss.txt"
cout << myTextB << endl; // prints "react.txt, scss.txt"
cout << stringHelper.GetSubstringPos(myText, "scss") << endl; // printsstring::npos
cout << stringHelper.GetSubstringPos(myTextB, "scss") << endl; // printsposition of 's' as expected
Here is my GetText function of the curlrequest.h in c++
string GetText(string ACTIONURL) {
CURL* curl;
CURLcode curlRes;
string res;
string url = templateCreator.APIURL + ACTIONURL;
curl_global_init(CURL_GLOBAL_DEFAULT);
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
curlRes = curl_easy_perform(curl);
res = curlRes;
if (curlRes == CURLE_HTTP_RETURNED_ERROR) {
res = "FAILED";
}
else if (curlRes != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(curlRes));
res = "FAILED";
}
curl_easy_cleanup(curl);
}
curl_global_cleanup();
return res;
}
Here is the route in expressjs (receiving the request and responding with a string)
router.get('/names', function(req, res, next) {
var str = "react.txt, scss.txt, example.txt";
res.send(str);
});
// I also tried sending a buffer but as its sended as string i face the same problem..
// C++ could decode the buffer ..
router.get('/buf', function(req, res, next) {
let data = 'hello world: (1 + 2 !== 4)';
let buff = new Buffer.from(data);
let base64data = buff.toString('base64');
console.log(base64data);
res.send(base64data);
});
Retrieving textfiles from my expressjs API is not a problem..
void GetFile(string ACTIONURL, string OUTDIR) {
CURL* curl;
FILE* fp;
CURLcode res;
string url = templateCreator.APIURL + ACTIONURL;
curl = curl_easy_init();
if (curl)
{
fopen_s(&fp, OUTDIR.c_str(), "wb");
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L);
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
fclose(fp);
}
return;
}
(After download I can read those line by line and store in a vector).
Still im hoping to get sending actual strings working..
Does anyone have a clue why im facing problems here?
I literally spent days on this unexpected 'small' issue already..
Thank you #n.'pronouns'm.
I Updated my function and realised that res = curlRes was a nobrainer..
Also the checks for valid response is gone for now.
//those 2 lines and a write_to_string function were missing and `res = curlRes` should do their job
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
The following 2 functions replace my GetText function now, and everything works as expected.
size_t write_to_string(void* ptr, size_t size, size_t count, void* stream) {
((string*)stream)->append((char*)ptr, 0, size * count);
return size * count;
}
string GetText(string ACTIONURL) {
CURL* curl;
CURLcode res;
string response;
curl = curl_easy_init();
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, templateCreator.APIURL + ACTIONURL.c_str());
curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1L);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_to_string);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);
res = curl_easy_perform(curl);
if (res != CURLE_OK) {
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
}
curl_easy_cleanup(curl);
}
return response;
}
Thank you! I found the fix on 1 or 2 questions earlier too but was not aware that this was the actual problem. Working with strings is possible now!

How to include an h file locally, without visual studio?

I have a simple cpp code that uses curl.h as it's header file.
I use vscode, and not interested on linking via gui or w/e
How can I make the include statement work properly with the single project?
I got the header file from https://github.com/curl/curl/blob/master/include/curl/curl.h
Folder structure is like this:
/Folder
+-> curl.h
+-> curli.cpp
the code:
#include "curl.h"
using namespace std;
#define YOUR_URL "https://www.insertyourURLhere.com/"
#define USER_AND_PWD "user:password"
static string gs_strLastResponse;
// Callback to gather the response from the server. Comes in chunks (typically 16384 characters at a time), so needs to be stitched together.
size_t function_pt(void *ptr, size_t size, size_t nmemb, void * /*stream*/)
{
gs_strLastResponse += (const char*)ptr;
return size * nmemb;
}
bool CallServerWithCurl(string strData1, strData2, string& strErrorDescription)
{
CURL* curl = curl_easy_init();
if (curl == NULL)
{
strErrorDescription = "Unable to initialise Curl";
return false;
}
curl_easy_setopt(curl, CURLOPT_URL, (YOUR_URL + "get_result").c_str());
curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_easy_setopt(curl, CURLOPT_USERPWD, MY_USER_AND_PWD); // set user name and password for the authentication
char* data1 = curl_easy_escape(curl, strData1.c_str(), 0);
char* data2 = curl_easy_escape(curl, strData2.c_str(), 0);
string strArguments = "id=" + data1 + "&task=" + data2;
const char* my_data = strArguments.c_str();
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, my_data);
curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, (long)strlen(my_data)); // if we don't provide POSTFIELDSIZE, libcurl will strlen() by itself
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); // enable verbose for easier tracing
gs_strLastResponse = "";
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, function_pt); // set a callback to capture the server's response
CURLcode res = curl_easy_perform(curl);
// we have to call twice, first call authenticates, second call does the work
res = curl_easy_perform(curl);
if (res != CURLE_OK)
{
strErrorDescription = "Curl call to server failed";
return false;
}
if (!DoSomethingWithServerResponse(gs_strLastResponse))
{
strErrorDescription = "Curl call to server returned an unexpected response";
return false;
}
// extract some transfer info
double speed_upload, total_time;
curl_easy_getinfo(curl, CURLINFO_SPEED_UPLOAD, &speed_upload);
curl_easy_getinfo(curl, CURLINFO_TOTAL_TIME, &total_time);
fprintf(stderr, "Speed: %.3f bytes/sec during %.3f seconds\n", speed_upload, total_time);
curl_easy_cleanup(curl);
return true;
}

libcurl - "curl_easy_cleanup()" is causing malloc error (C++)

When running my code (pertinent sections pasted below), I periodically get the following error:
program(34010,0x70000e58b000) malloc: *** error for object
0x7fc43d93fcf0: pointer being freed was not allocated set a breakpoint
in malloc_error_break to debug Signal: SIGABRT (signal SIGABRT)
I am running multi-threaded C++ code on a Macbook (OS-10.13) wherein different threads make use of the code in question simultaneously. To my knowledge, libcurl is indeed thread safe as long as I do not utilize the same "curl handle" (which I understand to be an instance of "CURL" aka "CURL *curl = curl_easy_init();") in two different threads at the same time. In my case, since each thread calls the function separately and initializes a new instance of a CURL object, I should be "safe", right? Hopefully there is something obvious that I'm missing that is causing me (or lib curl in this case) to attempt to free memory that has already been freed. If there is any more information I should have included (below) please don't hesitate to let me know.
The function that seg faults is
string http_lib::make_get_request(string url)
on the line that reads
curl_easy_cleanup(curl);
and sometimes (less often) on the line that reads
res = curl_easy_perform(curl);
Below is what I think would be the pertinent sections of my code:
size_t http_lib::CurlWrite_CallbackFunc_StdString(void *contents, size_t size, size_t nmemb, std::string *s)
{
size_t newLength = size*nmemb;
size_t oldLength = s->size();
try
{
s->resize(oldLength + newLength);
}
catch(std::bad_alloc &e)
{
//handle memory problem
return 0;
}
std::copy((char*)contents,(char*)contents+newLength,s->begin()+oldLength);
return size*nmemb;
}
string http_lib::make_post_request(string url, vector<string> headers, string post_params) {
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if(curl)
{
struct curl_slist *chunk = NULL;
for(int i=0; i<headers.size(); i++){
/* Add a custom header */
chunk = curl_slist_append(chunk, headers[i].c_str());
}
/* set our custom set of headers */
res = curl_easy_setopt(curl, CURLOPT_HTTPHEADER, chunk);
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, post_params.c_str());
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if(res != CURLE_OK)
{
fprintf(stderr, "curl_easy_perform() failed: %s\n",
curl_easy_strerror(res));
}
/* always cleanup */
curl_easy_cleanup(curl);
}
// Debug output
if (networking_debug){
cout<<"Response: " << s <<endl;
}
return s;
}
string http_lib::make_get_request(string url) {
//SslCurlWrapper sslObject;
CURL *curl;
CURLcode res;
curl = curl_easy_init();
string s;
if (curl) {
curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
//tell libcurl to follow redirection
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0L); //only for https
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, CurlWrite_CallbackFunc_StdString);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &s);
if(networking_debug){
curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); //verbose output
}
/* Perform the request, res will get the return code */
res = curl_easy_perform(curl);
/* Check for errors */
if (res != CURLE_OK)
fprintf(stderr, "curl_easy_perform() failed: %s\n", curl_easy_strerror(res));
/* always cleanup */
curl_easy_cleanup(curl);
}
if (networking_debug){
cout << "Response: " << s << endl;
}
return s;
}
In main() I have
int main(int argc, char *argv[]){
// Initialize http_lib (curl)
curl_global_init(CURL_GLOBAL_DEFAULT);
... spin up 10 or so threads that make get/post requests to https site (some requests utilize the make_post_request() function and others utilize make_get_requet() function).
}
CMAKE doesn't/didn't seem to want to use anything other than CURL_ROOT_DIR of "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include" for libcurl (aka curl).
Thus it was using the curl lib that mac (and/or Xcode) ships with. I haven't figured out what version that is, but I can say that not using it and instead using CURL version 7.57 is what fixed my issue.
I used "brew" package manager to
brew install curl
Doing so created /usr/local/Cellar/curl/7.57.0 directory and put all libs/includes in there.
Then I added
-I/usr/local/Cellar/curl/7.57.0/include -L/usr/local/Cellar/curl/7.57.0/lib
to my CMAKE CMAKE_CXX_FLAGS.
TLDR; Solution was to ensure I was using the newest version of the curl lib. Now that I am, no problem.

CURL - simple example returning "CURLE_WRITE_ERROR"

I am trying to run a simple example using libcurl, but just running this simple example gives me CURLE_WRITE_ERROR when I execute the curl_easy_perform(...) command. Does anyone have any idea what I am doing wrong? I have also tried other sites besides example.com.
CURL *curl = curl_easy_init();
if(curl)
{
CURLcode res;
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/");
res = curl_easy_perform(curl); // returns CURLE_WRITE_ERROR always!
curl_easy_cleanup(curl);
}
OK turns out Joachim is right. I did need a write callback
size_t CurlWriteCallback(char* buf, size_t size, size_t nmemb, void* up)
{
TRACE("CURL - Response received:\n%s", buf);
TRACE("CURL - Response handled %d bytes:\n%s", size*nmemb);
// tell curl how many bytes we handled
return size*nmemb;
}
// ...
CURL *curl = curl_easy_init();
if(curl)
{
CURLcode res;
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &CurlWriteCallback);
curl_easy_setopt(curl, CURLOPT_URL, "http://example.com/");
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}