I am using an Adafruit Feather Huzzah ESP8266 and the ArduinoJson library to parse the response of an HTTP request. I am successfully getting the expected responses but the deserialization process fails every other time it iterates through the loop. I think it may be a memory allocation issue but I cannot resolve the problem. Any help would be greatly appreciated.
I have tried using dynamic vs static documents and initialising them inside/outside the loop without success. I have also tried the doc.clear() method to release the memory but still no luck. Below is my loop with the connection parameters missing:
void loop() {
WiFiClientSecure client;
for (int i = 0; i <= numOfInstallations - 1; i++) {
String url = "/v2/installations/" + String(idSites[i]) +
"/widgets/Graph?attributeCodes[]=SOC&instance=" + String(instance[i]) +
"&start=" + String(startTime) +
"&end=" + String(endTime);
client.print(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: " + host + "\r\n" +
"X-Authorization: Token " + token + "\r\n" +
"Connection: keep-alive\r\n\r\n");
Serial.println(String("GET ") + url + " HTTP/1.1\r\n" +
"Host: " + host + "\r\n" +
"X-Authorization: Token " + token + "\r\n" +
"Connection: keep-alive\r\n\r\n");
Serial.println("request sent");
delay(500);
// Ignore the response headers
char endOfHeaders[] = "\r\n\r\n";
if (!client.find(endOfHeaders)) {
Serial.println(F("No headers"));
return;
}
const size_t capacity = 5*JSON_ARRAY_SIZE(2) + JSON_ARRAY_SIZE(5) + 2*JSON_OBJECT_SIZE(1) + 2*JSON_OBJECT_SIZE(2) + JSON_OBJECT_SIZE(4) + 130;
DynamicJsonDocument doc(capacity); // Json document setup
// Get the Json data from the response
DeserializationError error = deserializeJson(doc, client);
if (error) {
Serial.print(F("deserializeJson() failed: "));
Serial.println(error.c_str());
Serial.print("\r\n");
}
else {
Serial.println("deserializeJson() successful\r\n");
}
}
}
I am expecting the deserialization process to be successful each time but here is the output:
GET /v2/installations/xxxxx/widgets/Graph?attributeCodes[]=SOC&instance=215&start=1555070100&end=1555070400 HTTP/1.1
Host: vrmapi.victronenergy.com
X-Authorization: Token c324f8876e672ad1797cd69a9d9f62611507d25aa5a0b1ff40f9fb524d96f2fc
Connection: keep-alive
request sent
deserializeJson() successful
GET /v2/installations/xxxxx/widgets/Graph?attributeCodes[]=SOC&instance=258&start=1555070100&end=1555070400 HTTP/1.1
Host: vrmapi.victronenergy.com
X-Authorization: Token XXXXXXXXXX
Connection: keep-alive
request sent
deserializeJson() failed: InvalidInput
GET /v2/installations/xxxxx/widgets/Graph?attributeCodes[]=SOC&instance=258&start=1555070100&end=1555070400 HTTP/1.1
Host: vrmapi.victronenergy.com
X-Authorization: Token XXXXXXXXXX
Connection: keep-alive
request sent
deserializeJson() successful
GET /v2/installations/xxxxx/widgets/Graph?attributeCodes[]=SOC&instance=258&start=1555070100&end=1555070400 HTTP/1.1
Host: vrmapi.victronenergy.com
X-Authorization: Token XXXXXXXXXX
Connection: keep-alive
request sent
deserializeJson() failed: InvalidInput
GET /v2/installations/xxxxx/widgets/Graph?attributeCodes[]=SOC&instance=258&start=1555070100&end=1555070400 HTTP/1.1
Host: vrmapi.victronenergy.com
X-Authorization: Token XXXXXXXXXX
Connection: keep-alive
request sent
deserializeJson() successful
Related
The following code was able to perform a post request on my (unchanged) server until recently. Since a couple of weeks I got a 400 bad request response. What could be the problem? A POST send with command line curl works fine.
std::ostringstream oss;
oss << "mydata";
int size = oss.tellp();
std::string test = oss.str();
boost::asio::ip::tcp::iostream stream;
stream.connect("myserver.nl", "http");
stream << "POST /dir/newdata.php HTTP/1.1\r\n";
stream << "Host: myserver.nl\r\n";
stream << "Accept: */*\r\n";
stream << "Content-Type: application/x-www-form-urlencoded; charset=utf-8\r\n";
stream << "Content-Length: " << size << "\r\n";
stream << "Connection: close\r\n\r\n";
stream << oss.str();
stream.flush();
std::cout << stream.rdbuf();
It now results in the following error:
HTTP/1.1 400 Bad Request
Date: Tue, 20 Feb 2018 09:22:15 GMT
Server: Apache/2.4.29 (Unix)
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br/>
</p>
You have an extra space between the query and the HTTP version:
stream << "POST /dir/newdata.php HTTP/1.1\r\n";
I am trying to send a get request to acounts.google.com to be able to implement a library for C++ OAuth to learn it.
I get the following code from this post: Creating a HTTPS request using Boost Asio and OpenSSL and modified it as follow:
int main()
{
try
{
std::string request = "/o/oauth2/v2/auth";
boost::system::error_code ec;
using namespace boost::asio;
// what we need
io_service svc;
ssl::context ctx(svc, ssl::context::method::sslv23_client);
ssl::stream<ip::tcp::socket> ssock(svc, ctx);
ip::tcp::resolver resolver(svc);
auto it = resolver.resolve({ "accounts.google.com", "443" }); // https://accouts.google.com:443
boost::asio::connect(ssock.lowest_layer(), it);
ssock.handshake(ssl::stream_base::handshake_type::client);
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
// read response
std::string response;
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
std::cout << "Response received: '" << response << "'\n"; // I add this to see what I am getting from the server, so it should not be here.
} while (!ec);
// print and exit
std::cout << "Response received: '" << response << "'\n";
}
catch (const std::exception& e)
{
std::cerr << e.what() << std::endl;
if (std::string const * extra = boost::get_error_info<my_tag_error_info>(e))
{
std::cout << *extra << std::endl;
}
}
}
The problem that I have is as follow:
1- The results that I am getting is not what I am getting when I visit https://accounts.google.com/o/oauth2/v2/auth using a web browser. I essentially getting a message that they can not find the requested URL /o/oauth2/v2/auth
<p>The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>ThatÔÇÖs all we know.</ins>
How should I setup the GET commend so I can get the same result that I am getting with a browser?
2- The application hangs getting data from server, apparently the following loop is not right:
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
} while (!ec);
What is the correct way of reading responce from the web server which is fast and read all data?
Edit 1
For reference based on accepted answer, I fixed the problem using the correct GET header as shown below:
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n";
fullResuest+= "Host: " + server + "\r\n";
fullResuest += "Accept: */*\r\n";
fullResuest += "Connection: close\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
A HTTP/1.1 request must have a Host header. A simple experiment with OpenSSL will show the problem, i.e. the missing header:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
... The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>That’s all we know.</ins>
When adding the Host header instead we get a different response:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
Host: accounts.google.com
... >Required parameter is missing: response_type<
Apart from that HTTP/1.1 implicitly uses HTTP keep-alive, i.e. server and client might keep the connection open after the response is done. This means you should not read until the end of connection but should instead properly parse the HTTP header, extract the Content-length header and/or Transfer-Encoding header and behave according to their values. Or if you want it simpler use HTTP/1.0 instead.
For more information see the HTTP/1.1 standard.
i'm developing a small application.
For testing purpouse i'm tryng to "ping" google using libcurl while behind an ntlm proxy.
This is my c++ code:
CURLcode testConnection(void)
{
CURL *curl;
CURLcode res = CURLE_OK;
res = curl_global_init(CURL_GLOBAL_ALL);
curl = curl_easy_init();
if (curl)
{
cout << "Url: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_URL, "http://www.google.com")) << "\n";
cout << "T-out: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, 3)) << "\n";
cout << "No_body: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_NOBODY, true)) << "\n";
cout << "Proxy Url: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_PROXY, "proxyrm.wind.root.it")) << "\n";
cout << "Proxy Port: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_PROXYPORT, 8080)) << "\n";
cout << "Ntml: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_NTLM)) << "\n";
cout << "Verbose: " << curl_easy_strerror(curl_easy_setopt(curl, CURLOPT_VERBOSE, true)) << "\n";
res = curl_easy_perform(curl);
curl_easy_cleanup(curl);
}
else
res = CURLE_FAILED_INIT;
curl_global_cleanup();
return res;
}
All thi went good, since the verbose output is the following.
Url: No error
T-out: No error
No_body: No error
Proxy Url: No error
Proxy Port: No error
Ntml: No error
Verbose: No error
* About to connect() to proxy myroxy port 8080 (#0)
* Trying IP...
* Connected to myroxy (IP) port 8080 (#0)
> HEAD http://www.google.com HTTP/1.1
Host: www.google.com
Accept: */*
Proxy-Connection: Keep-Alive
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: BASIC realm="windroot"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
* HTTP/1.1 proxy connection set close!
< Proxy-Connection: close
< Set-Cookie: BCSI-CS-602d36a7505d346e=2; Path=/
< Connection: close
< Content-Length: 989
<
* Closing connection 0
Curl Final: No error
But, if i try to ping https://google.com the result become this
Url: No error
T-out: No error
No_body: No error
Proxy Url: No error
Proxy Port: No error
Ntml: No error
Ntml: No error
Ntml: No error
Verbose: No error
* About to connect() to proxy myroxy port 8080 (#0
* Trying IP...
* Connected to myroxy (IP) port 8080 (#0)
* Establish HTTP proxy tunnel to www.google.com:443
> CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Connection: Keep-Alive
< HTTP/1.1 407 Proxy Authentication Required
< Proxy-Authenticate: NTLM
< Proxy-Authenticate: BASIC realm="windroot"
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: close
< Set-Cookie: BCSI-CS-602d36a7505d346e=2; Path=/
< Connection: close
< Content-Length: 1135
<
* Ignore 1135 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* Connection #0 to host myroxy left intact
Curl Final: Failure when receiving data from the peer
Using curl in command line allow me to ping https (i have to specify the -k argument) but i don't know if this is really relevant. Someone can help me to figure out what's happening? And how to avoid that?
Solved: i was missing this instruction
curl_easy_setopt(ctx, CURLOPT_PROXYUSERPWD, ":");
This (i think) tell curl to use Username an Password retrieved from SO passing througt ntlm auth
I want to connect to an HTTPS server using boost::asio. I managed to successfully shake hands with the server, but I just can't manage to get the server to respond to my POST request.
This is the related code (I left out debugging and try-catch to save some space):
HTTPSClient::HTTPSClient()
{
ssl::context context(ssl::context::sslv23);
context.set_verify_mode(ssl::verify_peer);
context.set_default_verify_paths();
context.load_verify_file("certificate.pem");
mSSLSocket = new ssl::stream<ip::tcp::socket>(mIOService, context);
}
void HTTPSClient::SendRequest(const ptree &crPTree, const std::string cHost,
const std::string cURI)
{
tcp::resolver resolver(mIOService);
tcp::resolver::query query(cHost, "https");
resolver.async_resolve(query, boost::bind(&HTTPSClient::HandleResolve, this,
placeholders::error, placeholders::iterator, request));
}
void HTTPSClient::HandleResolve(const error_code &crError,
const iterator &criEndpoints, HTTPSRequest &rRequest)
{
async_connect(mSSLSocket->lowest_layer(), criEndpoints,
boost::bind(&HTTPSClient::HandleConnect, this, placeholders::error,
rRequest));
}
void HTTPSClient::HandleConnect(const error_code &crError, HTTPSRequest &rRequest)
{
mSSLSocket->lowest_layer().set_option(ip::tcp::no_delay(true));
mSSLSocket->set_verify_callback(ssl::rfc2818_verification(rRequest.mcHost));
mSSLSocket->handshake(ssl::stream_base::client);
// Write the json into a stringstream
std::ostringstream json;
boost::property_tree::write_json(json, rRequest.mcPTree);
std::string result;
result = json.str();
// Form the request
streambuf request;
std::ostream requestStream(&request);
requestStream << "POST " << rRequest.mcURI << " HTTP/1.1\r\n";
requestStream << "Host: " << rRequest.mcHost << "\r\n";
requestStream << "Accept: application/json\r\n";
requestStream << "Content-Type: application/json; charset=UTF-8\r\n";
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << result << "\r\n\r\n";
write(*mSSLSocket, request);
streambuf response;
read_until(*mSSLSocket, response, "\r\n");
std::istream responseStream(&response);
}
read_until hangs until it throws the error read_until: End of file. Everything before that goes successfully, including the SSL handshake (which I just recently figured out).
I used to do everything asynchronously until I started debugging, and started trying to backtrace to the problem, to no avail. It would be awesome if someone could help me out after two painful days of debugging.
EDIT
I just realized it might be useful to add the contents of requestStream after composing the header:
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
You need a double linefeed before the body (POST contents)
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
Otherwise, the content will have been received by the server as header lines and the server just keeps waiting for 136 bytes of content data (also make sure that Content-Length is accurate, which it isn't in this example)
So, basically:
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << "\r\n"; // THIS LINE ADDED
I managed to figure out what I was doing wrong. For some reason, I couldn't get boost to write data using the boost::asio::streambuf and std::ostream approach. Instead, I put the POST data in a std::string and sent it like this:
write(*mSSLSocket, boost::asio::buffer(requestString));
Which worked out fine.
I wrote an HTTP server in Java and a client in C++ with Poco. This is a part of the C++ client code:
URI uri("http://127.0.0.1:4444");
HTTPClientSession session(uri.getHost(), uri.getPort());
HTTPRequest req(HTTPRequest::HTTP_POST,
"/pages/page",
HTTPMessage::HTTP_1_1);
session.sendRequest(req);
HTTPResponse res;
std::istream &is = session.receiveResponse(res);
In the last line I get the following error:
terminate called after throwing an instance of 'Poco::Net::NoMessageException'
what(): No message received
But I don't understand why. The connection was established successfully and the page requested exists. I tried the same code with known websites (like Wikipedia) and it works without any exception.
I also tried to make the exact same request with cURL (to my server) in command-line and it shows the response of the server, so the server seems fine.
This is the original response of the server in a string form:
"HTTP/1.1 200 OK\r\n" +
"Server: [server name]\r\n" +
"Content-Type: text/xml; charset=utf-8\r\n" +
"Content-Length:" + bodyBytes.length + "\r\n" +
"Resource: " + job.resId + "\r\n\r\n" +
"<?xml version=\"1.0\" encoding=\"UTF-8\"?><JobRequest><InputRepresentation id=\"0\"/> <effectsList><cvtColor><code> CV_RGB2GRAY </code></cvtColor><resize><scaleFactorX> 0.5 </scaleFactorX><scaleFactorY> 0.5 </scaleFactorY><interpolation> INTER_LINEAR </interpolation></resize><GaussianBlur><kSize> 3 </kSize><sigmaX> 2 </sigmaX><sigmaY> 2 </sigmaY><borderType> BORDER_REPLICATE </borderType></GaussianBlur></effectsList></JobRequest>"
I have written a simple HTTP server which respond with a fixed response for every request, to test what's wrong. this is the code:
public class Test {
public static void main(String[] args) {
try {
ServerSocket serverSocket = new ServerSocket(4449);
Socket clientSocket = serverSocket.accept();
String body = "ab";
byte[] bodyBytes = body.getBytes("UTF-8");
String headers = "HTTP/1.1 200 OK\r\n" +
"Server: Foo\r\n" +
"Content-Type: text/plain\r\n" +
"Content-Length: " + bodyBytes.length + "\r\n\r\n";
byte[] headerBytes = headers.getBytes("UTF-8");
byte[] responseBytes = new byte[headerBytes.length + bodyBytes.length];
/* Fill responseBytes with the header and body bytes */
int i = 0;
for (int j = 0; j < headerBytes.length; ++j) {
responseBytes[i] = headerBytes[j];
++i;
}
for (int j = 0; j < bodyBytes.length; ++j) {
responseBytes[i] = bodyBytes[j];
++i;
}
clientSocket.getOutputStream().write(responseBytes);
} catch (IOException e) {}
}
}
I get the same exception even with this server. So what's wrong here?
Based on experimentation, I've found that it's necessary to include a Content-Length header if you are making an empty POST request. I.e.,
req.add("Content-Length", "0");
I had to use the following sequence in my handler code for a minimal response to be received without a No message received error:
resp.setStatus( Poco::Net::HTTPResponse::HTTP_OK );
resp.setContentType( "text/json" );
ostream &out = resp.send();
out << "[]";
out.flush();