Introduction What I want to do:
I want get text from file which is on server.
Imagine:
a pseudo code
connectTo("https://www.boost.org/LICENSE_1_0.txt");
std::string result = connectTo.GetResult();
std::cout << result;
and I should get:
Boost Software License - Version 1.0 - August 17th, 2003
Permission is here...
What I done:
I use Asio (not Boost Asio) and this example:
ip::tcp::iostream stream;
stream.connect("www.boost.org", "http");
if (!stream)
{
std::cout << "Error: " << stream.error().message() << "\n";
}
stream << "GET /LICENSE_1_0.txt HTTP/1.0\r\n\r\n";
stream << "Host: www.boost.org\r\n";
stream << "Accept: */*\r\n";
stream << "Connection: close\r\n\r\n";
for (std::string line; getline(stream, line);)
{
std::cout << line << std::endl;
}
and I get this:
HTTP/1.1 301 Moved Permanently
Date: Fri, 12 Jun 2020 12:54:59 GMT
Server: Apache/2.2.15 (CentOS)
Location: https://beta.boost.org/LICENSE_1_0.txt
Content-Length: 326
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://beta.boost.org/LICENSE_1_0.txt">here<
/a>.</p>
<hr>
<address>Apache/2.2.15 (CentOS) Server at beta.boost.org Port 80</address>
</body></html>
What I want:
instead html I want get text from LICENSE_1_0.txt file. How to do that?
Related
The following code was able to perform a post request on my (unchanged) server until recently. Since a couple of weeks I got a 400 bad request response. What could be the problem? A POST send with command line curl works fine.
std::ostringstream oss;
oss << "mydata";
int size = oss.tellp();
std::string test = oss.str();
boost::asio::ip::tcp::iostream stream;
stream.connect("myserver.nl", "http");
stream << "POST /dir/newdata.php HTTP/1.1\r\n";
stream << "Host: myserver.nl\r\n";
stream << "Accept: */*\r\n";
stream << "Content-Type: application/x-www-form-urlencoded; charset=utf-8\r\n";
stream << "Content-Length: " << size << "\r\n";
stream << "Connection: close\r\n\r\n";
stream << oss.str();
stream.flush();
std::cout << stream.rdbuf();
It now results in the following error:
HTTP/1.1 400 Bad Request
Date: Tue, 20 Feb 2018 09:22:15 GMT
Server: Apache/2.4.29 (Unix)
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br/>
</p>
You have an extra space between the query and the HTTP version:
stream << "POST /dir/newdata.php HTTP/1.1\r\n";
I am trying to send a get request to acounts.google.com to be able to implement a library for C++ OAuth to learn it.
I get the following code from this post: Creating a HTTPS request using Boost Asio and OpenSSL and modified it as follow:
int main()
{
try
{
std::string request = "/o/oauth2/v2/auth";
boost::system::error_code ec;
using namespace boost::asio;
// what we need
io_service svc;
ssl::context ctx(svc, ssl::context::method::sslv23_client);
ssl::stream<ip::tcp::socket> ssock(svc, ctx);
ip::tcp::resolver resolver(svc);
auto it = resolver.resolve({ "accounts.google.com", "443" }); // https://accouts.google.com:443
boost::asio::connect(ssock.lowest_layer(), it);
ssock.handshake(ssl::stream_base::handshake_type::client);
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
// read response
std::string response;
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
std::cout << "Response received: '" << response << "'\n"; // I add this to see what I am getting from the server, so it should not be here.
} while (!ec);
// print and exit
std::cout << "Response received: '" << response << "'\n";
}
catch (const std::exception& e)
{
std::cerr << e.what() << std::endl;
if (std::string const * extra = boost::get_error_info<my_tag_error_info>(e))
{
std::cout << *extra << std::endl;
}
}
}
The problem that I have is as follow:
1- The results that I am getting is not what I am getting when I visit https://accounts.google.com/o/oauth2/v2/auth using a web browser. I essentially getting a message that they can not find the requested URL /o/oauth2/v2/auth
<p>The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>ThatÔÇÖs all we know.</ins>
How should I setup the GET commend so I can get the same result that I am getting with a browser?
2- The application hangs getting data from server, apparently the following loop is not right:
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
} while (!ec);
What is the correct way of reading responce from the web server which is fast and read all data?
Edit 1
For reference based on accepted answer, I fixed the problem using the correct GET header as shown below:
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n";
fullResuest+= "Host: " + server + "\r\n";
fullResuest += "Accept: */*\r\n";
fullResuest += "Connection: close\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
A HTTP/1.1 request must have a Host header. A simple experiment with OpenSSL will show the problem, i.e. the missing header:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
... The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>That’s all we know.</ins>
When adding the Host header instead we get a different response:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
Host: accounts.google.com
... >Required parameter is missing: response_type<
Apart from that HTTP/1.1 implicitly uses HTTP keep-alive, i.e. server and client might keep the connection open after the response is done. This means you should not read until the end of connection but should instead properly parse the HTTP header, extract the Content-length header and/or Transfer-Encoding header and behave according to their values. Or if you want it simpler use HTTP/1.0 instead.
For more information see the HTTP/1.1 standard.
I want to connect to an HTTPS server using boost::asio. I managed to successfully shake hands with the server, but I just can't manage to get the server to respond to my POST request.
This is the related code (I left out debugging and try-catch to save some space):
HTTPSClient::HTTPSClient()
{
ssl::context context(ssl::context::sslv23);
context.set_verify_mode(ssl::verify_peer);
context.set_default_verify_paths();
context.load_verify_file("certificate.pem");
mSSLSocket = new ssl::stream<ip::tcp::socket>(mIOService, context);
}
void HTTPSClient::SendRequest(const ptree &crPTree, const std::string cHost,
const std::string cURI)
{
tcp::resolver resolver(mIOService);
tcp::resolver::query query(cHost, "https");
resolver.async_resolve(query, boost::bind(&HTTPSClient::HandleResolve, this,
placeholders::error, placeholders::iterator, request));
}
void HTTPSClient::HandleResolve(const error_code &crError,
const iterator &criEndpoints, HTTPSRequest &rRequest)
{
async_connect(mSSLSocket->lowest_layer(), criEndpoints,
boost::bind(&HTTPSClient::HandleConnect, this, placeholders::error,
rRequest));
}
void HTTPSClient::HandleConnect(const error_code &crError, HTTPSRequest &rRequest)
{
mSSLSocket->lowest_layer().set_option(ip::tcp::no_delay(true));
mSSLSocket->set_verify_callback(ssl::rfc2818_verification(rRequest.mcHost));
mSSLSocket->handshake(ssl::stream_base::client);
// Write the json into a stringstream
std::ostringstream json;
boost::property_tree::write_json(json, rRequest.mcPTree);
std::string result;
result = json.str();
// Form the request
streambuf request;
std::ostream requestStream(&request);
requestStream << "POST " << rRequest.mcURI << " HTTP/1.1\r\n";
requestStream << "Host: " << rRequest.mcHost << "\r\n";
requestStream << "Accept: application/json\r\n";
requestStream << "Content-Type: application/json; charset=UTF-8\r\n";
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << result << "\r\n\r\n";
write(*mSSLSocket, request);
streambuf response;
read_until(*mSSLSocket, response, "\r\n");
std::istream responseStream(&response);
}
read_until hangs until it throws the error read_until: End of file. Everything before that goes successfully, including the SSL handshake (which I just recently figured out).
I used to do everything asynchronously until I started debugging, and started trying to backtrace to the problem, to no avail. It would be awesome if someone could help me out after two painful days of debugging.
EDIT
I just realized it might be useful to add the contents of requestStream after composing the header:
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
You need a double linefeed before the body (POST contents)
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
Otherwise, the content will have been received by the server as header lines and the server just keeps waiting for 136 bytes of content data (also make sure that Content-Length is accurate, which it isn't in this example)
So, basically:
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << "\r\n"; // THIS LINE ADDED
I managed to figure out what I was doing wrong. For some reason, I couldn't get boost to write data using the boost::asio::streambuf and std::ostream approach. Instead, I put the POST data in a std::string and sent it like this:
write(*mSSLSocket, boost::asio::buffer(requestString));
Which worked out fine.
I am writing an application for work that needs to perform HTTP requests to a server, and get a response, in JSON back.
At the moment, my code connects to the server, and gets a response back, which is great. However, I also need to send data to the server, which will process it, and send me the jSON.
My problem is that I am able to send, thanks to POSTFIELDS, only a single "field", and won't get any response if I insert more that one.
The code is the following:
// writefunc works, so there is no point adding its code in here
size_t Simulator::writefunc(void *ptr, size_t size, size_t nmemb, struct string *buffer_in);
void Simulator::move(Player *player)
{
std::ostringstream ss_url;
ss_url << API_URL << "functionCalled";
char const *url = ss_url.str().c_str();
std::ostringstream ss_postfields;
// This will not work - And all values do NOT contain any space
// The server do NOT receive any data (in POST and GET)
ss_postfields << "user_name=" << player->getName()
<< "&user_secret=" << player->secret()
<< "&app_id=" << player->getApp_id()
<< "&lat=" << player->getLocation()->getLat()
<<"&lon=" << player->getLocation()->getLon();
// This will not work either
// ss_postfields << "user_name=nicolas&app_id=2";
// This works and will send the data to the server, which will receive and process it.
// ss_postfields << "user_name=" << player->getName();
const char *postfields = ss_postfields.str().c_str();
CURL *curl_handle;
curl_global_init(CURL_GLOBAL_ALL);
curl_handle = curl_easy_init();
if(curl_handle){
struct string s;
init_string(&s);
curl_easy_setopt(curl_handle, CURLOPT_URL, url);
curl_easy_setopt(curl_handle, CURLOPT_WRITEFUNCTION, writefunc);
curl_easy_setopt(curl_handle, CURLOPT_NOPROGRESS, 1L);
curl_easy_setopt(curl_handle, CURLOPT_WRITEDATA, &s);
curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDS, postfields);
curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDSIZE, strlen(postfields));
curl_easy_setopt(curl_handle, CURLOPT_POST, 1L);
CURLcode res = curl_easy_perform(curl_handle);
if(CURLE_OK != res)
{
printf("Error: %s\n", strerror(res));
exit(0);
}
printf("%s\n", s.ptr);
curl_easy_cleanup(curl_handle);
curl_global_cleanup();
}
}
I thought I would also give the output of CURLOPT_HEADER and CURLOPT_VERBOSE for when I send only 1 value, or multiple values:
When I Send One value only:
* About to connect() to localhost port 8888 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 8888 (#0)
> POST [api_url] HTTP/1.1
Host: localhost:8888
Accept: */*
Content-Length: 22
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 22 out of 22 bytes
< HTTP/1.1 200 OK
< Date: Sun, 24 Nov 2013 17:25:14 GMT
< Server: Apache
< X-Powered-By: PHP/5.3.20
< Cache-Control: no-cache
< X-Debug-Token: a41727
< Transfer-Encoding: chunked
< Content-Type: application/json
<
* Connection #0 to host localhost left intact
0
HTTP/1.1 200 OK
Date: Sun, 24 Nov 2013 17:25:14 GMT
Server: Apache
X-Powered-By: PHP/5.3.20
Cache-Control: no-cache
X-Debug-Token: a41727
Transfer-Encoding: chunked
Content-Type: application/json
[ OUTPUT FROM SERVER HERE ]
* Closing connection #0
And when I send multiple values:
* About to connect() to localhost port 8888 (#0)
* Trying ::1...
* connected
* Connected to localhost (::1) port 8888 (#0)
> POST [api_url] HTTP/1.1
Host: localhost:8888
Accept: */*
Content-Length: 1
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 1 out of 1 bytes
< HTTP/1.1 200 OK
< Date: Sun, 24 Nov 2013 17:30:13 GMT
< Server: Apache
< X-Powered-By: PHP/5.3.20
< Cache-Control: no-cache
< X-Debug-Token: d1947e
< Transfer-Encoding: chunked
< Content-Type: application/json
<
* Connection #0 to host localhost left intact
0
HTTP/1.1 200 OK
Date: Sun, 24 Nov 2013 17:30:13 GMT
Server: Apache
X-Powered-By: PHP/5.3.20
Cache-Control: no-cache
X-Debug-Token: d1947e
Transfer-Encoding: chunked
Content-Type: application/json
[ OUTPUT FROM SERVER HERE ]
* Closing connection #0
Well you seem to be missing an equals sign!
ss_postfields << "user_name=" << player->getName()
<< "&user_secret=" << player->secret()
<< "&app_id=" << player->getApp_id()
<< "&lat=" << player->getLocation()->getLat()
<<"&lon=" << player->getLocation()->getLon();
instead of
ss_postfields << "user_name=" << player->getName()
<< "&user_secret" << player->secret()
<< "&app_id=" << player->getApp_id()
<< "&lat=" << player->getLocation()->getLat()
<<"&lon=" << player->getLocation()->getLon();
Since I was in a rush, I rewrote everything using HTTP sockets. But I could not be done with this issue, so I went back to it, and finally found the problem. It seems like cUrl does not really like streams.
I thought I would update this post in the case anyone is having the same issue:
Therefore, something like the following will NOT work:
std::ostringstream ss_postfields;
// This will not work - And all values do NOT contain any space
// The server do NOT receive any data (in POST and GET)
ss_postfields << "user_name=" << player->getName()
<< "&user_secret=" << player->getSecret();
const char *postfields = ss_postfields.str().c_str();
// ...
curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDS, postfields);
CURLcode res = curl_easy_perform(curl_handle);
However, and bizarrely, this will work:
std::string str_postfields;
std::string user_name = player->getName();
std::string user_secret = player->getSecret();
std::string str_postfields = "user_name=" +user_name +"&user_secret=" +user_secret;
const char *postfields = str_postfields.c_str();
// ...
curl_easy_setopt(curl_handle, CURLOPT_POSTFIELDS, postfields);
CURLcode res = curl_easy_perform(curl_handle);
I've been trying to get this to work for a couple of days however I keep getting a 400 error from the server.
Basically, what I'm trying to do is send a http POST request to a server that requires a JSON request body with a couple of properties.
These are the libs I'm currently using
UPDATED --- 7/23/13 10:00am just noticed I'm using TCP instead of HTTP not sure how much this will effect an HTTP call but i can't find any examples of clients using pure HTTP with BOOST::ASIO
#include <iostream>
#include <istream>
#include <ostream>
#include <string>
#include <boost/asio.hpp>
#include <sstream>
#include <boost/property_tree/ptree.hpp>
#include <boost/property_tree/json_parser.hpp>
using boost::property_tree::ptree; using boost::property_tree::read_json; using boost::property_tree::write_json;
using boost::asio::ip::tcp;
SET UP CODE
// Get a list of endpoints corresponding to the server name.
tcp::resolver resolver(io_service);
tcp::resolver::query query(part1, "http");
tcp::resolver::iterator endpoint_iterator = resolver.resolve(query);
// Try each endpoint until we successfully establish a connection.
tcp::socket socket(io_service);
boost::asio::connect(socket, endpoint_iterator);
// Form the request. We specify the "Connection: close" header so that the
// server will close the socket after transmitting the response. This will
// allow us to treat all data up until the EOF as the content.
boost::asio::streambuf request;
std::ostream request_stream(&request);
JSON BODY
ptree root, info;
root.put ("some value", "8");
root.put ( "message", "value value: value!");
info.put("placeholder", "value");
info.put("value", "daf!");
info.put("module", "value");
root.put_child("exception", info);
std::ostringstream buf;
write_json (buf, root, false);
std::string json = buf.str();
HEADER AND CONNECTION REQUEST
request_stream << "POST /title/ HTTP/1.1 \r\n";
request_stream << "Host:" << some_host << "\r\n";
request_stream << "User-Agent: C/1.0";
request_stream << "Content-Type: application/json; charset=utf-8 \r\n";
request_stream << json << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Connection: close\r\n\r\n";
// Send the request.
boost::asio::write(socket, request);
I put place holder values however if you see anything that doesn't work in my code that jumps out please let me know I have no idea why i keep getting a 400, bad request.
info about the rig
C++
WIN7
VISUAL STUDIO
Although this question is very old I would like to post this answer for users who are facing similar problem for http POST.
The server is sending you HTTP 400 means "BAD REQUEST". It is because the way you are forming your request is bit wrong.
The following is the correct way to send the POST request containing JSON data.
#include<string> //for length()
request_stream << "POST /title/ HTTP/1.1 \r\n";
request_stream << "Host:" << some_host << "\r\n";
request_stream << "User-Agent: C/1.0\r\n";
request_stream << "Content-Type: application/json; charset=utf-8 \r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Content-Length: " << json.length() << "\r\n";
request_stream << "Connection: close\r\n\r\n"; //NOTE THE Double line feed
request_stream << json;
Whenever you are sending any data(json,string etc) with your POST request, make sure:
(1) Content-Length: is accurate.
(2) that you put the Data at the end of your request with a line gap.
(3) and for that (2nd point) to happen you MUST provide double line feed (i.e. \r\n\r\n) in the last header of your header request. This tells the header that HTTP request content is over and now it(server) will get the data.
If you don't do this then the server fails to understand that where the header is ending ? and where the data is beginning ? So, it keeps waiting for the promised data (it hangs).
Disclaimer: Feel free to edit for inaccuracies, if any.