I am trying to send a get request to acounts.google.com to be able to implement a library for C++ OAuth to learn it.
I get the following code from this post: Creating a HTTPS request using Boost Asio and OpenSSL and modified it as follow:
int main()
{
try
{
std::string request = "/o/oauth2/v2/auth";
boost::system::error_code ec;
using namespace boost::asio;
// what we need
io_service svc;
ssl::context ctx(svc, ssl::context::method::sslv23_client);
ssl::stream<ip::tcp::socket> ssock(svc, ctx);
ip::tcp::resolver resolver(svc);
auto it = resolver.resolve({ "accounts.google.com", "443" }); // https://accouts.google.com:443
boost::asio::connect(ssock.lowest_layer(), it);
ssock.handshake(ssl::stream_base::handshake_type::client);
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
// read response
std::string response;
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
std::cout << "Response received: '" << response << "'\n"; // I add this to see what I am getting from the server, so it should not be here.
} while (!ec);
// print and exit
std::cout << "Response received: '" << response << "'\n";
}
catch (const std::exception& e)
{
std::cerr << e.what() << std::endl;
if (std::string const * extra = boost::get_error_info<my_tag_error_info>(e))
{
std::cout << *extra << std::endl;
}
}
}
The problem that I have is as follow:
1- The results that I am getting is not what I am getting when I visit https://accounts.google.com/o/oauth2/v2/auth using a web browser. I essentially getting a message that they can not find the requested URL /o/oauth2/v2/auth
<p>The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>ThatÔÇÖs all we know.</ins>
How should I setup the GET commend so I can get the same result that I am getting with a browser?
2- The application hangs getting data from server, apparently the following loop is not right:
do
{
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
} while (!ec);
What is the correct way of reading responce from the web server which is fast and read all data?
Edit 1
For reference based on accepted answer, I fixed the problem using the correct GET header as shown below:
// send request
std::string fullResuest = "GET " + request + " HTTP/1.1\r\n";
fullResuest+= "Host: " + server + "\r\n";
fullResuest += "Accept: */*\r\n";
fullResuest += "Connection: close\r\n\r\n";
boost::asio::write(ssock, buffer(fullResuest));
A HTTP/1.1 request must have a Host header. A simple experiment with OpenSSL will show the problem, i.e. the missing header:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
... The requested URL <code>/o/oauth2/v2/auth</code> was not found on this server. <ins>That’s all we know.</ins>
When adding the Host header instead we get a different response:
$ openssl s_client -connect accounts.google.com:443
...
GET /o/oauth2/v2/auth HTTP/1.1
Host: accounts.google.com
... >Required parameter is missing: response_type<
Apart from that HTTP/1.1 implicitly uses HTTP keep-alive, i.e. server and client might keep the connection open after the response is done. This means you should not read until the end of connection but should instead properly parse the HTTP header, extract the Content-length header and/or Transfer-Encoding header and behave according to their values. Or if you want it simpler use HTTP/1.0 instead.
For more information see the HTTP/1.1 standard.
Related
I am trying to send HTTPS request to a server and receive the page contents by only using Boost.Asio(not Network.Ts or Beast or others) by these code :
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
#include <iostream>
int main() {
boost::system::error_code ec;
using namespace boost::asio;
// what we need
io_service svc;
ssl::context ctx(ssl::context::method::tlsv1);
ssl::stream<ip::tcp::socket> ssock(svc, ctx);
ip::tcp::endpoint endpoint(boost::asio::ip::make_address("157.90.94.153",ec),443);
ssock.lowest_layer().connect(endpoint);
ssock.handshake(ssl::stream_base::handshake_type::client);
// send request
std::string request("GET /index.html HTTP/1.1\r\n\r\n");
boost::asio::write(ssock, buffer(request));
// read response
std::string response;
do {
char buf[1024];
size_t bytes_transferred = ssock.read_some(buffer(buf), ec);
if (!ec) response.append(buf, buf + bytes_transferred);
} while (!ec);
// print and exit
std::cout << "Response received: '" << response << "'\n";
}
But I keep getting 405 Not Allowed on my local PC and 400 Bad Request on Coliru.
What did I do wrong?
... "GET /index.html HTTP/1.1\r\n\r\n"
This is not a valid HTTP/1.1 request. It must at least also contain a Host field and the value of the field must match the servers expectation, i.e.
"GET /index.html HTTP/1.1\r\nHost: example.com\r\n\r\n"
In general, HTTP might look easy but is actually complex and has several pitfalls. If you really need to do HTTP by your own please study the standard.
I'm trying to communicate with some server. To get username (with permissions), I need to do something like registration: HTTP request method POST send json-like body containing {"devicetype": "devicename"}. O tried to do it with ASIO library.
asio::error_code ec;
asio::io_context context;
asio::ip::tcp::endpoint endpoint(asio::ip::make_address("ipAddress", ec), 80);
asio::ip::tcp::socket socket(context);
if (!ec)
{
std::cout << "Succesfully connected\n";
}
else
{
std::cout << "Failed to connect to address: \n" << ec.message() << std::endl;
}
if (socket.is_open())
{
std::string sRequest =
"POST /api HTTP/1.1\r\n"
"Host: ipAddress \r\n"
"Body: {\"devicetype\": \"devicename\"}"
"Connection: close\r\n\r\n";
socket.write_some(asio::buffer(sRequest.data(), sRequest.size()), ec);
/*Reading received message and getting error message from server*/
}
Error says: "invalid/missing parameters in body". The parameters are correct. The problem is probably with message formatting I am sending (sRequest). How can I specify json body to message?
Thanks for help.
What you have shown is not a properly formatted HTTP request. There is no Body header in HTTP. The JSON data needs to go after the \r\n\r\n that terminates the headers. And you need to add Content-Type and Content-Length headers so the server knows what kind of data you are posting and how large it is.
Try this instead:
std::string json = "{\"devicetype\": \"devicename\"}";
std::string sRequest =
"POST /api HTTP/1.1\r\n"
"Host: ipAddress\r\n"
"Content-Type: application/json\r\n"
"Content-Length: " + std::to_string(json.size()) + "\r\n"
"Connection: close\r\n\r\n" + json;
Using boost-asio I prepared simple code:
asio::io_service io_service;
asio::ip::tcp::socket s(io_service);
asio::ip::tcp::resolver resolver(io_service);
asio::connect(s, resolver.resolve({ "aire.pl", "80" }));
cout << "connected" << endl;
string request = "GET http://aire.pl/ HTTP/1.1";
size_t request_length = std::strlen(request.c_str());
asio::write(s, asio::buffer(request, request_length));
cout << "packet sent" << endl;
char reply[1024];
size_t reply_length = asio::read(s, asio::buffer(reply, request_length));
std::cout << "Reply is: ";
std::cout.write(reply, reply_length);
std::cout << "\n";
Everything seems to work fine, because using tcp-dump I can see my packets that the program has sent:
But I don't have any response. The one interesting fact is that, if HTTP server is nginx it works ok! In this example, the HTTP server is Apache2. What's wrong?
It looks like you haven't sent a complete HTTP request. The GET line is followed by optional headers, followed by a blank line to indicate the end of the headers. Even if you don't want to send any headers, you need to send the blank line so that the server knows it's received the entire request.
Add \r\n\r\n to the end of your request string.
Hello I'm trying to download content from webpage that uses https via C++. My very basic client program taken from the Boost asio examples compiles and runs fine, but when I test it eg with Google: www.google.co.uk/?gws_rd=ssl, it gives me the error "handshake: certificate verify failed".
I think this is because ctx.set_default_verify_paths() doesn't contain a path with a certificate for Google (I'm on Windows).
I'm very new to SSL, please can you help me with the following questions:
1) When I installed openSSL, did it stick a list of trusted certifying authorities on my computer? If it did, what would cause Google's certificate not to be verified?
2) Is there anyway of saying I don't care about verification, proceed to connect anyway, like when you add an exception manually in firefox? I'm not particularly interested in whether the connection is trusted as I am not transmitting anything that needs to be secure.
Answers to either would be greatly appreciated!
#include <iostream>
#include <istream>
#include <ostream>
#include <fstream>
#include <string>
#include <boost/asio.hpp>
#include <boost/asio/ssl.hpp>
using boost::asio::ip::tcp;
namespace ssl = boost::asio::ssl;
typedef ssl::stream<tcp::socket> ssl_socket;
int main(int argc, char* argv[])
{
try
{
if (argc != 3)
{
std::cout << argc;
std::cout << "Usage: sync_client <server> <path>\n";
std::cout << "Example:\n";
std::cout << " sync_client www.boost.org /LICENSE_1_0.txt\n";
return 1;
}
boost::asio::io_service io_service;
// Create a context that uses the default paths for
// finding CA certificates.
ssl::context ctx(ssl::context::sslv23);
ctx.set_default_verify_paths();
// Get a list of endpoints corresponding to the server name.
tcp::resolver resolver(io_service);
tcp::resolver::query query(argv[1], "https");
tcp::resolver::iterator endpoint_iterator = resolver.resolve(query);
// Try each endpoint until we successfully establish a connection.
ssl_socket socket(io_service, ctx);
boost::asio::connect(socket.lowest_layer(), endpoint_iterator);
socket.lowest_layer().set_option(tcp::no_delay(true));
// Perform SSL handshake and verify the remote host's
// certificate.
socket.set_verify_mode(ssl::verify_peer);
socket.set_verify_callback(ssl::rfc2818_verification("host.name"));
socket.handshake(ssl_socket::client);
// Form the request. We specify the "Connection: close" header so that the
// server will close the socket after transmitting the response. This will
// allow us to treat all data up until the EOF as the content.
boost::asio::streambuf request;
std::ostream request_stream(&request);
request_stream << "GET " << argv[2] << " HTTP/1.0\r\n";
request_stream << "Host: " << argv[1] << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Connection: close\r\n\r\n";
// Send the request.
boost::asio::write(socket, request);
// Read the response status line. The response streambuf will automatically
// grow to accommodate the entire line. The growth may be limited by passing
// a maximum size to the streambuf constructor.
boost::asio::streambuf response;
boost::asio::read_until(socket, response, "\r\n");
// Check that response is OK.
std::istream response_stream(&response);
std::string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
std::string status_message;
std::getline(response_stream, status_message);
if (!response_stream || http_version.substr(0, 5) != "HTTP/")
{
std::cout << "Invalid response\n";
return 1;
}
if (status_code != 200)
{
std::cout << "Response returned with status code " << status_code << "\n";
std::cout << status_message << "\n";
// Read the response headers, which are terminated by a blank line.
boost::asio::read_until(socket, response, "\r\n\r\n");
// Process the response headers.
std::string header;
while (std::getline(response_stream, header) && header != "\r")
std::cout << header << "\n";
std::cout << "\n";
return 1;
}
//code to read the data goes here, which works fine for http pages
}
catch (std::exception& e)
{
std::cout << "Exception: " << e.what() << "\n";
}
return 0;
}
Trusted certificates are often installed or updated via the OS, browsers, or individual packages. For instance, in the *nix world, the certificates are often available through the ca-certificates package, and the certificates are installed to locations that boost::asio::ssl::context::set_default_verify_paths() will find.
The certification verification is failing because the the client is attempting to verify the peer's certificates with hostname verification (rfc2818), and is checking for the literal "host.name" to be in the certificate, and the server's certificates do not list "host.name" as a name. Try changing:
socket.set_verify_callback(ssl::rfc2818_verification("host.name"));
to:
socket.set_verify_callback(ssl::rfc2818_verification(argv[1]));
To disable peer verification, provide boost::asio::ssl::verify_none to the boost::asio::ssl::stream::set_verify_mode():
socket.set_verify_mode(boost::asio::ssl::verify_none);
Boost.Asio provides other peer verify_modes.
When peer verification is failing, it can be helpful to provide a custom callback to boost::asio::ssl::stream::set_verify_callback that provides diagnostic information. As noted in the documentation, the handler signature must be:
bool verify_callback(
bool preverified, // True if the certificate passed pre-verification.
verify_context& ctx // The peer certificate and other context.
);
Here is a custom functor that prints the certificate subject name:
///#brief Helper class that prints the current certificate's subject
/// name and the verification results.
template <typename Verifier>
class verbose_verification
{
public:
verbose_verification(Verifier verifier)
: verifier_(verifier)
{}
bool operator()(
bool preverified,
boost::asio::ssl::verify_context& ctx
)
{
char subject_name[256];
X509* cert = X509_STORE_CTX_get_current_cert(ctx.native_handle());
X509_NAME_oneline(X509_get_subject_name(cert), subject_name, 256);
bool verified = verifier_(preverified, ctx);
std::cout << "Verifying: " << subject_name << "\n"
"Verified: " << verified << std::endl;
return verified;
}
private:
Verifier verifier_;
};
///#brief Auxiliary function to make verbose_verification objects.
template <typename Verifier>
verbose_verification<Verifier>
make_verbose_verification(Verifier verifier)
{
return verbose_verification<Verifier>(verifier);
}
And its usage:
socket.set_verify_callback(make_verbose_verification(
boost::asio::ssl::rfc2818_verification(argv[1])));
On my machine, when using it and set_default_verify_paths() is not invoked, I get the following output:
$ ./a.out www.google.co.uk /?gws_rd=ssl
Verifying: /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
Verified: 0
Exception: handshake: certificate verify failed
And when set_default_verify_paths() is invoked:
$ ./a.out www.google.co.uk /?gws_rd=ssl
Verifying: /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
Verified: 1
Verifying: /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
Verified: 1
Verifying: /C=US/O=Google Inc/CN=Google Internet Authority G2
Verified: 1
Verifying: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=google.com
Verified: 1
And when rfc2818_verification("host.name") is used:
$ ./a.out www.google.co.uk /?gws_rd=ssl
Verifying: /C=US/O=Equifax/OU=Equifax Secure Certificate Authority
Verified: 1
Verifying: /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
Verified: 1
Verifying: /C=US/O=Google Inc/CN=Google Internet Authority G2
Verified: 1
Verifying: /C=US/ST=California/L=Mountain View/O=Google Inc/CN=google.com
Verified: 0
Exception: handshake: certificate verify failed
You said that" After setting this variable to point to Mozilla's cacert.pem file, everything worked as per your example". Can I know whether can use "load_verify_file(// here is the CA certificate path and file)" for your cert verification? Seems it's easier than change the environment variable points to single pem file.
I want to connect to an HTTPS server using boost::asio. I managed to successfully shake hands with the server, but I just can't manage to get the server to respond to my POST request.
This is the related code (I left out debugging and try-catch to save some space):
HTTPSClient::HTTPSClient()
{
ssl::context context(ssl::context::sslv23);
context.set_verify_mode(ssl::verify_peer);
context.set_default_verify_paths();
context.load_verify_file("certificate.pem");
mSSLSocket = new ssl::stream<ip::tcp::socket>(mIOService, context);
}
void HTTPSClient::SendRequest(const ptree &crPTree, const std::string cHost,
const std::string cURI)
{
tcp::resolver resolver(mIOService);
tcp::resolver::query query(cHost, "https");
resolver.async_resolve(query, boost::bind(&HTTPSClient::HandleResolve, this,
placeholders::error, placeholders::iterator, request));
}
void HTTPSClient::HandleResolve(const error_code &crError,
const iterator &criEndpoints, HTTPSRequest &rRequest)
{
async_connect(mSSLSocket->lowest_layer(), criEndpoints,
boost::bind(&HTTPSClient::HandleConnect, this, placeholders::error,
rRequest));
}
void HTTPSClient::HandleConnect(const error_code &crError, HTTPSRequest &rRequest)
{
mSSLSocket->lowest_layer().set_option(ip::tcp::no_delay(true));
mSSLSocket->set_verify_callback(ssl::rfc2818_verification(rRequest.mcHost));
mSSLSocket->handshake(ssl::stream_base::client);
// Write the json into a stringstream
std::ostringstream json;
boost::property_tree::write_json(json, rRequest.mcPTree);
std::string result;
result = json.str();
// Form the request
streambuf request;
std::ostream requestStream(&request);
requestStream << "POST " << rRequest.mcURI << " HTTP/1.1\r\n";
requestStream << "Host: " << rRequest.mcHost << "\r\n";
requestStream << "Accept: application/json\r\n";
requestStream << "Content-Type: application/json; charset=UTF-8\r\n";
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << result << "\r\n\r\n";
write(*mSSLSocket, request);
streambuf response;
read_until(*mSSLSocket, response, "\r\n");
std::istream responseStream(&response);
}
read_until hangs until it throws the error read_until: End of file. Everything before that goes successfully, including the SSL handshake (which I just recently figured out).
I used to do everything asynchronously until I started debugging, and started trying to backtrace to the problem, to no avail. It would be awesome if someone could help me out after two painful days of debugging.
EDIT
I just realized it might be useful to add the contents of requestStream after composing the header:
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
You need a double linefeed before the body (POST contents)
POST /authenticate HTTP/1.1
Host: <hostname>
Accept: application/json
Content-Type: application/json; charset=UTF-8
Content-Length: 136
{
"username": "vijfhoek",
"password": "test123",
<other json content>
}
Otherwise, the content will have been received by the server as header lines and the server just keeps waiting for 136 bytes of content data (also make sure that Content-Length is accurate, which it isn't in this example)
So, basically:
requestStream << "Content-Length: " << result.length() << "\r\n";
requestStream << "\r\n"; // THIS LINE ADDED
I managed to figure out what I was doing wrong. For some reason, I couldn't get boost to write data using the boost::asio::streambuf and std::ostream approach. Instead, I put the POST data in a std::string and sent it like this:
write(*mSSLSocket, boost::asio::buffer(requestString));
Which worked out fine.