Boost ASIO : connection rejected on someURL's - c++

Question

I'm trying to make http requests from c++ using boost asio library based on the example given on boost website : http://www.boost.org/doc/libs/1_36_0/doc/html/boost_asio/example/http/client/sync_client.cpp
The implementation is working fine on some URL's example : www.amazon.com, www.cnn.com etc.
On the other hand on some URL's the request is beingg rejected from the server
eg: www.stackoverflow.com, www.flipkart.com, www.soundcloud.com etc.
The below code creates a new thread: httpRequest and calls a function task with params as host,path,portnumber
#include <iostream>
#include <thread>
#include <boost/asio.hpp>
using boost::asio::ip::tcp;
void createRequest(std::string host,std::string path,std::string port) {
try { boost::asio::io_service io_service;
// Get a list of endpoints corresponding to the server name.
tcp::resolver resolver(io_service);
tcp::resolver::query query(host, port);
tcp::resolver::iterator endpoint_iterator = resolver.resolve(query);
tcp::resolver::iterator end;
// Try each endpoint until we successfully establish a connection.
tcp::socket socket(io_service);
boost::system::error_code error1 = boost::asio::error::host_not_found;
while (error1 && endpoint_iterator != end)
{
socket.close();
socket.connect(*endpoint_iterator++, error1);
}
if (error1)
throw boost::system::system_error(error1);
// Form the request. We specify the "Connection: close" header so that the
// server will close the socket after transmitting the response. This will
// allow us to treat all data up until the EOF as the content.
boost::asio::streambuf request;
std::ostream request_stream(&request);
request_stream << "GET " << path << " HTTP/1.1\r\n";
request_stream << "Host: " << host << "\r\n";
request_stream << "Accept: */*\r\n";
request_stream << "Connection: close\r\n\r\n";
// Send the request.
boost::asio::write(socket, request);
//std::cout << typeid(socket).name() << std::endl;
// Read the response status line. The response streambuf will automatically
// grow to accommodate the entire line. The growth may be limited by passing
// a maximum size to the streambuf constructor.
boost::asio::streambuf response;
boost::asio::read_until(socket, response, "\r\n");
// Check that response is OK.
std::istream response_stream(&response);
std::string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
std::string status_message;
std::getline(response_stream, status_message);
if (!response_stream || http_version.substr(0, 5) != "HTTP/") {
std::cout << "Invalid response\n";;
}
if (status_code != 200) {
std::cout << "Response returned with status code " << status_code << "\n";
}
// Read the response headers, which are terminated by a blank line.
boost::asio::read_until(socket, response, "\r\n\r\n");
// Process the response headers.
std::string header;
while (std::getline(response_stream, header) && header != "\r")
std::cout << header << "\n";
std::cout << "\n";
// Write whatever content we already have to output.
if (response.size() > 0)
std::cout << &response;
// Read until EOF, writing data to output as we go.
boost::system::error_code error;
while (boost::asio::read(socket, response, boost::asio::transfer_at_least(1), error))
std::cout << &response;
if (error != boost::asio::error::eof)
throw boost::system::system_error(error);
}
catch (std::exception& e) {
std::cout << "Exception: " << e.what() << "\n";
}
}
int main() {
std::thread httpThred(createRequest, "www.stackoverflow.com","/","80");
std::cout << "async task launched\n";
std::cout << "main done\n";
httpThred.join();
}
output:
Response returned with status code 301
Content-Type: text/html; charset=UTF-8
Location: http://stackoverflow.com/
Date: Sun, 05 Jul 2015 19:50:44 GMT
Connection: close
Content-Length: 148

Answer 1

On the other hand on some URL's the request is beingg rejected from the server eg: ...
Response returned with status code 301 ... Location: https://stackoverflow.com/
This is no rejection but a redirect, i.e. you should request the resource at the given URI.
// Form the request. We specify the "Connection: close" header so that the
// server will close the socket after transmitting the response. This will
// allow us to treat all data up until the EOF as the content.
Not true. You are making a HTTP/1.1 request so the response could be with Transfer-Encoding chunked. This kind of encoding needs special processing which you don't do. Note that the example code you refer too uses HTTP/1.0 so it is not affected by this problem.
I really recommend that you make yourself comfortable with the basics on how HTTP works before trying to implement it on your own.