Send GET to google through local proxy. C++. Linux. TCP - c++

Question

I'm trying to create a web crawler, and I want it to be able to connect to web sites through a local proxy.
So, let's say that we want to send a GET message to google and retrieve it's HTML code, all this through a local proxy (i'm working at my university and there is a proxy to connect to external sites like google).
This is my code:
#include <iostream>
#include <cstring> // Needed for memset
#include <sys/socket.h> // Needed for the socket functions
#include <netdb.h> // Needed for the socket functions
#include <cstdlib>
#include <string>
using namespace std;
int main(int argc, char* argv[])
{
addrinfo host_info; // The struct that getaddrinfo() fills up with data.
addrinfo *host_info_list;
int socketfd;
char* msg = NULL;
char* msg2 = NULL;
int status;
int len;
memset(&host_info, 0, sizeof host_info);
host_info.ai_family = AF_INET;//AF_UNSPEC;
host_info.ai_socktype = SOCK_STREAM;
//PROXY IP = proxy.feng.edu.uy ; PORT = 3318; //HTTP1.0 proxy
status = getaddrinfo("proxy.feng.edu.uy", "3318", &host_info, &host_info_list);
socketfd = socket(host_info_list->ai_family, host_info_list->ai_socktype,
host_info_list->ai_protocol);
if (socketfd == -1) std::cout << "ERROR: socket error " << std::endl ;
std::cout << "Connect()ing..." << std::endl;
status = connect(socketfd, host_info_list->ai_addr, host_info_list->ai_addrlen);
if (status == -1) std::cout << "ERROR: connect error" << std::endl ;
msg = new char[200];
strcpy(msg,"CONNECT www.google.com HTTP/1.0\r\n");
strcat(msg,"\r\n");
ssize_t bytes_sent;
len = strlen(msg);
bytes_sent = send(socketfd, msg, len, 0);
ssize_t bytes_recieved=0;
std::cout << "Waiting to recieve data..." << std::endl;
char* incoming_data_buffer = new char[200];
bytes_recieved = recv(socketfd, incoming_data_buffer,200, 0);
if (bytes_recieved == 0) std::cout << "host shut down." << std::endl ;
if (bytes_recieved == -1)std::cout << "ERROR: receive error!" << std::endl ;
std::cout << bytes_recieved << " bytes recieved" << std::endl ;
std::cout << incoming_data_buffer << std::endl;
msg2 = new char[300];
strcpy(msg2,"GET http://www.google.com/ HTTP/1.0\r\n\r\n");
std::cout << "Message sent to google: " << msg2 << std::endl;
len = strlen(msg2);
bytes_sent = send(socketfd, msg2, len, 0);
cout << "bytes_sent: " << bytes_sent << endl;
bytes_recieved=0;
std::cout << "Waiting to recieve data ..." << std::endl;
char* incoming_data_buffer2 = new char[1000];
bytes_recieved = recv(socketfd, incoming_data_buffer2,1000, 0);
if (bytes_recieved == 0) std::cout << "host shut down." << std::endl ;
if (bytes_recieved == -1)std::cout << "ERROR: recieve error!" << std::endl ;
std::cout << bytes_recieved << " bytes recieved" << std::endl ;
std::cout << incoming_data_buffer2 << std::endl;
return 0;
}
The problem I'm having is as follows..
First, incoming_data_buffer (which is the buffer from the "CONNECT") returns: "HTTP1.0 200 connection established", which is good, no problems until now.
Next I send the "GET" message to the proxy so that it forwards the message to google as expected(now that the connection is established), and it stays idle in the "recv()" for 1 minute or so, then it returns 0 (which means that the connection is closed i guess) and the buffer is empty...
My problem is that i have no clue why recv() returns 0... Any ideas?? It supposedly means the connection is closed, but why then? what else do i have to do in order for the proxy to maintain the connection? (supposing that a "connection closed" is the problem).
Thanks in advance!

Answer 1

The CONNECT method is an HTTP tunneling feature. A proxy that supports it may limit its use to connecting to HTTPS web sites (source: Wikipedia -- HTTP tunnel). You are trying to establish a connection with a standard HTTP server through CONNECT, which the proxy may be blocking.
After you establish a connection with your proxy, instead of establishing a tunnel, just send your request. This will work since you are using an absoluteURI to specify your GET target.