My sockets server is receiving a GET request for an image, the image is 2MB so it doesn't fit in a single send(), this is what I am sending in the first send():
std::stringstream wsss;
wsss << "HTTP/1.1 200 OK\r\n"
<< "Connection: keep-alive\r\n"
<< "Content-Type: image/x-icon\r\n"
<< "Content-Length: " << imageSize << "\r\n"
<< "\r\n";
wsss.write(imageData, imageSize);
Does every subsequent send() of this image needs the header fields?
I am sending a .ico image, are the header fields correct?
the image is 2MB so it doesn't fit in a single send()
send() is not guaranteed to send as many bytes as you ask it to send. It can send fewer bytes. Its return value tells you how many bytes it actually accepted for sending. So you should call send() in a loop until all bytes have been accepted. If you move this loop into its own reusable function, that will also allow you to send the icon data without having to first copy it into the std::stringstream.
Try something like this:
int sendData(int sckt, void *data, int datalen)
{
unsigned char *pdata = (unsigned char *) data;
int numSent;
// send() can send fewer bytes than requested,
// so call it in a loop until the specified data
// has been sent in full...
while (datalen > 0) {
numSent = send(sckt, pdata, datalen, 0);
if (numSent == -1) return -1;
pdata += numSent;
datalen -= numSent;
}
return 0;
}
std::stringstream wsss;
wsss << "HTTP/1.1 200 OK\r\n"
<< "Connection: keep-alive\r\n"
<< "Content-Type: image/x-icon\r\n"
<< "Content-Length: " << imageSize << "\r\n"
<< "\r\n";
// do not append the image data to the stringstream...
//wsss.write(imageData, imageSize);
// send the headers first...
std::string headers = wsss.str();
int res = sendData(TheSocket, headers.c_str(), headers.size());
if (res == -1) ...
// now send the image data...
res = sendData(TheSocket, imageData, imageSize);
if (res == -1) ...
Does every subsequent send() of this image needs the header fields?
Every HTTP response to every HTTP request for the same image needs to send the same headers1. But every send() for any particular response does not need to repeat the headers, they only need to be sent once. Just keep sending whatever bytes have not been sent yet. That is why you have to pay attention to the return value of send() so you know how many bytes have been sent so far and how many bytes are still need to be sent.
I am sending a .ico image, are the header fields correct?
In general, yes1.
1: assuming that either:
the client sent an HTTP 1.1 request without a Connection: close request header.
the client sent an HTTP 1.0 request with a Connection: keep-alive request header.
Otherwise, your Connection: keep-alive header would be erroneous, you should be sending a Connection: close header instead, and then close the socket after sending the complete response.
Related
This question already has an answer here:
Differ between header and content of http server response (sockets)
(1 answer)
Closed 1 year ago.
I'm making this socket HTTP client (very basic). When recv()'ing response data from example.com it works fine and writes it all to a buffer but when I try to revc any bigger amounts of data it stops at around 1500 bytes.
Right now all I'm trying to do is get the response written into the buffer (headers and all). Not trying to parse anything. But that isn't working. It works for a few iterations but then stops or hangs. I'm asking for help identifying the issue with this receive_response() function that causes these behaviors.
This is the function that revc's the HTTP response:
void tcp_client::receive_response(char *buffer) {
int bytes_recv = 0;
int total_bytes_recv = 0;
for (;;) {
bytes_recv = recv(sock, &buffer[total_bytes_recv], CHUNK_SIZE, 0);
if (bytes_recv <= 0) {
break;
} else {
total_bytes_recv += bytes_recv;
}
}
}
The main function:
int main(int argc, char **argv) {
http_client http;
char response[100000] = {0};
http.connect_to_host("go.com", 80);
http.send_request("GET / HTTP/1.1\r\n\r\n");
http.receive_response(response);
std::cout << response << std::endl;
return 0;
}
Thank you
You seem to expect the server to close the connection after the response is transmitted. A typical HTTP 1.1 server doesn't do that by default; they keep the connection open for further requests, unless the client explicitly asks otherwise via Connection: close header.
So, you receive all the data, and then the next recv call is sitting there, waiting for more data to arrive.
An HTTP 1.1 client is expected to detect the end of response via Content-Length header, or by decoding a chunked response as indicated by Transfer-Encoding: chunked header.
First, here is the C++ code I am using to make HTTP POST requests. It links to ws2_32.lib.
int POST_TEST(std::string URL, std::string DATA)
{
//**************************************************
std::string TEXT = "";
std::string DIRECTORY = URL.substr(URL.find(".com") + std::string(".com").length());
std::string HOST = URL.substr(0, URL.find(".com", std::string(".com").length())) + ".com";
std::string REQ = "POST " + DIRECTORY + " HTTP/1.1\r\nContent-Length: " + to_string(DATA.length()) + "\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: " + HOST + "\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:44.0) Gecko/20100101 Firefox/44.0\r\nConnection: keep-alive\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n" + DATA;
const char * REQUEST = REQ.c_str();
//**************************************************
WSADATA wsaData;
if(WSAStartup(MAKEWORD(2, 2), &wsaData) != 0)
return 1;
SOCKET Socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
struct hostent *host = gethostbyname(HOST.c_str());
SOCKADDR_IN SockAddr;
SockAddr.sin_port = htons(80);
SockAddr.sin_family = AF_INET;
SockAddr.sin_addr.s_addr = *((unsigned long*)host->h_addr);
if(connect(Socket,(SOCKADDR*)(&SockAddr), sizeof(SockAddr)) != 0)
return 1;
send(Socket, REQUEST, strlen(REQUEST), 0);
char buffer[1000];
int nDataLength;
while((nDataLength = recv(Socket, buffer, 1000, 0)) > 0)
{
int i = 0;
while(buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r')
{
TEXT += buffer[i];
i++;
}
}
closesocket(Socket);
WSACleanup();
MessageBoxA(NULL, TEXT.c_str(), "HTTP Response", MB_OK);
return 0;
}
POST_TEST("blah.com/a.php", "a=blah");
The code works well, but I am receiving only the HTTP Header response, and I don't see the response BODY. Making the POST request using Python to the same php page allows me to see the response body just fine. I'm not sure if I'm misunderstanding something, or there's an error in my code.
The PHP page I'm POSTing to is located on my own server. The PHP echoes some text, but I do not see that in the response. This is the response I get:
HTTP/1.1 200 OK
Server: nginx/1.8.1
Date: Fri, 08 Apr 2016 02:10:14 GMT
Content-Type: text/html
Content-Length: 286
Connection: keep-alive
Vary: Accept-Encoding
Content-Encoding: gzip
How would I be able to see the entire response?
There are two bugs in the shown code.
It doesn't check the return value from send(). send() does not guarantee that the number of bytes requested to be sent was actually sent. The return value from send() may return an indication that fewer bytes were written than what were requested. It is your responsibility to check that, and try again to send() the remaining bytes.
The logic for reading and collecting the response is wrong. recv() tells you how many bytes were received. This is ignored completely. Instead, the code scans the read buffer until it sees a control character as a byte.
Note that your HTTP response indicates that you're getting raw, gzip-compressed binary data. It is fairly likely that within the first few bytes of the actual response there are going to be some binary bytes that correspond to ASCII control characters, in value. Your loop will come to a screeching halt at that point.
Even if you were expecting a plain text response, this logic would still be wrong. It is the return value from recv() that tells you how many bytes were received, and nothing else.
I have been writing a simple web server (http 1.0) for a class, but whenever I try to get a file (wget 127.0.0.1 /filename) is is short a few bytes. The confusing thing is when I sum the number of sent bytes it matches the file size, but not the amount wget receives.
Why is wget not getting all of the data I write to the socket?
some wget output
wget:
--2012-10-27 19:02:00-- (try: 4) http://127.0.0.1:5555/
Connecting to 127.0.0.1:5555... connected.
HTTP request sent, awaiting response... 200 Document follows
Length: 5777 (5.6K) [text/html]
Saving to: `index.html.6'
99% [=====================================> ] 5,776 --.-K/s in 0s
2012-10-27 19:02:00 (322 MB/s) - Read error at byte 5776/5777 (Connection reset by peer). Retrying.
--2012-10-27 19:03:52-- (try: 4) http://127.0.0.1:5555/ZoEY8.jpg
Connecting to 127.0.0.1:5555... connected.
HTTP request sent, awaiting response... 200 Document follows
Length: 163972 (160K) [image/jpeg]
Saving to: `ZoEY8.jpg.4'
91% [==================================> ] 149,449 --.-K/s in 0.001s
2012-10-27 19:03:52 (98.8 MB/s) - Read error at byte 163917/163972 (Connection reset by peer). Retrying.
Get method:
void *
processGetRequest(requestParser request)
{
string resp= "HTTP/1.0 200 Document follows\r\nServer: lab5 \r\nContent-Length: ";
string path="";
path =request.path;
//find file
int page= open (path.c_str(),O_RDONLY);
FILE * pageF= fdopen(page,"rb");
//get size
fseek(pageF, 0L, SEEK_END);
int sz = ftell(pageF);
fseek(pageF, 0L, SEEK_SET);
//form content length
stringstream ss;
ss<<resp<<sz<<"\r\n";
resp=ss.str();
//make response
if(page<0){
cout<<"404 \n";
resp = "HTTP/1.0 404 File Not Found\r\nServer: lab5 \r\nContent-type: text/html \r\n \r\n";
write( request.fd, resp.c_str(), resp.length());
return 0;
}
if(path.find(".gif")!=string::npos)
resp += "Content-type: image/gif\r\n \r\n";
else if(path.find(".png")!=string::npos)
resp += "Content-type: image/png\r\n \r\n";
else if(path.find(".jpg")!=string::npos)
resp += "Content-type: image/jpeg\r\n \r\n";
else
resp += "Content-type: text/html \r\n \r\n";
//write response
write( request.fd, resp.c_str(), resp.length());
int total=0;
char buff[1024];
int readBytes = 0;
int er;
//send file
do{
readBytes= read(page, buff, 1024);
cout<<"read bytes "<<readBytes<<"\n";
if(readBytes<0){
perror("read");
break;
}
total+=readBytes;
er= send( request.fd, buff, readBytes,0 );
cout<<"sent bytes "<<er<<"\n";
if (er==-1){
perror("send");
}
else if( er != readBytes){
cout<<"Read write miss match\n";
}
}while(readBytes>0);
close(page);
return 0;
}
Edit:
I have been working at this while and I'm wondering if Im doing my sockets wrong
// Set the IP address and port for this server
struct sockaddr_in serverIPAddress;
memset( &serverIPAddress, 0, sizeof(serverIPAddress) );
serverIPAddress.sin_family = AF_INET;
serverIPAddress.sin_addr.s_addr = INADDR_ANY;
serverIPAddress.sin_port = htons((u_short) port);
// Allocate a socket
int masterSocket = socket(PF_INET, SOCK_STREAM, 0);
if ( masterSocket < 0) {
perror("socket");
exit( -1 );
}
while ( 1 ) {
// Accept incoming connections
struct sockaddr_in clientIPAddress;
int alen = sizeof( clientIPAddress );
int slaveSocket = accept( masterSocket,
(struct sockaddr *)&clientIPAddress,
(socklen_t*)&alen);
// send slaveSocket to get method
}
My first answer is below, but i just noticed something..
"Content-type: text/html \r\n \r\n";
The headers must be separated from the content with two CR/LF. It looks like you have space in there
you can try this:
"Content-type: text/html\r\n\r\n";
Is the output buffer being correctly flushed and closed after the last write? Try changing the size of your 1024 byte read buffer to something larger than your gif file. This isnt a fix, but you may get different results, and this may help track down the cause of the problem. Maybe also put some logging into the read write loop. See if the size of the last buffer write equals the number of bytes the response is missing.
I'm trying to upload a PNG file through Winsock2 HTTP Post. Here's my request string:
boundary = "-----rueldotme";
request += "POST " + uri + " HTTP/1.1\r\n";
request += "Host: " + hostname + "\r\n";
request += "User-Agent: " + UserAgent + "\r\n";
request += "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
request += "Accept-Language: en-us,en;q=0.5\r\n";
request += "Accept-Encoding: gzip,deflate\r\n";
request += "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n";
request += "Keep-Alive: 115\r\n";
request += "Connection: keep-alive\r\n";
request += "Content-Length: " + fileSize + "\r\n";
request += "Content-Type: multipart/form-data, boundary=" + boundary + "\r\n";
request += "\r\n";
request += "--" + boundary + "\r\n";
request += "Content-Disposition: form-data; name=\"filename\"; filename=\"" + fileName + "\"\r\n";
request += "Content-Type: image/png; charset=binary\r\n";
request += "Content-Transfer-Encoding: binary\r\n";
request += "\r\n";
request += "%s\r\n";
request += "\r\n";
The connection is OK, no errors and such, the fileCon by the way is from ReadFile(). And there's no error code. The number of bytes read is the same as the output of GetFileSize(). I tried displaying the contents of fileCon but only gave me this:
Don't mind the title "Error" (I set it).
Also, the request takes ages to complete, and gives me a blank response. Yep, blank with no HTTP headers. Am I doing the request right? Should I really have to include the file contents at the POST data?
Thanks in advance.
EDIT: The PNG size is about 256KB. I'm in a 1mbps connection.
EDIT: Sorry if the information was insufficient. Anyway, here's what I did lately:
int flz;
char bdata[BSIZE];
DWORD dwe, bytesRead = 0;
HANDLE fh = CreateFile(fileName.c_str(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
LPVOID fbuff = NULL;
flz = GetFileSize(fh, NULL);
fbuff = malloc(flz);
ReadFile(fh, fbuff, flz, &bytesRead, NULL));
...
sprintf_s(bdata, BSIZE, request.c_str(), reinterpret_cast<char *>(fbuff)); //BSIZE = 1024
...
send(sock, bdata, std::strlen(bdata), 0);
Not enough information to solve the problem, so I'll give a meta-answer instead:
Use a packet sniffer (e.g., wireshark) to check exactly what data is actually being sent and received. This will let you verify that the request is as it should be, and that the "blank response" you're getting really is blank.
One wild stab in the dark:
You haven't included any variable declarations in your code snippet, so I don't know what type "fileCon" is, but don't forget that the PNG data is likely to contain null bytes, which will mess up a default conversion from a char* to a std::string.
Edit:
Your modification contains the same bug that the std::string based version had, namely, that the PNG data is likely to contain null bytes. Perhaps this code will explain more clearly:
const char* data = "Hello\0world."; // some data that contains a null byte
std::string dataStr(data);
std::cout << dataStr << "\n"; // will print "Hello".
std::cout << dataStr.size() << "\n"; // will print "5"
char buf[512];
sprintf_s(buf, sizeof(buf), "Data: %s\n", data);
std::cout << buf; // will print "Data: Hello"
Both the conversion to std::string and formatting with sprintf will interpret the null byte as being the end of the data, and so the rest of the original data ("world.") will never be used.
I'm having a significant problem using a standard BSD-style socket in a C++ program. In the code below, I connect to a local web server, send a request, and simply create a loop waiting for data to return. I actually do receive the data, but then I get an endless stream of 0-length data as if it was a non-blocking socket. The web server presumably didn't kill the connection, because if so I would have received a length of -1.
Please ignore simple typos I make below, as I'm writing the code from memory, not a direct copy/paste. The code produces the same result on OSX and Windows.
int sock = socket(AF_INET, SOCK_STREAM, 0);
//assume serv_addr has been created correctly
connect(sock, (sockaddr*)&serv_addr, sizeof(serv_addr)) < 0);
std::string header = "GET / HTTP/1.1\r\n"
"Host: 127.0.0.1:80\r\n"
"Keep-Alive: 300\r\n"
"Connection: keep-alive\r\n\r\n";
send(sock, header.c_str(), header.length()+1, 0);
for (;;) {
char buffer[1024];
int len = recv(sock, buffer, 1024, 0);
cout << len << endl;
//this outputs two numbers around 200 and 500,
//which are the header and html, and then it
//outputs and endless stream of 0's
}
From the man page of recv
For TCP sockets, the return value 0 means the peer has closed its half
side of the connection.