I have been writing a simple web server (http 1.0) for a class, but whenever I try to get a file (wget 127.0.0.1 /filename) is is short a few bytes. The confusing thing is when I sum the number of sent bytes it matches the file size, but not the amount wget receives.
Why is wget not getting all of the data I write to the socket?
some wget output
wget:
--2012-10-27 19:02:00-- (try: 4) http://127.0.0.1:5555/
Connecting to 127.0.0.1:5555... connected.
HTTP request sent, awaiting response... 200 Document follows
Length: 5777 (5.6K) [text/html]
Saving to: `index.html.6'
99% [=====================================> ] 5,776 --.-K/s in 0s
2012-10-27 19:02:00 (322 MB/s) - Read error at byte 5776/5777 (Connection reset by peer). Retrying.
--2012-10-27 19:03:52-- (try: 4) http://127.0.0.1:5555/ZoEY8.jpg
Connecting to 127.0.0.1:5555... connected.
HTTP request sent, awaiting response... 200 Document follows
Length: 163972 (160K) [image/jpeg]
Saving to: `ZoEY8.jpg.4'
91% [==================================> ] 149,449 --.-K/s in 0.001s
2012-10-27 19:03:52 (98.8 MB/s) - Read error at byte 163917/163972 (Connection reset by peer). Retrying.
Get method:
void *
processGetRequest(requestParser request)
{
string resp= "HTTP/1.0 200 Document follows\r\nServer: lab5 \r\nContent-Length: ";
string path="";
path =request.path;
//find file
int page= open (path.c_str(),O_RDONLY);
FILE * pageF= fdopen(page,"rb");
//get size
fseek(pageF, 0L, SEEK_END);
int sz = ftell(pageF);
fseek(pageF, 0L, SEEK_SET);
//form content length
stringstream ss;
ss<<resp<<sz<<"\r\n";
resp=ss.str();
//make response
if(page<0){
cout<<"404 \n";
resp = "HTTP/1.0 404 File Not Found\r\nServer: lab5 \r\nContent-type: text/html \r\n \r\n";
write( request.fd, resp.c_str(), resp.length());
return 0;
}
if(path.find(".gif")!=string::npos)
resp += "Content-type: image/gif\r\n \r\n";
else if(path.find(".png")!=string::npos)
resp += "Content-type: image/png\r\n \r\n";
else if(path.find(".jpg")!=string::npos)
resp += "Content-type: image/jpeg\r\n \r\n";
else
resp += "Content-type: text/html \r\n \r\n";
//write response
write( request.fd, resp.c_str(), resp.length());
int total=0;
char buff[1024];
int readBytes = 0;
int er;
//send file
do{
readBytes= read(page, buff, 1024);
cout<<"read bytes "<<readBytes<<"\n";
if(readBytes<0){
perror("read");
break;
}
total+=readBytes;
er= send( request.fd, buff, readBytes,0 );
cout<<"sent bytes "<<er<<"\n";
if (er==-1){
perror("send");
}
else if( er != readBytes){
cout<<"Read write miss match\n";
}
}while(readBytes>0);
close(page);
return 0;
}
Edit:
I have been working at this while and I'm wondering if Im doing my sockets wrong
// Set the IP address and port for this server
struct sockaddr_in serverIPAddress;
memset( &serverIPAddress, 0, sizeof(serverIPAddress) );
serverIPAddress.sin_family = AF_INET;
serverIPAddress.sin_addr.s_addr = INADDR_ANY;
serverIPAddress.sin_port = htons((u_short) port);
// Allocate a socket
int masterSocket = socket(PF_INET, SOCK_STREAM, 0);
if ( masterSocket < 0) {
perror("socket");
exit( -1 );
}
while ( 1 ) {
// Accept incoming connections
struct sockaddr_in clientIPAddress;
int alen = sizeof( clientIPAddress );
int slaveSocket = accept( masterSocket,
(struct sockaddr *)&clientIPAddress,
(socklen_t*)&alen);
// send slaveSocket to get method
}
My first answer is below, but i just noticed something..
"Content-type: text/html \r\n \r\n";
The headers must be separated from the content with two CR/LF. It looks like you have space in there
you can try this:
"Content-type: text/html\r\n\r\n";
Is the output buffer being correctly flushed and closed after the last write? Try changing the size of your 1024 byte read buffer to something larger than your gif file. This isnt a fix, but you may get different results, and this may help track down the cause of the problem. Maybe also put some logging into the read write loop. See if the size of the last buffer write equals the number of bytes the response is missing.
Related
I normally write a bunch of Typescript/Javascript code, and I took the HTTP protocol for granted. I decided to write a C++ program that can make GET requests to a given URL with sockets to learn more about how HTTP works. I've coded up what I think is the basic structure of what I should be doing:
Create a socket
Connect the socket to host
Send request byte by byte
Received response byte by byte
The issue is that the request always returns status code 400:
>>>> Request sent:
GET /links HTTP/1.1
>>>> Response received:
HTTP/1.1 400 Bad Request
Server: cloudflare
Date: Sun, 11 Oct 2020 05:57:11 GMT
Content-Type: text/html
Content-Length: 155
Connection: close
CF-RAY: -
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>cloudflare</center>
</body>
</html>
Here's the code I'm using to create the socket and connect it to the host:
// ---- Create the socket
int sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd < 0) error("!!!! ERROR opening socket");
// ---- Look up (and parse) the server host, route
std::string host = parseHost(url);
std::string route = parseRoute(url);
struct hostent *server = gethostbyname(host.c_str()); // host.c_str())
if (server == NULL)
error("!!!! ERROR, no such host");
// ---- Fill in the server address structure
struct sockaddr_in serv_addr;
memset(&serv_addr, 0, sizeof(serv_addr));
serv_addr.sin_family = AF_INET;
serv_addr.sin_port = htons(PORT); // PORT
memcpy(&serv_addr.sin_addr.s_addr, server->h_addr, server->h_length);
// ---- Connect the socket to the given url's host
const int didConnect = connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
if (didConnect < 0) error("!!!! ERROR connecting to host");
If that logic seems correct, then here's the code that sends the request and receives the response:
// ---- Send the GET request byte by byte
int bytes, sent, received;
char response[MAX_CONTENT_LENGTH];
int total = sizeof(response) - 1;
sent = 0;
std::string message = "GET " + route + "HTTP/1.1" + "\r\n\r\n";
do
{
bytes = write(sockfd, message.c_str() + sent, total - sent);
if (bytes < 0) error("!!!! ERROR writing message to socket");
if (bytes == 0) break;
sent += bytes;
} while (sent < total);
std::cout << ">>>> Request sent: \n" << message << std::endl;
// ---- Receive the response byte by byte
memset(response, 0, sizeof(response));
total = sizeof(response) - 1;
received = 0;
do {
bytes = read(sockfd, response + received, total - received);
if (bytes < 0)
error("!!!! ERROR reading response from socket");
if (bytes == 0)
break;
received += bytes;
} while (received < total);
if (received == total)
error("!!!! ERROR storing complete response from socket (not enough space allocated)");
TLDR: Am I sending the right char array to the server? As of right now it has this format: "GET /route HTTP/1.1\r\n\r\n". Am I forgetting any headers/other information that lets the server know that it's a GET request?
>>>> Request sent:
GET /links HTTP/1.1
This is not a valid HTTP request. It is missing the Host header with the target domain name, i.e. it should be at least something like this
GET /links HTTP/1.1
Host: www.example.com
Apart from that your code simply tries to read data until the server closes the connection or the size of your internal buffer is reached. But HTTP/1.1 by default uses persistent connection. So the server might simply keep the connection open after the response is sent because it expects your next request, which means that your program will simply block doing nothing.
Note that HTTP is far more complex than it looks. Please study the actual standard if you want to implement it instead of using existing libraries. That's what standards are for.
How should this be done?
I want to receive a (rather long) HTTP request and cannot get this to work.
The problem: Without flags, recv does not read the whole message. I guess this is normal behavior. From what I understand using the MSG_WAITALL flag causes it to block until everything is received. However, in that case the call blocks forever (until I ctrl+c the client (curl) process.
Below there is a (still lengthy, but rather minimal) example snippet.
Sorry it mixes C and C++ style but I wanted to avoid own errors and sticked largely to example code with as few modifications as possible.
// Example largely based on: beej.us/guide/bgnet/output/html/singlepage/bgnet.html
// Mixes c-style and c++ due to my modifications.
// Only intended to reproduce a problem.
#include <unistd.h>
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <arpa/inet.h>
#include <sys/wait.h>
#include <sstream>
#include <iostream>
#include <cstring>
#define PORT "7004" // the port users will be connecting to
#define BACKLOG 10 // how many pending connections queue will hold
// get sockaddr, IPv4 or IPv6:
void *get_in_addr(struct sockaddr *sa)
{
if (sa->sa_family == AF_INET) {
return &(((struct sockaddr_in*)sa)->sin_addr);
}
return &(((struct sockaddr_in6*)sa)->sin6_addr);
}
int main(void)
{
int sockfd, new_fd; // listen on sock_fd, new connection on new_fd
struct addrinfo hints, *servinfo, *p;
struct sockaddr_storage their_addr; // connector's address information
socklen_t sin_size;
struct sigaction sa;
int yes=1;
char s[INET6_ADDRSTRLEN];
int rv;
memset(&hints, 0, sizeof hints);
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_flags = AI_PASSIVE; // use my IP
if ((rv = getaddrinfo(NULL, PORT, &hints, &servinfo)) != 0) {
fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rv));
return 1;
}
// loop through all the results and bind to the first we can
for(p = servinfo; p != NULL; p = p->ai_next) {
if ((sockfd = socket(p->ai_family, p->ai_socktype,
p->ai_protocol)) == -1) {
perror("server: socket");
continue;
}
if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes,
sizeof(int)) == -1) {
perror("setsockopt");
exit(1);
}
if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
close(sockfd);
perror("server: bind");
continue;
}
break;
}
freeaddrinfo(servinfo); // all done with this structure
if (p == NULL) {
fprintf(stderr, "server: failed to bind\n");
exit(1);
}
if (listen(sockfd, BACKLOG) == -1) {
perror("listen");
exit(1);
}
printf("server: waiting for connections...\n");
while(1) { // main accept() loop
sin_size = sizeof their_addr;
new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &sin_size);
inet_ntop(their_addr.ss_family,
get_in_addr((struct sockaddr *)&their_addr),
s, sizeof s);
printf("server: got connection from %s\n", s);
const uint EASILY_ENOUGH = 1000000;
char* buffer = new char[EASILY_ENOUGH + 1];
auto bytesRead = recv(new_fd, buffer, EASILY_ENOUGH, MSG_WAITALL); // FREEZES UNTIL I KILL THE CLIENT
// auto bytesRead = recv(new_fd, buffer, EASILY_ENOUGH, 0); // DOES NOT READ FULL REQUEST
if (bytesRead == -1) {
perror("recv");
close(new_fd);
continue;
}
buffer[bytesRead] = 0;
printf("bytes read: %ld\n", bytesRead);
printf("request: %s\n", buffer);
delete[] buffer;
std::string content = "some content.";
std::ostringstream os;
os << "HTTP/1.1 200 OK\r\n" << "Content-Length: " << content.size() << "\r\n"
<< "Connection: close\r\n" << "Content-Type: " << "plain/text"
<< "; charset=" << "utf-8" << "\r\n"
<< "Access-Control-Allow-Origin: *" << "\r\n" << "\r\n" << content;
std::string response = os.str();
printf("Response has size: %ld\n", response.size());
auto bytesSent = send(new_fd, response.c_str(), response.size(), 0);
printf("Method send claims it has sent %ld bytes\n", bytesSent);
close(new_fd);
}
return 0;
}
The curl resquest i used:
curl 'http://<myhost>:7004/___THE START___jsondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwa___THE END'
With MSG_WAITALL my server produces the output:
server: waiting for connections...
server: got connection from <my ip>
and hangs, once I CTRL+C curl it progresses with:
bytes read: 6379
request: GET /___THE START___jsondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwasondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwa___THE END HTTP/1.1
Host: <myhost>:7004
User-Agent: curl/7.49.0
Accept: */*
Response has size: 144
Method send claims it has sent 144 bytes
Without MSG_WAITALL and the same request, the server produces the output:
server: waiting for connections...
server: got connection from <my ip>
bytes read: 1448
request: GET /___THE START___jsondsadasdasdzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwaghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewjfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiwehfoewhfiuewhfiewhfhfiuewhfiwehfiuewwehgurwadzughjdssftzghuijerdftzghujiesdhufdewojfoewfhoiw
Response has size: 144
Method send claims it has sent 144 bytes
And curl does not receive the HTTP response.
However, a shorter request properly receives the response:
curl 'http://myhost:7004/___THE START___THE END__'
> some content.
And the server properly received the full request:
server: got connection from <my ip>
bytes read: 128
request: GET /___THE START___THE END__ HTTP/1.1
Host: <myhost>:7004
User-Agent: curl/7.49.0
Accept: */*
Response has size: 144
Method send claims it has sent 144 bytes
I do understand that it is normal that not all TCP packets arrive right away. However, what is the correct way for me to assemble the full request? I also tried non-blocking variants but usually ran into situations where no more data was ready for reading. If necessary I can produce sample code for this, similarly.
PS: The problem manifests when request length and connection badness exceed a threshold. I cannot reproduce it on my own machine with queries to localhost, and depending on my connection, the request length where problems start to manifest varies.
HTTP is a protocol, it has structure and rules to it. Read RFC 2616, particularly Section 4 "HTTP Message".
Your recv code is not doing anything to follow the protocol at all. You can't just blindly read an arbitrary buffer of data and expect it to be the complete HTTP request correctly. You have to read the request according to the rules of the protocol. Specifically, you need to:
read a CRLF delimited line of text. This will contain the requested method, resource, and HTTP version.
then read a variable length list of CRLF delimited request headers. The list is terminated by a CRLF CRLF sequence.
then analyze the request method and headers to determine whether the request has an entity body. If so, the headers will tell you how it is encoded over the connection (see Section 4.4 "Message Length"), so you know how it needs to be read and when you need to stop reading.
then process the completed request, and send your response.
then close the connection, unless either:
the request asked for HTTP 1.0 and Connection: keep-alive is present in the request headers.
the request asked for HTTP 1.1+ and Connection: close is not present in the request headers.
My sockets server is receiving a GET request for an image, the image is 2MB so it doesn't fit in a single send(), this is what I am sending in the first send():
std::stringstream wsss;
wsss << "HTTP/1.1 200 OK\r\n"
<< "Connection: keep-alive\r\n"
<< "Content-Type: image/x-icon\r\n"
<< "Content-Length: " << imageSize << "\r\n"
<< "\r\n";
wsss.write(imageData, imageSize);
Does every subsequent send() of this image needs the header fields?
I am sending a .ico image, are the header fields correct?
the image is 2MB so it doesn't fit in a single send()
send() is not guaranteed to send as many bytes as you ask it to send. It can send fewer bytes. Its return value tells you how many bytes it actually accepted for sending. So you should call send() in a loop until all bytes have been accepted. If you move this loop into its own reusable function, that will also allow you to send the icon data without having to first copy it into the std::stringstream.
Try something like this:
int sendData(int sckt, void *data, int datalen)
{
unsigned char *pdata = (unsigned char *) data;
int numSent;
// send() can send fewer bytes than requested,
// so call it in a loop until the specified data
// has been sent in full...
while (datalen > 0) {
numSent = send(sckt, pdata, datalen, 0);
if (numSent == -1) return -1;
pdata += numSent;
datalen -= numSent;
}
return 0;
}
std::stringstream wsss;
wsss << "HTTP/1.1 200 OK\r\n"
<< "Connection: keep-alive\r\n"
<< "Content-Type: image/x-icon\r\n"
<< "Content-Length: " << imageSize << "\r\n"
<< "\r\n";
// do not append the image data to the stringstream...
//wsss.write(imageData, imageSize);
// send the headers first...
std::string headers = wsss.str();
int res = sendData(TheSocket, headers.c_str(), headers.size());
if (res == -1) ...
// now send the image data...
res = sendData(TheSocket, imageData, imageSize);
if (res == -1) ...
Does every subsequent send() of this image needs the header fields?
Every HTTP response to every HTTP request for the same image needs to send the same headers1. But every send() for any particular response does not need to repeat the headers, they only need to be sent once. Just keep sending whatever bytes have not been sent yet. That is why you have to pay attention to the return value of send() so you know how many bytes have been sent so far and how many bytes are still need to be sent.
I am sending a .ico image, are the header fields correct?
In general, yes1.
1: assuming that either:
the client sent an HTTP 1.1 request without a Connection: close request header.
the client sent an HTTP 1.0 request with a Connection: keep-alive request header.
Otherwise, your Connection: keep-alive header would be erroneous, you should be sending a Connection: close header instead, and then close the socket after sending the complete response.
First, here is the C++ code I am using to make HTTP POST requests. It links to ws2_32.lib.
int POST_TEST(std::string URL, std::string DATA)
{
//**************************************************
std::string TEXT = "";
std::string DIRECTORY = URL.substr(URL.find(".com") + std::string(".com").length());
std::string HOST = URL.substr(0, URL.find(".com", std::string(".com").length())) + ".com";
std::string REQ = "POST " + DIRECTORY + " HTTP/1.1\r\nContent-Length: " + to_string(DATA.length()) + "\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: " + HOST + "\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:44.0) Gecko/20100101 Firefox/44.0\r\nConnection: keep-alive\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n" + DATA;
const char * REQUEST = REQ.c_str();
//**************************************************
WSADATA wsaData;
if(WSAStartup(MAKEWORD(2, 2), &wsaData) != 0)
return 1;
SOCKET Socket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
struct hostent *host = gethostbyname(HOST.c_str());
SOCKADDR_IN SockAddr;
SockAddr.sin_port = htons(80);
SockAddr.sin_family = AF_INET;
SockAddr.sin_addr.s_addr = *((unsigned long*)host->h_addr);
if(connect(Socket,(SOCKADDR*)(&SockAddr), sizeof(SockAddr)) != 0)
return 1;
send(Socket, REQUEST, strlen(REQUEST), 0);
char buffer[1000];
int nDataLength;
while((nDataLength = recv(Socket, buffer, 1000, 0)) > 0)
{
int i = 0;
while(buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r')
{
TEXT += buffer[i];
i++;
}
}
closesocket(Socket);
WSACleanup();
MessageBoxA(NULL, TEXT.c_str(), "HTTP Response", MB_OK);
return 0;
}
POST_TEST("blah.com/a.php", "a=blah");
The code works well, but I am receiving only the HTTP Header response, and I don't see the response BODY. Making the POST request using Python to the same php page allows me to see the response body just fine. I'm not sure if I'm misunderstanding something, or there's an error in my code.
The PHP page I'm POSTing to is located on my own server. The PHP echoes some text, but I do not see that in the response. This is the response I get:
HTTP/1.1 200 OK
Server: nginx/1.8.1
Date: Fri, 08 Apr 2016 02:10:14 GMT
Content-Type: text/html
Content-Length: 286
Connection: keep-alive
Vary: Accept-Encoding
Content-Encoding: gzip
How would I be able to see the entire response?
There are two bugs in the shown code.
It doesn't check the return value from send(). send() does not guarantee that the number of bytes requested to be sent was actually sent. The return value from send() may return an indication that fewer bytes were written than what were requested. It is your responsibility to check that, and try again to send() the remaining bytes.
The logic for reading and collecting the response is wrong. recv() tells you how many bytes were received. This is ignored completely. Instead, the code scans the read buffer until it sees a control character as a byte.
Note that your HTTP response indicates that you're getting raw, gzip-compressed binary data. It is fairly likely that within the first few bytes of the actual response there are going to be some binary bytes that correspond to ASCII control characters, in value. Your loop will come to a screeching halt at that point.
Even if you were expecting a plain text response, this logic would still be wrong. It is the return value from recv() that tells you how many bytes were received, and nothing else.
So I thought I would play with HTTP a bit and try to send simple plain(not encoded) text from my program to a server. However, something is not right and I don't know what.
Here is the server side PHP script, I have tested it by sending POST data from HTML form and it worked just great, so I guess there isn't anything wrong on server side.
<?php
$file = 'postData.txt';
$somecontent = $_POST['dat'];
$fp = fopen($file, 'w') or die('Could not open file!');
fwrite($fp, "$somecontent") or die('Could not write to file');
fclose($fp);
?>
Here is the program(this code includes some unused parts like reading file content in buffer etc., that's cuz I am playing with it all the time and changing stuff every 5 seconds, don't mind it):
#include <windows.h>
#include "WinSock2.h"
#include <stdio.h>
#include <stdint.h>
#include <iostream>
int main()
{
WSADATA wsa;
if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0)
return 0;
SOCKET fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (fd < 0) throw;
SOCKADDR_IN service;
service.sin_family = AF_INET;
service.sin_port = htons(80);
LPHOSTENT host = gethostbyname("123.17.25.123");
if (!host) throw;
service.sin_addr = *((LPIN_ADDR)*host->h_addr_list);
if (connect(fd, (SOCKADDR *)&service, sizeof(service)) < 0) throw;
FILE *f = fopen("file.txt", "rb");
if (!f) throw;
uint32_t len = 0;
fseek(f, 0x00, SEEK_END);
len = ftell(f);
fseek(f, 0x00, SEEK_SET);
char header[1024];
char *buffer = new char[len];
fread(buffer, sizeof(char), len, f);
sprintf(header,
"POST /recv.php HTTP/1.1\r\n"
"Host: 123.17.25.123\r\n"
"User-Agent: Mozilla Firefox/4.0\r\n"
"Content-Length: %d\r\n"
"Content-Type: application/x-www-form-urlencoded\r\n"
"Accept-Charset: utf-8\r\n\r\n",
len+4);
std::cout << header << std::endl;
send(fd, header, strlen(header), 0);
send(fd, "dat=", 4, 0);
send(fd, buffer, strlen(buffer), 0);
send(fd, "\r\n", 2, 0);
delete [] buffer;
fprintf(stderr, "Done\n");
closesocket(fd);
WSACleanup();
return 0;
}
So, what's wrong with it? Does anyone have any idea? :P
edit1: I monitored the traffic with wireshark and tried to run the program few times, but no packets were captured. Strange, it does not even send anything anywhere.
edit2: Thanks to TokenMacGuy got it working. Code above is lame, but it will read all file content and send it as POST data to your server, hopefully it will be useful for noobs like me to learn. Thank you once again!
You aren't recieving the data because you aren't actually sending any data. Although buffer appears in the sprintf, there's no format specifier to consume it (only the length is formatted).
Try removing the buffer from the sprintf call altogether, and then call send twice, once for the headers (as you already do) and again to send the actual data.
Or maybe you don't intend to send any data from the file you read. You just want to get that dat=somedatar. The problem is that you indicate the content-type as text/plain, in which case the server won't interpret it at all. The content type should probably be application/x-www-form-urlencoded. Since the dat parameter is part of the body, the content-length header must include it. If the content length doesn't match the actual number of bytes sent as content, conforming servers ignore the whole request (usually returning a 400-499 range response code).
POST should have two newlines before the data, not after it.
"Accept-Charset: utf-8\r\n\r\n"
"dat=somedata\r\n",