I'm serving some files locally via HTTP using QTcpSocket. My problem is that only wget downloads the file properly, firefox adds four extra bytes to the end. This is the header I send:
HTTP/1.0 200 Ok
Content-Length: 382917;
Content-Type: application/x-shockwave-flash;
Content-Disposition: attachment; filename=file.swf;
This is the code used to send the response:
QTextStream os(socket);
os.setAutoDetectUnicode(true);
QString name = tokens[1].right(tokens[1].length() - 1);
QString resname = ":/" + name; // the served file is a Qt resource
QFile f(resname); f.open(QIODevice::ReadOnly);
os << "HTTP/1.0 200 Ok\r\n" <<
"Content-Length: " << f.size() << ";\r\n" <<
"Content-Type: application/x-shockwave-flash;\r\n" <<
"Content-Disposition: attachment; filename=" << name <<
";\r\n\r\n";
os.flush();
QDataStream ds(socket);
ds << f.readAll();
socket->close();
if (socket->state() == QTcpSocket::UnconnectedState)
{
delete socket;
}
As I stated above, wget gets it right and downloads the file properly. The problem is that Firefox (and my target application, a Flash ActiveX instance) don't.
The four extra bytes are always the same: 4E E9 A5 F4
Hex dump http://www.freeimagehosting.net/uploads/a5711fd7af.gif
My question is what am I doing wrong, and what should I change to get it right? Thanks in advance.
You should not be terminating the lines with semicolons. At first glance this seems like the most likely problem.
I don't know much about QDataStream (or QT in general), however a quick look at the QDataStream documentation mentions operator<<(char const*). If you are passing a null terminated string to QDataStream, you are almost certainly going over the end of the final buffer.
Try using QDataStream::writeRawBytes().
If you remove the semicolons, then the clients should at least read the correct number of bytes for the response and ignore the last four bytes.
I'd leave out "Content-Disposition" too. That's a MIME thing, not an HTTP thing.
So I found the whole solution to the question, and I think someone might need it, so here it is:
The first problem were the four extra bytes. The reason for this is that according to the QDataStream documentation, "each item written to the stream is written in a predefined binary format that varies depending on the item's type". And as QFile.readAll() returned a QByteArray, QDataStream.operator<< wrote that object in the following format:
If the byte array is null: 0xFFFFFFFF (quint32)
Otherwise: the array size (quint32) followed by the array bytes, i.e. size bytes
(link)
So, the four extra bytes were the four bytes of quint32 that denoted the array size.
The solution, according to janm's answer was to use the writeRawBytes() function.
QDataStream ds(socket);
ds.writeRawData(f.readAll().data(), f.size());
Wget probably got it right the first time because it strictly enforces the Content-Length field of the HTTP header, while apparently firefox does not.
The second problem was that despite the right header and working sockets, the flashplayer did not display the desired content at all. I experimented with various fields to make it work, and noticed that by uploading to a real server, it works all right. I copied the header from server, and tadaa! it works. This is the header:
HTTP/1.1 200 OK
Server: Apache/2.2.15 (Fedora)
Accept-Ranges: bytes
Content-Length: 382917
Content-Type: application/x-shockwave-flash
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
At first I only tried setting the version to 1.1, but that didn't help. Probably it's the keepalive thing, but honestly, I don't care at all as long as it works :).
There shouldn't be any semicolons at the end of the line.
Related
I am working on a C++ project where i listen on sockets and generate HTTP responses based on the requests i get from my clients on my fds, in short i use my browser to send a request, i end up getting the raw request, i parse it and generate the corresponding http response.
However in the case of large POST requests, usually what happens is that i get partial requests, so in the first part i will usually only find the first line (version/method/uri), some headers but no body, and i guess am supposed to get the rest of the body somehow, however i am unable to figure out two things,
first of all how do i know if the request i am getting is partial or completed from just the first part ? i am not getting any information relating to range, here's the first part i get when my client sends me a POST request.
POST / HTTP/1.1
Host: localhost:8081
Connection: keep-alive
Content-Length: 8535833
Cache-Control: max-age=0
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Origin: http://127.0.0.1:8081
Upgrade-Insecure-Requests: 1
DNT: 1
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryOs6fsdbaegBIumqh
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Referer: http://127.0.0.1:8081/
Accept-Encoding: gzip, deflate, br
Accept-Language: fr,en-US;q=0.9,en;q=0.8
how can i figure out just from this whether or not am getting a partial request or just a faulty request (I need to generate a 400 error in the case of a request that says it has X content-length but the body size is different)
second question is, suppose i already know whether or not its partial, how do i proceed with storing the entire request in a buffer before sending it to my parser and generating a response ? here's my reception function (i already know the client's fd, so i just recv on it
void Client::receive_request(void)
{
char buffer[2024];
int ret;
ret = recv(_fd, buffer, 2024, 0);
buffer[ret] = 0;
_received_request += buffer;
_bytes_request += ret;
std::cout << "Raw Request:\n" << _received_request << std::endl;
if (buffer[ret-1] == '\n')
{
_ready_request = true;
_request.parse(_received_request, _server->get_config());
}
}
and here's the code that checks whether or not a client is attempting to send a request, parse and generate a response
int Connections::check_clients() {
int fd;
for (std::vector<Client*>::iterator client = clients.begin();
client != clients.end() && ready_fd != 0 ; client++)
{
fd = (*client)->get_fd();
if (FD_ISSET(fd, &ready_rset))
{
ready_fd--;
(*client)->receive_request();
if ((*client)->request_is_ready())
{
(*client)->wait_response();
close(fd);
FD_CLR(fd, &active_set);
fd_list.remove(fd);
max_fd = *std::max_element(fd_list.begin(), fd_list.end());
free(*client);
client = clients.erase(client);
}
}
}
return 0;
}
as you can see am coding everything in C++ (98) and would rather not get answers that just dismiss my questions and refer me to different technologies or libraries, unless it will help me understand what am doing wrong and how to handle partial requests.
for info, am only handling HTTP 1.1(GET/POST/DELETE only) and i usually only get this issue when am getting a large chunked file or a file upload that has a very large body. thank you
PS : if needed i can link up the github repo of the current project if you wanna look further into the code
how can i figure out just from this whether or not am getting a partial request or just a faulty request (I need to generate a 400 error in the case of a request that says it has X content-length but the body size is different)
The body size is, by definition, the size of the Content-Length field. Any bytes that you receive afterwards belong to the next HTTP request (see HTTP pipelining). If you do not receive Content-Length bytes within a reasonable time period, then you can make the server issue a 408 Request Timeout error.
second question is, suppose i already know whether or not its partial, how do i proceed with storing the entire request in a buffer before sending it to my parser and generating a response ? here's my reception function (i already know the client's fd, so i just recv on it
Your posted code has at least the following problems:
You should check the return value of recv to determine whether the function succeeded or failed, and if it failed, you should handle the error appropriately. In your current code, if recv fails with the return value -1, then you will write to the array buffer out of bounds, causing undefined behavior.
It does not seem appropriate to use the line if (buffer[ret-1] == '\n'). The HTTP request header will be over when you encounter a "\r\n\r\n", and the HTTP request body will be over when you have read Content-Length bytes of the body. The ends of the header and body will not necessarily occur at the end of the data read by recv, but can also occur in the middle. If you want to support HTTP pipelining, then the additional data should be handled by the handler for the next HTTP request. If you don't want to support HTTP pipelining, then you can simply discard the additional data and use Connection: close in the HTTP response header.
You seem to be using a null terminating character to mark the end of the data read by recv. However, this will not work if a byte with the value 0 is part of the HTTP request. It is probably safe to assume that such a byte should not be part of the HTTP request header, but it is probably not safe to assume that such a byte won't be part of the HTTP request body (for example when using POST with binary data).
I am trying to build a web server for videos and my client is a web-browser. I am using HTTP with header structure with content-Disposition: attachment
std::string make_header(std::string filename, int file_size)
{
std::ostringstream header;
header << "HTTP/1.1 200 OK\r\n";
header << "Content-Type: " << get_contenttype(filename) << "\r\n";
header << "Content-Disposition: attachment; filename =\"" << filename << "\"\r\n";
header << "Connection: close\r\n";
header << "Content-Length: " << file_size << "\r\n\r\n";
return header.str();
}
I am able to send small files but as soon as large files are given the server takes lot of time. I am using the following method to copy the video file to a local variable.
std::string content((std::istreambuf_iterator<char>(file)),
std::istreambuf_iterator<char>());
Is there any way to send file using HTTP without actually loading the entire file into memory.
The short answer is: Yes, of course. In the end, HTTP goes over TCP, which goes over IPv4/v6. IP is a sequence of small packets, and TCP is even a bytestream protocol. The other side won't notice that you are still reading bytes from disk when you're sending the first bytes.
In practice, that means the client can't see or won't care how many calls to send() you made. One bug call or a thousand small calls are equivalent.
I wouldn't bother with istreambuf_iterator. I'd just use fread, but I'd still use a std::vector<char> for the buffer. Just read 1 MB chunks and send those. Your OS isn't going to choke on either an 1MB disk read or a 1MB send call.
For a higher-end solution I'd use Boost::asio on Windows, or sendfile on Linux.
std::string HTTPrequest = "GET /index.html HTTP/1.1\r\nHost: www.yahoo.com\r\nConnection: close\r\n\r\n";
write(socket, HTTPrequest.c_str(), sizeof(HTTPrequest));
char pageReceived[4096];
int bytesReceived = read(socket, pageReceived, 4096);
I've got an HTTP client program that I run from my terminal. I've also got a webserver program. Using the webserver as a test, I can verify that the socket creation and attachment works correctly.
I create the request as shown above, then write to the socket. Using print statements, I can see that the code moves beyond the write call. However, it hangs on the read call.
I can't figure out what's going on - my formatting looks correct on the request.
Any ideas? Everything seems to work perfectly fine when I connect to my webserver, but both www.yahoo.com and www.google.com cause a hang. I'm on Linux.
In C and C++, sizeof gives you the number of bytes required to hold a type, regardless of its contents. So you are not sending the full request, only sizeof(std::string) bytes. You want HTTPRequest.size() (which gives you the number of bytes the value stored in HTTPRequest takes), not sizeof(HTTPrequest) (which gives you the number of bytes a std::string itself requires).
I have some code where in, there is a TCP connection between the client and the server. I need to send some xml data to the server and receive a response from it. I am trying to do it like this:
char request[MAX];
sprintf(request, "<attestationRequest><majorVersion>%d</majorVersion><minorVersion>%d</minorVersion></attestationRequest>", major, minor);
write(sockfd,request, length);
while(recv(sockfd, response, MAX, 0) >= 0)
{
cout << "response " << response;
//do something;
}
But the data received is (expected response + request).
If I try to fill the data using QString, the data received is proper.
QString request = QString("<attestationRequest>"
"<majorVersion>%1</majorVersion>"
"<minorVersion>%2</minorVersion>"
"</attestationRequest>")
.arg(major)
.arg(minor)
The rest of the code is same for both the cases. I receive more than 11000 bytes of data when I use sprintf, but 9000 bytes in case of QString. I am not able to understand if I am missing something.
IMO both solution are bad.
It is safer and better to use QXmlStreamWriter. Manual manipulation on text will in most cases end with incorrect encoding or missing escape sequence or other xml format violation.
Other problem is that you didn't gave any data how you perform communication. It is highly probable that there is an error causing your problem.
Most likely your problem is, that UTF-16 data is sent through the connection (that is QString.)
Maybe if you tried the w_char-versions of those commands, it might succeed.
Symptom
I think, I messed up something, because both Mozilla Firefox and Google Chrome produce the same error: they don't receive the whole response the webserver sends them. CURL never misses, the last line of the quick-scrolling response is always "</html>".
Reason
The reason is, that I send response in more part:
sendHeaders(); // is calls sendResponse with a fix header
sendResponse(html_opening_part);
for ( ...scan some data... ) {
sendResponse(the_data);
} // for
sendResponse(html_closing_part)
The browsers stop receiving data between sendResponse() calls. Also, the webserver does not close() the socket, just at the end.
(Why I'm doing this way: the program I write is designed for non-linux system, it will run on an embedded computer. It has not too much memory, which is mostly occupied by lwIP stack. So, avoid collecting the - relativelly - huge webpage, I send it in parts. Browsers like it, no broken HTML occurred as under Linux.)
Environment
The platform is GNU/Linux (Ubuntu 32-bit with 3.0 kernel). My small webserver sends the stuff back to the client standard way:
int sendResponse(char* data,int length) {
int x = send(fd,data,length,MSG_NOSIGNAL);
if (x == -1) {
perror("this message never printed, so there's no error \n");
if (errno == EPIPE) return 0;
if (errno == ECONNRESET) return 0;
... panic() ... (never happened) ...
} // if send()
} // sendResponse()
And here's the fixed header I am using:
sendResponse(
"HTTP/1.0 200 OK\n"
"Server: MyTinyWebServer\n"
"Content-Type: text/html; charset=UTF-8\n"
"Cache-Control: no-store, no-cache\n"
"Pragma: no-cache\n"
"Connection: close\n"
"\n"
);
Question
Is this normal? Do I have to send the whole response with a single send()? (Which I'm working on now, until a quick solution arrives.)
If you read RFC 2616, you'll see that you should be using CR+LF for the ends of lines.
Aside from that, open the browser developer tools to see the exact requests they are making. Use a tool like Netcat to duplicate the requests, then eliminate each header in turn until it starts working.
Gotcha!
As #Jim adviced, I've tried sending same headers with CURL, as Mozilla does: fail, broken pipe, etc. I've deleted half of headers: okay. I've added back one by one: fail. Deleted another half of headers: okay... So, there is error, only if header is too long. Bingo.
As I've said, there're very small amount of memory in the embedded device. So, I don't read the whole request header, only 256 bytes of them. I need only the GET params and "Host" header (even I don't need it really, just to perform redirects with the same "Host" instead of IP address).
So, if I don't recv() the whole request header, I can not send() back the whole response.
Thanks for your advices, dudes!