Forming a HTTP response with strings - c++

I am working on a HTTP server with boost, and I have some questions about forming the HTTP response, particularly the header.
Here's the code to assemble the GET response :
std::string h = statusCodes[200]; // The status code is already finished with a '\r\n'
std::string t = "Date: " + daytime_() + "\r\n";
std::string s = "Server: Muffin 1.0\r\n";
std::string content = search->second();
std::string type = "Content-Type: text/html\r\n";
std::string length = "Content-Length: " + std::to_string(content.size()) + "\r\n";
res = h + t + s + length + type + "\r\n" + content + "\r\n";
As they say on this website, here's the header spec :
The format of the request and response messages are similar, and
English-oriented. Both kinds of messages consist of:
an initial line, zero or more header lines,
a blank line (i.e. a CRLF by itself),
and an optional message body (e.g. a file, or query data, or query output).
But when I do a request on the server, only the date goes in the header, the rest is directly in the content
HTTP/1.1 200 OK // Header
Date: Tue May 24 10:28:58 2016 // Header
Server: Muffin 1.0 // Content
Content-Length: 31
Content-Type: text/html
This is supposed to be an ID
I don't know what's wrong in that, it's the first time I'm dealing with HTTP response.
Thanks for your help

I finally found the bug.
My daytime function was returning a string with a newline character.
This was the original function, which uses the depreciated ctime
std::string
Response::daytime_()
{
std::time_t now = std::time(0);
return std::ctime(&now);
}
And now the new function with strftime
std::string
Response::daytime_()
{
time_t rawtime;
struct tm * timeinfo;
char buffer[80];
time (&rawtime);
timeinfo = localtime(&rawtime);
strftime(buffer,80,"%a %b %d %H:%M:%S %Y",timeinfo);
std::string time(buffer);
return time;
}
And the new way to form the responses, using just a '\n'
std::string res = "";
std::string h = statusCodes[200];
std::string t = "Date: " + daytime_() + "\r\n";
std::string s = "Server: Muffin 1.0\r\n";
std::string content = search->second();
std::string type = "Content-Type: text/html\r\n";
std::string length = "Content-Length: " + std::to_string(content.size()) + "\r\n";
res = h + t + s + length + type + "\n" + content + "\r\n";

Related

Tried to parse chunked transfer encoding,it's not working though, the file which I decoded is totally unreadable

I tried to parse the data which was generated by chunked transfer encoding in a Rest API ,I did see the data has value when I tried to print the value in a string and I thought it should be working,but when I tried to assign the value to the file, the file is totally unreadable, the code below I used boost library and I gonna elaborate my thoughts in the code , we gonna get started from the response portion of my code, I have no idea what wrong I have done
// Send the request.
boost::asio::write(socket, request);
// Read the response status line. The response streambuf will automatically
// grow to accommodate the entire line. The growth may be limited by passing
// a maximum size to the streambuf constructor.
boost::asio::streambuf response;
boost::asio::read_until(socket, response, "\r\n");
// Check that response is OK.
std::istream response_stream(&response);
std::string http_version;
response_stream >> http_version;
unsigned int status_code;
response_stream >> status_code;
std::string status_message;
std::getline(response_stream, status_message);
if (!response_stream || http_version.substr(0, 5) != "HTTP/")
{
//std::cout << "Invalid response\n";
return 9002;
}
if (status_code != 200)
{
//std::cout << "Response returned with status code " << status_code << "\n";
return 9003;
}
// Read the response headers, which are terminated by a blank line.
boost::asio::read_until(socket, response, "\r\n\r\n");
// Process the response headers.
//this portion of code I tried to parse the file name in the header of response which the file name is in the content-disposition of header
std::string header;
std::string fullHeader = "";
string zipfilename="", txtfilename="";
bool foundfilename = false;
while (std::getline(response_stream, header) && header != "\r")
{
fullHeader.append(header).append("\n");
std::transform(header.begin(), header.end(), header.begin(),
[](unsigned char c){ return std::tolower(c); });
string containstr = "content-disposition";
string containstr2 = "filename";
string quotestr = "\"";
if (header.find(containstr) != std::string::npos && header.find(containstr2) != std::string::npos)
{
int countquotes = 0;
bool foundquote = true;
std::size_t startpos = 0, beginpos, endpos;
while (foundquote)
{
std::size_t myfound = header.find(quotestr, startpos);
if (myfound != std::string::npos)
{
if (countquotes % 2 == 0)
beginpos = myfound;
else
{
endpos = myfound;
foundfilename = true;
}
startpos = myfound + 1;
}
else
foundquote = false;
countquotes++;
}
if (endpos > beginpos && foundfilename)
{
size_t zipfileleng = endpos - beginpos;
zipfilename = header.substr(beginpos+1, zipfileleng-1);
txtfilename = header.substr(beginpos+1, zipfileleng-5);
}
else
return 9004;
}
}
if (foundfilename == false || zipfilename.length() == 0 || txtfilename.length() == 0)
return 9005;
//when the zipfilename has been found, we gonna get the data from the body of response, due to the response was chunked transfer encoding, I tried to parse it,it's not complicated due to I saw it on the Wikipedia, it just first line was length of data,the next line was data,and it's the loop which over and over again ,all I tried to do was spliting all the data from the body of response by "\r\n" into a vector<string>, and I gonna read the data from that vector
// Write whatever content we already have to output.
std::string fullResponse = "";
if (response.size() > 0)
{
std::stringstream ss;
ss << &response;
fullResponse = ss.str();
}
//tried split the entire body of response into a vector<string>
vector<string> allresponsedata;
split_regex(allresponsedata, fullResponse, boost::regex("(\r\n)+"));
//tried to merge the data of response
string zipfiledata;
int myindex = 0;
for (auto &x : allresponsedata) {
std::cout << "Split: " << x << std::endl;// I tried to print the data, I did see the value in the variable of x
if (myindex % 2 != 0)
{
zipfiledata = zipfiledata + x;//tried to accumulate the datas
}
myindex++;
}
//tried to write the data into a file
std::ofstream zipfilestream(zipfilename, ios::out | ios::binary);
zipfilestream.write(zipfiledata.c_str(), zipfiledata.length());
zipfilestream.close();
//afterward, the zipfile was built, but it's unreadable which it's not able to open,the zip utlities software says it's a damaged zip file though
I even tried something else ways like this slow http client based on boost::asio - (Chunked Transfer) ,but this way is not working as well ,VS says
1 IntelliSense: no instance of overloaded function "boost::asio::read" matches the argument list
argument types are: (boost::asio::ip::tcp::socket, boost::asio::streambuf, boost::asio::detail::transfer_exactly_t, std::error_code)
it just NOT able to compile in the line which is
size_t n = asio::read(socket, response, asio::transfer_exactly(chunk_bytes_to_read), error);
even I have read the example of asio::transfer_exactly, there's no exactly example like this though https://www.boost.org/doc/libs/1_57_0/doc/html/boost_asio/reference/transfer_exactly.html
any idea?
I don't see you read the format correctly: https://en.wikipedia.org/wiki/Chunked_transfer_encoding#Format
You need to read the chunk length (in hex) and any optional chunk extensions before accumulating the full response body.
It needs to be done before, because the sequence \r\n that you split on can easily appear inside the chunk data.
Again, I recommend to just use Beast's support, making it all a simple
http::response<http::string_body> response;
boost::asio::streambuf buf;
http::read(socket, buf, response);
And you will have the headers fully parsed, interpreted (including Trailer headers!) and the content in response.body() as a std::string.
It will do the right thing even if the server doesn't use chunked encoding or combines with different encoding options.
There's simply no reason to reinvent the wheel.
Full Demo
This demonstrates with the Chunked Encoding test url from https://jigsaw.w3.org/HTTP/:
#include <boost/process.hpp>
#include <boost/beast.hpp>
#include <iostream>
namespace http = boost::beast::http;
using boost::asio::ip::tcp;
int main() {
http::response<http::string_body> response;
boost::asio::io_context ctx;
tcp::socket socket(ctx);
connect(socket, tcp::resolver{ctx}.resolve("jigsaw.w3.org", "http"));
http::write(
socket,
http::request<http::empty_body>(
http::verb::get, "/HTTP/ChunkedScript", 11));
boost::asio::streambuf buf;
http::read(socket, buf, response);
std::cout << response.body() << "\n";
std::cout << "Effective headers are:" << response.base() << "\n";
}
Printing
This output will be chunked encoded by the server, if your client is HTTP/1.1
Below this line, is 1000 repeated lines of 0-9.
-------------------------------------------------------------------------
01234567890123456789012345678901234567890123456789012345678901234567890
01234567890123456789012345678901234567890123456789012345678901234567890
...996 lines removed ...
01234567890123456789012345678901234567890123456789012345678901234567890
01234567890123456789012345678901234567890123456789012345678901234567890
Effective headers are:HTTP/1.1 200 OK
cache-control: max-age=0
date: Wed, 31 Mar 2021 20:09:50 GMT
transfer-encoding: chunked
content-type: text/plain
etag: "1j3k6u8:tikt981g"
expires: Wed, 31 Mar 2021 20:09:49 GMT
last-modified: Mon, 18 Mar 2002 14:28:02 GMT
server: Jigsaw/2.3.0-beta3

Add strings to a file buffer, for message body of cURL smtp send email

I'm trying to modify a working cURL email send example to add a message body.
I'm unsure why all of the curl email-with-attachment examples I'm finding have no message body.
I'm needing to send short text emails with a PDF file attachment.
This is what I have tried so far, with the lines un-commented, it compiles and runs, but fails to send. I understand that the message body should be separated from the Subject by one "\r\n" (blank line), but this isn't the correct method.
//Create structure of email to be sent
fileBuf = new char[ADD_SIZE + no_of_rows][CHARS]; //ADD_SIZE for TO,FROM,SUBJECT,CONTENT-TYPE,CONTENT-TRANSFER-
//ENCODING,CONETNT-DISPOSITION and \r\n
strcpy(fileBuf[len++],"To: " TO "\r\n");
buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"From: " FROM "\r\n");
buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"Subject: SMTP TLS example message\r\n");
buffer_size += strlen(fileBuf[len-1]);
//strcpy(fileBuf[len++],"\r\n");
//buffer_size += strlen(fileBuf[len-1]);
//strcpy(fileBuf[len++],"Message goes here, hopefully...\r\n");
//buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"Content-Type: application/x-msdownload; name=\"" FILENAME "\"\r\n");
buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"Content-Transfer-Encoding: base64\r\n");
buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"Content-Disposition: attachment; filename=\"" FILENAME "\"\r\n");
buffer_size += strlen(fileBuf[len-1]);
strcpy(fileBuf[len++],"\r\n");
buffer_size += strlen(fileBuf[len-1]);
The full project code is here See Solution 7.
Any advice on how to accomplish this would be greatly appreciated.
[edit] Test using very simple cURL, produced 0 Byte attachment:
#define FILENAME "Rpt05162017.pdf"
static const char *payload_text[] = {
"To: " TO "\r\n",
"From: " FROM "(Example User)\r\n",
//"Cc: " CC "(Another example User)\r\n",
"Subject: SMTPS Example\r\n",
"Date: 17-May-2017\r\n",
"User-Agent: My eMail Client\r\n",
"MIME-Version: 1.0\r\n",
"Content-Type: multipart/mixed;\r\n",
" boundary=\"------------030203080101020302070708\"\r\n",
"\r\nThis is a multi-part message in MIME format.\r\n",
"--------------030203080101020302070708\r\n",
"Content-Type: text/plain; charset=utf-8; format=flowed\r\n",
"Content-Transfer-Encoding: 7bit\r\n",
"\r\n", // empty line to divide headers from body, see RFC5322
"The body of the message starts here.\r\n",
"\r\n",
"It could be a lot of lines, could be MIME encoded, whatever.\r\n",
"Check RFC5322.\r\n\r\n",
"--------------030203080101020302070708\r\n",
"Content-Type: application/x-msdownload; name=\"" FILENAME "\"\r\n",
"Content-Transfer-Encoding: base64\r\n",
"Content-Disposition: attachment; filename=\"" FILENAME "\"\r\n",
"\r\n--------------030203080101020302070708--",
NULL
};
It's not clear how you send all that stuff. You created 2d-array of strings and you strcpy into each your headers. All of them have padding bytes that will corrupt your message.
Try to simplify it maybe?
std:string header =
"To: " TO "\r\n"
"From: " FROM "\r\n"
"Subject: SMTP TLS example message\r\n"
"Content-Type: application/x-msdownload; name=\"" FILENAME "\"\r\n"
"Content-Transfer-Encoding: base64\r\n"
"Content-Disposition: attachment; filename=\"" FILENAME "\"\r\n"
"\r\n";
and then simply send your message header followed by the body of your message.
With your updated payload_text it's not a surprise that you get 0-size attachments because you are sending empty file. After you send your payload_text you have to send your body of the file and then after the body you need to append multipart closing suffix:
"\r\n--------------030203080101020302070708--"

Convert tiff image into String to be posted as binary application/octet-stream (C++)

I'm using C++ Sockets to make an HTTP Multipart POST containing a TIFF image to a server. This server expects a binary octet-stream.
What I've tried is this:
// Convert out data into a string that can be appended to the body
std::ifstream fin(fileName, std::ios::binary);
std::ostringstream ostrm;
ostrm << fin.rdbuf();
string data(ostrm.str());
Unfortunately, I just get II*, when the data should be much longer. My guess is that the data contains a NULL character, which makes C++ think the String is finished. Any ideas on how to resolve this issue?
If it helps, this is my body constructing code:
string constructBody(string batchId, string fileString, string fileName) {
string body;
string CRLF = "\r\n";
// first we add the args
body.append("--" + string(BOUNDARY) + CRLF);
body.append("Content-Disposition: form-data; name=\"batch_id\"" + CRLF);
body.append(CRLF);
body.append(batchId + CRLF);
body.append("--" + string(BOUNDARY) + CRLF);
// now we add the file
body.append("Content-Disposition: form-data; name=\"front\"; filename=\"" + string(fileName) + "\"" + CRLF);
body.append("Content-Type: application/octet-stream" + CRLF);
body.append("Content-Transfer-Encording: binary" + CRLF);
body.append(CRLF);
body.append(fileString + CRLF);
body.append("--" + string(BOUNDARY) + "--" + CRLF);
body.append(CRLF);
return body;
}
And here is the posting code:
string body = constructBody(batchId, data, "Front.tif");
char header[1024];
sprintf(header, "POST %s HTTP/1.1\r\n"
"Host: %s\r\n"
"Content-Length: %d\r\n"
"Connection: Keep-Alive\r\n"
"Content-Type: multipart/form-data; boundary=%s\r\n"
"Accept-Charset: utf-8\r\n\r\n", RECEIVER, IP, strlen(body.c_str()), BOUNDARY);
int p = send(dataSock, header, strlen(header), 0);
int k = send(dataSock, body.c_str(), strlen(body.c_str()), 0);
Thanks in advance!
You cannot use functions that uses the null as a string terminator:
int k = send(dataSock, body.c_str(), strlen(body.c_str()), 0);
You're using strlen above. This is not correct.
The fix for this is to use the length() function for std::string:
int k = send(dataSock, body.c_str(), body.length()), 0);
You make the same error in other places, such as when you create the header:
"Accept-Charset: utf-8\r\n\r\n", RECEIVER, IP, strlen(body.c_str()), BOUNDARY);
should be:
"Accept-Charset: utf-8\r\n\r\n", RECEIVER, IP, body.length(), BOUNDARY);

Poco::HttpClientSession.receiveResponse() throws NoMessageException without any apparent reason

I wrote an HTTP server in Java and a client in C++ with Poco. This is a part of the C++ client code:
URI uri("http://127.0.0.1:4444");
HTTPClientSession session(uri.getHost(), uri.getPort());
HTTPRequest req(HTTPRequest::HTTP_POST,
"/pages/page",
HTTPMessage::HTTP_1_1);
session.sendRequest(req);
HTTPResponse res;
std::istream &is = session.receiveResponse(res);
In the last line I get the following error:
terminate called after throwing an instance of 'Poco::Net::NoMessageException'
what(): No message received
But I don't understand why. The connection was established successfully and the page requested exists. I tried the same code with known websites (like Wikipedia) and it works without any exception.
I also tried to make the exact same request with cURL (to my server) in command-line and it shows the response of the server, so the server seems fine.
This is the original response of the server in a string form:
"HTTP/1.1 200 OK\r\n" +
"Server: [server name]\r\n" +
"Content-Type: text/xml; charset=utf-8\r\n" +
"Content-Length:" + bodyBytes.length + "\r\n" +
"Resource: " + job.resId + "\r\n\r\n" +
"<?xml version=\"1.0\" encoding=\"UTF-8\"?><JobRequest><InputRepresentation id=\"0\"/> <effectsList><cvtColor><code> CV_RGB2GRAY </code></cvtColor><resize><scaleFactorX> 0.5 </scaleFactorX><scaleFactorY> 0.5 </scaleFactorY><interpolation> INTER_LINEAR </interpolation></resize><GaussianBlur><kSize> 3 </kSize><sigmaX> 2 </sigmaX><sigmaY> 2 </sigmaY><borderType> BORDER_REPLICATE </borderType></GaussianBlur></effectsList></JobRequest>"
I have written a simple HTTP server which respond with a fixed response for every request, to test what's wrong. this is the code:
public class Test {
public static void main(String[] args) {
try {
ServerSocket serverSocket = new ServerSocket(4449);
Socket clientSocket = serverSocket.accept();
String body = "ab";
byte[] bodyBytes = body.getBytes("UTF-8");
String headers = "HTTP/1.1 200 OK\r\n" +
"Server: Foo\r\n" +
"Content-Type: text/plain\r\n" +
"Content-Length: " + bodyBytes.length + "\r\n\r\n";
byte[] headerBytes = headers.getBytes("UTF-8");
byte[] responseBytes = new byte[headerBytes.length + bodyBytes.length];
/* Fill responseBytes with the header and body bytes */
int i = 0;
for (int j = 0; j < headerBytes.length; ++j) {
responseBytes[i] = headerBytes[j];
++i;
}
for (int j = 0; j < bodyBytes.length; ++j) {
responseBytes[i] = bodyBytes[j];
++i;
}
clientSocket.getOutputStream().write(responseBytes);
} catch (IOException e) {}
}
}
I get the same exception even with this server. So what's wrong here?
Based on experimentation, I've found that it's necessary to include a Content-Length header if you are making an empty POST request. I.e.,
req.add("Content-Length", "0");
I had to use the following sequence in my handler code for a minimal response to be received without a No message received error:
resp.setStatus( Poco::Net::HTTPResponse::HTTP_OK );
resp.setContentType( "text/json" );
ostream &out = resp.send();
out << "[]";
out.flush();

Uploading PNG File (HTTP POST) with C++ Winsock API

I'm trying to upload a PNG file through Winsock2 HTTP Post. Here's my request string:
boundary = "-----rueldotme";
request += "POST " + uri + " HTTP/1.1\r\n";
request += "Host: " + hostname + "\r\n";
request += "User-Agent: " + UserAgent + "\r\n";
request += "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
request += "Accept-Language: en-us,en;q=0.5\r\n";
request += "Accept-Encoding: gzip,deflate\r\n";
request += "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n";
request += "Keep-Alive: 115\r\n";
request += "Connection: keep-alive\r\n";
request += "Content-Length: " + fileSize + "\r\n";
request += "Content-Type: multipart/form-data, boundary=" + boundary + "\r\n";
request += "\r\n";
request += "--" + boundary + "\r\n";
request += "Content-Disposition: form-data; name=\"filename\"; filename=\"" + fileName + "\"\r\n";
request += "Content-Type: image/png; charset=binary\r\n";
request += "Content-Transfer-Encoding: binary\r\n";
request += "\r\n";
request += "%s\r\n";
request += "\r\n";
The connection is OK, no errors and such, the fileCon by the way is from ReadFile(). And there's no error code. The number of bytes read is the same as the output of GetFileSize(). I tried displaying the contents of fileCon but only gave me this:
Don't mind the title "Error" (I set it).
Also, the request takes ages to complete, and gives me a blank response. Yep, blank with no HTTP headers. Am I doing the request right? Should I really have to include the file contents at the POST data?
Thanks in advance.
EDIT: The PNG size is about 256KB. I'm in a 1mbps connection.
EDIT: Sorry if the information was insufficient. Anyway, here's what I did lately:
int flz;
char bdata[BSIZE];
DWORD dwe, bytesRead = 0;
HANDLE fh = CreateFile(fileName.c_str(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
LPVOID fbuff = NULL;
flz = GetFileSize(fh, NULL);
fbuff = malloc(flz);
ReadFile(fh, fbuff, flz, &bytesRead, NULL));
...
sprintf_s(bdata, BSIZE, request.c_str(), reinterpret_cast<char *>(fbuff)); //BSIZE = 1024
...
send(sock, bdata, std::strlen(bdata), 0);
Not enough information to solve the problem, so I'll give a meta-answer instead:
Use a packet sniffer (e.g., wireshark) to check exactly what data is actually being sent and received. This will let you verify that the request is as it should be, and that the "blank response" you're getting really is blank.
One wild stab in the dark:
You haven't included any variable declarations in your code snippet, so I don't know what type "fileCon" is, but don't forget that the PNG data is likely to contain null bytes, which will mess up a default conversion from a char* to a std::string.
Edit:
Your modification contains the same bug that the std::string based version had, namely, that the PNG data is likely to contain null bytes. Perhaps this code will explain more clearly:
const char* data = "Hello\0world."; // some data that contains a null byte
std::string dataStr(data);
std::cout << dataStr << "\n"; // will print "Hello".
std::cout << dataStr.size() << "\n"; // will print "5"
char buf[512];
sprintf_s(buf, sizeof(buf), "Data: %s\n", data);
std::cout << buf; // will print "Data: Hello"
Both the conversion to std::string and formatting with sprintf will interpret the null byte as being the end of the data, and so the rest of the original data ("world.") will never be used.