now I started studying the sockets. But it's not working properly with my hands. I have cool method for logging in:
int LogIn(const string &name, const string &password)
{
char Buffer[1024];
string data = "login=" + name + "&password=" + password;
sprintf(Buffer, "POST /Test/login_cl.php HTTP/1.1\r
"\nContent-Length: %d\r"
"\nConnection: Keep-Alive\r"
"\nAccept-Encoding: gzip\r"
"\nAccept-Language: ru-RU,en,*\r"
"\nUser-Agent: Mozilla/5.0\r"
"\nHost: 127.0.0.1\r"
"\nContent-Type: application/x-www-form-urlencoded\r\n\r\n"
"login=%s&"
"password=%s&", data.length(), name.c_str(), password.c_str());
int BytesSent = send(MySocket, Buffer, sizeof(Buffer), 0);
string test;
char ans;
while (recv(MySocket, &ans, 1, 0))
test.push_back(ans);
cout << test << endl;
return 0;
}
And problem is that i can call this method only once. Other calls or calls for logout methods are not working, for example:
int result = LogIn(logn, paswd);
int result = LogIn(logn, paswd);
is not working, I receive only one answer (second is empty and recv is returning -1)
Please help, thanks.
int BytesSent = send(MySocket, Buffer, sizeof(Buffer), 0);
This sends too many bytes of data, leaving the other side with a whole bunch of junk at the end of the request that it thinks is part of the next request.
string data = "login=" + name + "&password=" + password;
This has the query with no & on the end.
"password=%s&", data.length(), name.c_str(), password.c_str());
This puts an & on the end of the query. Well, which is it?
You need to fix two things:
Be consistent about the & on the end of the query. (Why do you compose the query string twice anyway?)
Use strlen(Buffer) instead of sizeof(Buffer).
Really though, you should use an implementation of the actual protocol specification rather than code like this. For example, what happens if there's an & in the password? What happens if the server decides to use chunked encoding in the reply? You don't actually implement HTTP 1.1, yet you claim that you do. That will lead to lots and lots of pain, sooner or later.
You are sending too many bytes, you are not formatting the body of the POST message correctly, and you are not reading the response correctly at all.
If you are going to use C++ for some of your string handling then you should use C++ for all of your string handling, avoid mixing C string handling if you can.
Try something more like this instead:
const string SafeChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789*-._";
string Urlencode(const string &str)
{
if (str.length() == 0)
return string();
ostringstream buffer;
for (int I = 0; I < ASrc.length(); ++I)
{
char ch = ASrc[I];
if (ch == ' ')
buffer << '+';
else if (SafeChars.find(ch) != string::npos)
buffer << ch;
else
buffer << '%' << setw(2) << fillchar('0') << hex << ch;
}
return buffer.str();
}
int LogIn(const string &name, const string &password)
{
string data = "login=" + Urlencode(name) + "&password=" + Urlencode(password);
ostringstream buffer;
buffer << "POST /Test/login_cl.php HTTP/1.1\r\n"
<< "Content-Length: " << data.length() << "\r\n"
<< "Connection: Keep-Alive\r\n"
<< "Accept-Encoding: gzip\r\n"
<< "Accept-Language: ru-RU,en,*\r\n"
<< "User-Agent: Mozilla/5.0\r\n"
<< "Host: 127.0.0.1\r\n"
<< "Content-Type: application/x-www-form-urlencoded\r\n"
<< "\r\n"
<< data;
string request = buffer.str();
const char *req = request.c_str();
int reqlen = request.length();
do
{
int BytesSent = send(MySocket, request.c_str(), request.length(), 0);
if (BytesSent <= 0)
return -1;
req += BytesSent;
reqlen -= BytesSent;
}
while (reqlen > 0);
// you REALLY need to flesh out this reading logic!
// See RFC 2616 Section 4.4 for details
string response;
char ch;
while (recv(MySocket, &ch, 1, 0) > 0)
response += ch;
cout << response << endl;
return 0;
}
I will leave it as an exercise for you to learn the correct way to read an HTTP response (HINT: it is a LOT harder then you think - especially since you are including Accept-Encoding: gzip and Connection: Keep-Alive headers, which have big impacts on response handling. Read RFC 2616 Section 4.4 for details how to determine the response length and format).
With that said, HTTP is not a trivial protocol to implement by hand, so you really should use a premade HTTP library, such as libcurl, or use Microsoft's own WinInet or WinHTTP APIs.
Related
I'm writing Http-Client which takes URL on somefile, download it and save it on a disk. Like curl does.
I can use only C/C++ with std:: and libc. I have no problems with downloading text files like XML, CSV or txt, because they were saved like it should be and if to open them in editor - it's ok, there's that text which was expected. But when i download tar or pdf and trying to open them, it tells that files are corrupted.
Here's 2 main methods of my class HttpClient. HttpClient::get - send Http-request to the host, which is mentioned in URL, and calls the 2nd main method - HttpClient::receive which defines what kind of data there is - binary or text, and write whole Http-request body in a file using binary or text mode.
All other methods i decided not to show, but i can if someone needs.
HttpClient::get:
bool HttpClient::get() {
std::string protocol = getProtocol();
if (protocol != "http://") {
std::cerr << "Don't support no HTTP protocol" << std::endl;
return false;
}
std::string host_name = getHost();
std::string request = "GET ";
request += url + " HTTP/" + HTTP_VERSION + "\r\n";
request += "Host: " + host_name + "\r\n";
request += "Accept-Encoding: gzip\r\n";
request += "Connection: close\r\n";
request += "\r\n";
sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock < 0) {
std::cerr << "Can't create socket" << std::endl;
return false;
}
addr.sin_family = AF_INET;
addr.sin_port = htons(HTTP_PORT);
raw_host = gethostbyname(host_name.c_str());
if (raw_host == NULL) {
std::cerr << "No such host: " << host_name << std::endl;
return false;
}
if (!this->connect()) {
std::cerr << "Can't connect" << std::endl;
return false;
} else {
std::cout << "Connection established" << std::endl;
}
if (!sendAll(request)) {
std::cerr << "Error while sending HTTP request" << std::endl;
return false;
}
if (!receive()) {
std::cerr << "Error while receiving HTTP response" << std::endl;
return false;
}
close(sock);
return true;
}
HttpClient::receive:
bool HttpClient::receive() {
char buf[BUF_SIZE];
std::string response = "";
std::ofstream file;
FILE *fd = NULL;
while (1) {
size_t bytes_read = recv(sock, buf, BUF_SIZE - 1, 0);
if (bytes_read < 0)
return false;
buf[bytes_read] = '\0';
if (!file.is_open())
std::cout << buf;
if (!file.is_open()) {
response += buf;
std::string content = getHeader(response, "Content-Type");
if (!content.empty()) {
std::cout << "Content-Type: " << content << std::endl;
if (content.find("text/") == std::string::npos) {
std::cout << "Binary mode" << std::endl;
file.open(filename, std::ios::binary);
}
else {
std::cout << "Text mode" << std::endl;
file.open(filename);
}
std::string::size_type start_file = response.find("\r\n\r\n");
file << response.substr(start_file + 4);
}
}
else
file << buf;
if (bytes_read == 0) {
file.close();
break;
}
}
return true;
}
I can't find help, but i think that binary data is encoded in some way, but how to decode it?
I can't find help, but i think that binary data is encoded in some way, but how to decode it?
You don't explain why you think this way but the following line from your request might cause some encoding you don't handle:
request += "Accept-Encoding: gzip\r\n";
Here you explicitly say that you are willing to accept content encoded (compressed) with gzip. But looking at your code you are not even checking if the content es declared as encoded with gzip by analyzing the Content-Encoding header.
Apart from this the following line might cause a problem too:
request += url + " HTTP/" + HTTP_VERSION + "\r\n";
You don't show what HTTP_VERSION is but assuming that it is 1.1 you also have to deal with Transfer-Encoding: chunked too.
Thanks everyone.
I solved this problem by changing response += buf; to response.append(buf, bytes_read); and file << buf; to file.write(buf, bytes_read);.
It was stupid to write binary data like null-terminating string.
Salutations fellow programmers,
I am trying to write a program that allows you input what you want and the program will send your input to the server.
At the moment, my goal is sending HTTP requests to a web page. It connects fine. But when the while loop runs in immediately sends something through the cin.getline procedure without me inputting anything. I thought this was weird but it seemed to be work anyway.
Every time I send something like: "GET / HTTP/1.1\r\n\r\n" it will return the correct thing, but anything else I input, like "OPTIONS" returns the source code + "application blocked" (I am at school so it makes sense).
So, I connected to hotspot shield VPN and tested the application, but to my horror when I input something to send it returns nothing.
I searched through stack overflow and google but I haven't been able to find anything so far; probably because I'm searching for the wrong solutions to the problem.
Anyway, if you have time, please scan through the code send some help. It could just be a VPN and school issue and I could try at home if the code seems to be working for you, so just let me know.
SPECIFIC OUTLINE OF PROBLEM:
When I use this outside the school network nothing is returned and the while loop doesn't seem to execute. I can connect but the program seems to be in an endless time-out or something.
cout << "Connected to " << hostName << endl;
while (true) {
cout << ">";
cin.getline(sendBuf, sizeof(sendBuf));
string s(sendBuf);
cout << s.c_str() << endl;
send(connectSocket, s.c_str(), sizeof(s.c_str()), 0);
int rec = recv(connectSocket, recvBuf, sizeof(recvBuf), 0);
if (rec > 0) {
cout << recvBuf << endl;
}
else if (rec <= 0) {
cout << "nothing" << endl;
}
}
system("pause");
}
system("pause");
}
my goal is sending HTTP requests to a web page
The code you showed does not attempt to implement any semblance of the HTTP protocol, not even close.
For one thing, if you look at your own example more carefully, you will see that the GET request (which BTW, is missing a required Host header, due to your use of HTTP 1.1) contains 2 line breaks, but cin.getline() (why not std::getline()?) reads only 1 line at a time. So, you read in one line, send it, and wait for a response that doesn't arrive since you didn't finish sending a complete request yet. That would explain why your while loop is hanging.
If you want the user to type in a complete HTTP request and then you send it as-is, you have to read in the ENTIRE request from the user, and then send it entirely to the server, before you can then attempt to receive the server's response. That means you have to handle line breaks between individual message headers, handle the terminating line break that separates the message headers from the message body, and detect the end of the body data.
I would suggest not relying on the user typing in a complete HTTP request as-is. I suggest you prompt the user for relevant pieces and let the user type normal text, and then your code can format that text into a proper HTTP request as needed.
When you are reading the server's response, you can't just blindly read arbitrary chunks of data. You have to process what you read, per the rules of the HTTP protocol. This is particularly important in order to determine when you have reached the end of the response and need to stop reading. The end of the response can be signaled in one of many different ways, as outlined in RFC 2616 Section 4.4 Message Length.
You are also making some common newbie mistakes in your TCP handling in general. TCP is a streaming transport, you are not taking into account that send() and recv() can sent/receive fewer bytes than requested. Or that recv() does not return null-terminated data.
With that said, try something like this:
void sendAll(SOCKET sckt, const void *buf, int buflen)
{
// send all bytes until buflen has been sent,
// or an error occurs...
const char *pbuf = static_cast<const char*>(buf);
while (buflen > 0)
{
int numSent = send(sckt, pbuf, buflen, 0);
if (numSent < 0) {
std::ostringstream errMsg;
errMsg << "Error sending to socket: " << WSAGetLastError();
throw std::runtime_error(errMsg.str());
}
pbuf += numSent;
buflen -= numSent;
}
}
int readSome(SOCKET sckt, void *buf, int buflen)
{
// read as many bytes as possible until buflen has been received,
// the socket is disconnected, or an error occurs...
char *pbuf = static_cast<char*>(buf);
int total = 0;
while (buflen > 0)
{
int numRecvd = recv(sckt, pbuf, buflen, 0);
if (numRecvd < 0) {
std::ostringstream errMsg;
errMsg << "Error receiving from socket: " << WSAGetLastError();
throw std::runtime_error(errMsg.str());
}
if (numRecvd == 0) break;
pbuf += numRecvd;
buflen -= numRecvd;
total += numRecvd;
}
return total;
}
void readAll(SOCKET sckt, void *buf, int buflen)
{
// read all bytes until buflen has been received,
// or an error occurs...
if (readSome(sckt, buf, buflen) != buflen)
throw std::runtime_error("Socket disconnected unexpectedly");
}
std::string readLine(SOCKET sckt)
{
// read a line of characters until a line break is received...
std::string line;
char c;
do
{
readAll(sckt, &c, 1);
if (c == '\r')
{
readAll(sckt, &c, 1);
if (c == '\n') break;
line.push_back('\r');
}
else if (c == '\n') {
break;
}
line.push_back(c);
}
while (true);
return line;
}
...
inline void ltrim(std::string &s) {
// erase whitespace on the left side...
s.erase(s.begin(), std::find_if(s.begin(), s.end(), [](int ch) {
return !std::isspace(ch);
}));
}
inline void rtrim(std::string &s) {
// erase whitespace on the right side...
s.erase(std::find_if(s.rbegin(), s.rend(), [](int ch) {
return !std::isspace(ch);
}).base(), s.end());
}
inline void trim(std::string &s) {
// erase whitespace on both sides...
ltrim(s);
rtrim(s);
}
inline void upperCase(std::string &s)
{
// translate all characters to upper-case...
std::transform(s.begin(), s.end(), s.begin(), ::toupper);
}
...
std::string makeRequest(const std::string &host, const std::string &method, const std::string &resource, const std::vector<std::string> &extraHeaders, const void *body, int bodyLength)
{
std::ostringstream oss;
oss << method << " " << resource << " HTTP/1.1\r\n";
oss << "Host: " << host << "\r\n";
oss << "Content-Length: " << bodyLength << "\r\n";
for(auto &hdr : extraHeaders)
{
// TODO: ignore Host and Content-Length...
oss << hdr << "\r\n";
}
oss << "\r\n";
oss.write(static_cast<const char*>(body), bodyLength);
return oss.str();
}
bool getHeaderValue(const std::vector<std::string> &headers, const std::string &headerName, std::string &value)
{
value.clear();
std::string toFind = headerName;
upperCase(toFind);
// find the requested header by name...
for(auto &s : headers)
{
std::string::size_type pos = s.find(':');
if (pos != std::string::npos)
{
std::string name = s.substr(0, pos-1);
trim(name);
upperCase(name);
if (name == toFind)
{
// now return its value...
value = s.substr(pos+1);
trim(value);
return true;
}
}
}
// name not found
return false;
}
...
std::cout << "Connected to " << hostName << std::endl;
try
{
std::string method, resource, hdr, data;
std::string status, version, reason;
std::vector<std::string> headers;
int statusCode, rec;
do
{
headers.clear();
data.clear();
// get user input
std::cout << "Method > " << std::flush;
if (!std::getline(std::cin, method))
throw std::runtime_error("Error reading from stdin");
upperCase(method);
std::cout << "Resource > " << std::flush;
if (!std::getline(std::cin, resource))
throw std::runtime_error("Error reading from stdin");
std::cout << "Extra Headers > " << std::flush;
while (std::getline(std::cin, hdr) && !hdr.empty())
headers.push_back(hdr);
if (!std::cin)
throw std::runtime_error("Error reading from stdin");
std::cout << "Data > " << std::flush;
// use Ctrl-Z or Ctrl-D to end the data, depending on platform...
std::ios_base::fmtflags flags = std::cin.flags();
std::cin >> std::noskipws;
std::copy(std::istream_iterator<char>(std::cin), std::istream_iterator<char>(), std::back_inserter(data));
if (!std::cin)
throw std::runtime_error("Error reading from stdin");
std::cin.flags(flags);
std::cin.clear();
// send request
std::string request = makeRequest(hostName, method, resource, headers, data.c_str(), data.length());
std::cout << "Sending request: << std::endl << request << std::endl;
// TODO: reconnect to hostName if previous request disconnected...
sendAll(connectSocket, request.c_str(), request.length());
// receive response
headers.clear();
data.clear();
// read the status line and parse it...
status = readLine(connectSocket);
std::cout << status << std::endl;
std::getline(std::istringstream(status) >> version >> statusCode, reason);
upperCase(version);
// read the headers...
do
{
hdr = readLine(connectSocket);
std::cout << hdr << std::endl;
if (hdr.empty()) break;
headers.push_back(hdr);
}
while (true);
// The transfer-length of a message is the length of the message-body as
// it appears in the message; that is, after any transfer-codings have
// been applied. When a message-body is included with a message, the
// transfer-length of that body is determined by one of the following
// (in order of precedence):
// 1. Any response message which "MUST NOT" include a message-body (such
// as the 1xx, 204, and 304 responses and any response to a HEAD
// request) is always terminated by the first empty line after the
// header fields, regardless of the entity-header fields present in
// the message.
if (((statusCode / 100) != 1) &&
(statusCode != 204) &&
(statusCode != 304) &&
(method != "HEAD"))
{
// 2. If a Transfer-Encoding header field (section 14.41) is present and
// has any value other than "identity", then the transfer-length is
// defined by use of the "chunked" transfer-coding (section 3.6),
// unless the message is terminated by closing the connection.
if (getHeaderValue(headers, "Transfer-Encoding", hdr))
upperCase(hdr);
if (!hdr.empty() && (hdr != "IDENTITY"))
{
std::string chunk;
std::string::size_type oldSize, size;
do
{
chunk = readLine(connectSocket);
std::istringstream(chunk) >> std::hex >> size;
if (size == 0) break;
oldSize = data.size();
chunkData.resize(oldSize + size);
readAll(connectSocket, &data[oldSize], size);
std::cout.write(&data[oldSize], size);
readLine(connectSocket);
}
while (true);
std::cout << std::endl;
do
{
hdr = readLine(connectSocket);
std::cout << hdr << std::endl;
if (hdr.empty()) break;
headers.push_back(hdr);
}
while (true);
}
// 3. If a Content-Length header field (section 14.13) is present, its
// decimal value in OCTETs represents both the entity-length and the
// transfer-length. The Content-Length header field MUST NOT be sent
// if these two lengths are different (i.e., if a Transfer-Encoding
// header field is present). If a message is received with both a
// Transfer-Encoding header field and a Content-Length header field,
// the latter MUST be ignored.
else if (getHeaderValue(headers, "Content-Length", hdr))
{
std::string::size_type size;
if ((std::istringstream(hdr) >> size) && (size > 0))
{
data.resize(size);
readAll(connectSock, &data[0], size);
std::cout << data;
}
}
// 4. If the message uses the media type "multipart/byteranges", and the
// transfer-length is not otherwise specified, then this self-
// delimiting media type defines the transfer-length. This media type
// MUST NOT be used unless the sender knows that the recipient can parse
// it; the presence in a request of a Range header with multiple byte-
// range specifiers from a 1.1 client implies that the client can parse
// multipart/byteranges responses.
else if (getHeaderValue(headers, "Content-Type", hdr) &&
(hdr.compare(0, 10, "multipart/") == 0))
{
// TODO: extract 'boundary' attribute and read from
// socket until the terminating boundary is reached...
}
// 5. By the server closing the connection.
else
{
do
{
rec = readSome(connectSocket, recvBuf, sizeof(recvBuf));
if (rec == 0) break;
data.append(recvBuf, rec);
std::cout.write(recvBuf, rec);
}
while (rec == sizeof(recvBuf));
}
}
std::cout << std::endl;
// use status, headers, and data as needed ...
getHeaderValue(headers, "Connection", hdr);
upperCase(hdr);
if (version == "HTTP/1.0")
{
if (hdr != "KEEP-ALIVE")
break;
}
else
{
if (hdr == "CLOSE")
break;
}
}
while (true);
}
catch (const std::exception &e)
{
std::cerr << e.what() << std::endl;
}
closesocket(connectSocket);
std::cout << "Disconnected from " << hostName << std::endl;
std::system("pause");
Isn't HTTP fun? :-) This is, by far, not a complete HTTP implementation, but it should get you started. However, as you can see, HTTP can be quite complex to implement from scratch, and it has many rules and restrictions that you have to follow. You are better off not implementing HTTP manually at all. There are plenty of 3rd party HTTP libraries that are available for C++. Use one of them instead, and let them handle the hard work for you, so you can focus on your own business logic.
I am currently trying to transfer some JSON data over the network from a client to a server using the socket API of boost-asio. My client essentially does this:
int from = 1, to = 2;
boost::asio::streambuf buf;
ostream str(&buf);
str << "{"
<< "\"purpose\" : \"request\"" << "," << endl
<< "\"from\" : " << from << "," << endl
<< "\"to\" : " << to << "," << endl
<< "}" << endl;
// Start an asynchronous operation to send the message.
boost::asio::async_write(socket_, buf,
boost::bind(&client::handle_write, this, _1));
On the server side I have the choice between various boost::asio::async_read* functions.
I wanted to use JsonCpp to parse the received data. Studying the JsonCpp API (http://jsoncpp.sourceforge.net/class_json_1_1_reader.html) I found that the Reader operates on top of either a std::string, a char* array or a std::istream which I could operate from the boost::asio::streambuf passed to the functions.
The point is that as far as I know it is not necessarily the case that the entire content is transferred at once, so I would need some kind of confirmation that the buffer contains sufficient data to process the entire document using JsonCpp. How can I assure that the buffer contains enough data?
This is an area for application level protocol
Either
read until the stream end (the sender disconnects); this doesn't work with connections that are kept alive for more than a single message
supply a header like Content-Length: 12346\r\n to know in advance how much to read
supply a delimiter (a bit like mime boundaries, but you could use any sequence that is not allowed/supported as part of the JSON payload) (async_read_until)
Treat the payload as "binary-style" (BSON e.g.) and supply a (network-order) length field before the text transmission.
The ASIO Http server example contains a pretty nice pattern for parsing HTTP request/headers that you could use. This assumes that your parser can detect completeness and just 'soft-fails' until all information is present.
void connection::handle_read(const boost::system::error_code& e,
std::size_t bytes_transferred)
{
if (!e)
{
boost::tribool result;
boost::tie(result, boost::tuples::ignore) = request_parser_.parse(
request_, buffer_.data(), buffer_.data() + bytes_transferred);
if (result)
{
request_handler_.handle_request(request_, reply_);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else if (!result)
{
reply_ = reply::stock_reply(reply::bad_request);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else
{
socket_.async_read_some(boost::asio::buffer(buffer_),
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
else if (e != boost::asio::error::operation_aborted)
{
connection_manager_.stop(shared_from_this());
}
}
I've provided an answer that parses JSON using Boost Spirit earlier Parse a substring as JSON using QJsonDocument; you could use this to detect the end of a proper JSON document (and if it's incomplete, the end will coincide with the start)
2 problems here : 1) tell the server how many bytes to read; 2) read the JSON
for 1) you can make your own simple protocol
300#my message here
sends a 300 byte sized message; # is the delimiter between size and message
int write_request(socket_t &socket, const char* buf_json)
{
std::string buf;
size_t size_json = strlen(buf_json);
buf = std::to_string(static_cast<long long unsigned int>(size_json));
buf += "#";
buf += std::string(buf_json);
return (socket.write_all(buf.data(), buf.size()));
}
to read on the server
//parse header, one character at a time and look for for separator #
//assume size header lenght less than 20 digits
for (size_t idx = 0; idx < 20; idx++)
{
char c;
if ((recv_size = ::recv(socket.m_sockfd, &c, 1, 0)) == -1)
{
std::cout << "recv error: " << strerror(errno) << std::endl;
return str;
}
if (c == '#')
{
break;
}
else
{
str_header += c;
}
}
to read JSON, you can use
https://github.com/nlohmann/json
I am currently trying to transfer some JSON data over the network from a client to a server using the socket API of boost-asio. My client essentially does this:
int from = 1, to = 2;
boost::asio::streambuf buf;
ostream str(&buf);
str << "{"
<< "\"purpose\" : \"request\"" << "," << endl
<< "\"from\" : " << from << "," << endl
<< "\"to\" : " << to << "," << endl
<< "}" << endl;
// Start an asynchronous operation to send the message.
boost::asio::async_write(socket_, buf,
boost::bind(&client::handle_write, this, _1));
On the server side I have the choice between various boost::asio::async_read* functions.
I wanted to use JsonCpp to parse the received data. Studying the JsonCpp API (http://jsoncpp.sourceforge.net/class_json_1_1_reader.html) I found that the Reader operates on top of either a std::string, a char* array or a std::istream which I could operate from the boost::asio::streambuf passed to the functions.
The point is that as far as I know it is not necessarily the case that the entire content is transferred at once, so I would need some kind of confirmation that the buffer contains sufficient data to process the entire document using JsonCpp. How can I assure that the buffer contains enough data?
This is an area for application level protocol
Either
read until the stream end (the sender disconnects); this doesn't work with connections that are kept alive for more than a single message
supply a header like Content-Length: 12346\r\n to know in advance how much to read
supply a delimiter (a bit like mime boundaries, but you could use any sequence that is not allowed/supported as part of the JSON payload) (async_read_until)
Treat the payload as "binary-style" (BSON e.g.) and supply a (network-order) length field before the text transmission.
The ASIO Http server example contains a pretty nice pattern for parsing HTTP request/headers that you could use. This assumes that your parser can detect completeness and just 'soft-fails' until all information is present.
void connection::handle_read(const boost::system::error_code& e,
std::size_t bytes_transferred)
{
if (!e)
{
boost::tribool result;
boost::tie(result, boost::tuples::ignore) = request_parser_.parse(
request_, buffer_.data(), buffer_.data() + bytes_transferred);
if (result)
{
request_handler_.handle_request(request_, reply_);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else if (!result)
{
reply_ = reply::stock_reply(reply::bad_request);
boost::asio::async_write(socket_, reply_.to_buffers(),
boost::bind(&connection::handle_write, shared_from_this(),
boost::asio::placeholders::error));
}
else
{
socket_.async_read_some(boost::asio::buffer(buffer_),
boost::bind(&connection::handle_read, shared_from_this(),
boost::asio::placeholders::error,
boost::asio::placeholders::bytes_transferred));
}
}
else if (e != boost::asio::error::operation_aborted)
{
connection_manager_.stop(shared_from_this());
}
}
I've provided an answer that parses JSON using Boost Spirit earlier Parse a substring as JSON using QJsonDocument; you could use this to detect the end of a proper JSON document (and if it's incomplete, the end will coincide with the start)
2 problems here : 1) tell the server how many bytes to read; 2) read the JSON
for 1) you can make your own simple protocol
300#my message here
sends a 300 byte sized message; # is the delimiter between size and message
int write_request(socket_t &socket, const char* buf_json)
{
std::string buf;
size_t size_json = strlen(buf_json);
buf = std::to_string(static_cast<long long unsigned int>(size_json));
buf += "#";
buf += std::string(buf_json);
return (socket.write_all(buf.data(), buf.size()));
}
to read on the server
//parse header, one character at a time and look for for separator #
//assume size header lenght less than 20 digits
for (size_t idx = 0; idx < 20; idx++)
{
char c;
if ((recv_size = ::recv(socket.m_sockfd, &c, 1, 0)) == -1)
{
std::cout << "recv error: " << strerror(errno) << std::endl;
return str;
}
if (c == '#')
{
break;
}
else
{
str_header += c;
}
}
to read JSON, you can use
https://github.com/nlohmann/json
So i am trying to make a downloader based on http protocol. It works ok on small txt files, but if i try downloading something bigger, lets say a picture, it doesnt get all of the data.
void accel::getfile(int from, int to){
//send GET message
string msg = msg_get(fpath, true, from, to);
int back = send(socketC, (char*)&msg[0], msg.size(), 0);
//recv response
int buffsize = 512; //buffer size. should not be less than 512
string buff(buffsize, 0);//, resp; //buffer, response message
fstream output(fname, ios::out, ios::trunc); //file
bool found = false;
int dld = 0;
do {
//recv msg to buffer
back = recv(socketC, (char*)&buff[0], buffsize, 0);
dld += back;
//buff.insert(back, "\0");
//cout << buff.c_str();
//is the writing of the body started?
if (found){
output.write(buff.c_str(), back);
cout << ".";
}
//if not find where the body starts
else {
int point = buff.find("\r\n\r\n");
if (point != -1){
char *p = (char*)&buff[point+4];
output.write(p, back-point-4);
cout << ".";
found = true;
}
}
} while (back == buffsize);
output.close();
cout << "\nComplete\n";
cout << dld << "bytes downloaded\n";
}
Requests:
string msg_head(string fpath, bool keep_alive){
string msg;
//
msg = "HEAD ";
msg += fpath;
msg += " HTTP/1.0\r\n";
if (keep_alive)
msg += "Connection: keep-alive\r\n\r\n";
else
msg += "Connection: close\r\n\r\n";
return msg;
}
string msg_get(string fpath, bool keep_alive, int from, int to){
string msg;
char number[10];
msg = "GET ";
msg += fpath;
msg += " HTTP/1.0\r\n";
msg += "Range: bytes=";
sprintf(number, "%d", from);
msg += number;
msg += "-";
sprintf(number, "%d", to);
msg += number;
msg += "\r\n";
if (keep_alive)
msg += "Connection: keep-alive\r\n\r\n";
else
msg += "Connection: close\r\n\r\n";
return msg;
}
recv may not fill the whole buffer if there is not enough data available yet. You should detect the end of the message by one of the means specified in the HTTP protocol.
Your code appears to assume that recv always returns the maximum amount of data (ie. that it waits for n bytes to appear). recv may return early if a smaller amount of data is currently available.
Instead, you should continue until either disconnection (when recv returns 0, or an error such as WSAECONNRESET) or an expected number of bytes (which you should have once the header is fully received)