I am sending udp packages from server to client. At the server side I split data into packages by 500 bytes, and sent to client. The client receives the packages and accumulate received data and deserializes an object.
The problem is that client receive 133 packages maximum and stops like nothing else was sent to socket, but server send whole object (1238 packages). And this problem exists in Windows only, but works perfectly under OSX.
Here is a server code sending packages:
// sends #buffer of size #length to #endpoint
// #buffer already contains a header, and the method splits #buffer into chunks and send it one by one
void server::send_package(char* buffer, int length, udp::endpoint endpoint){
if (length > BUFFER){
protocol::header header;
int dataLength = length - sizeof (header);
// copy header from buffer
memcpy(&header, buffer, sizeof(header));
header.isEnd = false;
int position = 0;
// allocate memory to collect data to send
char* data_to_send = new char[dataLength];
// copy data
memcpy(data_to_send, &buffer[sizeof(header)], dataLength);
header.totalPackages = dataLength/(BUFFER-sizeof (header));
// create chucks of data and send
while (position < dataLength){
int frame_size = BUFFER;
header.currentPackage++;
if (dataLength-position+sizeof (header) <= BUFFER) {
header.isEnd = true;
frame_size = dataLength-position+sizeof (header);
}
char* temp_buffer = new char[frame_size];
header.length = frame_size-sizeof(header);
// set the header of a chunk
memcpy(temp_buffer, &header, sizeof(header));
// set data to chunk
memcpy(&temp_buffer[sizeof (header)], &data_to_send[position], frame_size-sizeof(header));
// send chunk
socket->send_to(boost::asio::buffer(temp_buffer, frame_size), endpoint);
socket->wait(boost::asio::ip::tcp::socket::wait_write);
position += frame_size-sizeof(header);
}
} else {
socket->async_send_to(boost::asio::buffer(buffer, length), endpoint,
boost::bind(&server::release_sent_buffer,
this,
buffer, length)
);
}
}
Here is the client receives packages:
void connectionManager::handle_receive( const boost::system::error_code &error,
std::size_t size,
udp::endpoint* ep) {
if (size > 0) {
// _lock.try_lock();
protocol::header header;
memcpy(&header, &recv_buffer, sizeof(header));
logg("response from server received " + boost::asio::ip::address_v4(header.ip).to_string());
logg("received header:");
logg(protocol::getHeaderInfo(header));
std::stringstream ss;
ss << "header.length = " << header.length;
logg(ss.str().c_str());
udp::endpoint endpoint(boost::asio::ip::address_v4(header.ip), _server_port);
switch (header.command) {
case protocol::commands::server_instance_instruments_state_response: {
package_chain chain(size-sizeof(header));
memcpy(chain.data, &recv_buffer[sizeof(header)], size-sizeof(header));
packages[header.id].push_back(chain);
// at Windows machine the last package is #133. But 1248 packages expected.
// WHY????...
int packs = (packages[header.id].size());
if (header.isEnd) {
char* buf = getDataFromPackages(header.id, header.length);
std::stringstream str;
str << buf;
boost::archive::text_iarchive ar(str);
instance_plugin_information* inst_inf;
inst_inf = new instance_plugin_information();
try {
ar & inst_inf;
if (onPluginStateResponse != nullptr) {
onPluginStateResponse(*inst_inf);
}
} catch (const std::exception& e) {
}
}
break;
}
}
// We will hang on this line when package #133 received.
socket->wait(boost::asio::ip::tcp::socket::wait_read);
connectionManager::start_receive();
}
I just don't understand what I am missing? Why client receives exactly 133 packages (133 x 500 bytes) and then drops?
I have changed the code in many ways, but with no luck. The last thing I added is
socket->wait(boost::asio::ip::tcp::socket::wait_read);
before I call start_receive() again, and the program hands on this line exactly when package #133 is received.
Please help. I am close to give up and become a pizza delivery guy.
I'm trying to read / write multiple Protocol Buffers messages from files, in both C++ and Java. Google suggests writing length prefixes before the messages, but there's no way to do that by default (that I could see).
However, the Java API in version 2.1.0 received a set of "Delimited" I/O functions which apparently do that job:
parseDelimitedFrom
mergeDelimitedFrom
writeDelimitedTo
Are there C++ equivalents? And if not, what's the wire format for the size prefixes the Java API attaches, so I can parse those messages in C++?
Update:
These now exist in google/protobuf/util/delimited_message_util.h as of v3.3.0.
I'm a bit late to the party here, but the below implementations include some optimizations missing from the other answers and will not fail after 64MB of input (though it still enforces the 64MB limit on each individual message, just not on the whole stream).
(I am the author of the C++ and Java protobuf libraries, but I no longer work for Google. Sorry that this code never made it into the official lib. This is what it would look like if it had.)
bool writeDelimitedTo(
const google::protobuf::MessageLite& message,
google::protobuf::io::ZeroCopyOutputStream* rawOutput) {
// We create a new coded stream for each message. Don't worry, this is fast.
google::protobuf::io::CodedOutputStream output(rawOutput);
// Write the size.
const int size = message.ByteSize();
output.WriteVarint32(size);
uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
if (buffer != NULL) {
// Optimization: The message fits in one buffer, so use the faster
// direct-to-array serialization path.
message.SerializeWithCachedSizesToArray(buffer);
} else {
// Slightly-slower path when the message is multiple buffers.
message.SerializeWithCachedSizes(&output);
if (output.HadError()) return false;
}
return true;
}
bool readDelimitedFrom(
google::protobuf::io::ZeroCopyInputStream* rawInput,
google::protobuf::MessageLite* message) {
// We create a new coded stream for each message. Don't worry, this is fast,
// and it makes sure the 64MB total size limit is imposed per-message rather
// than on the whole stream. (See the CodedInputStream interface for more
// info on this limit.)
google::protobuf::io::CodedInputStream input(rawInput);
// Read the size.
uint32_t size;
if (!input.ReadVarint32(&size)) return false;
// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit =
input.PushLimit(size);
// Parse the message.
if (!message->MergeFromCodedStream(&input)) return false;
if (!input.ConsumedEntireMessage()) return false;
// Release the limit.
input.PopLimit(limit);
return true;
}
Okay, so I haven't been able to find top-level C++ functions implementing what I need, but some spelunking through the Java API reference turned up the following, inside the MessageLite interface:
void writeDelimitedTo(OutputStream output)
/* Like writeTo(OutputStream), but writes the size of
the message as a varint before writing the data. */
So the Java size prefix is a (Protocol Buffers) varint!
Armed with that information, I went digging through the C++ API and found the CodedStream header, which has these:
bool CodedInputStream::ReadVarint32(uint32 * value)
void CodedOutputStream::WriteVarint32(uint32 value)
Using those, I should be able to roll my own C++ functions that do the job.
They should really add this to the main Message API though; it's missing functionality considering Java has it, and so does Marc Gravell's excellent protobuf-net C# port (via SerializeWithLengthPrefix and DeserializeWithLengthPrefix).
I solved the same problem using CodedOutputStream/ArrayOutputStream to write the message (with the size) and CodedInputStream/ArrayInputStream to read the message (with the size).
For example, the following pseudo-code writes the message size following by the message:
const unsigned bufLength = 256;
unsigned char buffer[bufLength];
Message protoMessage;
google::protobuf::io::ArrayOutputStream arrayOutput(buffer, bufLength);
google::protobuf::io::CodedOutputStream codedOutput(&arrayOutput);
codedOutput.WriteLittleEndian32(protoMessage.ByteSize());
protoMessage.SerializeToCodedStream(&codedOutput);
When writing you should also check that your buffer is large enough to fit the message (including the size). And when reading, you should check that your buffer contains a whole message (including the size).
It definitely would be handy if they added convenience methods to C++ API similar to those provided by the Java API.
IsteamInputStream is very fragile to eofs and other errors that easily occurs when used together with std::istream. After this the protobuf streams are permamently damaged and any already used buffer data is destroyed. There are proper support for reading from traditional streams in protobuf.
Implement google::protobuf::io::CopyingInputStream and use that together with CopyingInputStreamAdapter. Do the same for the output variants.
In practice a parsing call ends up in google::protobuf::io::CopyingInputStream::Read(void* buffer, int size) where a buffer is given. The only thing left to do is read into it somehow.
Here's an example for use with Asio synchronized streams (SyncReadStream/SyncWriteStream):
#include <google/protobuf/io/zero_copy_stream_impl_lite.h>
using namespace google::protobuf::io;
template <typename SyncReadStream>
class AsioInputStream : public CopyingInputStream {
public:
AsioInputStream(SyncReadStream& sock);
int Read(void* buffer, int size);
private:
SyncReadStream& m_Socket;
};
template <typename SyncReadStream>
AsioInputStream<SyncReadStream>::AsioInputStream(SyncReadStream& sock) :
m_Socket(sock) {}
template <typename SyncReadStream>
int
AsioInputStream<SyncReadStream>::Read(void* buffer, int size)
{
std::size_t bytes_read;
boost::system::error_code ec;
bytes_read = m_Socket.read_some(boost::asio::buffer(buffer, size), ec);
if(!ec) {
return bytes_read;
} else if (ec == boost::asio::error::eof) {
return 0;
} else {
return -1;
}
}
template <typename SyncWriteStream>
class AsioOutputStream : public CopyingOutputStream {
public:
AsioOutputStream(SyncWriteStream& sock);
bool Write(const void* buffer, int size);
private:
SyncWriteStream& m_Socket;
};
template <typename SyncWriteStream>
AsioOutputStream<SyncWriteStream>::AsioOutputStream(SyncWriteStream& sock) :
m_Socket(sock) {}
template <typename SyncWriteStream>
bool
AsioOutputStream<SyncWriteStream>::Write(const void* buffer, int size)
{
boost::system::error_code ec;
m_Socket.write_some(boost::asio::buffer(buffer, size), ec);
return !ec;
}
Usage:
AsioInputStream<boost::asio::ip::tcp::socket> ais(m_Socket); // Where m_Socket is a instance of boost::asio::ip::tcp::socket
CopyingInputStreamAdaptor cis_adp(&ais);
CodedInputStream cis(&cis_adp);
Message protoMessage;
uint32_t msg_size;
/* Read message size */
if(!cis.ReadVarint32(&msg_size)) {
// Handle error
}
/* Make sure not to read beyond limit of message */
CodedInputStream::Limit msg_limit = cis.PushLimit(msg_size);
if(!msg.ParseFromCodedStream(&cis)) {
// Handle error
}
/* Remove limit */
cis.PopLimit(msg_limit);
Here you go:
#include <google/protobuf/io/zero_copy_stream_impl.h>
#include <google/protobuf/io/coded_stream.h>
using namespace google::protobuf::io;
class FASWriter
{
std::ofstream mFs;
OstreamOutputStream *_OstreamOutputStream;
CodedOutputStream *_CodedOutputStream;
public:
FASWriter(const std::string &file) : mFs(file,std::ios::out | std::ios::binary)
{
assert(mFs.good());
_OstreamOutputStream = new OstreamOutputStream(&mFs);
_CodedOutputStream = new CodedOutputStream(_OstreamOutputStream);
}
inline void operator()(const ::google::protobuf::Message &msg)
{
_CodedOutputStream->WriteVarint32(msg.ByteSize());
if ( !msg.SerializeToCodedStream(_CodedOutputStream) )
std::cout << "SerializeToCodedStream error " << std::endl;
}
~FASWriter()
{
delete _CodedOutputStream;
delete _OstreamOutputStream;
mFs.close();
}
};
class FASReader
{
std::ifstream mFs;
IstreamInputStream *_IstreamInputStream;
CodedInputStream *_CodedInputStream;
public:
FASReader(const std::string &file), mFs(file,std::ios::in | std::ios::binary)
{
assert(mFs.good());
_IstreamInputStream = new IstreamInputStream(&mFs);
_CodedInputStream = new CodedInputStream(_IstreamInputStream);
}
template<class T>
bool ReadNext()
{
T msg;
unsigned __int32 size;
bool ret;
if ( ret = _CodedInputStream->ReadVarint32(&size) )
{
CodedInputStream::Limit msgLimit = _CodedInputStream->PushLimit(size);
if ( ret = msg.ParseFromCodedStream(_CodedInputStream) )
{
_CodedInputStream->PopLimit(msgLimit);
std::cout << mFeed << " FASReader ReadNext: " << msg.DebugString() << std::endl;
}
}
return ret;
}
~FASReader()
{
delete _CodedInputStream;
delete _IstreamInputStream;
mFs.close();
}
};
I ran into the same issue in both C++ and Python.
For the C++ version, I used a mix of the code Kenton Varda posted on this thread and the code from the pull request he sent to the protobuf team (because the version posted here doesn't handle EOF while the one he sent to github does).
#include <google/protobuf/message_lite.h>
#include <google/protobuf/io/zero_copy_stream.h>
#include <google/protobuf/io/coded_stream.h>
bool writeDelimitedTo(const google::protobuf::MessageLite& message,
google::protobuf::io::ZeroCopyOutputStream* rawOutput)
{
// We create a new coded stream for each message. Don't worry, this is fast.
google::protobuf::io::CodedOutputStream output(rawOutput);
// Write the size.
const int size = message.ByteSize();
output.WriteVarint32(size);
uint8_t* buffer = output.GetDirectBufferForNBytesAndAdvance(size);
if (buffer != NULL)
{
// Optimization: The message fits in one buffer, so use the faster
// direct-to-array serialization path.
message.SerializeWithCachedSizesToArray(buffer);
}
else
{
// Slightly-slower path when the message is multiple buffers.
message.SerializeWithCachedSizes(&output);
if (output.HadError())
return false;
}
return true;
}
bool readDelimitedFrom(google::protobuf::io::ZeroCopyInputStream* rawInput, google::protobuf::MessageLite* message, bool* clean_eof)
{
// We create a new coded stream for each message. Don't worry, this is fast,
// and it makes sure the 64MB total size limit is imposed per-message rather
// than on the whole stream. (See the CodedInputStream interface for more
// info on this limit.)
google::protobuf::io::CodedInputStream input(rawInput);
const int start = input.CurrentPosition();
if (clean_eof)
*clean_eof = false;
// Read the size.
uint32_t size;
if (!input.ReadVarint32(&size))
{
if (clean_eof)
*clean_eof = input.CurrentPosition() == start;
return false;
}
// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit = input.PushLimit(size);
// Parse the message.
if (!message->MergeFromCodedStream(&input)) return false;
if (!input.ConsumedEntireMessage()) return false;
// Release the limit.
input.PopLimit(limit);
return true;
}
And here is my python2 implementation:
from google.protobuf.internal import encoder
from google.protobuf.internal import decoder
#I had to implement this because the tools in google.protobuf.internal.decoder
#read from a buffer, not from a file-like objcet
def readRawVarint32(stream):
mask = 0x80 # (1 << 7)
raw_varint32 = []
while 1:
b = stream.read(1)
#eof
if b == "":
break
raw_varint32.append(b)
if not (ord(b) & mask):
#we found a byte starting with a 0, which means it's the last byte of this varint
break
return raw_varint32
def writeDelimitedTo(message, stream):
message_str = message.SerializeToString()
delimiter = encoder._VarintBytes(len(message_str))
stream.write(delimiter + message_str)
def readDelimitedFrom(MessageType, stream):
raw_varint32 = readRawVarint32(stream)
message = None
if raw_varint32:
size, _ = decoder._DecodeVarint32(raw_varint32, 0)
data = stream.read(size)
if len(data) < size:
raise Exception("Unexpected end of file")
message = MessageType()
message.ParseFromString(data)
return message
#In place version that takes an already built protobuf object
#In my tests, this is around 20% faster than the other version
#of readDelimitedFrom()
def readDelimitedFrom_inplace(message, stream):
raw_varint32 = readRawVarint32(stream)
if raw_varint32:
size, _ = decoder._DecodeVarint32(raw_varint32, 0)
data = stream.read(size)
if len(data) < size:
raise Exception("Unexpected end of file")
message.ParseFromString(data)
return message
else:
return None
It might not be the best looking code and I'm sure it can be refactored a fair bit, but at least that should show you one way to do it.
Now the big problem: It's SLOW.
Even when using the C++ implementation of python-protobuf, it's one order of magnitude slower than in pure C++. I have a benchmark where I read 10M protobuf messages of ~30 bytes each from a file. It takes ~0.9s in C++, and 35s in python.
One way to make it a bit faster would be to re-implement the varint decoder to make it read from a file and decode in one go, instead of reading from a file and then decoding as this code currently does. (profiling shows that a significant amount of time is spent in the varint encoder/decoder). But needless to say that alone is not enough to close the gap between the python version and the C++ version.
Any idea to make it faster is very welcome :)
Just for completeness, I post here an up-to-date version that works with the master version of protobuf and Python3
For the C++ version it is sufficient to use the utils in delimited_message_utils.h, here a MWE
#include <google/protobuf/io/zero_copy_stream_impl.h>
#include <google/protobuf/util/delimited_message_util.h>
#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>
template <typename T>
bool writeManyToFile(std::deque<T> messages, std::string filename) {
int outfd = open(filename.c_str(), O_WRONLY | O_CREAT | O_TRUNC);
google::protobuf::io::FileOutputStream fout(outfd);
bool success;
for (auto msg: messages) {
success = google::protobuf::util::SerializeDelimitedToZeroCopyStream(
msg, &fout);
if (! success) {
std::cout << "Writing Failed" << std::endl;
break;
}
}
fout.Close();
close(outfd);
return success;
}
template <typename T>
std::deque<T> readManyFromFile(std::string filename) {
int infd = open(filename.c_str(), O_RDONLY);
google::protobuf::io::FileInputStream fin(infd);
bool keep = true;
bool clean_eof = true;
std::deque<T> out;
while (keep) {
T msg;
keep = google::protobuf::util::ParseDelimitedFromZeroCopyStream(
&msg, &fin, nullptr);
if (keep)
out.push_back(msg);
}
fin.Close();
close(infd);
return out;
}
For the Python3 version, building on #fireboot 's answer, the only thing thing that needed modification is the decoding of raw_varint32
def getSize(raw_varint32):
result = 0
shift = 0
b = six.indexbytes(raw_varint32, 0)
result |= ((ord(b) & 0x7f) << shift)
return result
def readDelimitedFrom(MessageType, stream):
raw_varint32 = readRawVarint32(stream)
message = None
if raw_varint32:
size = getSize(raw_varint32)
data = stream.read(size)
if len(data) < size:
raise Exception("Unexpected end of file")
message = MessageType()
message.ParseFromString(data)
return message
Was also looking for a solution for this. Here's the core of our solution, assuming some java code wrote many MyRecord messages with writeDelimitedTo into a file. Open the file and loop, doing:
if(someCodedInputStream->ReadVarint32(&bytes)) {
CodedInputStream::Limit msgLimit = someCodedInputStream->PushLimit(bytes);
if(myRecord->ParseFromCodedStream(someCodedInputStream)) {
//do your stuff with the parsed MyRecord instance
} else {
//handle parse error
}
someCodedInputStream->PopLimit(msgLimit);
} else {
//maybe end of file
}
Hope it helps.
Working with an objective-c version of protocol-buffers, I ran into this exact issue. On sending from the iOS client to a Java based server that uses parseDelimitedFrom, which expects the length as the first byte, I needed to call writeRawByte to the CodedOutputStream first. Posting here to hopegully help others that run into this issue. While working through this issue, one would think that Google proto-bufs would come with a simply flag which does this for you...
Request* request = [rBuild build];
[self sendMessage:request];
}
- (void) sendMessage:(Request *) request {
//** get length
NSData* n = [request data];
uint8_t len = [n length];
PBCodedOutputStream* os = [PBCodedOutputStream streamWithOutputStream:outputStream];
//** prepend it to message, such that Request.parseDelimitedFrom(in) can parse it properly
[os writeRawByte:len];
[request writeToCodedOutputStream:os];
[os flush];
}
Since I'm not allowed to write this as a comment to Kenton Varda's answer above; I believe there is a bug in the code he posted (as well as in other answers which have been provided). The following code:
...
google::protobuf::io::CodedInputStream input(rawInput);
// Read the size.
uint32_t size;
if (!input.ReadVarint32(&size)) return false;
// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit =
input.PushLimit(size);
...
sets an incorrect limit because it does not take into account the size of the varint32 which has already been read from input. This can result in data loss/corruption as additional bytes are read from the stream which may be part of the next message. The usual way of handling this correctly is to delete the CodedInputStream used to read the size and create a new one for reading the payload:
...
uint32_t size;
{
google::protobuf::io::CodedInputStream input(rawInput);
// Read the size.
if (!input.ReadVarint32(&size)) return false;
}
google::protobuf::io::CodedInputStream input(rawInput);
// Tell the stream not to read beyond that size.
google::protobuf::io::CodedInputStream::Limit limit =
input.PushLimit(size);
...
You can use getline for reading a string from a stream, using the specified delimiter:
istream& getline ( istream& is, string& str, char delim );
(defined in the header)
I just started a project where i'm struggling since days now about serial ports. I wrote a static library that can handle all the serial routine and give an interface with "readLine()" and "writeLine()" functions.
Everything works flawlessly on the write and read (which are threaded by the way) except if the slave does not anwser after he gets the data, then, the data is sent back to me, and i read it.
I open my fd with O_NDELAY and configure my read system call as Non blocking with fcntl.
here are the two threaded loops that work perfectly beside that.
void *Serial_Port::readLoop(void *param)
{
Serial_Port *sp = static_cast<Serial_Port*>(param);
std::string *line = NULL;
char buffer[128];
while (1)
{
line = new std::string();
while ((line->find("\r\n")) == std::string::npos)
{
usleep(100);
bzero(buffer, 128);
pthread_mutex_lock(sp->getRLock());
if (read(sp->getDescriptor(), buffer, 127) > 0)
*line += buffer;
pthread_mutex_unlock(sp->getRLock());
}
pthread_mutex_lock(sp->getRLock());
sp->getRStack()->push(line->substr(0, line->find("\r\n")));
pthread_mutex_unlock(sp->getRLock());
delete (line);
}
return (param);
}
void *Serial_Port::writeLoop(void *param)
{
Serial_Port *sp = static_cast<Serial_Port*>(param);
std::string *line;
while (1)
{
line = NULL;
pthread_mutex_lock(sp->getWLock());
if (!sp->getWStack()->empty())
{
line = new std::string(sp->getWStack()->front());
sp->getWStack()->pop();
}
pthread_mutex_unlock(sp->getWLock());
if (line != NULL)
{
pthread_mutex_lock(sp->getWLock());
write(sp->getDescriptor(), line->c_str(), line->length());
// fsync(sp->getDescriptor());
pthread_mutex_unlock(sp->getWLock());
}
usleep(100);
}
return (param);
}
I tried to flush the file descriptor, but i can't manage to receive any data after doing that. How can I get rid of that duplicate, needless data?
Thanks.
After multiple tests and behavior analysis, I discovered it was the "Pulsar3" (the device i was using on serial) that kept giving me back what i sent as "Acknowledge". Nice to know!
Below code definitely works, but not fast as I expect.
I expect my program to read data at very good pace. There is another commercial app which connects to same server & retrives data at amazing speed. Server side is not a problem.
class A
{
//...
boost::asio::ip::tcp::socket* myPort;
}
void A::OpenPort()
{
if(myPort)
{
if(myPort->is_open())
{
return;
}
}
// make the connection
Connect();
if(! myPort->is_open())
{
return;
}
// set the protocol
static string init("INIT\r\n");
myPort->write_some(boost::asio::buffer(init.c_str(), init.length()));
}
void A::Read()
{
static string prev_msg = "";
try
{
OpenPort();
while(true)
{
boost::system::error_code error;
boost::asio::streambuf streamBuf;
boost::asio::streambuf::mutable_buffers_type mutableBuffer = streamBuf.prepare(614400);
size_t bytes_transferred = myPort->read_some(boost::asio::buffer(mutableBuffer), error);
if (error)
{
if (error != boost::asio::error::eof)
{
throw boost::system::system_error(error); // Some other error.
}
}
// add to any previous message we might not have processed
streamBuf.commit(bytes_transferred);
istreambuf_iterator<char> sbit(&streamBuf);
istreambuf_iterator<char> end;
string s(sbit, end);
prev_msg.append(s);
string delimiter1 = ",\r\n";
size_t pos1 = 0;
string response;
while ((pos1 = prev_msg.find(delimiter1)) != std::string::npos)
{
response = prev_msg.substr(0, pos1);
//SOME PROCESSING ON THE RESPONSE RECEIVED
}
}
}
catch (boost::system::system_error const& ex)
{
cout<<ex.what();
}
}
obviously, the problem is read_some(), program doesn't read complete data in one read operation, sometimes it receives 614000 bytes, sometimes very less.
I don't want to enforce any limit on size of the buffer, whatever server sends, program should read all that data in one go.
Hence, I decided to use just read(). But, now program is stuck at read(); read() call doesn't return.
boost::asio::streambuf streamBuf;
size_t bytes_transferred = read(*myPort, streamBuf, error);
if (error)
{
if (error != boost::asio::error::eof)
{
throw boost::system::system_error(error); // Some other error.
}
}
I must process the data received before requesting next data, Hence I can't use async_read().
Do not allocate a new buffer on each loop, do that only one time outside the loop.
while(true)
{
boost::system::error_code error;
boost::asio::streambuf streamBuf;
boost::asio::streambuf::mutable_buffers_type mutableBuffer = streamBuf.prepare(614400);
size_t bytes_transferred = myPort->read_some(boost::asio::buffer(mutableBuffer), error);
...
to be replaced by
boost::system::error_code error;
boost::asio::streambuf streamBuf;
boost::asio::streambuf::mutable_buffers_type mutableBuffer = streamBuf.prepare(614400);
while(true)
{
size_t bytes_transferred = myPort->read_some(boost::asio::buffer(mutableBuffer), error);
...
A few things:
With TCP you can never be sure that you will receive everything in one go.
Because you are reading up to a delimiter, read_until() is probably what you're after.
Make sure you're opening the socket with O_NDELAY, otherwise you will add 200ms to your write. Do this by calling myPort->set_option(tcp::no_delay(true)) where appropriate in your code.
The sleep is not a good idea. Design your code so that it isn't necessary.
Your code seems to go into an endless loop if the socket is closed.
You call write_some() without checking the return value. You should probably call write() to ensure all your data is written.
If you have many threads you will probably get an improvement from redesigning your code to be async.
I'll start with the code:
typedef std::vector<unsigned char> CharBuf;
static const int RCV_BUF_SIZE = 1024;
SOCKET m_socket = a connected and working socket;
// ...
CharBuf buf; // Declare buffer
buf.resize(RCV_BUF_SIZE); // resize buffer to 1024
char* p_buf = reinterpret_cast<char*>(&buf[0]); // change from unsigned char to char
//char p_buf[RCV_BUF_SIZE];
int ret = recv(m_socket, p_buf, RCV_BUF_SIZE, 0); // Does not work
for (int i=0; i<RCV_BUF_SIZE; ++i) // Works (does not crash, so the buffer is ok)
char c = p_buf[i];
//...
Now when I run this code ret becomes -1 and WSAGetLastError() returns 10014 which means the pointer is bad.
However I can't see why this shouldn't work? If I comment out the reinterpret_cast line and use the line below it works!
It could be argued that reinterpret_cast is risky, but I think it should be ok as both unsigned char and signed char has the exact same size.
std::vectors should be safe to address directly in memory as far as I know as well.
The funny part is that when I do the same thing with the same vector-type in send() it works! Send function:
void SendData(const CharBuf& buf)
{
buf.resize(RCV_BUF_SIZE); // resize buffer to 1024
const char* p_buf = reinterpret_cast<const char*>(&buf[0]); // change from unsigned char to char
int ret = send(m_socket, p_buf, (int)buf.size(), 0); // Works
}
As we see, no difference except CharBuf being const in this case, can that change anything?
Why is recv() more sensitive than send()? How can recv() even know the pointer is invalid (which it obviously isn't)?? all it should see is a char array!
As per request my whole receive function (bear in mind that I can't spell out every function in it, but I think they should be fairly self-explanatory.
bool TcpSocket::ReceiveData(CharBuf* pData)
{
if (!CheckInitialized("ReceiveData"))
return false;
if (m_status != CONNECTED_STAT)
{
AddToErrLog("Socket not connected", 1, "ReceiveData");
return false;
}
int ret;
pData->resize(RCV_BUF_SIZE);
char* p_buf = reinterpret_cast<char*>(&pData[0]);
ret = recv(m_socket, p_buf, RCV_BUF_SIZE, 0);
switch (ret)
{
case 0: // Gracefully closed
AddToLog("Connection gracefully closed", 2);
Shutdown(); // The connection is closed, no idea to keep running
return true;
case SOCKET_ERROR: // Error
ret = WSAGetLastError();
if (ret == 10004) // This indicates the socket was closed while we were waiting
AddToLog("Socket was shut down while waiting for data", 1, "ReceiveData(1)");
else
AddToErrLog("Receive data failed with code: " + CStr(ret));
AddToLog("Connection ended with error", 2);
Shutdown();
return false;
default: // Normal operation
pData->resize(ret); // Remove unused space
return true;
}
}
Never mind. I found it while I was pasting the function. Like always, you find your error when you try to explain it for someone else :)
I leave it up to the reader to figure out what was wrong, but I'll give &pData[0] as a hint.
Thanks for your help :D
Found the answer myself while pasting the whole function, &pData[0] is a hint.