Qt + protobuf, types? - c++

I would like to dip into Google's protocol buffers in Qt development, but I am having trouble figuring out how to incorporate them best.
Ultimately, I want to send with QUdpSocket and QTcpSocket using protocol buffers.
What is the best method for going between a protocol buffer message to sending the data over a socket (QByteArray) and then back again at the other side?

Creating a QByteArray from a protobuf object:
Person person; // a protobuf object
person.set_id(123);
person.set_name("Bob");
person.set_email("bob#example.com");
std::ostringstream out;
person.SerializeToOstream(&out);
QByteArray byteArray(out.str().c_str());
sendSerializedPersonOverQTcpSocket(byteArray);
Reading back a protobuf object from a QByteArray:
QByteArray byteArray = readSerializedPersonFromQTcpSocket();
Person person;
if (!person.ParseFromArray(byteArray, byteArray.size())) {
std::cerr << "Failed to parse person.pb." << std::endl;
}

Instead of:
std::ostringstream out;
person.SerializeToOstream(&out);
QByteArray byteArray(out.str().c_str());
you can also write:
QByteArray byteArray(person.SerializeAsString().c_str());
EDIT: Above two gives the same result, but I'm not sure wether it's correct. This one seems to work better:
QByteArray byteArray(QString::fromStdString(person.SerializeAsString()));
EDIT2: OK, now I know how it works: first two ways are wrong if there are \0 char in serialization - everything after it it's then lost. To correct it one can write:
QByteArray byteArray(person.SerializeAsString().c_str(), person.ByteSize());

Using the code below is really dangerous
std::ostringstream out;
person.SerializeToOstream(&out);
QByteArray byteArray(out.str().c_str());
sendSerializedPersonOverQTcpSocket(byteArray);
You can find a good explanation here In protobuf-c, can optional uint32 variable have value 0
A right way to create a QByteArray from a protobuf message is
QByteArray byteArray;
byteArray.resize(message.ByteSize());
message.SerializeToArray(byteArray.data(), byteArray.size());

#James: You can use ParseFromArray(), for example, as below: (Please note that ParseFromArray() is available only on proto-buf-lite versions of the libs).
void convertQByteArrayToUser(QByteArray& aByteArray)
{
com::your::name_space::User user;
if(!user.ParseFromArray(aByteArray.data(), aByteArray.size()))
{
//could not parse
}
else { //yayyyyy
if(user.has_userid())
{
//...
}
}
}

Related

External vs internal declaration of QByteArray

I have currently some problems with the QSerialPort: When I am using the function from an example which looks like
QKeyEvent *e;
emit getData(e->text().toLocal8Bit());
connect(console, SIGNAL(getData(QByteArray)), this, SLOT(writeData(QByteArray)));
void MainWindow::writeData(const QByteArray &data)
{
qDebug() << "Data is to write: " << data;
serial->write(data);
}
then the receiving device can work with the data. But when I change the function writeData() to
void MainWindow::writeData(const QByteArray &data)
{
QString a = "Q";
QByteArray b = a.toLocal8Bit();
serial->write(b);
}
the receiving device can not work with the received data. Where is the difference between those two approaches?
Update: I found out that apparently the data is only usefully transferred if I press Enter after typing the letters. Somehow the '\n' gets lost in the conversion from QString to QByteArray. How can I keep it?
you should add an enter to your Qstring like this
QString a = "Q\x00D";
In the example you have given, there is no "\n" in the QString! It is not getting lost, it is not there in the first place.
If a newline is necessary, then construct the String as QString a = "Q\n".
You can also construct the QByteArray directly from a character array rather than going through a char array -> QString -> QByteArrray conversion sequence, like so:
QByteArray b("Q\n");
EDIT: I realized that your contrived example where you are just sending the letter "Q" is probably a debug attempt, not your real code. In reality, you're getting data in as a QByteArray from some other signal that is emitting a QByteArray. That QByteArray that you are receiving must not include the newline character in it. If you are reading from a file or user input, then that is normal. Most readline-like functions strip off the trailing newline. If it is always necessary to have a newline, you can simply do something like this in your WriteData method:
void MainWindow::writeData(const QByteArray &data)
{
serial->write(data);
serial->write("\n");
}
If sometimes the passed-in QByteArray has a newline at the end and sometimes not, and your receiving device cannot handle redundant newlines, then you'd need to check whether data ends with a newline and only write the "\n" if it does not.
what if you make the QByteArray like this
QByteArray b(&a);

sending struct via Qt UDP

I am trying to setup two way communication via the QUdpSocket. I am trying to send a struct consisting of a C++ Eigenvector and a double. I have tried serializing into a QByteArray as follows:
MyStruct toSend;
QByteArray buf;
QDataStream s(&buf, QIODevice::WriteOnly);
if (false) s.setByteOrder(QDataStream::LittleEndian);
std::string vec_str = eigenToStr(toSend.vec);
s << (double)toSend.test1 << QString(vec_str.c_str());
Where eigenToStr() converts the Eigenvector to a string.
However, I am unable to read the message on the other end. When I convert back to a string before sending the QByteArray, I get #ffffff. So I assume it's an issue with the QByteArray/QDataStream conversion.
I would appreciate any suggestions as to how I might serialize my struct so that I can send it via UDP.
Thanks!
QByteArray and QDataStream cooperate very well:). If you really want write the raw data to QDataStream , please use the method :
int QDataStream::writeRawData(const char * s, int len)
And personally I prefer to overload << way to write user data to QDataStream, such as
QDataStream& operator <<(QDataStream& out,MyStruct & data)
Then probably code looks as :
MyStruct toSend;
QByteArray buf;
QDataStream s(&buf, QIODevice::WriteOnly);
if (false) s.setByteOrder(QDataStream::LittleEndian);//false ?
s << toSend;

ostringstream issues

when I use ostringstream, the only value that i get is : COM1
I have an application, which sends data.
I am using the code as :
std::ostringstream values;
values << someStruct.someValues;
...
...
std::string data
data << values.str();
But when I run this, all I get is an output saying COM1. My application is a DLL file.
But when I do this method below, I get the correct values
char *data;
char values[20];
sprintf(values, "%d",someStruct.someValue);
strcat(data,values);
But I don't want to use the above method as I have many variables that I need to fetch from the program. So someone please help.
std::string data;
data << values.str();
std::string is not a stream. It can't take operator<<. I'm surprised this code even compiles, but it almost certainly doesn't do something useful. What you want is this:
std::string data = values.str();

How to serialize an object to send over network

I'm trying to serialize objects to send over network through a socket using only STL. I'm not finding a way to keep objects' structure to be deserialized in the other host. I tried converting to string, to char* and I've spent a long time searching for tutorials on the internet and until now I have found nothing.
Is there a way to do it only with STL?
Are there any good tutorials?
I am almost trying boost, but if there is how to do it with STL I'd like to learn.
You can serialize with anything. All serialization means is that you are converting the object to bytes so that you can send it over a stream (like an std::ostream) and read it with another (like an std::istream). Just override operator <<(std::ostream&, const T&) and operator >>(std::istream&, T&) where T is each of your types. And all the types contained in your types.
However, you should probably just use an already-existing library (Boost is pretty nice). There are tons of things that a library like Boost does for you, like byte-ordering, taking care of common objects (like arrays and all the stuff from the standard library), providing a consistent means of performing serialization and tons of other stuff.
My first question will be: do you want serialization or messaging ?
It might seem stupid at first, since you asked for serialization, but then I have always distinguished the two terms.
Serialization is about taking a snapshot of your memory and restoring it later on. Each object is represented as a separate entity (though they might be composed)
Messaging is about sending information from one point to another. The message usually has its own grammar and may not reflect the organization of your Business Model.
Too often I've seen people using Serialization where Messaging should have been used. It does not mean that Serialization is useless, but it does mean that you should think ahead of times. It's quite difficult to alter the BOM once you have decided to serialize it, especially if you decide to relocate some part of information (move it from one object to another)... because how then are you going to decode the "old" serialized version ?
Now that that's been cleared up...
... I will recommend Google's Protocol Buffer.
You could perfectly rewrite your own using the STL, but you would end up doing work that has already been done, and unless you wish to learn from it, it's quite pointless.
One great thing about protobuf is that it's language agnostic in a way: ie you can generate the encoder/decoder of a given message for C++, Java or Python. The use of Python is nice for message injection (testing) or message decoding (to check the output of a logged message). It's not something that would come easy were you to use the STL.
Serializing C++ Objects over a Network Socket
This is 6 years late but I just recently had this problem and this was one of the threads that I came across in my search on how to serialize object through a network socket in C++. This solution uses just 2 or 3 lines of code. There are a lot of answers that I found work but the easiest that I found was to use reinterpret_cast<obj*>(target) to convert the class or structure into an array of characters and feed it through the socket. Here's an example.
Class to be serialized:
/* myclass.h */
#ifndef MYCLASS_H
#define MYCLASS_H
class MyClass
{
public:
int A;
int B;
MyClass(){A=1;B=2;}
~MyClass(){}
};
#endif
Server Program:
/* server.cpp */
#include "myclass.h"
int main (int argc, char** argv)
{
// Open socket connection.
// ...
// Loop continuously until terminated.
while(1)
{
// Read serialized data from socket.
char buf[sizeof(MyClass)];
read(newsockfd,buf, sizeof(MyClass));
MyClass *msg = reinterpret_cast<MyClass*>(buf);
std::cout << "A = " << std::to_string(msg->A) << std::endl;
std::cout << "B = " << std::to_string(msg->B) << std::endl;
}
// Close socket connection.
// ...
return 0;
}
Client Program:
/* client.cpp */
#include "myClass.h"
int main(int argc, char *argv[])
{
// Open socket connection.
// ...
while(1)
{
printf("Please enter the message: ");
bzero(buffer,256);
fgets(buffer,255,stdin);
MyClass msg;
msg.A = 1;
msg.B = 2;
// Write serialized data to socket.
char* tmp = reinterpret_cast<char*>(&msg);
write(sockfd,tmp, sizeof(MyClass));
}
// Close socket connection.
// ...
return 0;
}
Compile both server.cpp and client.cpp using g++ with -std=c++11 as an option. You can then open two terminals and run both programs, however, start the server program before the client so that it has something to connect to.
Hope this helps.
I got it!
I used strinstream to serialize objects and I sent it as a message using the stringstream's method str() and so string's c_str().
Look.
class Object {
public:
int a;
string b;
void methodSample1 ();
void methosSample2 ();
friend ostream& operator<< (ostream& out, Object& object) {
out << object.a << " " << object.b; //The space (" ") is necessari for separete elements
return out;
}
friend istream& operator>> (istream& in, Object& object) {
in >> object.a;
in >> object.b;
return in;
}
};
/* Server side */
int main () {
Object o;
stringstream ss;
o.a = 1;
o.b = 2;
ss << o; //serialize
write (socket, ss.str().c_str(), 20); //send - the buffer size must be adjusted, it's a sample
}
/* Client side */
int main () {
Object o2;
stringstream ss2;
char buffer[20];
string temp;
read (socket, buffer, 20); //receive
temp.assign(buffer);
ss << temp;
ss >> o2; //unserialize
}
I'm not sure if is necessary convert to string before to serialize (ss << o), maybe is possible directly from char.
I think you should use google Protocol Buffers in your project.In network transport Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:
are simpler
are 3 to 10 times smaller
are 20 to 100 times faster
are less ambiguous
generate data access classes that are easier to use programmaticall
and so on. I think you need read https://developers.google.com/protocol-buffers/docs/overview about protobuf

How to set binary data using setBlob() in C++ connector

I know this has been asked before, and I tried to implement the solution, but I just get exception errors when I call ps->executeUpdate(). Has anyone got an explicit example?
This post is a bit old, but I ran across the same question. I employed the method above and it didn't quite work right for my case, which was trying to take a vector and use that for the stream. What I was doing was taking a UUID and converting it into a 16 byte binary version to use in the table. Using the method above, I found that only half my buffer was being populated.
I ended up using a stringstream.
std::vector<unsigned char> convertedId;
std::stringstream stream;
// convertedId has been populated with the 16 byte binary version
stream = std::stringstream(std::string(convertedId.begin(), convertedId.end()));
// Parameter 1 is BINARY(16)
pStatement->setBlob(1, &stream);
A few other things to keep in mind. The stream is not accessed until one of the execute variants is called. So you'll need to keep the stream around until you have run execute.
Hopefully this will help someone and save them time.
Sorry Matthew - I assumed the previous answer to this question (by elrohin). Maybe I should have replied to that. Anyway heres the code he suggested:
class DataBuf : public streambuf
{
public:
DataBuf(char * d, size_t s) {
setg(d, d, d + s);
}
};
// prepare sql update statement etc. and set the data pointer
string* pData = ; // ...not part of the original answer
DataBuf buffer((char*)pData->data(), pData->length());
istream stream(&buffer);
ps->setBlob(1, &stream);
ps->executeUpdate(); // This causes an exception in free.c
I'm using VS9 with the latest (beta) connector/cpp debug libs. I've also tried using char* instead of string.
This code works fine for me:
Driver *driver;
Connection *conn;
driver = get_driver_instance();
conn = driver->connect("tcp://127.0.0.1:3306", "root", "root");
std::auto_ptr use_stmt(conn->createStatement());
use_stmt->execute("USE world");
std::auto_ptr stmt(conn->prepareStatement("INSERT INTO terrain_texture_tiles_0 (data) VALUES(?)"));
std::string value("A\0B", sizeof("A\0B") - 1);
std::istringstream tmp_blob(value);
stmt->setBlob(1, &tmp_blob);
stmt->execute();
hope it helps ... Jaroslav Pribyl