I am new to protocol buffers and I am having issues with multiple messages using Any data.
I am not sure if I am using it correctly.
I have the following in message.proto
import "google/protobuf/any.proto";
message config {
string ip = 1;
string username = 2;
string serialnumber = 3;
};
message userdata {
string username = 1;
};
message ServerMessage {
config = 1;
userdata = 2;
google.protobuf.Any data = 3;
};
I am trying to serialize and send the data like below
ServerMessage server_msg;
google::protobuf::Any any;
server_msg.mutable_userdata()->set_username("test");
any.PackFrom(server_msg);
boost::asio::streambuf sbuf;
ostream serialized(&sbuf);
any.SerializeToOstream(&serialized);
On the receiving side, I try to deserialize like below
userdata msg;
google::protobuf::Any any;
any.ParseFromArray(message.data(), message.size());
if (any.Is<userdata>()) {
std::cout<<"It is a userdata mesage"<<std::endl;
}
if (any.Is<config>()) {
std::cout<<"it is a config message"<<std::endl;
}
if (any.UnpackTo(&msg)) {
std::cout<<msg.username()<<std::endl;
} else {
std::cout<<"could not unpack"<<std::endl;
}
The packing doesn't complain. But it is not able to unpack or determine the message type.
Am I packing it wrong?
I really appreciate your help with this!
I am not an expert on Protobuf, but I think the problem is that you are packing a ServerMessage and unpacking a userdata.
You should unpack a ServerMessage and then access the userdata() member.
Related
I am using gRPC where I want to transfer a struct type object from server to client using protocol buffer. My desire is to directly dump the struct datatype in a message field.
Some of my sample code is given here
server.cpp where I am getting a struct type object from a function.
// sample struct
struct Dummy_Info{
int age;
bool presence;
};
// sample function to return struct
Dummy_Info Pass_Dummy_Info()
{
Dummy_Info obj;
obj.age = 54;
obj.presence = 1;
return obj;
}
sample.proto file which I desire to make(I am only using here the message which will be used for server purpose)
syntax = "proto3";
package abc_xyz;
// Response message from server to client
message Server_Response {
float val = 1;
// The following is my desire
struct variable_for_struct_object = 2;
}
// Request message from client to server
message Client_Request {
string name = 1;
}
service Service {
rpc GetAddress(Client_Request) returns (Server_Response) {}
}
server.cpp
// omitted the unnecessary part intentionally. Suppose I have passed the Server_Response as pointer type object and it is `response`
Dummy_Info result;
result = Pass_Dummy_Info();
// The following works flawlessly
response -> set_val(0.5);
// The following I want to prepare
response -> set_variable_for_struct_object(result);
client.cpp
// Here I need to fetch the struct type data
auto data = response_.variable_for_struct_object();
cout << data.presence << endl;
cout << data.age << endl;
I don't know this direct dumping is possible or not or maybe I am missing concept.
What I am doing right now:
sample.proto
syntax = "proto3";
package abc_xyz;
// The following I have added which mimics my struct type object
message DATA_TYPE{
int32 age = 1;
bool presence = 2;
}
// Response message from server to client
message Server_Response {
float val = 1;
// The following is my desire
DATA_TYPE variable_for_struct_object = 2;
}
// Request message from client to server
message Client_Request {
string name = 1;
}
service Service {
rpc GetAddress(Client_Request) returns (Server_Response) {}
}
After then I have formatted the cpp files as like as follows which is working
server.cpp
Dummy_Info result;
result = Pass_Dummy_Info();
response->mutable_variable_for_struct_object()->set_age(result.age);
client.cpp is as like as before.
Reason of my desire is to avoid declare the same struct 2 times (one time in cpp, later in proto message format). If here is any workaround would be grateful to have the direction.
protobuf holds similar data for struct. For more info about pb vs struct can be found at: https://developers.google.com/protocol-buffers/docs/cpptutorial
syntax = "proto3";
package communication;
message YourData {
int32 d1 = 1;
string d2 = 2;
}
// Response message from server to client
message Response {
int32 result = 1;
// struct obj_data = 2; // I know this will not work and which pushes me to ask the question.
YourData data = 2;
}
// Request message from client to server
message Request {
string name = 1
}
service Dummy_Service {
rpc GetAddress(Request) returns (Response) {}
}
server related code
YourData d;
d.set_d1(1);
d.set_d2("hello");
response->mutable_data()->CopyFrom(d)
client related code
auto data = response_.data();
cout << data.d1() << ":" << data.d2() << endl;
I tried to know what package I got in my tcp socket, so I use protobuf.
But when I SerializeToString my first protobuf class, the ParseFromString method of an other protobuf class returns true.
The two classes are differents
Here are my .proto
syntax = "proto3";
package protobuf;
message Message
{
string content = 1;
}
message Player
{
int32 id = 1;
string name = 2;
}
Here is my c++ code
auto messageProto = new protobuf::Message;
messageProto->set_content("Hello");
std::string data;
messageProto->SerializeToString(&data);
protobuf::Player player;
if (player.ParseFromString(data))
{
qDebug() << "parse player";
}
protobuf::Message message2;
if (message2.ParseFromString(data))
{
qDebug() << "parse message";
}
Output :
parse player
parse message
Why ?
My recommended solution to the problem of multiple different payloads:
syntax = "proto3";
package protobuf;
message RenameMe // the outer payload wrapper
{
oneof payload
{
Foo foo = 1;
Bar bar = 2;
}
}
message Foo // one type of content
{
string content = 1;
}
message Bar // another type of content
{
int32 id = 1;
string name = 2;
}
Now you can just deserialize everything as a RenameMe (naming is hard!) and check the payload discriminated union enum to see how you should interpret the data. Then just access foo or bar respectively.
This approach is clear, obvious, and readily and effectively extensible into additional message types. The testing can be implemented with switch in many programming languages. This style also works well with polymorphism in some environments - for example, in C# with protobuf-net that could be serialized/deserialized with:
[ProtoContract, ProtoInclude(1, typeof(Foo)), ProtoInclude(2, typeof(Bar))]
abstract class RenameMe {}
[ProtoContract]
class Foo : RenameMe {
[ProtoMember(1)] public string Content {get;set;}
}
[ProtoContract]
class Bar : RenameMe {
[ProtoMember(1)] public int Id {get;set;}
[ProtoMember(2)] public string Name {get;set;}
}
EDIT : So, now, is it the best approach ?
syntax = "proto3";
message Header
{
oneof payload
{
Message message = 1;
Player player = 2;
}
}
message Message
{
string content = 1;
}
message Player
{
int32 id = 1;
string name = 2;
}
When I write :
void SocketManager::sendData(Player& player)
{
Header header;
header.set_allocated_player(&player);
write(header);
}
void SocketManager::sendData(Message& message)
{
Header header;
header.set_allocated_message(&message);
write(header);
}
// etc... for each kind of message
When I read :
void read(const std::string& data)
{
protobuf::Header header;
header.ParseFromString(data);
switch (header.payload_case())
{
case protobuf::Header::kMessage:
emit messageProtoReceived(header.message());
break;
case protobuf::Header::kPlayer:
emit playerProtoReceived(header.player());
break;
case protobuf::Header::PAYLOAD_NOT_SET:
qDebug() << "Error, the payload isn't set, please create a header with a payload";
break;
}
}
I have the following proto file
package DS3DExcite.Cpp.ExternalVariantSwitcher.ProtocolBuffers;
message DGCommand
{
extensions 100 to max;
enum Type
{
AttachModel = 1;
AttachModelReply = 2;
....
...
SwitchVariants = 4;
}
required Type type = 1;
required uint32 id = 2;
optional uint32 replyTo = 3 [default = 0];
optional string message = 4;
}
message SwitchVariants
{
extend DGCommand
{
required SwitchVariants command = 103;
}
message VariantState
{
required string variant = 1;
required string state = 2;
}
repeated VariantState variants = 1;
repeated string variantNames = 2;
optional string state = 3;
}
I compiled the proto file with protobuf 2.4.1 version to generate .pb.h and .pb.cc files
Now I form the commands
DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand commandObj;
commandObj.set_type(DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand_Type_SwitchVariants);
commandObj.set_id(3);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants *objVarState;
objVarState = commandObj.MutableExtension(DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants::command);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants_VariantState *state = objVarState->add_variants();
state->set_state("OFF");
state->set_variant("M_Carpaint_3");
I serialise the message
int size = commandObj.ByteSize();
int sizeSize = sizeof(int);
std::vector<char> data(size ,0);
memcpy(data.data(), &size, sizeSize);
data.resize(size + sizeSize );
commandObj.SerializeToArray(static_cast<void*>(&(data[0])+sizeSize) ,size);
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
And I send this message on a Qtcp socket to server which deserializes the message and extract the information from the message .
At the server end this is the code to read
uint32 pendingData = 0;
rcvSocket->HasPendingData(pendingData); //rcvSocket is the serversside socket
if (pendingData == 0)
{
UE_LOG(PVCConnector, Warning, TEXT("Lost connection to client."));
break;
}
TArray<char> newData; //customized Array template
newData.InsertZeroed(0, pendingData);
int32 bytesRead = 0;
rcvSocket->Recv(reinterpret_cast<uint8*>(newData.GetData()), pendingData, bytesRead);
data += newData;
However at the server end the the desired information lands up in the unknown fields of ::google::protobuf::Message . What could be the reason ?
I had a similar problem whe I have been sending big enough messages. We decide, that this happens when message divided into several net packages. We use blobs to prevent that, and it works. About blobs, its technique to send message length, before message
I was able to solve the problem . There were 2 issues
1) The way I was converting to ByteArray
I replaced
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
with
QByteArray *byteArray = new QByteArray(reinterpret_cast<const char*>(data.data()), data.size());
2) The way I was sending the message on the socket . I just used
const int nbBytes = itM->second->write(qByteArray);
instead of using QTextStream
I'm using google protobuf to implement a simple Request/Response based protocol.
A peer can receive via socket both Request and Response, (of course serialized) as string.
I'm using my own c++ socket implementation, so I implemented operator>> this way (the same is for operator<<) to receive data from a socket object:
...
template<class M>
void operator>>(M& m) throw (socks::exception) {
std::string str;
if (!this->recv(str)) {
throw socks::exception(">> failed to retrieve stream via socket");
return;
}
if (!m.ParseFromString(str))
throw socks::exception(
"failed to parse the received stream via socket");
}
So template argument M can be objects Request and Response.
// some lines from req_res.proto
message Request {
required action_t action = 1;
}
enum result_t {
...
}
message Response {
required result_t result = 1;
...
}
How can I determine whether I received a Response or a Request using operator>> this way?
my_socket_object s;
...
for (;;) {
Request|Response r;
s >> r;
...
}
...
You can have one basic Message object and extend all other types used in your protocol from it:
message Message {
extensions 100 to max;
}
message Request {
extends Message {
optional Request request = 100;
}
required action_t action = 1;
}
message Response {
extends Message {
optional Response response = 101;
}
required result_t result = 1;
}
This is a bit more elegant, self contained and easier to extend, than the discriminator/union solution proposed in the other answer IMHO.
You can use this technique even further to structure e.g. your Request/Response messages like this
message Request {
extends Message {
optional Request request = 100;
}
extensions 100 to max;
}
message Action1 {
extends Request {
optional Action1 action1 = 100;
}
optional int32 param1 = 1;
optional int32 param2 = 2;
}
message Action2 {
extends Request {
optional Action2 action2 = 101;
}
optional int32 param1 = 1;
optional int32 param2 = 2;
}
One possible approach is to put any possible message inside another high level message as a kind of tagged-union:
enum protocolmessage_t {
Request = 1;
Response = 2;
}
message ProtocolMessage {
required protocolmessage_t type = 1;
optional Request request = 10;
optional Response response = 11;
}
Then you would provide this ProtocolMessage as the M parameter to the >> operator and you can check the type and extract the corresponding value element.
An alternative way is to prefix every message with 1 byte for the type, make a switch on that type and then call your >> operator with the corresponding type.
I wanted to know if it was possible to use a map with a google protocol buffer.
I currently have something like this in my .proto file
message MsgA
{
required string symbol = 1 ;
optional int32 freq = 2 [default = 0];
}
message MsgB
{
//What should I do to make a map<int,MsgA>
}
My question is in MsgB i would like to create a type that would be a map::
Any suggestion on how I could accomplish this ?
Do this:
message MapEntry
{
required int32 mapKey = 1;
required MsgA mapValue = 2;
}
message MsgB
{
repeated MapEntry = 1;
}
You will have to write your own code to convert the map to and from a MsgB, but that should be basically trivial.