I wanted to know if it was possible to use a map with a google protocol buffer.
I currently have something like this in my .proto file
message MsgA
{
required string symbol = 1 ;
optional int32 freq = 2 [default = 0];
}
message MsgB
{
//What should I do to make a map<int,MsgA>
}
My question is in MsgB i would like to create a type that would be a map::
Any suggestion on how I could accomplish this ?
Do this:
message MapEntry
{
required int32 mapKey = 1;
required MsgA mapValue = 2;
}
message MsgB
{
repeated MapEntry = 1;
}
You will have to write your own code to convert the map to and from a MsgB, but that should be basically trivial.
Related
I'm working with Protobuf3 in my C++14 project. There have been some functions, which returns the google::protobuf::Message*s as a rpc request, what I need to do is to set their fields. So I need to use the reflection of Protobuf3.
Here is a proto file:
syntax="proto3";
package srv.user;
option cc_generic_services = true;
message BatchGetUserInfosRequest {
uint64 my_uid = 1;
repeated uint64 peer_uids = 2;
map<string, string> infos = 3;
}
message BatchGetUserInfosResponse {
uint64 my_uid = 1;
string info = 2;
}
Service UserSrv {
rpc BatchGetUserInfos(BatchGetUserInfosRequest) returns (BatchGetUserInfosResponse);
};
Now I called a function, which returns a google::protobuf::Message*, pointing an object BatchGetUserInfosRequest and I try to set its fields.
// msg is a Message*, pointing to an object of BatchGetUserInfosRequest
auto descriptor = msg->GetDescriptor();
auto reflection = msg->GetReflection();
auto field = descriptor->FindFieldByName("my_uid");
reflection->SetUInt64(msg, field, 1234);
auto field2 = descriptor->FindFieldByName("peer_uids");
reflection->GetMutableRepeatedFieldRef<uint64_t>(msg, field2).CopyFrom(peerUids); // peerUids is a std::vector<uint64_t>
As you see, I can set my_uid and peer_uids as above, but for the field infos, which is a google::protobuf::Map, I don't know how to set it with the reflection mechanism.
If you dig deep into the source code, you would find out the map in proto3 is implemented on the RepeatedField:
// Whether the message is an automatically generated map entry type for the
// maps field.
//
// For maps fields:
// map<KeyType, ValueType> map_field = 1;
// The parsed descriptor looks like:
// message MapFieldEntry {
// option map_entry = true;
// optional KeyType key = 1;
// optional ValueType value = 2;
// }
// repeated MapFieldEntry map_field = 1;
//
// Implementations may choose not to generate the map_entry=true message, but
// use a native map in the target language to hold the keys and values.
// The reflection APIs in such implementations still need to work as
// if the field is a repeated message field.
//
// NOTE: Do not set the option in .proto files. Always use the maps syntax
// instead. The option should only be implicitly set by the proto compiler
// parser.
optional bool map_entry = 7;
Inspired by the test code from protobuf, this works for me:
BatchGetUserInfosRequest message;
auto *descriptor = message.GetDescriptor();
auto *reflection = message.GetReflection();
const google::protobuf::FieldDescriptor *fd_map_string_string =
descriptor->FindFieldByName("infos");
const google::protobuf::FieldDescriptor *fd_map_string_string_key =
fd_map_string_string->message_type()->map_key();
const google::protobuf::FieldDescriptor *fd_map_string_string_value =
fd_map_string_string->message_type()->map_value();
const google::protobuf::MutableRepeatedFieldRef<google::protobuf::Message>
mmf_string_string =
reflection->GetMutableRepeatedFieldRef<google::protobuf::Message>(
&message, fd_map_string_string);
std::unique_ptr<google::protobuf::Message> entry_string_string(
google::protobuf::MessageFactory::generated_factory()
->GetPrototype(fd_map_string_string->message_type())
->New(message.GetArena()));
entry_string_string->GetReflection()->SetString(
entry_string_string.get(), fd_map_string_string->message_type()->field(0),
"1234");
entry_string_string->GetReflection()->SetString(
entry_string_string.get(), fd_map_string_string->message_type()->field(1),
std::to_string(10));
mmf_string_string.Add(*entry_string_string);
std::cout << "1234: " << message.infos().at("1234") << '\n';
The output:
1234: 10
I am new to protocol buffers and I am having issues with multiple messages using Any data.
I am not sure if I am using it correctly.
I have the following in message.proto
import "google/protobuf/any.proto";
message config {
string ip = 1;
string username = 2;
string serialnumber = 3;
};
message userdata {
string username = 1;
};
message ServerMessage {
config = 1;
userdata = 2;
google.protobuf.Any data = 3;
};
I am trying to serialize and send the data like below
ServerMessage server_msg;
google::protobuf::Any any;
server_msg.mutable_userdata()->set_username("test");
any.PackFrom(server_msg);
boost::asio::streambuf sbuf;
ostream serialized(&sbuf);
any.SerializeToOstream(&serialized);
On the receiving side, I try to deserialize like below
userdata msg;
google::protobuf::Any any;
any.ParseFromArray(message.data(), message.size());
if (any.Is<userdata>()) {
std::cout<<"It is a userdata mesage"<<std::endl;
}
if (any.Is<config>()) {
std::cout<<"it is a config message"<<std::endl;
}
if (any.UnpackTo(&msg)) {
std::cout<<msg.username()<<std::endl;
} else {
std::cout<<"could not unpack"<<std::endl;
}
The packing doesn't complain. But it is not able to unpack or determine the message type.
Am I packing it wrong?
I really appreciate your help with this!
I am not an expert on Protobuf, but I think the problem is that you are packing a ServerMessage and unpacking a userdata.
You should unpack a ServerMessage and then access the userdata() member.
I have a requirement to build the following Metadata message using serialized key value pairs in C++.
message MetadataValue {
string key = 1;
google.protobuf.Value value = 2;
}
message Metadata {
repeated MetadataValue metadata = 1;
}
So I can have the values for MetadataValue from the following for statement in C++.
Metadata metadata;
if (buffer.has_value()) {
auto pairs = buffer.value()->pairs();
for (auto &p : pairs) {
MetadataValue* metadataValue = metadata.add_metadata();
metadataValue->set_key(std::string(p.first));
// I don't know how to set the value for google.protobuf.Value
}
}
My question is whether my approach is correct ? Are there better alternatives and how to set the google.protobuf.Value in that above scenario ? A simple code snippet with the answer is much appreciated.
I think this code works, I just checked the generated APIs by protoc.
If the typeof(p.second) is not a google::protobuf::Value, you need to add conversion like
auto v = google::protobuf::Value();
v.set_number_value(p.second);
// or p.second is string
// v.set_string_value(p.second);
Metadata metadata;
if (buffer.has_value()) {
auto pairs = buffer.value()->pairs();
for (auto &p : pairs) {
MetadataValue* metadataValue = metadata.add_metadata();
metadataValue->set_key(std::string(p.first));
*metadataValue->mutable_value() = p.second;
// I don't know how to set the value for google.protobuf.Value
}
}
And I am using protoc version 3
syntax = "proto3";
import "google/protobuf/struct.proto";
message MetadataValue {
string key = 1;
google.protobuf.Value value = 2;
}
message Metadata {
repeated MetadataValue metadata = 1;
}
I have the following proto file
package DS3DExcite.Cpp.ExternalVariantSwitcher.ProtocolBuffers;
message DGCommand
{
extensions 100 to max;
enum Type
{
AttachModel = 1;
AttachModelReply = 2;
....
...
SwitchVariants = 4;
}
required Type type = 1;
required uint32 id = 2;
optional uint32 replyTo = 3 [default = 0];
optional string message = 4;
}
message SwitchVariants
{
extend DGCommand
{
required SwitchVariants command = 103;
}
message VariantState
{
required string variant = 1;
required string state = 2;
}
repeated VariantState variants = 1;
repeated string variantNames = 2;
optional string state = 3;
}
I compiled the proto file with protobuf 2.4.1 version to generate .pb.h and .pb.cc files
Now I form the commands
DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand commandObj;
commandObj.set_type(DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand_Type_SwitchVariants);
commandObj.set_id(3);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants *objVarState;
objVarState = commandObj.MutableExtension(DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants::command);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants_VariantState *state = objVarState->add_variants();
state->set_state("OFF");
state->set_variant("M_Carpaint_3");
I serialise the message
int size = commandObj.ByteSize();
int sizeSize = sizeof(int);
std::vector<char> data(size ,0);
memcpy(data.data(), &size, sizeSize);
data.resize(size + sizeSize );
commandObj.SerializeToArray(static_cast<void*>(&(data[0])+sizeSize) ,size);
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
And I send this message on a Qtcp socket to server which deserializes the message and extract the information from the message .
At the server end this is the code to read
uint32 pendingData = 0;
rcvSocket->HasPendingData(pendingData); //rcvSocket is the serversside socket
if (pendingData == 0)
{
UE_LOG(PVCConnector, Warning, TEXT("Lost connection to client."));
break;
}
TArray<char> newData; //customized Array template
newData.InsertZeroed(0, pendingData);
int32 bytesRead = 0;
rcvSocket->Recv(reinterpret_cast<uint8*>(newData.GetData()), pendingData, bytesRead);
data += newData;
However at the server end the the desired information lands up in the unknown fields of ::google::protobuf::Message . What could be the reason ?
I had a similar problem whe I have been sending big enough messages. We decide, that this happens when message divided into several net packages. We use blobs to prevent that, and it works. About blobs, its technique to send message length, before message
I was able to solve the problem . There were 2 issues
1) The way I was converting to ByteArray
I replaced
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
with
QByteArray *byteArray = new QByteArray(reinterpret_cast<const char*>(data.data()), data.size());
2) The way I was sending the message on the socket . I just used
const int nbBytes = itM->second->write(qByteArray);
instead of using QTextStream
I'm new to protobuf and I'm stuck with simple task: I need to iterate through fields of message and check it's type. If type is message I will do same recursively for this message.
For example, I have such messages:
package MyTool;
message Configuration {
required GloablSettings globalSettings = 1;
optional string option1 = 2;
optional int32 option2 = 3;
optional bool option3 = 4;
}
message GloablSettings {
required bool option1 = 1;
required bool option2 = 2;
required bool option3 = 3;
}
Now, to explicitly access a field value in C++ I can do this:
MyTool::Configuration config;
fstream input("config", ios::in | ios::binary);
config.ParseFromIstream(&input);
bool option1val = config.globalSettings().option1();
bool option2val = config.globalSettings().option2();
and so on. This approach is not convenient in case when have big amount of fields.
Can I do this with iteration and get field's name and type? I know there are descriptors of type and somewhat called reflection, but I didn't have success in my attempts.
Can some one give me example of code if it's possible?
Thanks!
This is old but maybe someone will benefit. Here is a method that prints the contents of a protobuf message:
void Example::printMessageContents(std::shared_ptr<google::protobuf::Message> m)
{
const Descriptor *desc = m->GetDescriptor();
const Reflection *refl = m->GetReflection();
int fieldCount= desc->field_count();
fprintf(stderr, "The fullname of the message is %s \n", desc->full_name().c_str());
for(int i=0;i<fieldCount;i++)
{
const FieldDescriptor *field = desc->field(i);
fprintf(stderr, "The name of the %i th element is %s and the type is %s \n",i,field->name().c_str(),field->type_name());
}
}
You can find in FieldDescriptor Enum Values the possible values you get from field->type. For example for the message type you would have to check if type is equal to FieldDescriptor::TYPE_MESSAGE.
This function prints all the "metadata" of the protobuf message. However you need to check separately for each value what the type is and then call the corresponding getter function using Reflection.
So using this condition we could extract the strings :
if(field->type() == FieldDescriptor::TYPE_STRING && !field->is_repeated())
{
std::string g= refl->GetString(*m, field);
fprintf(stderr, "The value is %s ",g.c_str());
}
However fields can be either repeated or not repeated and different methods are used for both field types. So a check is used here to assure that we are using the right method. For repeated fields we have for example this method for strings :
GetRepeatedString(const Message & message, const FieldDescriptor * field, int index)
So it takes the index of the repeated field into consideration.
In the case of FieldDescriptor of type Message, the function provided will only print the name of the message, we better print its contents too.
if(field->type()==FieldDescriptor::TYPE_MESSAGE)
{
if(!field->is_repeated())
{
const Message &mfield = refl->GetMessage(*m, field);
Message *mcopy = mfield.New();
mcopy->CopyFrom(mfield);
void *ptr = new std::shared_ptr<Message>(mcopy);
std::shared_ptr<Message> *m =
static_cast<std::shared_ptr<Message> *>(ptr);
printMessageContents(*m);
}
}
And finally if the field is repeated you will have to call the FieldSize method on the reflection and iterate all repeated fields.
Take a look at how the Protobuf library implements the TextFormat::Printer class, which uses descriptors and reflection to iterate over fields and convert them to text:
https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1473