How to use reflection of Protobuf to modify a Map - c++

I'm working with Protobuf3 in my C++14 project. There have been some functions, which returns the google::protobuf::Message*s as a rpc request, what I need to do is to set their fields. So I need to use the reflection of Protobuf3.
Here is a proto file:
syntax="proto3";
package srv.user;
option cc_generic_services = true;
message BatchGetUserInfosRequest {
uint64 my_uid = 1;
repeated uint64 peer_uids = 2;
map<string, string> infos = 3;
}
message BatchGetUserInfosResponse {
uint64 my_uid = 1;
string info = 2;
}
Service UserSrv {
rpc BatchGetUserInfos(BatchGetUserInfosRequest) returns (BatchGetUserInfosResponse);
};
Now I called a function, which returns a google::protobuf::Message*, pointing an object BatchGetUserInfosRequest and I try to set its fields.
// msg is a Message*, pointing to an object of BatchGetUserInfosRequest
auto descriptor = msg->GetDescriptor();
auto reflection = msg->GetReflection();
auto field = descriptor->FindFieldByName("my_uid");
reflection->SetUInt64(msg, field, 1234);
auto field2 = descriptor->FindFieldByName("peer_uids");
reflection->GetMutableRepeatedFieldRef<uint64_t>(msg, field2).CopyFrom(peerUids); // peerUids is a std::vector<uint64_t>
As you see, I can set my_uid and peer_uids as above, but for the field infos, which is a google::protobuf::Map, I don't know how to set it with the reflection mechanism.

If you dig deep into the source code, you would find out the map in proto3 is implemented on the RepeatedField:
// Whether the message is an automatically generated map entry type for the
// maps field.
//
// For maps fields:
// map<KeyType, ValueType> map_field = 1;
// The parsed descriptor looks like:
// message MapFieldEntry {
// option map_entry = true;
// optional KeyType key = 1;
// optional ValueType value = 2;
// }
// repeated MapFieldEntry map_field = 1;
//
// Implementations may choose not to generate the map_entry=true message, but
// use a native map in the target language to hold the keys and values.
// The reflection APIs in such implementations still need to work as
// if the field is a repeated message field.
//
// NOTE: Do not set the option in .proto files. Always use the maps syntax
// instead. The option should only be implicitly set by the proto compiler
// parser.
optional bool map_entry = 7;
Inspired by the test code from protobuf, this works for me:
BatchGetUserInfosRequest message;
auto *descriptor = message.GetDescriptor();
auto *reflection = message.GetReflection();
const google::protobuf::FieldDescriptor *fd_map_string_string =
descriptor->FindFieldByName("infos");
const google::protobuf::FieldDescriptor *fd_map_string_string_key =
fd_map_string_string->message_type()->map_key();
const google::protobuf::FieldDescriptor *fd_map_string_string_value =
fd_map_string_string->message_type()->map_value();
const google::protobuf::MutableRepeatedFieldRef<google::protobuf::Message>
mmf_string_string =
reflection->GetMutableRepeatedFieldRef<google::protobuf::Message>(
&message, fd_map_string_string);
std::unique_ptr<google::protobuf::Message> entry_string_string(
google::protobuf::MessageFactory::generated_factory()
->GetPrototype(fd_map_string_string->message_type())
->New(message.GetArena()));
entry_string_string->GetReflection()->SetString(
entry_string_string.get(), fd_map_string_string->message_type()->field(0),
"1234");
entry_string_string->GetReflection()->SetString(
entry_string_string.get(), fd_map_string_string->message_type()->field(1),
std::to_string(10));
mmf_string_string.Add(*entry_string_string);
std::cout << "1234: " << message.infos().at("1234") << '\n';
The output:
1234: 10

Related

How to set a protobuf message to a oneof struct

Assuming I have this proto
message inner_body1{
... // some attr
}
message inner_body2{
... // some attr
}
message body {
oneof inner{
inner_body1 = 1;
inner_body2 = 2;
}
}
message head {
... // some attr
}
message pkg{
head h = 1;
body b = 2;
}
And I design a function like this
void SendPkg(proto::Message& data)
{
pkg p;
auto head = p.mutable_head();
head->fillsomething(); // not important
// My question is, if 'data' is definitely one of the message type defined in 'body'(e.g. 'inner_body1')
// How can I put 'data' into pkg's body field?
}
My question is, if 'data' is definitely one of the message type defined in 'body'(e.g. 'inner_body1')
How can I put 'data' into pkg's body field?
update:
I have tried this way
void SendPkg(proto::Message& data)
{
pkg p;
auto head = p.mutable_head();
head->fillsomething(); // not important
auto body = p.mutable_body();
const Descriptor* desc = data.GetDescriptor();
if (desc.name() == "inner_body1")
{
auto body1 = body->mutable_innerbody1();
body1.CopyFrom(data);
}
else
{
auto body2 = body->mutable_innerbody2();
body2.CopyFrom(data);
}
}
this may works. But the fallback is obviously. I have to maintain this ugly string mapping and it running effienciency is low.
Is there any way could achieve this more elegant?

Google protobuf repeated fields with C++

I have a requirement to build the following Metadata message using serialized key value pairs in C++.
message MetadataValue {
string key = 1;
google.protobuf.Value value = 2;
}
message Metadata {
repeated MetadataValue metadata = 1;
}
So I can have the values for MetadataValue from the following for statement in C++.
Metadata metadata;
if (buffer.has_value()) {
auto pairs = buffer.value()->pairs();
for (auto &p : pairs) {
MetadataValue* metadataValue = metadata.add_metadata();
metadataValue->set_key(std::string(p.first));
// I don't know how to set the value for google.protobuf.Value
}
}
My question is whether my approach is correct ? Are there better alternatives and how to set the google.protobuf.Value in that above scenario ? A simple code snippet with the answer is much appreciated.
I think this code works, I just checked the generated APIs by protoc.
If the typeof(p.second) is not a google::protobuf::Value, you need to add conversion like
auto v = google::protobuf::Value();
v.set_number_value(p.second);
// or p.second is string
// v.set_string_value(p.second);
Metadata metadata;
if (buffer.has_value()) {
auto pairs = buffer.value()->pairs();
for (auto &p : pairs) {
MetadataValue* metadataValue = metadata.add_metadata();
metadataValue->set_key(std::string(p.first));
*metadataValue->mutable_value() = p.second;
// I don't know how to set the value for google.protobuf.Value
}
}
And I am using protoc version 3
syntax = "proto3";
import "google/protobuf/struct.proto";
message MetadataValue {
string key = 1;
google.protobuf.Value value = 2;
}
message Metadata {
repeated MetadataValue metadata = 1;
}

Setting a protobuf item's value replaces a previously set variable as well

I have created a simple Protobuf based config file and have been using it without any issues until now. The issue is that I added two new items to my settings (Config.proto) and now whatever value I set for the last variable is reflected in the previous one.
The following snapshot demonstrates this better. As you can see below, the value of fv_shape_predictor_path and fv_eyenet_path depend solely on order of being set. the one that is set last changes the others value.
I made sure the cpp files related to Config.proto are built afresh. I also tested this under Linux and there it works just fine. It seems its only happening in windows! it also doesn't affect any other items in the same settings. its just these two new ones.
I have no idea what is causing this or how to go about it. For reference this is how the protobuf looks like:
syntax = "proto3";
package FVConfig;
message Config {
FV Configuration = 4 ;
message FAN_MODELS_WEIGHTS{
string fan_2DFAN_4 = 1;
string fan_3DFAN_4 = 2;
string fan_depth = 3;
}
message S3FD_MODELS_WEIGHTS{
string s3fd = 1;
}
message DLIB_MODELS_WEIGHTS{
string dlib_default = 1;
}
message MTCNN_MODELS_WEIGHTS {
string mt_onet = 1;
string mt_pnet = 2;
string mt_rnet = 3;
}
message FV_MODEL_WEIGHTS {
string r18 = 1;
string r50 = 2;
string r101 = 3;
repeated ModelContainer new_models_weights = 4;
message ModelContainer{
string model_name = 1;
string model_weight_path = 2;
string description = 3;
}
}
message FV {
MTCNNDetectorSettings mtcnn = 1 ;
FaceVerificationSettings fv = 2 ;
}
message MTCNNDetectorSettings {
Settings settings = 1;
MTCNN_MODELS_WEIGHTS model_weights = 4;
message Settings {
string mt_device = 2;
int32 mt_webcam_source = 100;
int32 mt_upper_threshold = 600;
int32 mt_hop = 700;
}
}
message FaceVerificationSettings {
Settings settings = 1;
FV_MODEL_WEIGHTS model_weights = 2;
message Settings {
string fv_model_name = 1;
string fv_model_checkpoint_path = 2;
bool fv_rebuild_cache = 3;
bool fv_short_circut = 6;
bool fv_accumulate_score = 7;
string fv_config_file_path = 10;
string fv_img_bank_folder_root = 11;
string fv_cache_folder = 12;
string fv_postfix = 13;
string fv_device = 14;
int32 fv_idle_interval = 15;
bool fv_show_dbg_info = 16;
// these are the new ones
string fv_shape_predictor_path = 17;
string fv_eyenet_path = 18;
}
}
} //end of Config message
What am I missing here? How should I be going about this? Restarting Windows and Visual Studio didn't do any good either. I'm using protobuf 3.11.4 both on Linux and Windows.
This issue seems to only exist in the Windows version of Protobuf 3.11.4 (didn't test with any newer version though).
Basically what happened was that I use to first create a Config object and initialize it with some default values. When I added these two entries to my Config.proto, I forgot to also add an initialization entry like other entries, thinking I'm fine with the default (which I assumed would be "").
This doesn't pose any issues under Linux/G++ and the program builds and runs just fine and works as intended. However under Windows this results in the behavior you just witnessed i.e. setting any of those newly added entries, would also set the other entries values. So basically I either had to create a whole new Config object or had to explicitly set their values when using the load_default_config.
To be more concrete this is the snippet I used for setting some default values in my Protobuf configs.
These reside in a separate header called Utility.h:
inline FVConfig::Config load_default_config()
{
FVConfig::Config baseCfg;
auto config = baseCfg.mutable_configuration();
load_fv_default_settings(config->mutable_fv());
load_mtcnn_default_settings(config->mutable_mtcnn());
return baseCfg;
}
inline void load_fv_default_settings(FVConfig::Config_FaceVerificationSettings* fv)
{
fv->mutable_settings()->set_fv_config_file_path(fv::config_file_path);
fv->mutable_settings()->set_fv_device(fv::device);
fv->mutable_settings()->set_fv_rebuild_cache(fv::rebuild_cache);
...
// these two lines were missing previously and to my surprise, this was indeed
// the cause of the weird behavior.
fv->mutable_settings()->set_fv_shape_predictor_path(fv::shape_predictor_path);
fv->mutable_settings()->set_fv_eyenet_path(fv::eyenet_path);
auto new_model_list = fv->mutable_model_weights()->mutable_new_models_weights()->Add();
new_model_list->set_model_name("r18");
new_model_list->set_description("default");
new_model_list->set_model_weight_path(fv::model_weights_r18);
}
inline void load_mtcnn_default_settings(FVConfig::Config_MTCNNDetectorSettings* mt)
{
mt->mutable_settings()->set_mt_device(mtcnn::device);
mt->mutable_settings()->set_mt_hop(mtcnn::hop);
....
}
Not sure this counts as a bug in Protobuf, or my wrong approach here.

c++ protobuf: how to iterate through fields of message?

I'm new to protobuf and I'm stuck with simple task: I need to iterate through fields of message and check it's type. If type is message I will do same recursively for this message.
For example, I have such messages:
package MyTool;
message Configuration {
required GloablSettings globalSettings = 1;
optional string option1 = 2;
optional int32 option2 = 3;
optional bool option3 = 4;
}
message GloablSettings {
required bool option1 = 1;
required bool option2 = 2;
required bool option3 = 3;
}
Now, to explicitly access a field value in C++ I can do this:
MyTool::Configuration config;
fstream input("config", ios::in | ios::binary);
config.ParseFromIstream(&input);
bool option1val = config.globalSettings().option1();
bool option2val = config.globalSettings().option2();
and so on. This approach is not convenient in case when have big amount of fields.
Can I do this with iteration and get field's name and type? I know there are descriptors of type and somewhat called reflection, but I didn't have success in my attempts.
Can some one give me example of code if it's possible?
Thanks!
This is old but maybe someone will benefit. Here is a method that prints the contents of a protobuf message:
void Example::printMessageContents(std::shared_ptr<google::protobuf::Message> m)
{
const Descriptor *desc = m->GetDescriptor();
const Reflection *refl = m->GetReflection();
int fieldCount= desc->field_count();
fprintf(stderr, "The fullname of the message is %s \n", desc->full_name().c_str());
for(int i=0;i<fieldCount;i++)
{
const FieldDescriptor *field = desc->field(i);
fprintf(stderr, "The name of the %i th element is %s and the type is %s \n",i,field->name().c_str(),field->type_name());
}
}
You can find in FieldDescriptor Enum Values the possible values you get from field->type. For example for the message type you would have to check if type is equal to FieldDescriptor::TYPE_MESSAGE.
This function prints all the "metadata" of the protobuf message. However you need to check separately for each value what the type is and then call the corresponding getter function using Reflection.
So using this condition we could extract the strings :
if(field->type() == FieldDescriptor::TYPE_STRING && !field->is_repeated())
{
std::string g= refl->GetString(*m, field);
fprintf(stderr, "The value is %s ",g.c_str());
}
However fields can be either repeated or not repeated and different methods are used for both field types. So a check is used here to assure that we are using the right method. For repeated fields we have for example this method for strings :
GetRepeatedString(const Message & message, const FieldDescriptor * field, int index)
So it takes the index of the repeated field into consideration.
In the case of FieldDescriptor of type Message, the function provided will only print the name of the message, we better print its contents too.
if(field->type()==FieldDescriptor::TYPE_MESSAGE)
{
if(!field->is_repeated())
{
const Message &mfield = refl->GetMessage(*m, field);
Message *mcopy = mfield.New();
mcopy->CopyFrom(mfield);
void *ptr = new std::shared_ptr<Message>(mcopy);
std::shared_ptr<Message> *m =
static_cast<std::shared_ptr<Message> *>(ptr);
printMessageContents(*m);
}
}
And finally if the field is repeated you will have to call the FieldSize method on the reflection and iterate all repeated fields.
Take a look at how the Protobuf library implements the TextFormat::Printer class, which uses descriptors and reflection to iterate over fields and convert them to text:
https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1473

using std::map<int , std::string> with Google Protocol Buffer

I wanted to know if it was possible to use a map with a google protocol buffer.
I currently have something like this in my .proto file
message MsgA
{
required string symbol = 1 ;
optional int32 freq = 2 [default = 0];
}
message MsgB
{
//What should I do to make a map<int,MsgA>
}
My question is in MsgB i would like to create a type that would be a map::
Any suggestion on how I could accomplish this ?
Do this:
message MapEntry
{
required int32 mapKey = 1;
required MsgA mapValue = 2;
}
message MsgB
{
repeated MapEntry = 1;
}
You will have to write your own code to convert the map to and from a MsgB, but that should be basically trivial.