In my existing ZeroMQ application am playing to change my payload to google protobuf. During Initial analysis problem I am facing is i will have a multiple proto classes,which can be written to log file also. Now in runtime how do I determine which proto is received. And also later at some point if I read the log file here also I will face the same problem. Is there any solution to this? In both proto2 and proto3
Protocol Buffer does support multiple class definition. The way you do it is by declaring all the types of classes you have in your Protocol Format file.
In this example we declare two different classes and their properties:
syntax = "proto2";
package tutorial;
message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;
}
message House {
repeated string owner = 1;
}
If you are wailing to use protocol buffer as your standard communication protocols I guess you want to send one of the classes. See the official docs where they explain how to deal with this situation in this post.
Another good pattern is to create a unique message (Wrapper) that holds common properties and one of the classes:
message WrapperMessage {
required int64 timestmap = 1;
oneof data {
Person person = 2;
House house = 3;
}
}
Then before decoding the class you can check which type of class the wrapper holds by calling HasField.
Note: Protocol Buffers 3 implements a new feature for this purpose based on the FileDescriptorSet and Any property.
Related
I'd like to understand how to transmit the contents of a C++ class between processes or across a network.
I'm reading the Google Protobuf tutorial:
https://developers.google.com/protocol-buffers/docs/cpptutorial
and it seems you must create an abstracted, non-C++ interface to represent your class:
syntax = "proto2";
package tutorial;
message Person {
optional string name = 1;
optional int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
}
However, I'd prefer to specify my class via C++ code (rather than the abstraction) and just add something like serialize() and deserialize() methods.
Is this possible with Google Protobuf? Or is this how Protobuf works and I'd need to use a different serialization technique?
UPDATE
The reason for this is I don't want to have to maintain two interfaces. I'd prefer to have one C++ class, update it and not have to worry about a second .proto interface/definition. Code maintainability.
That's how Protobuf works. You have to use something else if you want to serialize your manually-written C++ classes. However, I'm not sure you really want that, because you then will have to either restrict yourself to very simple fields with no invariants (just like in Protobuf) or write custom (de)serialization logic yourself.
You could make a simple protocol buffer to hold binary information, but it sort of breaks the point of using Protocol buffers.
You can sort of cheat the system by using SerializeToString() and ParseFromString() to simply serialize binary information into a string.
There is also SerializeToOstream() and ParseFromIstream().
The real value of protocol buffers is being able to use messages across programs, systems and languages while using a single definition. If you aren't making messages using the protocol they've defined; this is more work than simply using native C++ capabilities.
In google protocol buffer, there exists a textual version of message. When parsing this textual message, can we define ourselves the callback functions in order that we could store the information parsed into our own data structure?
For example, if we have defined .proto:
message A {
required string name = 1;
optional string value =2;
repeated B bList =3;
}
message B {
required string name =1;
optional string value =2;
}
And we have textformat message:
A {
name: "x"
value: "123"
B {
name: "y"
value: "987"
}
B {
name: "z"
value: "965"
}
}
The protobuf compiler generates the corresponding class named "A", class named "B". The parser can parse this text format into the instance of A. However, if user want to defined our own version of class "A", or there exists a version of A used before. Now as we would like to replace the old exchange format by google protocol buffer, we are willing to parse the google protocol buffer text format version directly into the old data structure. If not, we will have to first of all have the generated data structure (class "A") filled then adapt the generated data structure to the legacy data structure. It occupies two times the memory than necessary. It can be much less efficient than we wanted.
The traditional method used for integrating a parser is to have a parser that can callback self-defined functors to be accustomed to the new data structure.
So, does there exist a way to inject the self-defined callback function into the text format parser?
No, the protobuf TextFormat implementation does not support such extensions.
That said, TextFormat (in at least C++, Java, and Python) is implemented as a self-contained module that operates only on public interfaces (mainly, the reflection interface). You can easily clone it and then make your own modifications to the format, or even write a whole new module in the same style that implements any arbitrary format. For example, many people have written JSON parsers / encoders based on Protobuf reflection, using the TextFormat implementation as a guide.
I made a ProtocolBuffer object from the proto class I usually use and I need to Serialize it. Now, I take the object and call SerializeToArray() on it like this:
int size = messageObject.ByteSize();
void* buffer = malloc(size);
messageObject.SerializeToArray(buffer, size);
As far as I know there is no problem with this since the object has data in it (I checked it by breaking right before the Serialize line).
When the method calls however it triggers an abort() which I don't know anything about.
I have no idea what it could be. The only data that is included in this object is a "type" enumerator (which I can set to the type of data that is being used in this object since it can include different sorts of messages) and it holds one message object of the repeatable type.
message MessageID
{
enum Type { LOGINDATA = 1; PLAYERDATA = 2; WORLDDATA = 3; }
// Identifies which field is filled in.
required Type type = 1;
// One of the following will be filled in.
repeated PlayerData playerData = 2;
optional WorldData worldData = 3;
optional LoginData loginData = 10;
}
This is the base message. So, Type is 2 in this case which stands for PLAYERDATA. Also, playerData is being set with a single object of the type PlayerData.
An help is appreciated.
Any time that the protobuf library aborts (which, again, should only be in debug mode or in sever circumstances), it will print information about the problem to the console. If your app doesn't have a console, you can use google::protobuf::SetLogHandler to direct the information somewhere else:
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.common#SetLogHandler.details
typedef void LogHandler(LogLevel level, const char* filename,
int line, const std::string& message);
LogHandler* SetLogHandler(LogHandler* new_func);
The protobuf library sometimes writes warning and error messages to stderr.
These messages are primarily useful for developers, but may also help end users figure out a problem. If you would prefer that these messages be sent somewhere other than stderr, call SetLogHandler() to set your own handler. This returns the old handler. Set the handler to NULL to ignore log messages (but see also LogSilencer, below).
Obviously, SetLogHandler is not thread-safe. You should only call it at initialization time, and probably not from library code. If you simply want to suppress log messages temporarily (e.g. because you have some code that tends to trigger them frequently and you know the warnings are not important to you), use the LogSilencer class below.
The only reason for an abort that I know of (which only applies in debug builds) is if some required field isn't set. You say that the type field is set, so there must be a required field in PlayerData which is not set.
Here's what I am trying to figure out, their docs don't explain this well enough, at least to me..
Senario:
I have 5 proto files that I generate with protoc for C++. My application needs to receive a message and then be able to iterate through all the fields while accessing their values and names.
What I would like to do is parse a message into the DynamicMessage class and then do the iteration through the fields. This way I don't have to know exactly what message it is and I can handle them all in a single generic way.
I know it's possible to handle the messages by parsing them to their specific type then treating them as their Message base class but for my application that is not desirable.
It looks like what I want to do should be possible via the "--descriptor_set_out" and dynamic message class.
What I've Tried (And Failed With):
I moved the descriptor.proto into the folder with my protos and included it along side my others in the compilation step. I also set the--descriptor_set_out flag to print to a file "my_descriptors.pb.ds"
I have no idea where to proceed from there.
Here's what i've referenced, although there isn't much...
Sorry for the long post, and somewhat vague topic naming schema.
Also, incase it wasn't clear, I assume the messages aren't "Unknown." I assume there will still be the requirement of including the respective headers for each proto so my code knows about the 'unknown' message its handling.
The most common way is to use message composition. Something like:
message Foo {...}
message Bar {...}
message GenericMessage {
enum Type {FOO = 1, BAR = 2};
optional Foo foo = 1;
optional Bar bar = 2;
}
If you make sure that exactly one of either Foo or Bar is present in each GenericMessage, you get the desired behaviour. You read one GenericMessage and then process it as one of several specific messages.
Think about refactoring the protocol. If all you need to do is iterate over the fields, maybe you'd be better off with something like a simple key-value map:
message ValueMessage {
required string key = 1;
optional int IntValue = 2;
optional string StringValue = 3;
optional bool BoolValue = 4;
...
}
message GenericMessage{
repeated ValueMessage = 1;
}
Or maybe you can refactor you protocol some other way.
Warning: my answer is not completely correct I am having some compilation errors regarding conflicts, i will edit when I fix it :). but this is a starting point
It might have been a long time since this question was posted, but I faced something similar this days now working with Protocol Buffers.
First of all the reference is wrong the option on the command that must be added is:
--descriptor_set_out=<Directory>
where Directory is where your compiled version of the descriptor.proto (or .proto compiled that describes your file) is located.
after this you will have to add the reference to the Descriptor.proto file in your autodescriving .proto file.
message MyMessage
{
required google.protobuf.FileDescriptorSet proto_files = 1;
...
}
There are frameworks for Java and other languages that help connect protocol buffers to JSON, but I have not seen a native solution in C++.
Is there a library/framework that I can use to connect C++ protocol buffer objects to JSON?
I'm developing one. I'm using the protobuf's reflection mechanism to parse any generated protobuf. Here http://corbasim.googlecode.com/svn/trunk/protobuf2json_exported.zip you can find an initial implementation of this idea. It currently just parse string fields, but I want to support any type as soon as possible.
For a message Foo:
message Foo {
optional string text = 1;
}
it can parse instances of Foo by this way:
Foo foo;
const std::string json_foo = "{\"text\": \"Hello world\"}";
protobuf2json::json::parse(foo, json_foo)
By the same way, I want to write a JSON serializer from protobuf generated types.
There is a similar question here:
C++ Protobuf to/from JSON conversion
pb2json is another library that can be used.