I made a ProtocolBuffer object from the proto class I usually use and I need to Serialize it. Now, I take the object and call SerializeToArray() on it like this:
int size = messageObject.ByteSize();
void* buffer = malloc(size);
messageObject.SerializeToArray(buffer, size);
As far as I know there is no problem with this since the object has data in it (I checked it by breaking right before the Serialize line).
When the method calls however it triggers an abort() which I don't know anything about.
I have no idea what it could be. The only data that is included in this object is a "type" enumerator (which I can set to the type of data that is being used in this object since it can include different sorts of messages) and it holds one message object of the repeatable type.
message MessageID
{
enum Type { LOGINDATA = 1; PLAYERDATA = 2; WORLDDATA = 3; }
// Identifies which field is filled in.
required Type type = 1;
// One of the following will be filled in.
repeated PlayerData playerData = 2;
optional WorldData worldData = 3;
optional LoginData loginData = 10;
}
This is the base message. So, Type is 2 in this case which stands for PLAYERDATA. Also, playerData is being set with a single object of the type PlayerData.
An help is appreciated.
Any time that the protobuf library aborts (which, again, should only be in debug mode or in sever circumstances), it will print information about the problem to the console. If your app doesn't have a console, you can use google::protobuf::SetLogHandler to direct the information somewhere else:
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.common#SetLogHandler.details
typedef void LogHandler(LogLevel level, const char* filename,
int line, const std::string& message);
LogHandler* SetLogHandler(LogHandler* new_func);
The protobuf library sometimes writes warning and error messages to stderr.
These messages are primarily useful for developers, but may also help end users figure out a problem. If you would prefer that these messages be sent somewhere other than stderr, call SetLogHandler() to set your own handler. This returns the old handler. Set the handler to NULL to ignore log messages (but see also LogSilencer, below).
Obviously, SetLogHandler is not thread-safe. You should only call it at initialization time, and probably not from library code. If you simply want to suppress log messages temporarily (e.g. because you have some code that tends to trigger them frequently and you know the warnings are not important to you), use the LogSilencer class below.
The only reason for an abort that I know of (which only applies in debug builds) is if some required field isn't set. You say that the type field is set, so there must be a required field in PlayerData which is not set.
Related
I have a problem on getting message id and set it on reply's correlation id.
Here's the piece of code:
MQBYTE msgID;
request_msg.messageId().copyOut(msgID, MQ_MSG_ID_LENGTH, 0);
response_msg.setCorrelationId(msgID);
when I checked the correlation Id of the reply, the correlation id is 0.
How can I copy/get the messageId of the request and put it on the correlation id of the reply?
Thanks in advance. :)
The Infocenter page Manipulating binary strings in C++ shows the data types involved as ImqBinary and MQBYTE24. As Shashi notes in the comments, MQBYTE is a single byte and incapable of containing a 24-byte message ID. The page linked above provides a reference example:
#include <imqi.hpp> // C++ classes
ImqMessage message ;
ImqBinary id, correlationId ;
MQBYTE24 byteId ;
correlationId.set( byteId, sizeof( byteId ) ); // Set.
id = message.id( ); // Assign.
if ( correlationId == id ) { // Compare.
...
One tool to diagnose issues like this is the very useful SupportPac MA0W which is an API exit. It can show you the exact contents of the structures passed to MQ before and after an API call. Very often what is seen is that what is expected (in this case having copied the MQMD.MsgID to the MQMD.CorrelID is not what actually happened. In this case, I believe a trace such as that provided by MA0W would reveal that the msgID was accurately passed from the app to MQ but consisted of only one character.
UPDATE
OP asks:
Variable id is an ImqBinary and message is an ImqMessage
object, from what I know id is not a member of ImqMessage like
messageId, correlationId, groupId and etc. so how can
message.id() passed it's value on id?
You are coorrect that id is a declared variable of type ImqBinary and not a member of the ImqMessage class. Please see the Infocenter page for the ImqBinary C++ class which explains that:
This class encapsulates a binary byte array that can be used for
ImqMessage accounting token, correlation id, and message id values. It
allows easy assignment, copying, and comparison.
The intent of the ImqBinary class is to provide a variable type to encapsulate a byte array with overloaded methods such that "normal" variable manipulations work as expected. Rather than copying the byte array one byte at a time, it can be an LVAL or RVAL in an assignment. Rather than comparing the array a byte at a time, you can just use comparison operators like ==.
So if your code were modified to use the example, it might look like this:
#include <imqi.hpp> // C++ classes
ImqMessage request_msg, response_msg ;
ImqBinary id ;
id = request_msg.id( );
response_msg.setCorrelationId(id);
That said, I'm not sure you even need an intermediate variable. You might be able to assign the correlation ID using the output of the getter call for the message ID from the source message. Something like....
response_msg.setCorrelationId( request_msg.id( ) );
...might do it. I don't code C or C++ anymore so I probably got the syntax wrong or didn't do it as elegantly as it might be coded, but you should get the idea.
Is there a way to create a protocol buffer message in C++ that contains a pre-encoded inner message, without parsing and then re-serializing the inner message?
To clarify, consider the following message definitions:
message Inner {
required int i = 1;
// ... more fields ...
}
message Outer {
repeated Inner inners = 1;
// ... more fields ...
}
Suppose you have a collection of 10 byte arrays, each of which contains an encoded version of an Inner. You'd like to create an Outer that contains the 10 Inners. You don't want to hand-encode because Outer has other fields and may itself be included in other messages. Is there a way to get protocol buffers to directly copy the pre-encoded Inner?
There is no a clean way, but there are a few hacky ways. One is to define a second message like this:
message RawOuter {
repeated bytes inners = 1;
// ... same fields as Outer ...
}
RawOuter is identical to Outer except that the inners repeated field has been changed from type Inner to type bytes. If you populate inners with the encoded instances of Inner, then serialize the RawOuter, you get exactly the same result as if you had built an Outer with the parsed verisons. That is to say, the wire format for a nested message is identical to the wire format for a bytes field containing the serialization of that nested message. This is one of those funny exploitable quirks of the protobuf encoding.
This hack has some problems, though. In particular, it doesn't work well if you're trying to build an Outer instance that is embedded in some other proto, since you probably don't want to maintain two copies of every containing message, one using Outer and one using RawOuter.
Another, even hackier option is to inject the encoded messages into the Outer instance's UnknownFieldSet.
Outer outer;
for (auto& inner: inners) {
outer.mutable_unknown_fields()
->AddLengthDelimited(1, inner);
}
The UnknownFieldSet is intended to store fields seen while parsing that do not match any known field number defined in the .proto file. The idea is that this allows you to write a proxy server that simply receives messages and forwards them to another server without having to re-compile the proxy every time you add a new field to the protocol. Here, we're abusing it by sticking a value into it that actually corresponds to a known field, but the implementation will not notice, and so it will write out these fields just fine.
The main problem with this approach is that if anyone else inspects your Outer instance in the meantime, it will appear to them as if the inners list is empty, since the values are actually hidden somewhere else. This is a pretty ugly hack that will probably come back to haunt you later. I would only recommend it if you have measured the performance difference and found it to be large.
Also note that the serialization code always writes unknown fields last, whereas known fields are written in order by field number. Parsers are supposed to accept any order, but occasionally you'll find someone who is using the unparsed data as a hash map key or something and that totally breaks if the fields are re-ordered.
By the way, you can improve performance of both of these approaches by swapping the strings into place rather than copying, i.e.
raw_outer->add_inners()->swap(inner);
or
outer->mutable_unknown_fields()->AddLengthDelimited(1)->swap(inner);
I am new to protobuf and I have started considering the following trivial example
message Entry {
required int32 id = 1;
}
used by the c++ code
#include <iostream>
#include "example.pb.h"
int main() {
std::string mySerialized;
{
Entry myEntry;
std::cout << "Serialization succesfull "
<< myEntry.SerializeToString(&mySerialized) << std::endl;
std::cout << mySerialized.size() << std::endl;
}
Entry myEntry;
std::cout << "Deserialization successfull "
<< myEntry.ParseFromString(mySerialized) << std::endl;
}
Even if the "id" field is required, since it has not been set, the size of the serialization buffer is 0 (??).
When I deserialize the message an error occurs:
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of type "Entry" because it is missing required fields: id
Is it a normal behavior?
Francesco
ps- If I initialize "id" with the value 0, the behavior is different
pps- The SerializeToString function returns true, the ParseFromString returns false
I dont think I exactly understand your question, but I'll have a go at the answer anyways. Hope this helps you in some way or the other :)
Yes this is normal behavior. You should add required only if the field is important to the message. It makes sense semantically. (why would you skip a required field). To enforce this, protobuf would not parse the message.
It sees that the field marked with number 1 is required, and the has_id() method is returning false. So it wont parse the message at all.
In the developer guide it is advised not to use required fields.
Required Is Forever You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field – old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some engineers at Google have come to the conclusion that using required does more harm than good; they prefer to use only optional and repeated. However, this view is not universal.
Also
Any new fields that you add should be optional or repeated. This means that any messages serialized by code using your "old" message format can be parsed by your new generated code, as they won't be missing any required elements. You should set up sensible default values for these elements so that new code can properly interact with messages generated by old code. Similarly, messages created by your new code can be parsed by your old code: old binaries simply ignore the new field when parsing. However, the unknown fields are not discarded, and if the message is later serialized, the unknown fields are serialized along with it – so if the message is passed on to new code, the new fields are still available. Note that preservation of unknown fields is currently not available for Python
Here's what I am trying to figure out, their docs don't explain this well enough, at least to me..
Senario:
I have 5 proto files that I generate with protoc for C++. My application needs to receive a message and then be able to iterate through all the fields while accessing their values and names.
What I would like to do is parse a message into the DynamicMessage class and then do the iteration through the fields. This way I don't have to know exactly what message it is and I can handle them all in a single generic way.
I know it's possible to handle the messages by parsing them to their specific type then treating them as their Message base class but for my application that is not desirable.
It looks like what I want to do should be possible via the "--descriptor_set_out" and dynamic message class.
What I've Tried (And Failed With):
I moved the descriptor.proto into the folder with my protos and included it along side my others in the compilation step. I also set the--descriptor_set_out flag to print to a file "my_descriptors.pb.ds"
I have no idea where to proceed from there.
Here's what i've referenced, although there isn't much...
Sorry for the long post, and somewhat vague topic naming schema.
Also, incase it wasn't clear, I assume the messages aren't "Unknown." I assume there will still be the requirement of including the respective headers for each proto so my code knows about the 'unknown' message its handling.
The most common way is to use message composition. Something like:
message Foo {...}
message Bar {...}
message GenericMessage {
enum Type {FOO = 1, BAR = 2};
optional Foo foo = 1;
optional Bar bar = 2;
}
If you make sure that exactly one of either Foo or Bar is present in each GenericMessage, you get the desired behaviour. You read one GenericMessage and then process it as one of several specific messages.
Think about refactoring the protocol. If all you need to do is iterate over the fields, maybe you'd be better off with something like a simple key-value map:
message ValueMessage {
required string key = 1;
optional int IntValue = 2;
optional string StringValue = 3;
optional bool BoolValue = 4;
...
}
message GenericMessage{
repeated ValueMessage = 1;
}
Or maybe you can refactor you protocol some other way.
Warning: my answer is not completely correct I am having some compilation errors regarding conflicts, i will edit when I fix it :). but this is a starting point
It might have been a long time since this question was posted, but I faced something similar this days now working with Protocol Buffers.
First of all the reference is wrong the option on the command that must be added is:
--descriptor_set_out=<Directory>
where Directory is where your compiled version of the descriptor.proto (or .proto compiled that describes your file) is located.
after this you will have to add the reference to the Descriptor.proto file in your autodescriving .proto file.
message MyMessage
{
required google.protobuf.FileDescriptorSet proto_files = 1;
...
}
Edit:
For personn interested in a cleaner way to implemenent that, have a look to that answer.
In my job I often need to use third-made API to access remote system.
For instance to create a request and send it to the remote system:
#include "external_lib.h"
void SendRequest(UserRequest user_request)
{
try
{
external_lib::Request my_request;
my_request.SetPrice(user_request.price);
my_request.SetVolume(user_request.quantity);
my_request.SetVisibleVolume(user_request.quantity);
my_request.SetReference(user_request.instrument);
my_request.SetUserID(user_request.user_name);
my_request.SetUserPassword(user_request.user_name);
// Meny other member affectations ...
}
catch(external_lib::out_of_range_error& e)
{
// Price , volume ????
}
catch(external_lib::error_t& e)
{
// Here I need to tell the user what was going wrong
}
}
Each lib's setter do checks the values that the end user has provided, and may thow an exception when the user does not comply with remote system needs. For instance a specific user may be disallowed to send a too big volume. That's an example, and actually many times users tries does not comply: no long valid instrument, the prices is out of the limit, etc, etc.
Conseqently, our end user need an explicit error message to tell him what to modify in its request to get a second chance to compose a valid request. I have to provide hiim such hints
Whatever , external lib's exceptions (mostly) never specifies which field is the source
of aborting the request.
What is the best way, according to you, to handle those exceptions?
My first try at handling those exceptions was to "wrap" the Request class with mine. Each setters are then wrapped in a method which does only one thing : a try/catch block. The catch block then throws a new exceptions of mine : my_out_of_range_volume, or my_out_of_range_price depending on the setter. For instance SetVolume() will be wrapped this way:
My_Request::SetVolume(const int volume)
{
try
{
m_Request.SetVolume(volume);
}
catch(external_lib::out_range_error& e)
{
throw my_out_of_range_volume(volume, e);
}
}
What do you think of it? What do you think about the exception handling overhead it implies? ... :/
Well the question is open, I need new idea to get rid of that lib constraints!
If there really are a lot of methods you need to call, you could cut down on the code using a reflection library, by creating just one method to do the calling and exception handling, and passing in the name of the method/property to call/set as an argument. You'd still have the same amount of try/catch calls, but the code would be simpler and you'd already know the name of the method that failed.
Alternatively, depending on the type of exception object that they throw back, it may contain stack information or you could use another library to walk the stack trace to get the name of the last method that it failed on. This depends on the platform you're using.
I always prefer a wrapper whenever I'm using third party library.
It allows me to define my own exception handling mechanism avoiding users of my class to know about external library.
Also, if later the third party changes the exception handling to return codes then my users need not be affected.
But rather than throwing the exception back to my users I would implement the error codes. Something like this:
class MyRequest
{
enum RequestErrorCode
{
PRICE_OUT_OF_LIMIT,
VOLUME_OUT_OF_LIMIT,
...
...
...
};
bool SetPrice(const int price , RequestErrorCode& ErrorCode_out);
...
private:
external_lib::Request mRequest;
};
bool MyRequest::SetPrice(const int price , RequestErrorCode& ErrorCode_out)
{
bool bReturn = true;
try
{
bReturn = mRequest.SetPrice(price);
}
catch(external_lib::out_of_range_error& e)
{
ErrorCode_out = PRICE_OUT_OF_LIMIT;
bReturn = false;
}
return bReturn;
}
bool SendRequest(UserRequest user_request)
{
MyRequest my_request;
MyRequest::RequestErrorCode anErrorCode;
bool bReturn = my_request.SetPrice(user_request.price, anErrorCode);
if( false == bReturn)
{
//Get the error code and process
//ex:PRICE_OUT_OF_LIMIT
}
}
I think in this case I might dare a macro. Something like (not tested, backslashes omitted):
#define SET( ins, setfun, value, msg )
try {
ins.setfun( value );
}
catch( external::error & ) {
throw my_explanation( msg, value );
}
and in use:
Instrument i;
SET( i, SetExpiry, "01-01-2010", "Invalid expiry date" );
SET( i, SetPeriod, 6, "Period out of range" );
You get the idea.
Although this is not really the answer you are looking for, but i think that your external lib, or you usage of it, somehow abuses exceptions. An exception should not be used to alter the general process flow. If it is the general case, that the input does not match the specification, than it is up to your app to valid the parameter before passing it to the external lib. Exceptions should only be thrown if an "exceptional" case occurrs, and i think whenever it comes to doing something with user input, you usually have to deal with everything and not rely on 'the user has to provide the correct data, otherwise we handle it with exceptions'.
nevertheless, an alternative to Neil's suggestions could be using boost::lambda, if you want to avoid macros.
In your first version, you could report the number of operations that succeeded provided the SetXXX functions return some value. You could also keep a counter (which increases after every SetXXX call in that try block) to note what all calls succeeded and based on that counter value, return an appropriate error message.
The major problem with validating each and every step is, in a real-time system -- you are probably introducing too much latency.
Otherwise, your second option looks like the only way. Now, if you have to write a wrapper for every library function and why not add the validation logic, if you can, instead of making the actual call to the said library? This IMO, is more efficient.