In my protobuf schema, I have a type that contains binary data (already defined in an existing schema, I can't change this):
// message BinaryKey { bytes data = 1; }
let BinaryKey = new Type('BinaryKey')
BinaryKey.add(new Field('data', 1, 'bytes'))
In my application JSON, I have a human readable string format of this field/type and would like to use this string for my field that is passed to encode(). What is the correct way to have encode() and decode() use custom conversion functions between string and binary format?
A full example of my code (using protobuf.js reflection):
let BinaryKey = new Type('BinaryKey')
BinaryKey.add(new Field('data', 1, 'bytes'))
let Message = new Type('CustomMessage')
Message.add(new Field('balance', 1, 'uint32'))
Message.add(BinaryKey)
Message.add(new Field('bin_key', 2, 'BinaryKey'))
let object = { balance: 100, bin_key: '<String representation>' }; // <-- *** pass in the data as a string to be converted
Message.encode(object).finish();
I would need to specify conversion functions between my string format and the binary field of the schema, but don't know how to add these to my Type (BinaryKey)
Related
I'd like to make a POC of using leveldb in order to store key-value table of different data types in protobuf format.
So far I was able to open the database file, and I also saw the get function with the following signature :
virtual Status Get(const ReadOptions& options, const Slice& key, std::string* value)=0
I understand that the value is actually refers to a binary string like vector and not regular alphanumeric string, so I guess it can fit for multi type primitives like string, uint, enum) but how can it support struct/class that represent protobuf layout in c++ ?
So this is my proto file that I'd like to store in the leveldb:
message agentStatus {
string ip = 1;
uint32 port = 2;
string url = 3;
google.protobuf.Timestamp last_seen = 4;
google.protobuf.Timestamp last_keepalive = 5;
bool status = 6;
}
and this is my current POC code. How can I use the get method to access any of the variables from the table above ?
#include <leveldb/db.h>
void main () {
std::string db_file_path = "/tmp/data.db";
leveldb::DB* db;
leveldb::Status status;
leveldb::Options options;
options.create_if_missing = false;
status_ = leveldb::DB::Open(options, db_file_path, &db);
if (!status_.ok()) {
throw std::logic_error("unable to open db");
}
Thanks !
You need to serialize the protobuf message into a binary string, i.e. SerilaizeToString, and use the Put method to write the binary string to LevelDB with a key.
Then you can use the Get method to retrieve the binary value with the given key, and parse the binary string to a protobuf message, i.e. ParseFromString.
Finally, you can get fields of the message.
I am currently working on a project in C++ using rapidjson.
My program receives some JSON data on a socket which includes some authentication details. I log the incoming message, but I want to hide the password so it can't be seen in the log file. So I am trying to get the JSON object, and replace each character of the string and put this replaced string back into the json object where the password was.
Below is the code that I have:
rapidjson::Document jsonObject;
jsonObject.Parse(command.c_str());
string method = jsonObject["method"].GetString();
if (jsonObject.HasMember("sshDetails"))
{
Value& sshDetails = jsonObject["sshDetails"];
string sshPassword = sshDetails["sshPassword"].GetString();
for (int i = 0; i < sshPassword.length(); i++)
{
sshPassword[i] = '*';
}
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->name.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
//Convert it back to a string
rapidjson::StringBuffer buffer;
buffer.Clear();
rapidjson::Writer<rapidjson::StringBuffer>writer(buffer);
Document jsonDoc;
jsonDoc.Accept(writer);
string jsonString = string(buffer.GetString());
I'm getting an error on the following line:
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
The error I am getting is:
No suitable conversion function from rapidjson::GenericMemberIterator<false, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CtrlAllocator>> to rapidjson::GenericMember::UTF8<char>, myProject...SocketProcessor.cpp
rapidjson::MemoryPoolAllocator<rapidjson::CtrlAllocator>>*exists
I took the above from an example on another question on SO which was an accepted answer from rapidjson - change key to another value, so what am I missing.
I've managed to find the answer to this with a bit of playing round and luck.
I changed
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->name.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
to be
rapidjson::Value::MemberIterator sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->value.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
using rapidjson in my project I found out that many of such problems can be omitted by the use of auto instead of specifying the type
Related to using protobuf as a textual configuraton file I'd like to use protobuf for configuration file.
I expect that protobuf allows me to use simple parser with exact structure.
My configuration structure looks like
//my.proto
package my_config;
message MyConfigItem {
required string type = 1;
required string name = 2;
repeated string inputNames = 3 [packed=true];
repeated string outputNames = 4 [packed=true];
}
And bunch of different items in config files like
MyConfigItem {
type = "type1";
name = "name1";
inputNames = {"input1", "input2"};
}
What's the best way for organizing that?
You may write config file like:
type : "type1"
name : "name1"
inputNames : {"input1", "input2"}
So I think that you must use ':' instead of '=', and "do not" write the items wrapped in the brackets.
I have a tsv file which has fixed rows but each row is mapped to different Java Class.
For example.
recordType recordValue1
recordType recordValue1 recordValue2
for First row I have follofing class:
public class FirstRow implements ItsvRecord {
#Parsed(index = 0)
private String recordType;
#Parsed(index = 1)
private String recordValue1;
public FirstRow() {
}
}
and for second row I have:
public class SecondRow implements ItsvRecord {
#Parsed(index = 0)
private String recordType;
#Parsed(index = 1)
private String recordValue1;
public SecondRow() {
}
}
I want to parse the TSV file directly to the respective objects but I am falling short of ideas.
Use an InputValueSwitch. This will match a value in a particular column of each row to determine what RowProcessor to use. Example:
Create two (or more) processors for each type of record you need to process:
final BeanListProcessor<FirstRow> firstProcessor = new BeanListProcessor<FirstRow>(FirstRow.class);
final BeanListProcessor<SecondRow> secondProcessor = new BeanListProcessor<SecondRow>(SecondRow.class);
Create an InputValueSwitch:
//0 means that the first column of each row has a value that
//identifies what is the type of record you are dealing with
InputValueSwitch valueSwitch = new InputValueSwitch(0);
//assigns the first processor to rows whose first column contain the 'firstRowType' value
valueSwitch.addSwitchForValue("firstRowType", firstProcessor);
//assigns the second processor to rows whose first column contain the 'secondRowType' value
valueSwitch.addSwitchForValue("secondRowType", secondProcessor);
Parse as usual:
TsvParserSettings settings = new TsvParserSettings(); //configure...
// your row processor is the switch
settings.setProcessor(valueSwitch);
TsvParser parser = new TsvParser(settings);
Reader input = new StringReader(""+
"firstRowType\trecordValue1\n" +
"secondRowType\trecordValue1\trecordValue2");
parser.parse(input);
Get the parsed objects from your processors:
List<FirstRow> firstTypeObjects = firstProcessor.getBeans();
List<SecondRow> secondTypeObjects = secondProcessor.getBeans();
The output will be*:
[FirstRow{recordType='firstRowType', recordValue1='recordValue1'}]
[SecondRow{recordType='secondRowType', recordValue1='recordValue1', recordValue2='recordValue2'}]
Assuming you have a sane toString() implemented in your classes
If you want to manage associations among the objects that are parsed:
If your FirstRow should contain the elements parsed for records of type SecondRow, simply override the rowProcessorSwitched method:
InputValueSwitch valueSwitch = new InputValueSwitch(0) {
#Override
public void rowProcessorSwitched(RowProcessor from, RowProcessor to) {
if (from == secondProcessor) {
List<FirstRow> firstRows = firstProcessor.getBeans();
FirstRow mostRecentRow = firstRows.get(firstRows.size() - 1);
mostRecentRow.addRowsOfOtherType(secondProcessor.getBeans());
secondProcessor.getBeans().clear();
}
}
};
The above assumes your FirstRow class has a addRowsOfOtherType method that takes a list of SecondRow as parameter.
And that's it!
You can even mix and match other types of RowProcessor. There's another example here that demonstrates this.
Hope this helps.
I'm using Qjson to parse a json object that is returned from a web service. I'm stuck on handling an array of complex ojects.
At the first level the web service returns a map consisting of "error", "id", and "return". If there are no errors I can get the first level value by using
nestedMap = m_jsonObject["result"].toMap();
group = new Group();
group->Caption = nestedMap["Caption"].toString();
group->CollectionCount = nestedMap["CollectionCount"].toInt();
I can even get a date item value that is at the second level using
group->ModifiedOn = nestedMap["ModifiedOn"].toMap()["Value"].toDateTime();
I have an object called "Elements" that consists of 29 key-value pairs. The web service is returning an array of these "Elements" and I am unable to find the right way to parse it. In the header file the container for the elements is defined as
QList<GroupElement> Elements;
The line
group->Elements = nestedMap["Elements"].toList();
causes the compiler to throw an error 'error: no match for 'operator=' in '((MyClass*)this)->MyClass::group->Group::Elements = QVariant::toMap() const()'
I would like to learn the correct syntax to put this element into the class.
Update: I wrote another function to convert the QVariantMap object to a
first:
The group-> Elements object was changed to a
class ParentClass{
QList<SharedDataPointer<Address> > Elements;
other class memmbers...
};
Second:
A method to convert the QMap object to an Address object was created
QSharedDataPointer<Address>
API_1_6::mapToAddress(QVariantMap o)
{
QSharedDataPointer<Address> address (new Address());
address-> FirstName = o["FirstName"].toString();
address->LastName = o["LastName"].toString();
address->CompanyName = o["CompanyName"].toString();
address->Street = o["Street"].toString();
address->Street2 = o["Street2"].toString();
address->City = o["City"].toString();
address->Zip = o["Zip"].toString();
address-> State = o["State"].toString();
address->Country = o["Country"].toString();
address->Phone = o["Phone"].toString();
address->Phone2 = o["Phone2"].toString();
address-> Fax = o["Fax"].toString();
address-> Url = o["Url"].toString();
address->Email = o["Email"].toString();
address->Other = o["Other"].toString();
return address;
}
third: In the code, foreach is used to walk through the list and create and store the new objects
// get the list of the elements
elementsList = nestedMap["Elements"].toList();
// Add the element, converted to the new type, to the Elements object of the'parent' class
foreach(QVariant qElement, elementsList){
group-> Elements.append(mapToAddress(qElement))
}