Passing Protocol buffer serialized datas from C++ to Python via LevelDB - c++

Though I've followed the excellent Protocol Buffer documentation and tutorials for C++ and Python, I can't achieve my goal which is :
- to serialize datas from a C++ process.
- insert it into LevelDB from that same process.
- extract the serialized datas from a Python process
- Deseralize it from this same Python process
- Use those deseralized datas in Python
I can serialize my datas using protocol buffer in C++ (using a std::string container). I can insert it into LevelDB. But, when I levelDB->Get my serialized datas, though Python seems to recognize it as a String, and showing me their raw content, whenever I deserialize it into a Python String, it is empty!
Here is how I serialize and insert my datas in C++ :
int main(int arg, char** argv)
{
GOOGLE_PROTOBUF_VERIFY_VERSION;
leveldb::DB* db;
leveldb::Options options;
leveldb::Status status;
tutorial::AddressBook address_book;
tutorial::Person* person1;
tutorial::Person* person2;
options.create_if_missing = true;
status = leveldb::DB::Open(options, "test_db", &db);
assert(status.ok());
person1 = address_book.add_person();
person1->set_id(1);
person1->set_name("ME");
person1->set_email("me#me.com");
person2 = address_book.add_person();
person2->set_id(2);
person2->set_name("SHE");
person2->set_email("she#she.com");
std::string test;
if (!address_book.SerializeToString(&test))
{
std::cerr << "Failed to write address book" << std::endl;
return -1;
}
if (status.ok()) status = db->Put(leveldb::WriteOptions(), "Test", test);
And here is how I try to deserialize it in Python:
address_book = addressbook_pb2.AddressBook()
db = leveldb.LevelDB('test_db')
ab = address_book.ParseFromString(db.Get("Test"))
ad var type is NoneType
Edit :
before the db.Get(), ab.ByteSize() returns 0, 76 after the ParseFromString(), I assume it's a Type problem then...
+
ab.ListFields() returns a unexploitable list of the contained field: succesfully couting two person instances, but unable to let me acces to it.
Any clues, any ideas of what I didn't understand, what I'm doing wrong here?
Many thanks!

Ok, so this was my bad.
I went back into the Protocol Buffers Python documentation, and the fact is that even if the AdressBook object I was retrieving did not showed any description, it was still able to be iterated over and even had a .str() method.
so, if anyone comes to that problem again, just try to explore your ProtocolBuffers object using iPython like I did, and you'll find that every of your proto elements are fields of your object.
Using my example:
ab = adress_book.ParseFromString(db.Get('Test'))
ab.__str__() # Shows a readable version of my object
for person in adress_book.person: # I'm even able to iterate over any of my ab fields values
print person.id
print person.name

Try using ' instead of ":
ab = address_book.ParseFromString(db->Get('Test'))

Related

Using C++ protobuf formatted structure in leveldb. set/get operations

I'd like to make a POC of using leveldb in order to store key-value table of different data types in protobuf format.
So far I was able to open the database file, and I also saw the get function with the following signature :
virtual Status Get(const ReadOptions& options, const Slice& key, std::string* value)=0
I understand that the value is actually refers to a binary string like vector and not regular alphanumeric string, so I guess it can fit for multi type primitives like string, uint, enum) but how can it support struct/class that represent protobuf layout in c++ ?
So this is my proto file that I'd like to store in the leveldb:
message agentStatus {
string ip = 1;
uint32 port = 2;
string url = 3;
google.protobuf.Timestamp last_seen = 4;
google.protobuf.Timestamp last_keepalive = 5;
bool status = 6;
}
and this is my current POC code. How can I use the get method to access any of the variables from the table above ?
#include <leveldb/db.h>
void main () {
std::string db_file_path = "/tmp/data.db";
leveldb::DB* db;
leveldb::Status status;
leveldb::Options options;
options.create_if_missing = false;
status_ = leveldb::DB::Open(options, db_file_path, &db);
if (!status_.ok()) {
throw std::logic_error("unable to open db");
}
Thanks !
You need to serialize the protobuf message into a binary string, i.e. SerilaizeToString, and use the Put method to write the binary string to LevelDB with a key.
Then you can use the Get method to retrieve the binary value with the given key, and parse the binary string to a protobuf message, i.e. ParseFromString.
Finally, you can get fields of the message.

findAndGetString() in DCMTK returns null for the tag

I am developing a quick DICOM viewer using DCMTK library and I am following the example provided in this link.
The buffer from the API always returns null for any tag ID, eg: DCM_PatientName.
But the findAndGetOFString() API works fine but returns only the first character of the tag in ASCII, is this how this API should work?
Can someone let me know why the buffer is empty the former API?
Also the DicomImage API also the same issue.
Snippet 1:
DcmFileFormat fileformat;
OFCondition status = fileformat.loadFile(test_data_file_path.toStdString().c_str());
if (status.good())
{
OFString patientName;
char* name;
if (fileformat.getDataset()->findAndGetOFString(DCM_PatientName, patientName).good())
{
name = new char[patientName.length()];
strcpy(name, patientName.c_str());
}
else
{
qDebug() << "Error: cannot access Patient's Name!";
}
}
else
{
qDebug() << "Error: cannot read DICOM file (" << status.text() << ")";
}
In the above snippet name has the ASCII value "50" and the actual name is "PATIENT".
Snippet 2:
DcmFileFormat file_format;
OFCondition status = file_format.loadFile(test_data_file_path.toStdString().c_str());
std::shared_ptr<DcmDataset> dataset(file_format.getDataset());
qDebug() << "\nInformation extracted from DICOM file: \n";
const char* buffer = nullptr;
DcmTagKey key = DCM_PatientName;
dataset->findAndGetString(key,buffer);
std::string tag_value = buffer;
qDebug() << "Patient name: " << tag_value.c_str();
In the above snippet, the buffer is null. It doesn't read the name.
NOTE:
This is only a sample. I am just playing around the APIs for learning
purpose.
The following sample method reads the patient name from a DcmDataset object:
std::string getPatientName(DcmDataset& dataset)
{
// Get the tag's value in ofstring
OFString ofstring;
OFCondition condition = dataset.findAndGetOFString(DCM_PatientName, ofstring);
if(condition.good())
{
// Tag found. Put it in a std::string and return it
return std::string(ofstring.c_str());
}
// Tag not found
return ""; // or throw if you need the tag
}
I have tried your code with your datasets. I just replaced the output to QT console classes to std::cout. It works for me - i.e. it prints the correct patient name (e.g. "PATIENT2" for scan2.dcm). Everything seems correct, except for the fact that you apparently want to transfer the ownership for the dataset to a smart pointer.
To obtain the ownership for the DcmDataset from the DcmFileFormat, you must call getAndRemoveDataset() instead of getDataset(). However, I do not think that your issue is related that. You may want to try my modified snippet:
DcmFileFormat file_format;
OFCondition status = file_format.loadFile("d:\\temp\\StackOverflow\\scan2.dcm");
std::shared_ptr<DcmDataset> dataset(file_format.getAndRemoveDataset());
std::cout << "\nInformation extracted from DICOM file: \n";
const char* buffer = nullptr;
DcmTagKey key = DCM_PatientName;
dataset->findAndGetString(key, buffer);
std::string tag_value = buffer;
std::cout << "Patient name: " << tag_value.c_str();
It probably helps you to know that your code and the dcmtk methods you use are correct, but that does not solve your problem. Another thing I would recommend is to verify the result returned by file_format.loadFile(). Maybe there is a surprise in there.
Not sure if I can help you more, but my next step would be to verify your build environment, e.g. the options that you use for building dcmtk. Are you using CMake to build dcmtk?

Replacing value of a member in rapidjson

I am currently working on a project in C++ using rapidjson.
My program receives some JSON data on a socket which includes some authentication details. I log the incoming message, but I want to hide the password so it can't be seen in the log file. So I am trying to get the JSON object, and replace each character of the string and put this replaced string back into the json object where the password was.
Below is the code that I have:
rapidjson::Document jsonObject;
jsonObject.Parse(command.c_str());
string method = jsonObject["method"].GetString();
if (jsonObject.HasMember("sshDetails"))
{
Value& sshDetails = jsonObject["sshDetails"];
string sshPassword = sshDetails["sshPassword"].GetString();
for (int i = 0; i < sshPassword.length(); i++)
{
sshPassword[i] = '*';
}
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->name.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
//Convert it back to a string
rapidjson::StringBuffer buffer;
buffer.Clear();
rapidjson::Writer<rapidjson::StringBuffer>writer(buffer);
Document jsonDoc;
jsonDoc.Accept(writer);
string jsonString = string(buffer.GetString());
I'm getting an error on the following line:
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
The error I am getting is:
No suitable conversion function from rapidjson::GenericMemberIterator<false, rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CtrlAllocator>> to rapidjson::GenericMember::UTF8<char>, myProject...SocketProcessor.cpp
rapidjson::MemoryPoolAllocator<rapidjson::CtrlAllocator>>*exists
I took the above from an example on another question on SO which was an accepted answer from rapidjson - change key to another value, so what am I missing.
I've managed to find the answer to this with a bit of playing round and luck.
I changed
rapidjson::Value::Member* sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->name.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
to be
rapidjson::Value::MemberIterator sshPasswordMember = sshDetails.FindMember("sshPassword");
sshPasswordMember->value.SetString(sshPassword.c_str(), jsonObject.GetAllocator());
using rapidjson in my project I found out that many of such problems can be omitted by the use of auto instead of specifying the type

Creating json string using json lib

I am using jsonc-libjson to create a json string like below.
{ "author-details": {
"name" : "Joys of Programming",
"Number of Posts" : 10
}
}
My code looks like below
json_object *jobj = json_object_new_object();
json_object *jStr1 = json_object_new_string("Joys of Programming");
json_object *jstr2 = json_object_new_int("10");
json_object_object_add(jobj,"name", jStr1 );
json_object_object_add(jobj,"Number of Posts", jstr2 );
this gives me json string
{
"name" : "Joys of Programming",
"Number of Posts" : 10
}
How do I add the top part associated with author details?
To paraphrase an old advertisement, "libjson users would rather fight than switch."
At least I assume you must like fighting with the library. Using nlohmann's JSON library, you could use code like this:
nlohmann::json j {
{ "author-details", {
{ "name", "Joys of Programming" },
{ "Number of Posts", 10 }
}
}
};
At least to me, this seems somewhat simpler and more readable.
Parsing is about equally straightforward. For example, let's assume we had a file named somefile.json that contained the JSON data shown above. To read and parse it, we could do something like this:
nlohmann::json j;
std::ifstream in("somefile.json");
in >> j; // Read the file and parse it into a json object
// Let's start by retrieving and printing the name.
std::cout << j["author-details"]["name"];
Or, let's assume we found a post, so we want to increment the count of posts. This is one place that things get...less tasteful--we can't increment the value as directly as we'd like; we have to obtain the value, add one, then assign the result (like we would in lesser languages that lack ++):
j["author-details"]["Number of Posts"] = j["author-details"]["Number of Posts"] + 1;
Then we want to write out the result. If we want it "dense" (e.g., we're going to transmit it over a network for some other machine to read it) we can just use <<:
somestream << j;
On the other hand, we might want to pretty-print it so a person can read it more easily. The library respects the width we set with setw, so to have it print out indented with 4-column tab stops, we can do:
somestream << std::setw(4) << j;
Create a new JSON object and add the one you already created as a child.
Just insert code like this after what you've already written:
json_object* root = json_object_new_object();
json_object_object_add(root, "author-details", jobj); // This is the same "jobj" as original code snippet.
Based on the comment from Dominic, I was able to figure out the correct answer.
json_object *jobj = json_object_new_object();
json_object* root = json_object_new_object();
json_object_object_add(jobj, "author-details", root);
json_object *jStr1 = json_object_new_string("Joys of Programming");
json_object *jstr2 = json_object_new_int(10);
json_object_object_add(root,"name", jStr1 );
json_object_object_add(root,"Number of Posts", jstr2 );

How to get data from Photon eventContent dictionary

We are receiving this callback using ExitGames Photon Realtime engine when an event is fired
customEventAction(int playerNr,
nByte eventCode,
const ExitGames::Common::Object& eventContent)
If the object is a string we use this code to extract it
ExitGames::Common::JString str =
ExitGames::Common::ValueObject<ExitGames::Common::JString>(eventContent).getDataCopy();
However, the object being sent is a dictionary. It's being sent from the server using BroadcastEvent.
How do we get data out of it ?
We've tried this, but it doesn't make any sense:
ExitGames::Common::Dictionary<byte,ExitGames::Common::Object> pdic
= ExitGames::Common::ValueObject<ExitGames::Common::Dictionary<byte,ExitGames::Common::Object>>(eventContent).getDataCopy();
I've found code to get the data from a hashtable, but that doesn't work either.
thanks
Shaun
ExitGames::Common::Dictionary<nByte, ExitGames::Common::Object> dic = ExitGames::Common::ValueObject<ExitGames::Common::Dictionary<nByte, ExitGames::Common::Object> >(eventContent).getDataCopy();
is absolutely correct and works for me.
The cause of your problem must be inside another line.
When you replace the implementations of sendEvent() and customEventAction() in demo_loadBalancing inside one of the Photon C++ client SDKs with the following snippets, then that demo successfully sends and receives a Dictionary:
send:
void NetworkLogic::sendEvent(void)
{
ExitGames::Common::ValueObject<ExitGames::Common::JString> obj(L"test");
ExitGames::Common::Dictionary<nByte, ExitGames::Common::Object> dic;
dic.put(1, obj);
mLoadBalancingClient.opRaiseEvent(false, dic, 0);
}
receive:
void NetworkLogic::customEventAction(int /*playerNr*/, nByte /*eventCode*/, const ExitGames::Common::Object& eventContent)
{
EGLOG(ExitGames::Common::DebugLevel::ALL, L"");
ExitGames::Common::Dictionary<nByte, ExitGames::Common::Object> dic = ExitGames::Common::ValueObject<ExitGames::Common::Dictionary<nByte, ExitGames::Common::Object> >(eventContent).getDataCopy();
const ExitGames::Common::Object* pObj = dic.getValue(1);
ExitGames::Common::JString str = ExitGames::Common::ValueObject<ExitGames::Common::JString>(pObj).getDataCopy();
mpOutputListener->write(L"received the following string as Dictionary value: " + str);
}
This gives me the following line of output on the receiving client:
received the following string as Dictionary value: test