Can protocol buffer be partial updated? - c++

I am new to protobuf and here is my question: Can protocol buffer support partial update?
For example, I have such messages:
package model.test;
message Person{
required int32 id = 1;
required string name = 2;
repeated PhoneNumber phone = 3;
}
enum PhoneType{
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber{
required string number = 1;
optional PhoneType type = 2 [default = HOME];
}
Now the data I have like that:
model::test::Person person;
person.set_id(1);
person.set_name("Jack");
model::test::PhoneNumber* _phone3 = person.add_phone();
_phone3->set_number("123567");
_phone3->set_type(model::test::MOBILE);
model::test::PhoneNumber* _phone4 = person.add_phone();
_phone4->set_number("347890");
_phone4->set_type(model::test::WORK);
The case is that when only work phone number is changed, I have to update the whole person object with the following codes.
fstream out("User.txt", ios::out | ios::binary | ios::trunc);
person.SerializePartialToOstream(&out);
But it is not efficient to do that. I want to only update the PhoneNumber, Is there any partial update in protolbuf or something like that?

Protocol buffers are actually designed such that concatenation is the same as merge, and that the last field wins when merging (except for repeated, which are added). In your case, you should actually be able to serialize a blob containing just the phone-number set, and append this data, and it will over-ride the earlier value. This, however, only works well for the root object. Which yours: isn't. And it doesn't work for repeated, which yours: is.

I don't think there is any support for what you want to do. If you think about it, it doesn't really make sense that some sort of partial update serialization would exist in the first place. For protobuf to be able to manipulate an object that is serialized in a file on disk, it needs to read and deserialize the whole object so it knows what fields have been previously populated. Then when serializing and writing the updated object back to disk, you're going to have to overwrite the old file no matter what you do (i.e. you can't shove extra bytes into a file on the file system without overwriting the original file completely).

Related

Should the Protobufs from two different repositories be aligned

Problem Description
We have two codes on different repositories. One is in Java and the Other is in C++. We share a common protobuf. The problem is that on our side which is the C++ side we have less members that the one on the JAVA side. As you can see, on our work is assigned id 4, whereas on Java side it is assigned id 5. Both members have the same name which is work.
Question
If the protobufs are not aligned what problems can we have? is it ok for the protobufs not be aligned?
message CPPContext {
optional string date = 1;
optional string time = 2;
optional string hour = 3;
optional string work = 4;
}
message JAVAContext {
optional string date = 1;
optional string time = 2;
optional string hour = 3;
optional string currency = 4;
optional string work = 5;
}
Protobuf serialize and deserialize messages based on field numbers not field names.
For example if CPPContext message gets deserialized on the other side as JAVAContext then your work field will be treated as currency field on the other side.
It is better to use same proto files on both communicating sides. Or at least (backward-)compatible proto files. For example it is fine to add new optional fields with new field ids in proto files on one side first (they will be ignored on the other side) but it is not fine to change id of a field or remove required field.

Arrow Java ListVector writeBatch and read get empty list

I have a ListVector and value [[1,2,3,4,5]] in VectorSchemaRoot and I could see its value in IDEA.
I use following code to write the VectorSchemaRoot variable and get the byte array
val out = new ByteArrayOutputStream()
val writer = new ArrowStreamWriter(vectorSchemaRoot, null, out)
writer.start()
writer.writeBatch()
writer.end()
out.close()
val byteArr = out.toByteArray
And read back
val allocator = new RootAllocator(Int.MaxValue)
val reader = new ArrowStreamReader(new ByteArrayInputStream(byteArr), allocator)
while (reader.loadNextBatch()) {
val schemaRoot = reader.getVectorSchemaRoot
schemaRoot
}
The schema is correct, but the list is empty []
However, I use other types of values, like char, bit, the result read from the byteArr is correct(non-empty).
How to fix the ListVector empty issue?
Finally I used just basic classes.
The StructVector, ListVector are complex classes, and according to my test, they do not bring speed or memory benefit over just using basic classes. And the documents for complex classes are very few.
Thus basic classes is recommended. And just use List of Fields to make the schema of them, could also get the structured vector.

<Binary> in sql

I want to select all the binary data from a column of a SQL database (SQL Server Enterprise) using C++ query. I'm not sure what is in the binary data, and all it says is .
I tried this (it's been passed onto me to study off from) and I honestly don't 100% understand the code at some parts, as I commented):
SqlConnection^ cn = gcnew SqlConnection();
SqlCommand^ cmd;
SqlDataAdapter^ da;
DataTable^ dt;
cn->ConnectionString = "Server = localhost; Database=portable; User ID = glitch; Pwd = 1234";
cn->Open();
cmd=gcnew SqlCommand("SELECT BinaryColumn FROM RawData", cn);
da = gcnew SqlDataAdapter(cmd);
dt = gcnew DataTable("BinaryTemp"); //I'm confused about this piece of code, is it supposed to create a new table in the database or a temp one in the code?
da->Fill(dt);
for(int i = 0; i < dt->Rows->Count-1; i++)
{
String^ value_string;
value_string=dt->Rows[i]->ToString();
Console::WriteLine(value_string);
}
cn->Close();
Console::ReadLine();
but it only returns a lot of "System.Data.DataRow".
Can someone help me?
(I need to put it into a matrix form after I extract the binary data, so if anyone could provide help for that part as well, it'd be highly appreciated!)
dt->Rows[i] is indeed a DataRow ^. To extract a specific field from it, use its indexer:
array<char> ^blob=dt->Rows[i][0];
This extracts the first column (since you have only one) and returns an array representation of it.
To answer the question in your code, the way SqlDataAdapter works is like this:
you build a DataTable to hold the data to retrieve. You can fill in its columns, but you're not required to. Neither are you required to give it a name.
you build the adapter object, giving it a query and a connection object
you call the Fill method on the adapter, giving it the previously created DataTable to fill with whatever your query returns.
and you're done with the adapter. At this point you can dispose of it (for example inside a using statement if you're using C#).

How to read semicolon separated certain values from a QString?

I am developing an application using Qt/KDE. While writing code for this, I need to read a QString that contains values like ( ; delimited)
<http://example.com/example.ext.torrent>; rel=describedby; type="application/x-bittorrent"; name="differentname.ext"
I need to read every attribute like rel, type and name into a different QString. The apporach I have taken so far is something like this
if (line.contains("describedby")) {
m_reltype = "describedby" ;
}
if (line.contains("duplicate")) {
m_reltype = "duplicate";
}
That is if I need to be bothered only by the presence of an attribute (and not its value) I am manually looking for the text and setting if the attribute is present. This approach however fails for attributes like "type" and name whose actual values need to be stored in a QString. Although I know this can be done by splitting the entire string at the delimiter ; and then searching for the attribute or its value, I wanted to know is there a cleaner and a more efficient way of doing it.
As I understand, the data is not always an URL.
So,
1: Split the string
2: For each substring, separate the identifier from the value:
id = str.mid(0,str.indexOf("="));
value = str.mid(str.indexOf("=")+1);
You can also use a RegExp:
regexp = "^([a-z]+)\s*=\s*(.*)$";
id = \1 of the regexp;
value = \2 of the regexp;
I need to read every attribute like rel, type and name into a different QString.
Is there a gurantee that this string will always be a URL?
I wanted to know is there a cleaner and a more efficient way of doing it.
Don't reinvent the wheel! You can use QURL::queryItems which would parse these query variables and return a map of name-value pairs.
However, make sure that your string is a well-formed URL (so that QURL does not reject it).

String extraction

Currently I am working very basic game using the C++ environment. The game used to be a school project but now that I am done with that programming class, I wanted to expand my skills and put some more flourish on this old assignment.
I have already made a lot of changes that I am pleased with. I have centralized all the data into folder hierarchies and I have gotten the code to read those locations.
However my problem stems from a very fundamental flaw that has been stumping me.
In order to access the image data that I am using I have used the code:
string imageLocation = "..\\DATA\\Images\\";
string bowImage = imageLocation + "bow.png";
The problem is that when the player picks up an item on the gameboard my code is supposed to use the code:
hud.addLine("You picked up a " + (*itt)->name() + "!");
to print to the command line, "You picked up a Bow!". But instead it shows "You picked up a ..\DATA\Images\!".
Before I centralized my data I used to use:
name_(item_name.substr(0, item_name.find('.')))
in my Item class constructor to chop the item name to just something like bow or candle. After I changed how my data was structured I realized that I would have to change how I chop the name down to the same simple 'bow' or 'candle'.
I have changed the above code to reflect my changes in data structure to be:
name_(item_name.substr(item_name.find("..\\DATA\\Images\\"), item_name.find(".png")))
but unfortunately as I alluded to earlier this change of code is not working as well as I planned it to be.
So now that I have given that real long winded introduction to what my problem is, here is my question.
How do you extract the middle of a string between two sections that you do not want? Also that middle part that is your target is of an unknown length.
Thank you so very much for any help you guys can give. If you need anymore information please ask; I will be more than happy to upload part or even my entire code for more help. Again thank you very much.
In all honeasty, you're probably approaching this from the wrong end.
Your item class should have a string "bow", in a private member. The function Item::GetFilePath would then (at runtime) do "..\DATA\Images\" + this->name + ".png".
The fundamental property of the "bow" item object isn't the filename bow.png, but the fact that it's a "bow". The filename is just a derived proerty.
Assuming I understand you correctly, the short version of your question is: how do I split a string containing a file path so I have removed the path and the extension, leaving just the "title"?
You need the find_last_of method. This gets rid of the path:
std::size_type lastSlash = filePath.find_last_of('\\');
if (lastSlash == std::string::npos)
fileName = filePath;
else
fileName = filePath.substr(lastSlash + 1);
Note that you might want to define a constant as \\ in case you need to change it for other platforms. Not all OS file systems use \\ to separate path segments.
Also note that you also need to use find_last_of for the extension dot as well, because filenames in general can contain dots, throughout their paths. Only the very last one indicates the start of the extension:
std::size_type lastDot = fileName.find_last_of('.');
if (lastDot == std::string::npos)
{
title = fileName;
}
else
{
title = fileName.substr(0, lastDot);
extension = fileName.substr(lastDot + 1);
}
See http://msdn.microsoft.com/en-us/library/3y5atza0(VS.80).aspx
using boost filesystem:
#include "boost/filesystem.hpp"
namespace fs = boost::filesystem;
void some_function(void)
{
string imageLocation = "..\\DATA\\Images\\";
string bowImage = imageLocation + "bow.png";
fs::path image_path( bowImage );
hud.addLine("You picked up a " + image_path.filename() + "!"); //prints: You picked up a bow!
So combining Paul's and my thoughts, try something like this (broken down for readability):
string extn = item_name.substr(item_name.find_last_of(".png"));
string path = item_name.substr(0, item_name.find("..\\DATA\\Images\\"));
name_ = item_name.substr( path.size(), item_name.size() - extn.size() );
You could simplify it a bit if you know that item name always starts with "..DATA" etc (you could store it in a constant and not need to search for it in the string)
Edit: Changed extension finding part to use find_last_of, as suggested by EarWicker, (this avoids the case where your path includes '.png' somewhere before the extension)
item_name.find("..\DATA\Images\") will return the index at which the substring "..\DATA\Images\" starts but it seems like you'd want the index where it ends, so you should add the length of "..\DATA\Images\" to the index returned by find.
Also, as hamishmcn pointed out, the second argument to substr should be the number of chars to return, which would be the index where ".png" starts minus the index where "..\DATA\Images\" ends, I think.
One thing that looks wrong is that the second parameter to substr should be the number of chars to copy, not the position.