I am currently trying to rewrite the network code of a MFC application to fit it to a new server, (written in c#) later on.
I'm experimenting with protobuf to ensure that all data models are the same.
My problem is, that the Scalar Value Types, protobuf works with is not understood by the MFC type system. It uses DWORD instead of uint32, BYTE instead of byte or bool and so on. That requires custom type mapping/casting and that's not necessarily taking work from my shoulders.
Is there a way to change those datatypes used in protobuf?
A sample protobuf model would be:
syntax = "proto3";
package CInterop;
option csharp_namespace = "CCX.Protobuf.Login";
message LoginAck {
LoginResultStatus Status = 1;
uint32 dwUserID = 2;
uint32 dwCharID = 3;
uint32 dwKEY = 4;
uint32 dwMapIP = 5;
uint32 wPort = 6;
uint32 bCharCreateCount = 7;
bool IsInPcRoom = 8;
uint32 PremiumPcRoomValue = 9;
uint64 dCurrentTime = 10;
uint64 dKey = 11;
string Reason = 12;
uint64 iDuration = 13;
bool bEternal = 14;
}
The current usage would be
CInterop::LoginAck loginAck;
(*pPacket) >> loginAck; // Reads the packet from the buffer
m_dwUserID = (DWORD)loginAck.dwuserid();
m_dwKickID = (DWORD)loginAck.dwcharid();
m_dwKEY = (DWORD)loginAck.dwkey();
vADDR.sin_addr.S_un.S_addr = (ULONG)loginAck.dwmapip();
vADDR.sin_port = (USHORT)loginAck.wport();
m_bCreateCardCnt = (byte)loginAck.bcharcreatecount();
m_bInPCROOM = (BYTE)loginAck.isinpcroom();
m_dwPrmPCROOM = (DWORD)loginAck.premiumpcroomvalue();
But here, i don't want to cast the types, this should happen automatically somehow.
Related
I got following warnings for the enum value xxxxx920P4 building a qt C++ project. And I found out in the debugging that xxxxx920P4 is now equals to xxxxxundefine (0x00000000), which causes the unexpected result. How should I resolve this problem?
I'm using Qt version: 4.8.7 and Visual studio 2010.
The following loop will not be executed. gVersions[0].verNum is "xxxxx920P4", so I guess "xxxxx920P4"'s value has been truncated and now it equals to 0x00000000.
for(int i=0; gVersions[i].verNum != xxxxxundefine ; i++)
{
}
Doc for Qt Qflags class: https://doc.qt.io/qt-5/qflags.html
warning C4341: 'xxxxx920P4' : signed value is out of range for enum constant
warning C4309: 'initializing' : truncation of constant value
enum Version
{
xxxxxundefine = 0x00000000,
xxxxx400 = 0x00000001,
xxxxx401 = 0x00000002,
xxxxx410 = 0x00000004,
xxxxx411 = 0x00000008,
xxxxx412 = 0x00000010,
xxxxx420 = 0x00000020,
xxxxx430 = 0x00000040,
xxxxx431 = 0x00000080,
xxxxx432 = 0x00000100,
xxxxx440 = 0x00000200,
xxxxx500 = 0x00000400,
xxxxx510 = 0x00000800,
xxxxx520 = 0x00001000,
xxxxx521 = 0x00002000,
xxxxx600 = 0x00004000,
xxxxx611 = 0x00008000,
xxxxx620 = 0x00010000,
xxxxx621 = 0x00020000,
xxxxx700 = 0x00040000,
xxxxx910 = 0x00080000,
xxxxx910P5 = 0x00100000,
xxxxx910P6 = 0x00200000,
xxxxx910P11 = 0x00400000,
xxxxx910P12 = 0x00800000,
xxxxx910P13 = 0x01000000,
xxxxx910P14 = 0x02000000,
xxxxx910P15 = 0x04000000,
xxxxx910P16 = 0x08000000,
xxxxx920 = 0x10000000,
xxxxx920P1 = 0x20000000,
xxxxx920P2 = 0x40000000,
xxxxx920P3 = 0x80000000,
xxxxx920P4 = 0x100000000,
}; Q_DECLARE_FLAGS(Versions, Version)
The last value is too long to fit into a a non-long int. While newer compilers may try to choose a larger integral type for it, some older ones would complain. GCC tends to do the former unless you choose type explicitly. Note that second to last value 0x80000000 or any combination of masks involving it would not fit into int, it's essentially a negative -2147483648.
enum Version : long long
{
//...
xxxxx920P4 = 0x100000000,
};
PS. Apparently Qt5 doesn't support something different from (unsigned)int: https://doc.qt.io/qt-5/qflags.html
I want to correctly extend .text section in PE file according to a File Alignment or Section Alignment.
Details:
Let's start from scratch.
I have .net v4.8 project with several small classes like
public class A1
{
public int IntProperty {get;set;}
}
public class A2
{
public string StringProperty {get;set;}
}
It doesn't really matter what these classes are.
After my compilation using MSBuild v16 I have a PE file. (exe or dll dosn't matter either).
Then I expand the .text section with File- or Section alignment(I tried both).
So, if compiled PE file have 512, 1024 or 2048 alignment - that works fine.
But when it have 4096 or 8192 bytes - it doesn't work.
This is the code example, how I do it:
BYTE* dos_header_ptr = reinterpret_cast<BYTE*>(dosHeader2);
IMAGE_NT_HEADERS64* nt_header2 = reinterpret_cast<TNT_HEADERS*>(dos_header_ptr + dosHeader2->e_lfanew);
const IMAGE_FILE_HEADER* file_header = &nt_header2->FileHeader;
IMAGE_OPTIONAL_HEADER64* optional_headers2 = &nt_header2->OptionalHeader;
DWORD file_alignment = optional_headers2->FileAlignment;
file_alignment = optional_headers2->SectionAlignment;
std::vector<IMAGE_SECTION_HEADER*> sections = ExtractSections(nt_header2);
IMAGE_SECTION_HEADER* section_header_text2 = sections[0];
IMAGE_SECTION_HEADER* section_header_rsrc2 = sections[1];
file_info* old_file_info = file_info2;
file_info2 = RelocateBufferWithDisplacement(old_file_info, file_alignment, section_header_text2);
// Update pointers and VA
dosHeader2 = reinterpret_cast<IMAGE_DOS_HEADER*>(file_info2->buffer);
dos_header_ptr = reinterpret_cast<BYTE*>(dosHeader2);
nt_header2 = reinterpret_cast<IMAGE_NT_HEADERS64*>(dos_header_ptr + dosHeader2->e_lfanew);
file_header = &nt_header2->FileHeader;
sections.clear();
sections = ExtractSections<IMAGE_NT_HEADERS64>(nt_header2);
section_header_text2 = sections[0];
section_header_rsrc2 = sections[1];
section_header_text2->SizeOfRawData += file_alignment;
section_header_text2->Misc.VirtualSize = section_header_text2->SizeOfRawData;
for(int i=1; i<sections.size(); i++)
sections[i]->PointerToRawData += file_alignment;
optional_headers2 = &nt_header2->OptionalHeader;
optional_headers2->SizeOfCode += file_alignment;
delete old_file_info;
FlushToFile(input_params.file_name_output, file_info2);
RelocateBufferWithDisplacement function
file_info* RelocateBufferWithDisplacement(file_info* file_info2, DWORD file_alignment, IMAGE_SECTION_HEADER* text_section_header)
{
// COPY BUFFERS
DWORD new_file_size = file_info2->file_size + file_alignment;
byte* new_buffer = new byte[new_file_size];
DWORD first_part_of_file_size = text_section_header->PointerToRawData + text_section_header->SizeOfRawData;
byte* end_first_part_old = file_info2->buffer + first_part_of_file_size;
byte* start_free_zone_of_file_ptr_new = new_buffer + first_part_of_file_size;
byte* second_part_of_file_ptr_new = start_free_zone_of_file_ptr_new + file_alignment;
std::copy(file_info2->buffer, end_first_part_old, new_buffer);
std::fill(start_free_zone_of_file_ptr_new, second_part_of_file_ptr_new, 0);
std::copy(end_first_part_old, file_info2->buffer + file_info2->file_size, second_part_of_file_ptr_new);
return new file_info(new_file_size, new_buffer);
}
ExtractSections method
std::vector<IMAGE_SECTION_HEADER*>& ExtractSections(IMAGE_NT_HEADERS64* nt_headers)
{
std::vector<IMAGE_SECTION_HEADER*>* result = new std::vector<IMAGE_SECTION_HEADER*>();
for(int i=0; i < nt_headers->FileHeader.NumberOfSections; i++)
result->push_back(reinterpret_cast<IMAGE_SECTION_HEADER*>(reinterpret_cast<BYTE*>(nt_headers) + sizeof(TNT_HEADERS) + i * sizeof(IMAGE_SECTION_HEADER)));
return *result;
}
That works the same way of both x32/x64 files, but now only x64 version is considered to simplification.
I also noticed:
If I use a larger project (e.g. up to 30000 code lines) I can't do it even for 2048 file alignment. Only for 512 and 1024. But if the project is almost clear - only 8192 fails.
I expect that something should be also updated, but I don't know what exactly.
Just a high level vision of the code:
So, what I'm doing wrong?
Thanks.
I have created a simple Protobuf based config file and have been using it without any issues until now. The issue is that I added two new items to my settings (Config.proto) and now whatever value I set for the last variable is reflected in the previous one.
The following snapshot demonstrates this better. As you can see below, the value of fv_shape_predictor_path and fv_eyenet_path depend solely on order of being set. the one that is set last changes the others value.
I made sure the cpp files related to Config.proto are built afresh. I also tested this under Linux and there it works just fine. It seems its only happening in windows! it also doesn't affect any other items in the same settings. its just these two new ones.
I have no idea what is causing this or how to go about it. For reference this is how the protobuf looks like:
syntax = "proto3";
package FVConfig;
message Config {
FV Configuration = 4 ;
message FAN_MODELS_WEIGHTS{
string fan_2DFAN_4 = 1;
string fan_3DFAN_4 = 2;
string fan_depth = 3;
}
message S3FD_MODELS_WEIGHTS{
string s3fd = 1;
}
message DLIB_MODELS_WEIGHTS{
string dlib_default = 1;
}
message MTCNN_MODELS_WEIGHTS {
string mt_onet = 1;
string mt_pnet = 2;
string mt_rnet = 3;
}
message FV_MODEL_WEIGHTS {
string r18 = 1;
string r50 = 2;
string r101 = 3;
repeated ModelContainer new_models_weights = 4;
message ModelContainer{
string model_name = 1;
string model_weight_path = 2;
string description = 3;
}
}
message FV {
MTCNNDetectorSettings mtcnn = 1 ;
FaceVerificationSettings fv = 2 ;
}
message MTCNNDetectorSettings {
Settings settings = 1;
MTCNN_MODELS_WEIGHTS model_weights = 4;
message Settings {
string mt_device = 2;
int32 mt_webcam_source = 100;
int32 mt_upper_threshold = 600;
int32 mt_hop = 700;
}
}
message FaceVerificationSettings {
Settings settings = 1;
FV_MODEL_WEIGHTS model_weights = 2;
message Settings {
string fv_model_name = 1;
string fv_model_checkpoint_path = 2;
bool fv_rebuild_cache = 3;
bool fv_short_circut = 6;
bool fv_accumulate_score = 7;
string fv_config_file_path = 10;
string fv_img_bank_folder_root = 11;
string fv_cache_folder = 12;
string fv_postfix = 13;
string fv_device = 14;
int32 fv_idle_interval = 15;
bool fv_show_dbg_info = 16;
// these are the new ones
string fv_shape_predictor_path = 17;
string fv_eyenet_path = 18;
}
}
} //end of Config message
What am I missing here? How should I be going about this? Restarting Windows and Visual Studio didn't do any good either. I'm using protobuf 3.11.4 both on Linux and Windows.
This issue seems to only exist in the Windows version of Protobuf 3.11.4 (didn't test with any newer version though).
Basically what happened was that I use to first create a Config object and initialize it with some default values. When I added these two entries to my Config.proto, I forgot to also add an initialization entry like other entries, thinking I'm fine with the default (which I assumed would be "").
This doesn't pose any issues under Linux/G++ and the program builds and runs just fine and works as intended. However under Windows this results in the behavior you just witnessed i.e. setting any of those newly added entries, would also set the other entries values. So basically I either had to create a whole new Config object or had to explicitly set their values when using the load_default_config.
To be more concrete this is the snippet I used for setting some default values in my Protobuf configs.
These reside in a separate header called Utility.h:
inline FVConfig::Config load_default_config()
{
FVConfig::Config baseCfg;
auto config = baseCfg.mutable_configuration();
load_fv_default_settings(config->mutable_fv());
load_mtcnn_default_settings(config->mutable_mtcnn());
return baseCfg;
}
inline void load_fv_default_settings(FVConfig::Config_FaceVerificationSettings* fv)
{
fv->mutable_settings()->set_fv_config_file_path(fv::config_file_path);
fv->mutable_settings()->set_fv_device(fv::device);
fv->mutable_settings()->set_fv_rebuild_cache(fv::rebuild_cache);
...
// these two lines were missing previously and to my surprise, this was indeed
// the cause of the weird behavior.
fv->mutable_settings()->set_fv_shape_predictor_path(fv::shape_predictor_path);
fv->mutable_settings()->set_fv_eyenet_path(fv::eyenet_path);
auto new_model_list = fv->mutable_model_weights()->mutable_new_models_weights()->Add();
new_model_list->set_model_name("r18");
new_model_list->set_description("default");
new_model_list->set_model_weight_path(fv::model_weights_r18);
}
inline void load_mtcnn_default_settings(FVConfig::Config_MTCNNDetectorSettings* mt)
{
mt->mutable_settings()->set_mt_device(mtcnn::device);
mt->mutable_settings()->set_mt_hop(mtcnn::hop);
....
}
Not sure this counts as a bug in Protobuf, or my wrong approach here.
I have the following proto file
package DS3DExcite.Cpp.ExternalVariantSwitcher.ProtocolBuffers;
message DGCommand
{
extensions 100 to max;
enum Type
{
AttachModel = 1;
AttachModelReply = 2;
....
...
SwitchVariants = 4;
}
required Type type = 1;
required uint32 id = 2;
optional uint32 replyTo = 3 [default = 0];
optional string message = 4;
}
message SwitchVariants
{
extend DGCommand
{
required SwitchVariants command = 103;
}
message VariantState
{
required string variant = 1;
required string state = 2;
}
repeated VariantState variants = 1;
repeated string variantNames = 2;
optional string state = 3;
}
I compiled the proto file with protobuf 2.4.1 version to generate .pb.h and .pb.cc files
Now I form the commands
DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand commandObj;
commandObj.set_type(DS3DExcite::Net::PVCConnector::ProtocolBuffers::DGCommand_Type_SwitchVariants);
commandObj.set_id(3);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants *objVarState;
objVarState = commandObj.MutableExtension(DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants::command);
DS3DExcite::Net::PVCConnector::ProtocolBuffers::SwitchVariants_VariantState *state = objVarState->add_variants();
state->set_state("OFF");
state->set_variant("M_Carpaint_3");
I serialise the message
int size = commandObj.ByteSize();
int sizeSize = sizeof(int);
std::vector<char> data(size ,0);
memcpy(data.data(), &size, sizeSize);
data.resize(size + sizeSize );
commandObj.SerializeToArray(static_cast<void*>(&(data[0])+sizeSize) ,size);
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
And I send this message on a Qtcp socket to server which deserializes the message and extract the information from the message .
At the server end this is the code to read
uint32 pendingData = 0;
rcvSocket->HasPendingData(pendingData); //rcvSocket is the serversside socket
if (pendingData == 0)
{
UE_LOG(PVCConnector, Warning, TEXT("Lost connection to client."));
break;
}
TArray<char> newData; //customized Array template
newData.InsertZeroed(0, pendingData);
int32 bytesRead = 0;
rcvSocket->Recv(reinterpret_cast<uint8*>(newData.GetData()), pendingData, bytesRead);
data += newData;
However at the server end the the desired information lands up in the unknown fields of ::google::protobuf::Message . What could be the reason ?
I had a similar problem whe I have been sending big enough messages. We decide, that this happens when message divided into several net packages. We use blobs to prevent that, and it works. About blobs, its technique to send message length, before message
I was able to solve the problem . There were 2 issues
1) The way I was converting to ByteArray
I replaced
QByteArray byteArray = QByteArray::fromRawData(static_cast<const char*>(data.data()), data.size());
with
QByteArray *byteArray = new QByteArray(reinterpret_cast<const char*>(data.data()), data.size());
2) The way I was sending the message on the socket . I just used
const int nbBytes = itM->second->write(qByteArray);
instead of using QTextStream
My Vulkan implementation returns VK_ERROR_INITIALIZATION_FAILED whenever i call vkCreateImage. I checked output of vkGetPhysicalDeviceImageFormatProperties as well as output of vkGetPhysicalDeviceFormatProperties and made sure that format of the image i'm creating (with regard to it's creation parameters) is supported.
I've successfully initialized a Vulkan instance and a logical device, so the problem must lie somewhere else.
The Vulkan Spec mentions no VK_ERROR_INITIALIZATION_FAILED as a possible return value from vkCreateImage, which makes it more hard to uproot the problem.
The VkImageCreateInfo structure i push to vkCreateImage is as follows:
imageCreateInfo.sType = VK_STRUCTURE_TYPE_IMAGE_CREATE_INFO;
imageCreateInfo.pNext = nullptr;
imageCreateInfo.flags = VK_IMAGE_CREATE_MUTABLE_FORMAT_BIT;
imageCreateInfo.imageType = VK_IMAGE_TYPE_2D;
imageCreateInfo.format = VK_FORMAT_R8G8B8A8_UNORM;
imageCreateInfo.extent.width = 1024;
imageCreateInfo.extent.height = 1024;
imageCreateInfo.extent.depth = 1;
imageCreateInfo.arrayLayers = 1;
imageCreateInfo.initialLayout = VK_IMAGE_LAYOUT_UNDEFINED;
imageCreateInfo.samples = VK_SAMPLE_COUNT_1_BIT;
imageCreateInfo.usage = VK_IMAGE_USAGE_TRANSFER_SRC_BIT | VK_IMAGE_USAGE_TRANSFER_DST_BIT | VK_IMAGE_USAGE_SAMPLED_BIT;
imageCreateInfo.mipLevels = 0;
imageCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
imageCreateInfo.tiling = VK_IMAGE_TILING_OPTIMAL;
imageCreateInfo.queueFamilyIndexCount = 0;
imageCreateInfo.pQueueFamilyIndices = nullptr;
As described in the specification:
mipLevels must be greater than 0
So judging by Your code, providing 0 value for number of mip levels may be the reason (or one of the reasons) of failure.