Send struct with char[] in MPI - c++

I'm trying to send some data from workers to a master (ranked 0) in a MPI program in C++. The goal is to pass 2 strings and an integer. To do that, I've created a struct.
The struct
It is called word and defined as follows:
struct word
{
char word_name[str_size];
char url[str_size];
int counter;
};
/* Some important variables to the question */
MPI_Datatype mpi_word_type;
const int str_size = 200;
Previously I was trying this through char * but it doesn't work, since the processes don't share the same memory space.
By now, I'm able to send the struct if I change the variables from char[] to a simple char and try with an example. With this as it is above, I cannot get rid of Segmentation fault errors.
Sending Part - Workers
I start by creating and filling an example struct and then sending the size of the struct first and the struct itself, secondly. Like this:
word word_col;
std::string tmp = "somename";
strcpy(word_col.word_name, tmp.c_str());
std::string tmp2 = "someurl";
strcpy(word_col.url, tmp2.c_str());
word_col.counter = 10;
int size = sizeof(word_col);
MPI::COMM_WORLD.Send(&size, 1, MPI::INT, 0, 1);
MPI::COMM_WORLD.Send(&word_col, size, mpi_word_type, 0, 1);
Receiving Part - Master
const int nitems = 3;
int blocklengths[3] = { str_size, str_size, 1 };
MPI_Datatype types[3] = { MPI::CHAR, MPI::CHAR, MPI::INT };
MPI_Aint offsets[3];
offsets[0] = (MPI_Aint) offsetof(struct word, word_name);
offsets[1] = (MPI_Aint) offsetof(struct word, url);
offsets[2] = (MPI_Aint) offsetof(struct word, counter);
MPI_Type_create_struct(nitems, blocklengths, offsets, types, &mpi_word_type);
MPI_Type_commit(&mpi_word_type);
...
for(...)
{
word word_col;
int size;
MPI::COMM_WORLD.Recv(&size, 1, MPI::INT, MPI::ANY_TAG, 1, status);
MPI::COMM_WORLD.Recv(&word_col, size, mpi_word_type, MPI::ANY_TAG, 1, status);
}
I've been struggling with this for hours and I've seen many examples and another questions about this, but I couldn't figure out what is the problem here.

This is erroneous programming. You have unallocated and uninitialized pointer and you are trying to push data to that.
You have two options:
Either you define your structure as:
const int str_size = 200;
struct word
{
char word_name[str_size]; // fixed sized char array
char url[str_size]; // fixed sized char array
int counter;
};
Or,
const int str_size = 200;
struct word
{
char *word_name; /
char *url;
int counter;
Word() {
word_name = new char[str_size];
url = new char[str_size];
}
~Word() {
delete [] word_name;
delete [] url;
}
};
The idea is that you need to have allocated memory for these variables
Also, while receiving, you have used:
MPI::COMM_WORLD.Recv(&word_col, size, mpi_word_type, MPI::ANY_TAG, 1, status);
Shouldn't it be like below?
MPI::COMM_WORLD.Recv(&word_col, sizeof(word_col), mpi_word_type, MPI::ANY_TAG, 1, status);

Related

Standard way of overlay flexible array member

So the server sends the data just as packed structures, so what only need to decode is to overlay the structure pointer on the buffer. However one of the structure is a dynamic array kind of data, but I learned that flexible array member is not a C++ standard feature. How can I do it in standard C++ way, but without copying like a vector?
// on wire format: | field a | length | length of struct b |
// the sturcts are defined packed
__pragma(pack(1))
struct B {
//...
};
struct Msg {
int32_t a;
uint32_t length;
B *data; // how to declare this?
};
__pragma(pack())
char *buf = readIO();
// overlay, without copy and assignments of each field
const Msg *m = reinterpret_cast<const Msg *>(buf);
// access m->data[i] from 0 to length
The common way to do this in C was to declare data as an array of length one as the last struct member. You then allocate the space needed as if the array was larger.
Seems to work fine in C++ as well. You should perhaps wrap access to the data in a span or equivalent, so the implementation details don't leak outside your class.
#include <string>
#include <span>
struct B {
float x;
float y;
};
struct Msg {
int a;
std::size_t length;
B data[1];
};
char* readIO()
{
constexpr int numData = 3;
char* out = new char[sizeof(Msg) + sizeof(B) * (numData - 1)];
return out;
}
int main(){
char *buf = readIO();
// overlay, without copy and assignments of each field
const Msg *m = reinterpret_cast<const Msg *>(buf);
// access m->data[i] from 0 to length
std::span<const B> data(m->data, m->length);
for(auto& b: data)
{
// do something
}
return 0;
}
https://godbolt.org/z/EoMbeE8or
A standard solution is to not represent the array as a member of the message, but rather as a separate object.
struct Msg {
int a;
size_t length;
};
const Msg& m = *reinterpret_cast<const Msg*>(buf);
span<const B> data = {
reinterpret_cast<const B*>(buf + sizeof(Msg)),
m.length,
};
Note that reinterpretation / copying of bytes is not portable between systems with different representations (byte endianness, integer sizes, alignments, subobject packing etc.), and same representation is typically not something that can be assumed in network communication.
// on wire format: | field a | length | length of struct b |
You can't overlay the struct, because you can't guarantee that the binary representation of Msg will match the on wire format. Also int is at least 16 bits, can be any number of bits greater than 16, and size_t has various size depending on architecture.
Write actual accessors to the data. Use fixed width integer types. It will only work if the data actually point to a properly aligned region. This method allows you to write assertions and throw exceptions when stuff goes bad (for example, you can throw on out-of-bounds access to the array).
struct Msg {
constexpr static size_t your_required_alignment = alingof(uint32_t);
char *buf;
Msg (char *buf) : buf(buf) {
assert((uintptr_t)buf % your_required_alignment == 0);
}
int32_t& get_a() { return *reinterpret_cast<int32_t*>(buf); }
uint32_t& length() { return *reinterpret_cast<uint32_t *>(buf + sizeof(int32_t)); }
struct Barray {
char *buf;
Barray(char *buf) : buf(buf) {}
int16_t &operator[](size_t idx) {
return *reinterpret_cast<int16_t*>(buf + idx * sizeof(int16_t));
}
}
Barray data() {
return buf + sizeof(int32_t) + sizoef(uint32_t);
}
};
int main() {
Msg msg(readIO());
std::cout << msg.a() << msg.length();
msg.data()[1] = 5;
// or maybe even implement straight operator[]:
// msg[1] = 5;
}
If the data do not point to a properly aligned region, you have to copy the data, there is no possibility to access them using other types then char.

Get int from byte array by offset

I am new to C++. Can't get int from byte array by offset.
When i read directly from memory all works fine and i am gettings 100 - this is correct value
int base = 0x100;
int offset = 0x256;
int easy = memory->ReadMemory<int>(base + offset); // easy = 100
But if i try to get a chunk of bytes and read from them, here problem comes
template<class T>
T FromBuffer(uint8_t* buffer, size_t offset)
{
T t_buf = 0;
memcpy(&t_buf, buffer + offset, sizeof(T));
return t_buf;
}
uint8_t* ReadBytes(DWORD Address, int Size)
{
auto arr = new uint8_t[Size];
ReadProcessMemory(TargetProcess, (LPVOID)Address, arr, sizeof(arr), 0);
return arr;
}
auto bytes = memory->ReadBytes(base, 2500);
int hard = *((unsigned int *)&bytes[offset]); // hard = -842150451
uint32_t hard2 = memory->FromBuffer<uint32_t>(bytes, offset); // hard2 = 3452816845
With C# it would easy like this
int hard = BitConverter.ToInt32(bytes, offset);
Converting this type of C# code to C++ doesn't make any sense, you were forced to do some whacky stuff in C# because doing this type of operation is not what C# was intended for.
You do not need to create a dynamic buffer and do any of that wizardry. Just do:
template <class T>
T RPM(void* addr)
{
T t;
ReadProcessMemory(handle, addr, &t, sizeof(t), nullptr);
return t;
}
int RPM(addr + offset);

Bsearch function issue in C

I am trying to use bsearch but the result of search keep staying NULL.My array of names is sorted alphabetically.I am new in C and need your help to understand where am I wrong.
Here is an array
// arrays of children's names
char *chBamba[] = { "Bambale1", "Bamb2", "Bamb3", "Bambook4", "Bambookle5" };
char *chBisli[] = { "Bislile1", "Bischick2", "Bislile3", "Bis4" };
int nBamba = sizeof(chBamba) / sizeof(*chBamba);
int nBisli = sizeof(chBisli) / sizeof(*chBisli);
// array of workers
Worker a[] = { { "Printf", 10, NULL, 0 } ,{ "Bisli", 20, chBisli, nBisli },
{ "TapooChips", 3, chBamba, nBamba },{ "Bamba", 19, chBamba, nBamba } };
This is struct
typedef struct{
char name[LEN]; // worker's name
int salary; // worker's salary
char **childArr; // array of children names
int numChildren; // number of children}Worker;
Compare function
int compareNames(const void* child1, const void* child2){
char* ch1 = (char*)child1;
char* ch2 = (char*)child2;
return strcmp(ch1, ch2);
}
And the bsearch
char childName[10];
char* nameFound;
printf("Please enter a child name to search for :");
scanf("%s",childName);
for (i = 0; i < 4; i++)
{
nameFound =(char*) bsearch(childName,a[i].childArr, a[i].numChildren, sizeof(a[i].childArr)/ sizeof(a[i].childArr[0]), compareNames);
}
It is C not C++.That what I am learning right now.I am using VS 2015 and it works for both C and C++.I am editing some changes to get it work correctly and compile.
So compare is
int compareNames(const void * s1, const void * s2){
const char *key = (char*)s1;
const char * const *arg = (char**)s2;
return strcmp(key, *arg);
}
and the bsearch is
nameFound = (char*)bsearch(childName,a[i].childArr, a[i].numChildren, sizeof(char*), compareNames);
The compare function is being passed pointers to the array's elements.
From man bsearch:
The compar routine is expected to have two arguments which point to the key
object and to an array member ....
The array's elements are of type char *, so the compare function should look like this:
int compareNames(const void * pvchild1, const void * pvchild2)
{
const char ** ppc1 = (const char **) pvchild1; // casting necessary with C++
const char ** ppc2 = (const char **) pvchild2; // casting necessary with C++
return strcmp(*ppc1, *ppc2);
}
To additionally cover the case that the array contained null-pointer values you could do:
...
return strcmp(ppc1 ?*ppc1 :"", ppc2 ?*ppc2 :"");
}
Call bsearch() like this:
char * p = childName;
nameFound =(char*) bsearch( // casting necessary with C++
&p,
a[i].childArr,
a[i].numChildren,
sizeof *(a[i].childArr),
compareNames
);
Also as a side note: The code you show defintily isn't C but C++, as this
Worker a[] = {
{ "Printf", 10, NULL, 0 } ,{ "Bisli", 20, chBisli, nBisli },
{ "TapooChips", 3, chBamba, nBamba },{ "Bamba", 19, chBamba, nBamba }
};
won't compile in C, but give the error:
error: initializer element is not constant

Difficulties deleting memory

I'm having some trouble working with memory: I have to keep a copy of some data in a new class. The main problem is that the first 9 bytes of this data should be thrown away. Whenever the object gets deleted though, I either get a segmentation fault or SIGABRT (it's not even consistent)
class Frame
{
public:
Frame();
~Frame();
void setFirstData(uint8_t *data, size_t dataLength);
void setSecondData(uint8_t *data, size_t dataLength);
void setThirdData(uint8_t *data, size_t dataLength);
void setFourthData(uint8_t *data, size_t dataLength);
...
private:
unsigned char *_firstData;
bool _firstDataSet;
size_t _firstDataLength;
unsigned char *_secondData;
bool _secondDataSet;
size_t _secondDataLength;
unsigned char *_thirdData;
bool _thirdDataSet;
size_t _thirdDataLength;
unsigned char *_fourthData;
bool _fourthDataSet;
size_t _fourthDataLength;
};
Frame::Frame()
{
_firstDataSet = false;
_secondDataSet = false;
_thirdDataSet = false;
_fourthDataSet = false;
}
Frame::~Frame()
{
if (_firstDataSet)
delete [] _firstData;
if (_secondDataSet)
delete[] _secondData;
if (_thirdDataSet)
delete[] _thirdData;
if (_fourthDataSet)
delete[] _fourthData;
}
void Frame::setFirstData(uint8_t *data, size_t dataLength)
{
//copy all the data in a unsigned char*, except for the first 9 bytes
_firstDataLength = dataLength - 9;
_firstData = new unsigned char[_firstDataLength];
memcpy(_firstData, data + 9, _firstDataLength*sizeof(*_firstData));
/*for (int i = 0; i < dataLength - 9; i++)
{
_firstData[i] = (unsigned char) data[i + 9];
}*/
_firstDataSet = true;
}
The other setData functions are identical to setFirstData, but with the correct arrays.
Am I supposed to use something else than memcpy? Or is the usage wrong? The commented for loop was my original method of 'copying' the data but I dont think it actually copies the data (original array will be deleted when the copied data still has to be available).
EDIT: I added the qt tag because i'm working in a Qt environment and using some Qt classes for GUI. I don't think qt has anything to do with these basic C++ functions.
What with setting firstData:
_firstData = new unsigned char[_dataLength];
memcpy(_firstData, data + 9, _dataLength*sizeof(*_firstData));

How to cast from char pointer to custom object pointer

I'm using leveldb to store key-value pairs of integer and MyClass objects. Actually, a key can contain more then one of theses objects.
The problem I have appears when retrieving the data from the database. It compiles, however the values of the MyClass members are not the one I put into the database.
std::string value;
leveldb::Slice keySlice = ANYKEY;
levelDBObj->Get(leveldb::ReadOptions(), keySlice, &value);
The std::string value1 can now contain only one MyClass object or more. So how do I get them?
I already tried the following which didn't work;
1.) directly typecasting and memcpy
std::vector<MyClass> vObjects;
MyClass* obj = (MyClass*)malloc( value.size());
memcpy((void*)obj, (void*) (value.c_str()), value.size());
MyClass dummyObj;
int numValues = value.size()/sizeof(MyClass);
for( int i=0; i<numValues; ++i) {
dummyObj = *(obj+i);
vObjects.push_back(dummyObj);
}
2.) reinterpret_cast to void pointer
MyClass* obj = (MyClass*)malloc( value.size());
const void* vobj = reinterpret_cast<const void*>( value.c_str() );
int numValues = value.size()/sizeof(MyClass);
for( int i=0; i<numValues; ++i) {
const MyClass dummyObj = *(reinterpret_cast<const MyClass*>(vobj)+i);
vObjects.push_back(dummyObj);
}
MyClass is a collection of several public members, e.g. unsigned int and unsigned char and it has a stable size.
I know that there are similar problems with only one object. But in my case the vector can contain more then one and it comes from the leveldb database.
EDIT: SOLUTION
I wrote (de)serialization method for MyClass which then made it working. Thanks for the hint!
void MyClass::serialize( char* outBuff ) {
memcpy(outBuff, (const void*) &aVar, sizeof(aVar));
unsigned int c = sizeof(aVar);
memcpy(outBuff+c, (const void*) &bVar, sizeof(bVar));
c += sizeof(bVAr);
/* and so on */
}
void MyClass::deserialize( const char* inBuff ) {
memcpy((void*) &aVar, inBuff, sizeof(aVar));
unsigned int c = sizeof(aVar);
memcpy((void*) &aVar, inBuff+c, sizeof(aVar));
c += sizeof(aVar);
/* and so on */
}
The get method is as follows (put analogously):
int getValues(leveldb::Slice keySlice, std::vector<MyObj>& values) const {
std::string value;
leveldb::Status status = levelDBObj->Get(leveldb::ReadOptions(), keySlice, &value);
if (!status.ok()) {
values.clear();
return -1;
}
int nValues = value1.size()/sizeof(CHit);
MyObj dummyObj;
for( int i=0; i<nValues; ++i) {
dummyObj.deserialize(value.c_str()+i*sizeof(MyObj));
values.push_back(dummyObj);
}
return 0;
}
You have to serialize your class... otherwise, you're just taking some memory and writing it in leveldb. Whatever you get back is not only going to be different, but it will probably be completely useless too. Check out this question for more info on serialization: How do you serialize an object in C++?
LevelDB does support multiple objects under one key, however, try to avoid doing that unless you have a really good reason. I would recommend that you hash each object with a unique hash (see Google's CityHash if you want a hashing function) and store the serialized objects with their corresponding hash. If your objects is a collection in itself, then you have to serialize all of your objects to an array of bytes and have some method that allows you to determine where each object begins/ends.
Update
A serializable class would look something like this:
class MyClass
{
private:
int _numeric;
string _text;
public:
// constructors
// mutators
void SetNumeric(int num);
void SetText(string text);
static unsigned int SerializableSize()
{
// returns the serializable size of the class with the schema:
// 4 bytes for the numeric (integer)
// 4 bytes for the unsigned int (the size of the text)
// n bytes for the text (it has a variable size)
return sizeof(int) + sizeof(unsigned int) + _text.size();
}
// serialization
int Serialize(const char* buffer, const unsigned int bufferLen, const unsigned int position)
{
// check if the object can be serialized in the available buffer space
if(position+SerializableSize()>bufferLen)
{
// don't write anything and return -1 signaling that there was an error
return -1;
}
unsigned int finalPosition = position;
// write the numeric value
*(int*)(buffer + finalPosition) = _numeric;
// move the final position past the numeric value
finalPosition += sizeof(int);
// write the size of the text
*(unsigned int*)(buffer + finalPosition) = (unsigned int)_text.size();
// move the final position past the size of the string
finalPosition += sizeof(unsigned int);
// write the string
memcpy((void*)(buffer+finalPosition), _text.c_str(), (unsigned int)_text.size());
// move the final position past the end of the string
finalPosition += (unsigned int)_text.size();
// return the number of bytes written to the buffer
return finalPosition-position;
}
// deserialization
static int Deserialize(MyClass& myObject,
const char* buffer,
const unsigned int buffSize,
const unsigned int position)
{
insigned int currPosition = position;
// copy the numeric value
int numeric = *(int*)(buffer + currentPosition);
// increment the current position past the numeric value
currentPosition += sizeof(int);
// copy the size of the text
unsigned int textSize = *(unsigned int*)(buffer + currentPosition);
// increment the current position past the size of the text
currentPosition += sizeof(unsigned int);
// copy the text
string text((buffer+currentPosition), textSize);
if(currentPosition > buffSize)
{
// you decide what to do here
}
// Set your object's values
myObject.SetNumeric(numeric);
myObject.SetText(text);
// return the number of bytes deserialized
return currentPosition - position;
}
};