Get int from byte array by offset - c++

I am new to C++. Can't get int from byte array by offset.
When i read directly from memory all works fine and i am gettings 100 - this is correct value
int base = 0x100;
int offset = 0x256;
int easy = memory->ReadMemory<int>(base + offset); // easy = 100
But if i try to get a chunk of bytes and read from them, here problem comes
template<class T>
T FromBuffer(uint8_t* buffer, size_t offset)
{
T t_buf = 0;
memcpy(&t_buf, buffer + offset, sizeof(T));
return t_buf;
}
uint8_t* ReadBytes(DWORD Address, int Size)
{
auto arr = new uint8_t[Size];
ReadProcessMemory(TargetProcess, (LPVOID)Address, arr, sizeof(arr), 0);
return arr;
}
auto bytes = memory->ReadBytes(base, 2500);
int hard = *((unsigned int *)&bytes[offset]); // hard = -842150451
uint32_t hard2 = memory->FromBuffer<uint32_t>(bytes, offset); // hard2 = 3452816845
With C# it would easy like this
int hard = BitConverter.ToInt32(bytes, offset);

Converting this type of C# code to C++ doesn't make any sense, you were forced to do some whacky stuff in C# because doing this type of operation is not what C# was intended for.
You do not need to create a dynamic buffer and do any of that wizardry. Just do:
template <class T>
T RPM(void* addr)
{
T t;
ReadProcessMemory(handle, addr, &t, sizeof(t), nullptr);
return t;
}
int RPM(addr + offset);

Related

Standard way of overlay flexible array member

So the server sends the data just as packed structures, so what only need to decode is to overlay the structure pointer on the buffer. However one of the structure is a dynamic array kind of data, but I learned that flexible array member is not a C++ standard feature. How can I do it in standard C++ way, but without copying like a vector?
// on wire format: | field a | length | length of struct b |
// the sturcts are defined packed
__pragma(pack(1))
struct B {
//...
};
struct Msg {
int32_t a;
uint32_t length;
B *data; // how to declare this?
};
__pragma(pack())
char *buf = readIO();
// overlay, without copy and assignments of each field
const Msg *m = reinterpret_cast<const Msg *>(buf);
// access m->data[i] from 0 to length
The common way to do this in C was to declare data as an array of length one as the last struct member. You then allocate the space needed as if the array was larger.
Seems to work fine in C++ as well. You should perhaps wrap access to the data in a span or equivalent, so the implementation details don't leak outside your class.
#include <string>
#include <span>
struct B {
float x;
float y;
};
struct Msg {
int a;
std::size_t length;
B data[1];
};
char* readIO()
{
constexpr int numData = 3;
char* out = new char[sizeof(Msg) + sizeof(B) * (numData - 1)];
return out;
}
int main(){
char *buf = readIO();
// overlay, without copy and assignments of each field
const Msg *m = reinterpret_cast<const Msg *>(buf);
// access m->data[i] from 0 to length
std::span<const B> data(m->data, m->length);
for(auto& b: data)
{
// do something
}
return 0;
}
https://godbolt.org/z/EoMbeE8or
A standard solution is to not represent the array as a member of the message, but rather as a separate object.
struct Msg {
int a;
size_t length;
};
const Msg& m = *reinterpret_cast<const Msg*>(buf);
span<const B> data = {
reinterpret_cast<const B*>(buf + sizeof(Msg)),
m.length,
};
Note that reinterpretation / copying of bytes is not portable between systems with different representations (byte endianness, integer sizes, alignments, subobject packing etc.), and same representation is typically not something that can be assumed in network communication.
// on wire format: | field a | length | length of struct b |
You can't overlay the struct, because you can't guarantee that the binary representation of Msg will match the on wire format. Also int is at least 16 bits, can be any number of bits greater than 16, and size_t has various size depending on architecture.
Write actual accessors to the data. Use fixed width integer types. It will only work if the data actually point to a properly aligned region. This method allows you to write assertions and throw exceptions when stuff goes bad (for example, you can throw on out-of-bounds access to the array).
struct Msg {
constexpr static size_t your_required_alignment = alingof(uint32_t);
char *buf;
Msg (char *buf) : buf(buf) {
assert((uintptr_t)buf % your_required_alignment == 0);
}
int32_t& get_a() { return *reinterpret_cast<int32_t*>(buf); }
uint32_t& length() { return *reinterpret_cast<uint32_t *>(buf + sizeof(int32_t)); }
struct Barray {
char *buf;
Barray(char *buf) : buf(buf) {}
int16_t &operator[](size_t idx) {
return *reinterpret_cast<int16_t*>(buf + idx * sizeof(int16_t));
}
}
Barray data() {
return buf + sizeof(int32_t) + sizoef(uint32_t);
}
};
int main() {
Msg msg(readIO());
std::cout << msg.a() << msg.length();
msg.data()[1] = 5;
// or maybe even implement straight operator[]:
// msg[1] = 5;
}
If the data do not point to a properly aligned region, you have to copy the data, there is no possibility to access them using other types then char.

Difficulties deleting memory

I'm having some trouble working with memory: I have to keep a copy of some data in a new class. The main problem is that the first 9 bytes of this data should be thrown away. Whenever the object gets deleted though, I either get a segmentation fault or SIGABRT (it's not even consistent)
class Frame
{
public:
Frame();
~Frame();
void setFirstData(uint8_t *data, size_t dataLength);
void setSecondData(uint8_t *data, size_t dataLength);
void setThirdData(uint8_t *data, size_t dataLength);
void setFourthData(uint8_t *data, size_t dataLength);
...
private:
unsigned char *_firstData;
bool _firstDataSet;
size_t _firstDataLength;
unsigned char *_secondData;
bool _secondDataSet;
size_t _secondDataLength;
unsigned char *_thirdData;
bool _thirdDataSet;
size_t _thirdDataLength;
unsigned char *_fourthData;
bool _fourthDataSet;
size_t _fourthDataLength;
};
Frame::Frame()
{
_firstDataSet = false;
_secondDataSet = false;
_thirdDataSet = false;
_fourthDataSet = false;
}
Frame::~Frame()
{
if (_firstDataSet)
delete [] _firstData;
if (_secondDataSet)
delete[] _secondData;
if (_thirdDataSet)
delete[] _thirdData;
if (_fourthDataSet)
delete[] _fourthData;
}
void Frame::setFirstData(uint8_t *data, size_t dataLength)
{
//copy all the data in a unsigned char*, except for the first 9 bytes
_firstDataLength = dataLength - 9;
_firstData = new unsigned char[_firstDataLength];
memcpy(_firstData, data + 9, _firstDataLength*sizeof(*_firstData));
/*for (int i = 0; i < dataLength - 9; i++)
{
_firstData[i] = (unsigned char) data[i + 9];
}*/
_firstDataSet = true;
}
The other setData functions are identical to setFirstData, but with the correct arrays.
Am I supposed to use something else than memcpy? Or is the usage wrong? The commented for loop was my original method of 'copying' the data but I dont think it actually copies the data (original array will be deleted when the copied data still has to be available).
EDIT: I added the qt tag because i'm working in a Qt environment and using some Qt classes for GUI. I don't think qt has anything to do with these basic C++ functions.
What with setting firstData:
_firstData = new unsigned char[_dataLength];
memcpy(_firstData, data + 9, _dataLength*sizeof(*_firstData));

Send struct with char[] in MPI

I'm trying to send some data from workers to a master (ranked 0) in a MPI program in C++. The goal is to pass 2 strings and an integer. To do that, I've created a struct.
The struct
It is called word and defined as follows:
struct word
{
char word_name[str_size];
char url[str_size];
int counter;
};
/* Some important variables to the question */
MPI_Datatype mpi_word_type;
const int str_size = 200;
Previously I was trying this through char * but it doesn't work, since the processes don't share the same memory space.
By now, I'm able to send the struct if I change the variables from char[] to a simple char and try with an example. With this as it is above, I cannot get rid of Segmentation fault errors.
Sending Part - Workers
I start by creating and filling an example struct and then sending the size of the struct first and the struct itself, secondly. Like this:
word word_col;
std::string tmp = "somename";
strcpy(word_col.word_name, tmp.c_str());
std::string tmp2 = "someurl";
strcpy(word_col.url, tmp2.c_str());
word_col.counter = 10;
int size = sizeof(word_col);
MPI::COMM_WORLD.Send(&size, 1, MPI::INT, 0, 1);
MPI::COMM_WORLD.Send(&word_col, size, mpi_word_type, 0, 1);
Receiving Part - Master
const int nitems = 3;
int blocklengths[3] = { str_size, str_size, 1 };
MPI_Datatype types[3] = { MPI::CHAR, MPI::CHAR, MPI::INT };
MPI_Aint offsets[3];
offsets[0] = (MPI_Aint) offsetof(struct word, word_name);
offsets[1] = (MPI_Aint) offsetof(struct word, url);
offsets[2] = (MPI_Aint) offsetof(struct word, counter);
MPI_Type_create_struct(nitems, blocklengths, offsets, types, &mpi_word_type);
MPI_Type_commit(&mpi_word_type);
...
for(...)
{
word word_col;
int size;
MPI::COMM_WORLD.Recv(&size, 1, MPI::INT, MPI::ANY_TAG, 1, status);
MPI::COMM_WORLD.Recv(&word_col, size, mpi_word_type, MPI::ANY_TAG, 1, status);
}
I've been struggling with this for hours and I've seen many examples and another questions about this, but I couldn't figure out what is the problem here.
This is erroneous programming. You have unallocated and uninitialized pointer and you are trying to push data to that.
You have two options:
Either you define your structure as:
const int str_size = 200;
struct word
{
char word_name[str_size]; // fixed sized char array
char url[str_size]; // fixed sized char array
int counter;
};
Or,
const int str_size = 200;
struct word
{
char *word_name; /
char *url;
int counter;
Word() {
word_name = new char[str_size];
url = new char[str_size];
}
~Word() {
delete [] word_name;
delete [] url;
}
};
The idea is that you need to have allocated memory for these variables
Also, while receiving, you have used:
MPI::COMM_WORLD.Recv(&word_col, size, mpi_word_type, MPI::ANY_TAG, 1, status);
Shouldn't it be like below?
MPI::COMM_WORLD.Recv(&word_col, sizeof(word_col), mpi_word_type, MPI::ANY_TAG, 1, status);

Pointer type casting altering unintended memory

#define ARRAY_SIZE 20
float DataSource[ARRAY_SIZE];
void Read(unsigned char const *Source, unsigned char *Destination, unsigned long DataSize)
{
for ( unsigned long i = 0; i < DataSize; i++)
{
*(Destination + i*DataSize) = *(Source + i*DataSize);
}
}
void fun()
{
int Index;
float Dest;
for ( Index = 0; Index < ARRAY_SIZE; Index++ )
{
Read((unsigned char *)&DataSource[Index], (unsigned char *)&Dest, sizeof(DataSource[Index]));
}
}
I'm having an issue with the above code where upon calling Read(), my Index variable gets overwritten and I am certain the ugly pointer casting is the culprit, but I'm having trouble understanding exactly what is happening here.
The unsigned char pointer types are mandatory because the above code is intended to simulate some driver level software and maintain the same prototype.
Can someone help me to understand the issue here? All the above code is changeable except for the prototype of Read().
The error is here:
for ( unsigned long i = 0; i < DataSize; i++)
{
// vvvvvvvvvv vvvvvvvvvv
*(Destination + i*DataSize) = *(Source + i*DataSize);
}
i * DataSize is always greater than i => "out of bound" access.
Replace with:
for ( unsigned long i = 0; i < DataSize; i++)
{
*(Destination + i) = *(Source + i);
}
You pass in a single float's address to Read (&Dest) and then proceed to write many valuese to consecutive memory locations. Since you're writing random memory at that point it's not unlikely that it could have overwritten index (and other stuff) because stacks usually grow downwards.
This is wrong:
*(Destination + i*DataSize) = *(Source + i*DataSize);
You want to copy DataSize adjacent bytes, not bytes DataSize apart (total span DataSize*DataSize)
Just say
Destination[i] = Source[i];
An amusing (to me) C++ way.
template<typename Data>
struct MemBlockRefHelper {
typedef Data value_type;
Data* data;
size_t size;
MemBlockRefHelper( Data* d, size_t s):data(d), size(s) {}
template<typename Target, typename Other=typename Target::value_type>
Target& Assign( MemBlockRefHelper<Other> const& other ) {
Assert(size == other.size);
for (size_t i = 0; i < size; ++i) {
if (i < other.size) {
data[i] = other.data[i];
} else {
data[i] = 0;
}
}
Target* self = static_cast<Target*>(this);
return *self;
}
};
struct MemBlockRef;
struct MemBlockCRef:MemBlockRefHelper<const unsigned char> {
MemBlockCRef( const unsigned char* d, size_t s ):MemBlockRefHelper<const unsigned char>( d, s ) {}
MemBlockCRef( const MemBlockRef& other );
};
struct MemBlockRef:MemBlockRefHelper<unsigned char> {
MemBlockRef( unsigned char* d, size_t s ):MemBlockRefHelper<unsigned char>( d, s ) {}
MemBlockRef& operator=( MemBlockRef const& other ) {
return Assign< MemBlockRef >( other );
}
MemBlockRef& operator=( MemBlockCRef const& other ) {
return Assign< MemBlockRef, const unsigned char >( other );
}
};
inline MemBlockCRef::MemBlockCRef( const MemBlockRef& other ): MemBlockRefHelper<const unsigned char>( other.data, other.size ) {}
void Read( unsigned char const* Source, unsigned char* Dest, unsigned long DataSize ) {
MemBlockCRef src( Source, DataSize );
MemBlockRef dest( Dest, DataSize );
dest = src;
}
massively over engineered, but the idea is to wrap up the idea of a block of POD memory of a certain size, and provide reference semantics to its contents (initialization is creating a new reference to the same data, assignment does a copy over the referred to data).
Once you have such classes, the code for Read becomes a 3 liner. Well, you can do it in one:
MemBlockRef( Dest, DataSize ) = MemBlockCRef( Source, DataSize );
but that is needless.
Well, so it this entire framework.
But I was amused by writing it.
Let's take a closer look at your Read(): i changes from 0 to DataSize-1; each time you access memory by an offset of i*DataSize... that is, by an offset from 0 to DataSize*(DataSize-1). Looks wrong, as DataSize**2-DataSize makes no sense.
Unlike other answers, I don't want to guess what you wanted. Just showing a kind of "dimensional analysis" that can help spotting the wrongest part of code without reading the author's mind.
You are treating the scalar variable Dest declared inside fun() as an array inside Read(). It seems that both Dest and your Index variable are placed adjacent on the stack which explains that Index gets overwritten exactly when the loop inside Read() is executed for i==1.
So the solution is: declare Dest as an array, too:
float Dest[ARRAY_SIZE];

How to byteswap a double?

I'm trying to write a byteswap routine for a C++ program running on Win XP. I'm compiling with Visual Studio 2008. This is what I've come up with:
int byteswap(int v) // This is good
{
return _byteswap_ulong(v);
}
double byteswap(double v) // This doesn't work for some values
{
union { // This trick is first used in Quake2 source I believe :D
__int64 i;
double d;
} conv;
conv.d = v;
conv.i = _byteswap_uint64(conv.i);
return conv.d;
}
And a function to test:
void testit() {
double a, b, c;
CString str;
for (a = -100; a < 100; a += 0.01) {
b = byteswap(a);
c = byteswap(b);
if (a != c) {
str.Format("%15.15f %15.15f %15.15f", a, c, a - c);
}
}
}
Getting these numbers not matching:
-76.789999999988126 -76.790000000017230 0.000000000029104
-30.499999999987718 -30.499999999994994 0.000000000007276
41.790000000014508 41.790000000029060 -0.000000000014552
90.330000000023560 90.330000000052664 -0.000000000029104
This is after having read through:
How do I convert between big-endian and little-endian values in C++?
Little Endian - Big Endian Problem
You can't use << and >> on double, by the way (unless I'm mistaken?)
Although a double in main memory is 64 bits, on x86 CPUs double-precision registers are 80 bits wide. So if one of your values is stored in a register throughout, but the other makes a round-trip through main memory and is truncated to 64 bits, this could explain the small differences you're seeing.
Maybe you can force variables to live in main memory by taking their address (and printing it, to prevent the compiler from optimizing it out), but I'm not certain that this is guaranteed to work.
b = byteswap(a);
That's a problem. After swapping the bytes, the value is no longer a proper double. Storing it back to a double is going to cause subtle problems when the FPU normalizes the value. You have to store it back into an __int64 (long long). Modify the return type of the method.
Try 3
Okay, found out there's a better way. The other way you have to worry about the order you pack/unpack stuff. This way you don't:
// int and float
static void swap4(void *v)
{
char in[4], out[4];
memcpy(in, v, 4);
out[0] = in[3];
out[1] = in[2];
out[2] = in[1];
out[3] = in[0];
memcpy(v, out, 4);
}
// double
static void swap8(void *v)
{
char in[8], out[8];
memcpy(in, v, 8);
out[0] = in[7];
out[1] = in[6];
out[2] = in[5];
out[3] = in[4];
out[4] = in[3];
out[5] = in[2];
out[6] = in[1];
out[7] = in[0];
memcpy(v, out, 8);
}
typedef struct
{
int theint;
float thefloat;
double thedouble;
} mystruct;
static void swap_mystruct(void *buf)
{
mystruct *ps = (mystruct *) buf;
swap4(&ps->theint);
swap4(&ps->thefloat);
swap8(&ps->thedouble);
}
Send:
char buf[sizeof (mystruct)];
memcpy(buf, &s, sizeof (mystruct));
swap_mystruct(buf);
Recv:
mystruct s;
swap_mystruct(buf);
memcpy(&s, buf, sizeof (mystruct));
Try 2
Okay, got it working! Hans Passant was right. They got me thinking with the "no longer a proper double" comment. So you can't byteswap a float into another float because then it might be in an improper format, so you have to byteswap to a char array and unswap back. This is the code I used:
int pack(int value, char *buf)
{
union temp {
int value;
char c[4];
} in, out;
in.value = value;
out.c[0] = in.c[3];
out.c[1] = in.c[2];
out.c[2] = in.c[1];
out.c[3] = in.c[0];
memcpy(buf, out.c, 4);
return 4;
}
int pack(float value, char *buf)
{
union temp {
float value;
char c[4];
} in, out;
in.value = value;
out.c[0] = in.c[3];
out.c[1] = in.c[2];
out.c[2] = in.c[1];
out.c[3] = in.c[0];
memcpy(buf, out.c, 4);
return 4;
}
int pack(double value, char *buf)
{
union temp {
double value;
char c[8];
} in, out;
in.value = value;
out.c[0] = in.c[7];
out.c[1] = in.c[6];
out.c[2] = in.c[5];
out.c[3] = in.c[4];
out.c[4] = in.c[3];
out.c[5] = in.c[2];
out.c[6] = in.c[1];
out.c[7] = in.c[0];
memcpy(buf, out.c, 8);
return 8;
}
int unpack(char *buf, int *value)
{
union temp {
int value;
char c[4];
} in, out;
memcpy(in.c, buf, 4);
out.c[0] = in.c[3];
out.c[1] = in.c[2];
out.c[2] = in.c[1];
out.c[3] = in.c[0];
memcpy(value, &out.value, 4);
return 4;
}
int unpack(char *buf, float *value)
{
union temp {
float value;
char c[4];
} in, out;
memcpy(in.c, buf, 4);
out.c[0] = in.c[3];
out.c[1] = in.c[2];
out.c[2] = in.c[1];
out.c[3] = in.c[0];
memcpy(value, &out.value, 4);
return 4;
}
int unpack(char *buf, double *value)
{
union temp {
double value;
char c[8];
} in, out;
memcpy(in.c, buf, 8);
out.c[0] = in.c[7];
out.c[1] = in.c[6];
out.c[2] = in.c[5];
out.c[3] = in.c[4];
out.c[4] = in.c[3];
out.c[5] = in.c[2];
out.c[6] = in.c[1];
out.c[7] = in.c[0];
memcpy(value, &out.value, 8);
return 8;
}
And a simple test function:
typedef struct
{
int theint;
float thefloat;
double thedouble;
} mystruct;
void PackStruct()
{
char buf[sizeof (mystruct)];
char *p;
p = buf;
mystruct foo, foo2;
foo.theint = 1;
foo.thefloat = 3.14f;
foo.thedouble = 400.5;
p += pack(foo.theint, p);
p += pack(foo.thefloat, p);
p += pack(foo.thedouble, p);
// Send or recv char array
p = buf;
p += unpack(p, &foo2.theint);
p += unpack(p, &foo2.thefloat);
p += unpack(p, &foo2.thedouble);
}
How to swap the bytes in any basic data type or array of bytes
ie: How to swap the bytes in place in any array, variable, or any other memory block, such as an int16_t, uint16_t, uint32_t, float, double, etc.:
Here's a way to improve the efficiency from 3 entire copy operations of the array to 1.5 entire copy operations of the array. See also the comments I left under your answer. I said:
Get rid of this: memcpy(in, v, 4); and just copy-swap straight into out from v, then memcpy the swapped values back from out into v. This saves you an entire unnecessary copy, reducing your copies of the entire array from 3 to 2.
There's also a further optimization to reduce the copies of the entire array from 2 to 1.5: copy the left half of the array into temporary variables, and the right-half of the array straight into the left-half, swapping as appropriately. Then copy from the temporary variables, which contain the old left-half of the array, into the right-half of the array, swapping as appropriately. This results in the equivalent of only 1.5 copy operations of the entire array, to be more efficient. Do all this in-place in the original array, aside from the temp variables you require for half of the array.
1. Here is my general C and C++ solution:
/// \brief Swap all the bytes in an array to convert from little-endian
/// byte order to big-endian byte order, or vice versa.
/// \note Works for arrays of any size. Swaps the bytes **in place**
/// in the array.
/// \param[in,out] byte_array The array in which to swap the bytes in-place.
/// \param[in] len The length (in bytes) of the array.
/// \return None
void swap_bytes_in_array(uint8_t * byte_array, size_t len)
{
size_t i_left = 0; // index for left side of the array
size_t i_right = len - 1; // index for right side of the array
while (i_left < i_right)
{
// swap left and right bytes
uint8_t left_copy = byte_array[i_left];
byte_array[i_left] = byte_array[i_right];
byte_array[i_right] = left_copy;
i_left++;
i_right--;
}
}
Usage:
// array of bytes
uint8_t bytes_array[16];
// Swap the bytes in this array of bytes in place
swap_bytes_in_array(bytes_array, sizeof(bytes_array));
double d;
// Swap the bytes in the double in place
swap_bytes_in_array((uint8_t*)(&d), sizeof(d));
uint64_t u64;
// swap the bytes in a uint64_t in place
swap_bytes_in_array((uint8_t*)(&u64), sizeof(u64));
2. And here is an optional C++ template wrapper around that to make it even easier to use in C++:
template <typename T>
void swap_bytes(T *var)
{
// Note that `sizeof(*var)` is the exact same thing as `sizeof(T)`
swap_bytes_in_array((uint8_t*)var, sizeof(*var));
}
Usage:
double d;
// Swap the bytes in the double in place
swap_bytes(&d);
uint64_t u64;
// swap the bytes in a uint64_t in place
swap_bytes(&u64);
Notes & unanswered questions
Note, however, that #Hans Passant seems to be onto something here. Although the above works perfectly on any signed or unsigned integer type, and seems to work on float and double for me too, it seems to be broken on long double. I think it's because when I store the swapped long double back into a long double variable, if it is determined to be not-a-valid long double representation anymore, something automatically changes a few of the swapped bytes or something. I'm not entirely sure.
On many 64-bit systems, long double is 16 bytes, so perhaps the solution is to keep the swapped version of the long double inside a 16-byte array and NOT attempt to use it or cast it back to a long double from the uint8_t 16-byte array until either A) it has been sent to the receiver (where the endianness of the system is opposite, so it's in good shape now) and/or B) byte-swapped back again so it's a valid long double again.
Keep the above in mind in case you see problems with float or double types too, as I see with only long double types.
Linux byteswap and endianness and host-to-network byte order utilities
Linux also has a bunch of built-in utilities via gcc GNU extensions that you can use. See:
https://man7.org/linux/man-pages/man3/bswap.3.html - #include <byteswap.h>
https://man7.org/linux/man-pages/man3/endian.3.html - #include <endian.h>
https://man7.org/linux/man-pages/man3/byteorder.3.html - #include <arpa/inet.h> - generally used for network sockets (Ethernet packets) and things; inet stands for "internet"