I am trying to access the data that is serialized using boost buffer function and would like to fill it into two vectors. I am having problem with address to fill the second vector. Following class shows the two vectors and how they are filled.
class LidarMeasurement {
private:
std::vector<uint32_t> _header;
std::vector<float> _azimuth;
public:
//The header consists of an array of uint32_t's in the following layout
enum Index : size_t {
HorizontalAngle,
ChannelCount,
SIZE
};
explicit LidarMeasurement(uint32_t NumOfChannels = 0u): _header(Index::SIZE + NumOfChannels, 0u) {
_header[Index::ChannelCount] = NumOfChannels;
}
// called before filling vectors
void Reset(uint32_t total_point_count) {
std::memset(_header.data() + Index::SIZE, 0, sizeof(uint32_t) * GetChannelCount());
_azimuth.clear();
_azimuth.reserve(total_point_count);
}
// after reset,Write point function starts filling vectors.. following function is called 104 times (not constant) before next reset
void WritePoint(uint32_t channel, float angle_hor) {
_header[Index::SIZE + channel] += 1u;
_azimuth.emplace_back(angle_hor);
}
uint32_t GetChannelCount() const {
return _header[Index::ChannelCount];
}
}
Once they are filled, its serialized and sent to a client. its serialized using the function below:
template <typename Sensor>
inline Buffer LidarSerializer::Serialize(
const Sensor &,
const LidarMeasurement &measurement,
Buffer &&output) {
std::array<boost::asio::const_buffer, 2u> seq = {
boost::asio::buffer(measurement._header),
boost::asio::buffer(measurement._azimuth)};
output.copy_from(seq);
return std::move(output);
}
Once I receive the serialized data, I need to put azimuth back to vector.
I am using the following function to get the vector. _begin is the address to the buffer.
std::vector<float> GetAzimuth(const uint32_t* _begin) const{
std::vector<float> localAzimuthMemCopy;
begin_azi = const_cast<float*>(reinterpret_cast<const float*>(_begin )) + (sizeof(uint32_t) * (GetChannelCount() + Index::SIZE));
end_azi = begin_azi + GetTotalPointCount();//Total point count is the addition of individual channel point counts (not shown here)
for(float* i = begin_azi; i < end_azi; i++){
localAzimuthMemCopy.emplace_back(*i);
}
return localAzimuthMemCopy;
}
However, the result i get has a memory offset. I am getting 104 values but the last 18 values are junk. vector is read from a wrong start address. What is wrong with the code?
The problem is caused bt wrong begin adress calculation.
begin_azi = const_cast<float*>(reinterpret_cast<const float*>(_begin )) + (sizeof(uint32_t) * (GetChannelCount() + Index::SIZE));
1) Pointer arithmetic requires only the pointer and number of elements to advance. Number of bytes the compiler should deduct by himself, based on the pointers type. So multiplication at sizeof(uint32_t) is redundant. The correct way of pointer advance is shown at float* end_azi = begin_azi + GetTotalPointCount();
2) Adress offset should be calculated for pointer to uint32_t type, and only then converted to pointer to float type.
So correct way of begin_azi should look this way:
begin_azi = const_cast<float*>(reinterpret_cast<const float*>(_begin + GetChannelCount() + Index::SIZE));
Why did it partially worked earlier? from cppreference
Pointer arithmetic
If the pointer P points at an element of an array with index I, then
P+N and N+P are pointers that point at an element of the same array with index I+N
P-N is a pointer that points at an element of the same array with index {tt|I-N}}
The behavior is defined only if both the original pointer and the result pointer are pointing at elements of the same array or one past the end of that array.
Noone knows where did the pointed begin_azi pointed after wrong calculation. So noone guarantee that the program will execute in correct or wrong way.
Related
I need to store and restore a vector with elements of different size and want to adjust my solution to get this working. The goal is to get a size of the data in bytes + get a void pointer to the beginning of the data.
I use a template class in my code and combine solution 1 and 2 in it, but for the sake of simplicity I split the code up in this question.
1) Solution for a vector of elements where all elements have the same size:
1.1) Save a vector as raw data - with fixed size elements, e.g. int:
std::vector<int> data;
Sint32 size = sizeof(int) * vec.size();
void* tdata = static_cast<void*>(&vec[0]);
// tdata points to the raw data => I can work with this now and save it to my memory structure (tdata + size is saved there)
1.2) Restore a vector from raw data - with fixed size elements, e.g. int:
std::vector<int> vec;
size_t size = ...; // received from my memory structure
void* tdata = ...; // received from my memory structure
int count = size / sizeof(int);
vec.resize(count);
memcpy(&vec[0], tdata, size);
// vec contains count elements of type int now
2) Solution for a vector of elements where all elements have dynamic sizes:
I simply define two helper interfaces/classes to get the size of all elements inside the vector:
class ObjectWithSize
{
public:
// eg. for an int + char*: sizeof(int) + strlen(text) + 1
virtual int getRawSize() = 0;
};
class RawDataWithCount
{
public:
int count;
void* data;
};
2.1) Save a vector of dynamic elements:
Sint32 datasize = 0;
for (int i = 0; i < vec.size(); i++)
datasize += vec.at(i).getRawSize();
RawDataWithCount data = RawDataWithCount();
data.count = vec.size();
data.data = static_cast<void*>(&vec[0]);
void* tdata = static_cast<void*>(&data);
// tdata points to the raw data => I can work with this now and save it to my memory structure (tdata + datasize is saved there)
2.2) Restore a vector of dynamic elements
std::vector<T> vec;
size_t size = ...; // received from my memory structure
void* tdata = ...; // received from my memory structure
RawDataWithCount* data = reinterpret_cast<RawDataWithCount*>(tdata);
int count = data->count;
vec.resize(count);
memcpy(&vec[0], data->data, size);
Information
Solution 1 is used for a long time already and works quite good, solution 2 will be an extension of it for the future. I think, 2.1 is save to use, but when it comes to 2.2 I'm not sure anymore, so here comes the question. Tests show that solution 2 seems to work, but I want to know if it is save.
Question
Is it save to do following in 2.2
vec.resize(count);
memcpy(&vec[0], data->data, size);
if the elements of vec do not all have the same size? I assume not, but I'm unsure here. If this is not save, maybe someone has a better suggestion? (I know I can manually solve this by simple creating a raw data in 2.1 with number of elements followed by a list of size/element, then parsing back elements 1 by 1 in 2.2 and fill up the vector with this data, but maybe (like in case 1)
there is a more simple solution with the help of a vector?)
I have the address of the first char in my byte array, and it's size:
const char *rawImageBytes, int size
And I want to copy the content to a different byte array. and then modify that one a bit.
This is whay I am doing now:
LOGI("FrameReceived will reach here 1");
modifiedRawImageBytes = rawImageBytes;
jint sizeWH = width * height;
jint quarter = sizeWH/4;
jint v0 = sizeWH + quarter;
for (int u = sizeWH, v = v0, o = sizeWH; u < v0; u++, v++, o += 2) {
modifiedRawImageBytes[o] = rawImageBytes[v]; // For NV21, V first
modifiedRawImageBytes[o + 1] = rawImageBytes[u]; // For NV21, U second
}
But I don't get the correct colours, as if I would to this in Java, instead of c++.
And I am assuming this happens, because I just do modifiedRawImageBytes = rawImageBytes; instead of actually copying the whole byte array, so that it can start in memory from another address pointer.
A bit of a beginner with c, so I'm lost at this, can someone help me understand what is done wrong?
PS: I am assuming that, because even if I send the rawImageBytes and not the modifiedRawImageBytes, it will still be modified
This is because const char * is a pointer. This mean it represent an address. So you guessed right, the new variable represent the same datas.
To avoid this you should create a copy.
char modifiedRawImageBytes[size];
//if the pointer come from function's param don't redeclare it ;)
std::memcpy(modifiedRawImageBytes, rawImageBytes, size*sizeof(char));
This code will allocate a new char array and then memcpy will copy in the previous array data in the new array.
Note that you need to includecstdio
I have attached my code below. I do not see what I am doing wrong. I have a struct that I am trying to serialize into a byte array. I have wrote some some simple code to test it. It all appears to work during runtime when I print out the values of objects, but once I hit return 0 it throws the error:
Run-Time Check Failure #2 - Stack around the variable 'command' was corrupted.
I do not see the issue. I appreciate all help.
namespace CommIO
{
enum Direction {READ, WRITE};
struct CommCommand
{
int command;
Direction dir;
int rwSize;
BYTE* wData;
CommCommand(BYTE* bytes)
{
int offset = 0;
int intsize = sizeof(int);
command = 0;
dir = READ;
rwSize = 0;
memcpy(&command, bytes + offset, intsize);
offset += intsize;
memcpy(&dir, bytes + offset, intsize);
offset += intsize;
memcpy(&rwSize, bytes + offset, intsize);
offset += intsize;
wData = new BYTE[rwSize];
if (dir == WRITE)
{
memcpy(&wData, bytes + offset, rwSize);
}
}
CommCommand() {}
}
int main()
{
CommIO::CommCommand command;
command.command = 0x6AEA6BEB;
command.dir = CommIO::WRITE;
command.rwSize = 128;
command.wData = new BYTE[command.rwSize];
for (int i = 0; i < command.rwSize; i++)
{
command.wData[i] = i;
}
command.print();
CommIO::CommCommand command2(reinterpret_cast<BYTE*>(&command));
command2.print();
cin.get();
return 0;
}
The following points mentioned in comments are most likely the causes of your problem.
You seem to be assuming that the size of Direction is the same as the size of an int. That may indeed be the case, but C++ does not guarantee it.
You also seem to be assuming that the members of CommIO::CommCommand will be laid out in memory without any padding between, which again may happen to be the case, but is not guaranteed.
There are couple of ways to fix the that.
Make sure that you fill up the BYTE array in the calling function with matching objects, or
Simply cast the BYTE* to CommCommand* and access the members directly.
For (1), you can use:
int command = 0x6AEA6BEB;
int dir = CommIO::WRITE;
int rwSize = 128;
totatlSize = rwSize + 3*sizeof(int);
BYTE* data = new BYTE[totalSize];
int offset = 0;
memcpy(data + offset, &comand, sizeof(int));
offset += sizeof(int);
memcpy(data + offset, &dir, sizeof(int));
offset += sizeof(int);
memcpy(data + offset, &rwSize, sizeof(int));
offset += sizeof(int);
for (int i = 0; i < rwSize; i++)
{
data[i + offset] = i;
}
CommIO::CommCommand command2(data);
For (2), you can use:
CommCommand(BYTE* bytes)
{
CommCommand* in = reinterpret_cast<CommCommand*>(bytes);
command = in->command;
dir = in->dir;
rwSize = in->size;
wData = new BYTE[rwSize];
if (dir == WRITE)
{
memcpy(wData, in->wData, rwSize);
}
}
The other error is that you are using
memcpy(&wData, bytes + offset, rwSize);
That is incorrect since you are treating the address of the variable as though it can hold the data. It cannot.
You need to use:
memcpy(wData, bytes + offset, rwSize);
The memory for your struct is laid out without padding, this can be rectified by adding the macro #pragma pack(1) at the start of the struct and #pragma pop() at the end of the struct - check its syntax though.
For your struct to byte conversion, I would use something simple as:
template<typename T, typename IteratorForBytes>
void ConvertToBytes(const T& t, IteratorForBytes bytes, std::size_t pos = 0)
{
std::advance(bytes, pos);
const std::size_t length = sizeof(t);
const uint8_t* temp = reinterpret_cast<const uint8_t*>(&t);
for (std::size_t i = 0; i < length; ++i)
{
(*bytes) = (*temp);
++temp;
++bytes;
}
}
Where T is the is the struct in your case your Command struct and bytes would be the array.
CommIO::CommCommand command;
command.wData = new BYTE[command.rwSize];
ConvertToBytes(command, command.wData);
The resulting array would contain the expected bytes You could specify the offset as well as an extra parameter if you want to start filling your byte array from a particular location
The main problem is here:
memcpy(&wData, bytes + offset, rwSize);
Member wData is a BYTE *, and you seem to mean to copy bytes into the space to which it points. Instead, you are copying data into the memory where the pointer value itself is stored. Therefore, if you copy more bytes than the size of the pointer then you will overrun its bounds and produce undefined behavior. In any case, you are trashing the original pointer value. You probably want this, instead:
memcpy(wData, bytes + offset, rwSize);
Additionally, although the rest of the deserialization code may be right for your actual serialization format, it is not safe to assume that it is right for the byte sequence you present to it in your test program via
CommIO::CommCommand command2(reinterpret_cast<BYTE*>(&command));
As detailed in comments, you are making assumptions about the layout in memory of a CommIO::CommCommand that C++ does not guarantee will hold.
At
memcpy(&wData, bytes + offset, rwSize);
you copy from the location of the wData pointer and to the location of the wData pointer of the new CommCommand. But you want to copy from and to the location that the pointer points to. You need to dereference. You corrupt the heap, because you have only sizeof(BYTE*) space (plus some extra, because heap blocks cannot be arbitrarily small), but you copy rwSize bytes, which is 128 bytes. What you probably meant to write is:
memcpy(wData, *(BYTE*)(bytes + offset), rwSize);
which would take use the pointer stored at bytes + offset, rather than the value of bytes + offset itself.
You also assume that your struct is tightly packed. However, C++ does not guarantee that. Is there a reason why you do not override the default copy constructor rather than write this function?
Is there a way to cross over all elements in integer array using pointer ( similiar to using pointer to cross over string elements).I know that integer array is not NULL terminated so when I try to cross over array using pointer it overflows.So I added NULL as a last element of an array and it worked just fine.
int array[7]={1,12,41,45,58,68,NULL};
int *i;
for(i=array;*i;i++)
printf("%d ",*i);
But what if one of the elements in array is 0 ,that will behave just as NULL.Is there any other way that will implement pointer in crossing over all elements in integer array?
In general, no unless you pick a sentinel value that's not part of the valid range of the data. For example, the valid range might be positive numbers, so you can use a negative number like -1 as a sentinel value that indicates the end of the array. This how C-style strings work; the NULL terminator is used because it's outside of the valid range of integers that could represent a character.
However, it's usually better to somehow pair up the array pointer with another variable that indicates the size of the array, or another pointer that points one-past-the-end of the array.
In your specific case, you can do something like this:
// Note that you don't have to specify the length of the array.
int array[] = {1,12,41,45,58,68};
// Let the compiler count the number of elements for us.
int arraySize = sizeof(array)/sizeof(int);
// or int arraySize = sizeof(array)/sizeof(array[0]);
int main()
{
int* i;
for(i = array; i != array + arraySize; i++)
printf("%d ",*i);
}
You can also do this:
int arrayBegin[] = {1,12,41,45,58,68};
int* arrayEnd = arrayBegin + sizeof(arrayBegin)/sizeof(arrayBegin[0]);
int main()
{
int* i;
for(i = arrayBegin; i != arrayEnd; i++)
printf("%d ",*i);
}
But given only a pointer, no you can't know how long the array it points to is. In fact, you can't even tell if the pointer points to an array or a single object! (At least not portably.)
If you have functions that must accept an array, either have your function require:
the pointer and the size of the array pointed by the pointer,
or two pointers with one pointing to the first element of the array and one pointing one-past-the-end of the array.
I'd like to give some additional advice: Never use some kind of sentinel/termination value in arrays for determining their bounds. This makes your programs prone to error and is often the cause for security issues. You should always store the length of arrays to limit all operations to their bounds and test against that value.
In C++ you have the STL and its containers.
In C you'll effectively end up using structures like
typedef struct t_int_array
{
size_t length;
int data[1]; /* note the 1 (one) */
} int_array;
and a set of manipulation functions like this
int_array * new_int_array(size_t length)
{
int_array * array;
/* we're allocating the size of basic t_int_array
(which already contains space for one int)
and additional space for length-1 ints */
array = malloc( sizeof(t_int_array) + sizeof(int) * (length - 1) );
if(!array)
return 0;
array->length = length;
return array;
}
int_array * concat_int_arrays(int_array const * const A, int_array const * const B);
int_array * int_array_push_back(int_array const * const A, int const value);
/* and so on */
This method will make the compiler align the t_int_array struct in a way, that it's optimal for the targeted architecture (also with malloc allocation), and just allocating more space in quantities of element sizes of the data array element will keep it that way.
The reason that you can iterate across a C-style string using pointers is that of the 256 different character values, one has been specifically reserved to be interpreted as "this is the end of the string." Because of this, C-style strings can't store null characters anywhere in them.
When you're trying to use a similar trick for integer arrays, you're noticing the same problem. If you want to be able to stop at some point, you'll have to pick some integer and reserve it to mean "this is not an integer; it's really the end of the sequence of integers." So no, there is no general way to take an array of integers and demarcate the end by a special value unless you're willing to pick some value that can't normally appear in the string.
C++ opted for a different approach than C to delineate sequences. Instead of storing the elements with some sort of null terminator, C++-style ranges (like you'd find in a vector, string, or list) store two iterators, begin() and end(), that indicate the first element and first element past the end. You can iterate over these ranges by writing
for (iterator itr = begin; itr != end; ++itr)
/* ... visit *itr here ... */
This approach is much more flexible than the C-string approach to defining ranges as it doesn't rely on specific properties of any values in the range. I would suggest opting to use something like this if you want to iterate over a range of integer values. It's more explicit about the bounds of the range and doesn't run into weird issues where certain values can't be stored in the range.
Apart from the usual suggestion that you should go and use the STL, you can find the length of a fixed array like this:
int array[6]={1,12,41,45,58,68};
for (int i = 0; i < sizeof(array) / sizeof(array[0]); ++i)
{ }
If you use a templated function, you can implicitly derive the length like this:
template<size_t len> void func(int (&array)[len])
{
for (int i = 0; i < len; ++i) { }
}
int array[6]={1,12,41,45,58,68};
func(array);
If 0 is a value that may occur in a normal array of integers, you can specify a different value:
const int END_OF_ARRAY = 0x80000000;
int array[8]={0,1,12,41,45,58,68,END_OF_ARRAY};
for (int i = 0; array[i] != END_OF_ARRAY; ++i)
{ }
If every value is a possibility, or if none of the other approaches will work (for example, a dynamic array) then you have to manage the length separately. This is how strings that allow embedded null characters work (such as BSTR).
In your example you are using (or rather abusing) the NULL macro as a sentinel value; this is the function of the NUL('\0') character in a C string, but in the case of a C string NUL is not a valid character anywhere other than as the terminal (or sentinel) value .
The NULL macro is intended to represent an invalid pointer not an integer value (although in C++ when implicitly or explicitly cast to an int, its value is guaranteed to be zero, and in C this is also almost invariably the case). In this case if you want to use zero as the sentinel value you should use a literal zero not NULL. The problem is of course that if in this application zero is a valid data value it is not suitable for use as a sentinel.
So for example the following might suit:
static const int SENTINEL_VALUE = -1 ;
int array[7] = { 1, 12, 41, 45, 58, 68, SENTINEL_VALUE } ;
int* i ;
for( i = array; *i != SENTINEL_VALUE; i++ )
{
printf( "%d ", *i ) ;
}
If all integer values are are valid data values then you will not be able to use a sentinel value at all, and will have to use either a container class (which knows its length) or iterate for the known length of the array (from sizeof()).
Just to pedanticize and expand a little on a previous answer: in dealing with integer arrays in C, it's vanishingly rare to rely on a sentinel value in the array itself. No(1) sane programmer does that. Why not? Because by definition an integer can hold any value within predefined negative/positive limits, or (for the nowadays-not-unusual 32-bit integer) 0 to 0xffffff. It's not a good thing to redefine the notion of "integer" by stealing one of its possible values for a sentinel.
Instead, one always(1) must(1) rely on a controlling up-to-date count of integers that are in the array. Suppose we are to write a C function
that returns an int pointer to the first array member whose value is greater than the function's argument or, if there's no such member, returns NULL (all code is untested):`
int my_int_array[10]; // maximum of 10 integers in my_int_array[], which must be static
int member_count = 0; // varies from 0 to 10, always holds number of ints in my_int_array[]
int *
first_greater_than ( int val ) {
int i;
int *p;
for ( i = 0, p = my_int_array; i < member_count; ++i, ++p ) {
if ( *p > val ) {
return p;
}
}
return NULL;
}
Even better is also to limit the value of i to never count past the last possible member of my_int_array[], i.e., it never gets bigger than 9, and p never points at my_int_array[10] and beyond:
int my_int_array[10]; // maximum of 10 integers in my_int_array[], which must be static
int member_count = 0; // varies from 0 to 10, always holds number of ints in my_int_array[]
int *
first_greater_than ( int val ) {
#define MAX_COUNT sizeof(my_int_array)/sizeof(int)
int i;
int* p;
for ( i = 0, p = my_int_array; i < member_count && i < MAX_COUNT; ++i, ++p ) {
if ( *p > val ) {
return p;
}
}
return NULL;
}
HTH and I apologize if this is just too, too elementary.
--pete
Not strictly true but believe it for now
In ANSI C it's very easy and shorter than solution before:
int array[]={1,12,41,45,58,68}, *i=array;
size_t numelems = sizeof array/sizeof*array;
while( numelems-- )
printf("%d ",*i++);
Another way is to manage array of pointers to int:
#include <stdlib.h>
#include <stdio.h>
#define MAX_ELEMENTS 10
int main() {
int * array[MAX_ELEMENTS];
int ** i;
int k;
// initialize MAX_ELEMENTS,1 matrix
for (k=0;k<MAX_ELEMENTS;k++) {
array[k] = malloc(sizeof(int*));
// last element of array will be NULL pointer
if (k==MAX_ELEMENTS-1)
array[k] = NULL;
else
array[k][0] = k;
}
// now loop until you get NULL pointer
for (i=array;*i;i++) {
printf("value %i\n",**i);
}
// free memory
for (k=0;k<MAX_ELEMENTS;k++) {
free(array[k]);
}
return 0;
}
In this way loop condition is totally independent from the values of integers. But... for this to work you must use 2D array (matrix) instead of ordinary 1D array. Hope that helps.
CODE:
struct Stringdata
{
// Length of data in buffer.
size_t len;
// Allocated size of buffer.
size_t alc;
// Buffer.
char data[1];
};
typedef std::list<Stringdata*> Stringdata_list;
Stringdata_list strings_;
Stringdata *psd = this->strings_.front();
//...
if (len > psd->alc - psd->len)
alc = sizeof(Stringdata) + buffer_size;
else
{
char* ret = psd->data + psd->len;
memcpy(ret, s, len - sizeof(Stringpool_char));
memset(ret + len - sizeof(Stringpool_char), 0,
sizeof(Stringpool_char));
psd->len += len;
return reinterpret_cast<const Stringpool_char*>(ret);
}
In the code sample above, I have confused about the operations in the else
branch.
Does it create a new element and insert it after the front element or
just place a new element after within the first element of list?
Your code appears to do neither. The code in the else branch does not modify the strings_ structure at all. The code is only modifying the element return from the front of the list. This should have no affect on the actual list structure.
It doesn't create a new element -- just appends data from s to the data that's already in the front element, if there's space. Very confusingly written code, though.
As far as I can tell (some important code is missing from your excerpt), you have a block of data, which is essentially an array of Stringdata object, and a list<> of pointers into that block. The else block is expanding that array.
You probably would be better off with a vector<Stringdata> rather than a list<Stringdata*>