Ok, i remade my question a bit, i do not understand how TO properly deflate the content into the DYNAMIC buffer, who needs to be constantly reallocating. I re-wrote my code a bit, and it work ONLY if no reallocating of the buffer happens, so on the small amount of data, reallocating breaks somehow the output stream.
void test_deflate_dynamic(char*str)
{
if(store == NULL) // first call to the function allocate some memory etc
{
gzip_stream.zalloc = Z_NULL;
gzip_stream.zfree = Z_NULL;
gzip_stream.opaque = Z_NULL;
result = deflateInit(&gzip_stream,9);
if(result!=Z_OK) { printf("bad\r\n"); exit(0); }
total_buf_size =deflateBound(&gzip_stream,strlen(str));
printf("d_bound_init=%d\r\n",total_buf_size);
store = realloc(store,total_buf_size); // first allocation
gzip_stream.avail_out = total_buf_size;
gzip_stream.next_out = store;
gzip_stream.avail_in = strlen(str)+1;
gzip_stream.next_in = str;
result = deflate(&gzip_stream,Z_NO_FLUSH);
}
else
{
gzip_stream.avail_in = strlen(str)+1;
gzip_stream.next_in = str;
int t_size;
printf ("avail_out=%d\r\n",gzip_stream.avail_out);
t_size = deflateBound(&gzip_stream,strlen(str));
printf("d_bound=%d\r\n",t_size);
total_buf_size += t_size;
gzip_stream.avail_out = total_buf_size;
store = realloc(store,total_buf_size);
gzip_stream.next_out = store;
result = deflate(&gzip_stream,Z_NO_FLUSH);
}
}
As you can see i'm using the function deflateBound to detect how much more data i need to allocate, so first, is it correct to use deflateBound? Second, is the pointer which modified by realloc and then re-assigned to the z_stream, still points to the beginning of the data? So basically if i'm using multiple reallocations the end data is broken. End: How do i proper detect, how much data do i need to allocate for my output deflate buffer, and is it correct to use the dynamic re-allocating buffer in the z_stream?
realloc ensures the data is copied to the new location when the new size didn't fit into the old memory position. calloc just zeros the allocated memory, and doesn't copy over the old data. Therefore you must pass bigger values to calloc. You probably only need one call to calloc when you pass it that big numbers whereas realloc can accept smaller increments. Make sense?
Cheers
Related
Hello I am having the following difficulty,
I am trying to read in a table of doubles (1 entry per line) and store it in an array, while dynamically changing this array's size (for each line/entry). This is for a school assignment and it forbids the use of vectors(would be much easier...). The main idea that I had is to have a main array which stores the value, then store the previous values and the next one into a new array and do this iteratively. Currently, the problem that I am having is that only the last value of the table is being stored. I am aware, that somehow I need to be passing the data by refference to the global function and that the pointers that I am working with become null ater they exit the following iteration of the while. However, since the exact length of the data is unknown, this seems impossible since intializing an array in the main() is impossible (exact length not known). Any help would be appreciated.
Code posted below.
EDIT: after consideration of the two comments I made the following changes to the code, however I am not sure, whether they will behave appropriately. I added a new function called add_new_datapoint, that should globally change the values of the pointer/length and this is done by passing the values by refference. Called in the problematic else statement as add_new_datapoint(data_ptr, data_len, new_dp). Also, I am not sure that reallocating new memory to the pointer variable, will not result in a memory leak. In essence (after I reallocate data_ptr is the memory that was 'being pointed to' released or do I have to delete it and then re-inialise it in the . In such case, can I refference the pointer 'data_ptr' again in the next iteration of the loop?
I think it will be easier to simplify your posted code than trying to find all the places where you could have errors.
If you expect to see only double values in your file, you can simplify the code for reading data from the file to:
while ( data_file >> new_data_pt )
{
// Use new_data_pt
}
If you expect that there might be values other than doubles, then you can use:
while ( getline(data_file, line) )
{
std::istringstream str(line);
while ( str >> new_data_pt )
{
// Use new_data_pt
}
}
but then you have to understand the code will not read any more values from a line after it encounters an error. If your line contains
10.2 K 25.4
the code will read 10.2, encounter an error at K, and will not process 25.4.
The code to process new_data_pt is that it needs to be stored in a dynamically allocated array. I would suggest putting that in a function.
double* add_point(double* data_ptr, int data_len, double new_data_pt)
Call that function as:
data_ptr = add_point(data_ptr, data_len, new_data_pt);
Assuming the first while loop, the contents of main become:
int main()
{
std::fstream data_file{ "millikan2.dat" };
// It is possible that the file has nothing in it.
// In that case, data_len needs to be zero.
int data_len{ 0 };
// There is no need to allocate memory when there is nothing in the file.
// Allocate memory only when data_len is greater than zero.
double* data_ptr = nullptr;
double new_data_pt;
if (!data_file.good()) {
std::cerr << "Cannot open file";
return 1;
}
while ( data_file >> new_data_pt )
{
++data_len;
data_ptr = add_point(data_ptr, data_len, new_data_pt);
}
// No need of this.
// The file will be closed when the function returns.
// data_file.close();
}
add_point can be implemented as:
double* add_point(double* data_ptr, int data_len, double new_data_pt)
{
double* new_data_ptr = new double[data_len];
// This works even when data_ptr is nullptr.
// When data_ptr is null_ptr, (data_len - 1) is zero. Hence,
// the call to std::copy becomes a noop.
std::copy(data_ptr, data_ptr + (data_len - 1); new_data_ptr);
// Deallocate old memory.
if ( data_ptr != nullptr )
{
delete [] data_ptr;
}
new_data_ptr[data_len-1] = new_data_pt;
return new_data_ptr;
}
The code to track the number of bad points is a lot more complex. Unless you are required to do it, I would advise to ignore it.
You already got an excellent answer but I figured it may be helpful to point out a few mistakes in your code, so you can understand why it won't work.
In the second else scope you declare data_ptr again, even though it is visible from the outer scope. (delete[] doesn't delete the pointer itself, it just deallocates the memory the pointer points to.)
else {
double* data_temp { new double[data_len] };
std::copy(data_ptr, data_ptr + data_len - 2, data_temp);
*(data_temp + data_len - 1) = new_data_pt;
delete[] data_ptr;
double* data_ptr{ new double[data_len] }; // <- Right here
//for (int j{1}; j < data_len; j++) *(data_ptr + j) = *(data_temp + j);
std::cout << std::endl;
}
Instead you could just write data_ptr = new double[data_len]. However, that alone won't make this work.
All of your data disappears because on every iteration you create a new array, pointed to by data_temp and copy the data there, and on the next iteration you set data_temp to point to a new array again. This means that on every iteration you lose all data from previous iterations. This also causes a memory leak, since you allocate more memory every time you hit this line:
double* data_temp { new double[data_len] };
but you don't call delete[] data_temp afterwards.
I hope this helps to understand why it doesn't work.
I get this error :
"crt detected that the application wrote to memory after end of heap
buffer"
this program should write every second sign...
char x = 1;
while ( x != 0) {
char *ptr(0);
ptr = new char;
fgets(ptr, 100001, stdin);
for (char i = 0; i < sizeof(ptr); i++) {
cout << *(ptr + i);
i++;
}
cout << endl;
delete ptr;
ptr = 0;
}
ptr = new char; allocates exactly one byte for the pointer. Than fgets(ptr, 100001, stdin); attempts to read up to 100001 characters into it, obviously well beyond it's allocated boundary.
The immediate solution would be to make sure you allocate the buffer with adequate size using array form of new, i.e. ptr = new char[100001];. This, in turn, will lead to changing the delete into it's array form as well: delete[] ptr
Also, keep in mind that sizeof(ptr) will give you the size of the pointer (4 or 8 bytes on most platforms). You would want to use strlen or similar to find out when the string ends.
All suggestions above should be taken in light of the learning exercise. In real life, you should not do this, and instead use std::string and modern ways of string-oriented reading. (std::getline would be the most readily available).
sizeof returns size in bytes of the object representation of type. https://en.cppreference.com/w/cpp/language/sizeof
One other point major point to note is that ptr = new char will only allocate exactly one byte from the heap. But in your code you are trying to read 100001 bytes and obviously this is beyond the accessible memory range of that pointer.
you can modify the error as ptr = new char[100001]; When you do this, there is another change that you must do in order to avoid memory leaks.
That being, delete[] ptr. Because you have allocated 100001 adjacent bytes, you have to de-allocate them all.
I have a class with a std::vector<unsigned char> mPacket as a packet buffer (for sending UDP strings). There is a corresponding member variable mPacketNumber that keeps track of how many packets have been sent so far.
The first thing I do in the class is reserve space:
mPacket.reserve(400);
and then later, in a loop that runs while I want packets to get sent:
mPacket.clear(); //empty out the vector
long packetLength = 0; //keep track of packetLength for sending udp strings
memcpy(&mPacket[0], &&mPacketNumber, 4); //4 bytes because it's a long
packetLength += 4; //add 4 bytes to the packet length
memcpy(&mPacket[packetLength], &data, dataLength);
packetLength += dataLength;
udp.send(mPacket.data(), packetLength);
Except I realized that nothing was getting sent! How peculiar.
So I dug a bit deeper, and found that mPacket.size() returns zero, while packetLength returns the size I think the packet should be.
I can't think of a reason for mPacket to have zero length -- even if I'm mishandling the data, the header with mPacketNumber should have been written just fine.
Can anyone suggest why I'm running into this problem?
thanks!
The elements you reserve are not for normal use. The elements are created only if you resize the vector. While it might somehow look it works, it would be a different situation with types having constructors - you could see that the constructors were not called. This is undefined behaviour - you're accessing elements which you aren't allowed in this situation.
The .reserve() operation is normally used together with .push_back() to avoid reallocations, but this is not the case here.
The .size() is not modified if you use .reserve(). You should use .resize() instead.
Alternatively, you can use your copy operation together with .push_back() and .reserve(), but you need to drop the usage of memcpy, and instead use the std::copy together with std::back_inserter, which uses .push_back() to push the elements to the other container:
std::copy(reinterpret_cast<unsigned char*>(&mPacketNumber), reinterpret_cast<unsigned char*>(&mPacketNumber) + sizeof(mPacketNumber), std::back_inserter(mPacket))
std::copy(reinterpret_cast<unsigned char*>(&data), reinterpret_cast<unsigned char*>(&data) + dataLength, std::back_inserter(mPacket));
These reinterpret_casts are vexing, but the code still has one advantage - you won't get buffer overrun in case your estimate was too low.
vector, apparently, doesn't count the elements when you call size(). There's a counter variable inside the vector that holds that information, because vector has plenty of memory allocated and can't really know where the end of your data is. It changes counter variable as you add/remove elements using methods of vector object, because they are programmed to do so.
You added data directly to its array pointer, which awakens no reaction of your vector object because it does not use any of its methods. Data is there, but vector doesn't acknowledge it, so counter remains at 0 and size() returns 0.
You should either replace all size() calls with packageLength, or use methods inside your vector to add/remove/read data, or use a dynamically allocated array instead of a vector, or create your own class for containing array and managing it the way you like it. To be honest, using a vector in a situation like this doesn't really make sense.
Vector is a conventional high-level object-oriented component and in most os the cases it should be used that way.
Example of one's own Array class:
If you used your own dynamically allocated array, you'd have to remember its length all the time in order to use it. So lets create a class that will cut us some slack in that. This example has element transfer based on memcpy, and the [] notation works perfectly. It has an original max length, but extends itself when necessary.
Also, this is an in-line class. certain IDEs may ask of you to actually seperate it in header and source file, so you may have to do that yourself.
Add more methods yourself if necessary. When applying this, do not use memcpy unless you're going to change arraySize attribute manually. You've got integrated addFrom and addBytesFrom methods that use memcpy inside (assuming calling array being the destination) and separately increase arraySize. If you do want to use memcpy, setSize method can be used for forcing new array size without modifying the array.
#include <cstring>
//this way you can easily change types during coding in case you change your mind
//more conventional object-oriented method would use templates and generic programming, but lets not complicate too much now
typedef unsigned char type;
class Array {
private:
type *array;
long arraySize;
long allocAmount; //number of allocated bytes
long currentMaxSize; //number of allocated elements
//private call that extends memory taken by the array
bool reallocMore()
{
//preserve old data
type *temp = new type[currentMaxSize];
memcpy(temp, array, allocAmount);
long oldAmount = allocAmount;
//calculate new max size and number of allocation bytes
currentMaxSize *= 16;
allocAmount = currentMaxSize * sizeof(type);
//reallocate array and copy its elements back into it
delete[] array;
array = new type[currentMaxSize];
memcpy(array, temp, oldAmount);
//we no longer need temp to take space in out heap
delete[] temp;
//check if space was successfully allocated
if(array) return true;
else return false;
}
public:
//constructor
Array(bool huge)
{
if(huge) currentMaxSize = 1024 * 1024;
else currentMaxSize = 1024;
allocAmount = currentMaxSize * sizeof(type);
array = new type[currentMaxSize];
arraySize = 0;
}
//copy elements from another array and add to this one, updating arraySize
bool addFrom(void *src, long howMany)
{
//predict new array size and extend if larger than currentMaxSize
long newSize = howMany + arraySize;
while(true)
{
if(newSize > currentMaxSize)
{
bool result = reallocMore();
if(!result) return false;
}
else break;
}
//add new elements
memcpy(&array[arraySize], src, howMany * sizeof(type));
arraySize = newSize;
return true;
}
//copy BYTES from another array and add to this one, updating arraySize
bool addBytesFrom(void *src, long byteNumber)
{
//predict new array size and extend if larger than currentMaxSize
int typeSize = sizeof(type);
long howMany = byteNumber / typeSize;
if(byteNumber % typeSize != 0) howMany++;
long newSize = howMany + arraySize;
while(true)
{
if(newSize > currentMaxSize)
{
bool result = reallocMore();
if(!result) return false;
}
else break;
}
//add new elements
memcpy(&array[arraySize], src, byteNumber);
arraySize = newSize;
return true;
}
//clear the array as if it's just been made
bool clear(bool huge)
{
//huge >>> 1MB, not huge >>> 1KB
if(huge) currentMaxSize = 1024 * 1024;
else currentMaxSize = 1024;
allocAmount = currentMaxSize * sizeof(type);
delete[] array;
array = new type[currentMaxSize];
arraySize = 0;
}
//if you modify this array out of class, you must manually set the correct size
bool setSize(long newSize) {
while(true)
{
if(newSize > currentMaxSize)
{
bool result = reallocMore();
if(!result) return false;
}
else break;
}
arraySize = newSize;
}
//current number of elements
long size() {
return arraySize;
}
//current number of elements
long sizeInBytes() {
return arraySize * sizeof(type);
}
//this enables the usage of [] as in yourArray[i]
type& operator[](long i)
{
return array[i];
}
};
mPacket.reserve();
mPacket.resize(4 + dataLength); //call this first and copy into, you can get what you want
mPacket.clear(); //empty out the vector
long packetLength = 0; //keep track of packetLength for sending udp strings
memcpy(&mPacket[0], &&mPacketNumber, 4); //4 bytes because it's a long
packetLength += 4; //add 4 bytes to the packet length
memcpy(&mPacket[packetLength], &data, dataLength);
packetLength += dataLength;
udp.send(mPacket, packetLength);
I have a class that parses some incoming serial data. After the parsing a method should return a byte array with some of the parsed data. The incoming data is of unknown length so my return array will always be different.
So far my method allocates an array bigger than what I need to return and fills it up with my data bytes and I keep an index so that I know how much data I put in the byte array. My problem is that I don't know how to return this from an instance method.
void HEXParser::getParsedData()
{
byte data[HEX_PARSER_MAX_DATA_SIZE];
int dataIndex = 0;
// fetch data, do stuff
// etc, etc...
data[dataIndex] = incomingByte;
_dataIndex++;
// At the very end of the method I know that all the bytes I need to return
// are stored in data, and the data size is dataIndex - 1
}
On other languages this is trivial to do but I'm not very proficient in C++ and I'm completely stuck.
Thanks!
You are working on a microcontroller with just a little bit of RAM. You need to carefully evaluate if "unknown length" also implies unbounded length. You cannot deal with unbounded length. Your best approach for reliable operation is to use fixed buffers setup for the maximum size.
A common pattern for this type of action is to pass the buffer to the function, and return what has been used. Your function would then look much like many of the C character string functions:
const size_t HEX_PARSER_MAX_DATA_SIZE = 20;
byte data[HEX_PARSER_MAX_DATA_SIZE];
n = oHexP.getParsedData(data, HEX_PARSER_MAX_DATA_SIZE);
int HEXParser::getParsedData(byte* data, size_t sizeData)
{
int dataIndex = 0;
// fetch data, do stuff
// etc, etc...
data[dataIndex] = incomingByte;
dataIndex++;
if (dataIndex >= sizeData) {
// stop
}
// At the very end of the method I know that all the bytes I need to return
// are stored in data, and the data size is dataIndex - 1
return dataIndex;
}
I want to successfully allocate an Array in my Memory Manager. I am having a hard time getting the data setup successfully in my Heap. I don't know how to instantiate the elements of the array, and then set the pointer that is passed in to that Array. Any help would be greatly appreciated. =)
Basically to sum it up, I want to write my own new[#] function using my own Heap block instead of the normal heap. Don't even want to think about what would be required for a dynamic array. o.O
// Parameter 1: Pointer that you want to pointer to the Array.
// Parameter 2: Amount of Array Elements requested.
// Return: true if Allocation was successful, false if it failed.
template <typename T>
bool AllocateArray(T*& data, unsigned int count)
{
if((m_Heap.m_Pool == nullptr) || count <= 0)
return false;
unsigned int allocSize = sizeof(T)*count;
// If we have an array, pad an extra 16 bytes so that it will start the data on a 16 byte boundary and have room to store
// the number of items allocated within this pad space, and the size of the original data type so in a delete call we can move
// the pointer by the appropriate size and call a destructor(potentially a base class destructor) on each element in the array
allocSize += 16;
unsigned int* mem = (unsigned int*)(m_Heap.Allocate(allocSize));
if(!mem)
{
return false;
}
mem[2] = count;
mem[3] = sizeof(T);
T* iter = (T*)(&(mem[4]));
data = iter;
iter++;
for(unsigned int i = 0; i < count; ++i,++iter)
{
// I have tried a bunch of stuff, not sure what to do. :(
}
return true;
}
Heap Allocate function:
void* Heap::Allocate(unsigned int allocSize)
{
Header* HeadPtr = FindBlock(allocSize);
Footer* FootPtr = (Footer*)HeadPtr;
FootPtr = (Footer*)((char*)FootPtr + (HeadPtr->size + sizeof(Header)));
// Right Split Free Memory if there is enough to make another block.
if((HeadPtr->size - allocSize) >= MINBLOCKSIZE)
{
// Create the Header for the Allocated Block and Update it's Footer
Header* NewHead = (Header*)FootPtr;
NewHead = (Header*)((char*)NewHead - (allocSize + sizeof(Header)));
NewHead->size = allocSize;
NewHead->next = NewHead;
NewHead->prev = NewHead;
FootPtr->size = NewHead->size;
// Create the Footer for the remaining Free Block and update it's size
Footer* NewFoot = (Footer*)NewHead;
NewFoot = (Footer*)((char*)NewFoot - sizeof(Footer));
HeadPtr->size -= (allocSize + HEADANDFOOTSIZE);
NewFoot->size = HeadPtr->size;
// Turn new Header and Old Footer High Bits On
(NewHead->size |= (1 << 31));
(FootPtr->size |= (1 << 31));
// Return actual allocated memory's location
void* MemAddress = NewHead;
MemAddress = ((char*)MemAddress + sizeof(Header));
m_PoolSizeTotal = HeadPtr->size;
return MemAddress;
}
else
{
// Updating descriptors
HeadPtr->prev->next = HeadPtr->next;
HeadPtr->next->prev = HeadPtr->prev;
HeadPtr->next = NULL;
HeadPtr->prev = NULL;
// Turning Header and Footer High Bits On
(HeadPtr->size |= (1 << 31));
(FootPtr->size |= (1 << 31));
// Return actual allocated memory's location
void* MemAddress = HeadPtr;
MemAddress = ((char*)MemAddress + sizeof(Header));
m_PoolSizeTotal = HeadPtr->size;
return MemAddress;
}
}
Main.cpp
int* TestArray;
MemoryManager::GetInstance()->CreateHeap(1); // Allocates 1MB
MemoryManager::GetInstance()->AllocateArray(TestArray, 3);
MemoryManager::GetInstance()->DeallocateArray(TestArray);
MemoryManager::GetInstance()->DestroyHeap();
As far as these two specific points:
Instantiate the elements of the array
Set the pointer that is passed in to that Array.
For (1): there is no definitive notion of "initializing" the elements of the array in C++. There are at least two reasonable behaviors, this depends on the semantics you want. The first is to simply zero the array (see memset). The other would be to call the default constructor for each element of the array -- I would not recommend this option as the default (zero argument) constructor may not exist.
EDIT: Example initialization using inplace-new
for (i = 0; i < len; i++)
new (&arr[i]) T();
For (2): It is not exactly clear what you mean by "and then set the pointer that is passed in to that Array." You could "set" the memory returned as data = static_cast<T*>(&mem[4]), which you already do.
A few other words of cautioning (having written my own memory managers), be very careful about byte alignment (reinterpret_cast(mem) % 16); you'll want to ensure you are returning points that are word (or even 16 byte) aligned. Also, I would recommend using inttypes.h to explicitly use uint64_t to be explicit about sizing -- current it looks like this allocator will break for >4GB allocations.
EDIT:
Speaking from experiment -- writing a memory allocator is a very difficult thing to do, and it is even more painful to debug. As commenters have stated, a memory allocator is specific to the kernel -- so information about your platform would be very helpful.