Is there an internal size limit of TIdtcpserver buffer? How come whatever method I use, It reaches a limit of 65535?
I have encountered this buffer issue of TidTCPSever these days. My code is very basic: preset the size of buffer array, extract server byte from InputBuffer, and copy buffer array to workspace. Here is the code
TByteDynArray buffer; // decliared in private
void __fastcall TmodWifiCom::IdServerExecute(TIdContext *AContext)
{
long readLength;
int c, s;
byte b;
DataH->FDataReceivedBytes=0;
AContext->Connection->IOHandler->CheckForDataOnSource(10);
while (!AContext->Connection->IOHandler->InputBufferIsEmpty()) {
// get hint of size of buffer
s = AContext->Connection->IOHandler->InputBuffer->Size;
buffer.set_length(s);
AContext->Connection->IOHandler->InputBuffer->ExtractToBytes(buffer,-1,false);
readLength = buffer.Length;
for (long i = 0; i < readLength; i++) {
b = buffer[i];
DataH->FDataBuffer[DataH->FDataReceivedBytes++]=b; // copy buffer bytes to workspace
}
// process workspace
}
}
The code appears to work fine, readLength and s are equal. FDataBuffer appears to recieve every bytes. However, as TidTCPSever reaches a limit deque fails.
// private of head file in other class
frameQue_Type frameQue0;
deque<frameQue_Type> frameQue;
// cpp file in other class
frameQue.push_back(frameQue0);
...
frameQue0 = DataH->frameQue.pop_front(); // [ERROR STOPS HERE]
The error message was: access volation #0048D893
I don't understand:
TidTCPSever and deque are in different classes
struct values in deque seem fine
Error occured as soon as buffer size reaches 65535 bytes
Am I using the buffer right?
Related
I am working on an Arduino with a WiFi shield. Currently, I am making a web request on the shield. I am trying to use malloc() to write the response to heap. Right now, when printing out _buffer, I get the response with all the headers followed by a bunch of garbage as pictured below:
char *NetConn::capture()
{
if (_client.available())
{
if (_buffer) // RELEASE ANY DATA HELD IN THE BUFFER IF THE BUFFER EXISTS
{
free(_buffer);
} // KEEP TRACK OF HOW LARGE THE BUFFER CURRENTLY IS TO PREPARE FOR ANY OVERFLOW
_buffer = (char *)malloc(_client.available() * sizeof(char)); // INITIALIZE THE BUFFER
Serial.println(_client.available()); // _client.available() returns 550 (size of payload)
char c;
int index = 0;
while (_client.available())
{
c = _client.read();
_buffer[index] = c;
index++;
}
int size = sizeof(_buffer) / sizeof(char);
Serial.println(size); // This is writing 2 for some reason
}
return _buffer;
}
I should mention that _client is an instance of an Arduino WiFi client.
I'm very very new to low level memory management and am trying to get a grasp of the whole memory management thing. I'm used to working with the comforts of high-level languages; However I am trying to branch out. Please help me understand what is going on here with an explanation as I am very new. Thank you!
I am saving settings to the flash memory and reading them back again. 2 of the values always comes back empty. However, the data IS written to flash, since after a reset the values read are the new saved values and not empty.
I started experiencing this problem after I did some code-refactoring after taking the code over from another company.
Saving and reading the settings back works when you actually do the following (old inefficient way):
save setting 0 - read setting 0
save setting 1 - read setting 1
...
save setting 13 read setting 13
This is EXTREMELY inefficient and slow since the same page with all the settings are read from flash, the whole block of flash cleared, the new setting put into the read buffer and then the whole block (with only 1 changed setting) are written to flash. And this happens for all 14 settings!! But it works ...
unsigned char Save_One_Setting(unsigned char Setting_Number, unsigned char* value, unsigned char length)
{
/* Program the user Flash area word by word
(area defined by FLASH_USER_START_ADDR and FLASH_USER_END_ADDR) ***********/
unsigned int a;
Address = FLASH_USER_START_ADDR;
a = 0;
while (Address < FLASH_USER_END_ADDR)
{
buf[a++] = *(__IO uint32_t *)Address;
Address = Address + 4;
}
memset(&buf[Setting_Number * 60], 0, 60); // Clear setting value
memcpy(&buf[Setting_Number * 60], &value[0], length); // Set setting value
Erase_User_Flash_Memory();
HAL_FLASH_Unlock();
Address = FLASH_USER_START_ADDR;
a = 0;
while (Address < FLASH_USER_END_ADDR)
{
if (HAL_FLASH_Program(FLASH_TYPEPROGRAM_WORD, Address, buf[a++]) == HAL_OK)
{
Address = Address + 4;
}
else
{
/* Error occurred while writing data in Flash memory.
User can add here some code to deal with this error */
while (1)
{
/* Make LED1 blink (100ms on, 2s off) to indicate error in Write operation */
BSP_LED_On(LED1);
HAL_Delay(100);
BSP_LED_Off(LED1);
HAL_Delay(2000);
}
}
}
/* Lock the Flash to disable the flash control register access (recommended
to protect the FLASH memory against possible unwanted operation) *********/
HAL_FLASH_Lock();
}
I changed this by actually, after reading the settings from the flash into a buffer, update all the changed settings in the buffer, then erase the flash block and write the buffer back to flash. Downside: my first and 4th values always comes back as NULL after saving this buffer to flash.
However, after a system reset the correct values are read from flash.
unsigned char Save_Settings(Save_Settings_struct* newSettings)
{
/* Program the user Flash area word by word
(area defined by FLASH_USER_START_ADDR and FLASH_USER_END_ADDR) ***********/
unsigned int a;
unsigned char readBack[60];
Address = FLASH_USER_START_ADDR;
a = 0;
while (Address < FLASH_USER_END_ADDR)
{
buf[a++] = *(__IO uint32_t *)Address;
Address = Address + 4;
}
a = 0;
while (a < S_MAXSETTING)
{
if (newSettings[a].settingNumber < S_MAXSETTING)
{
memset(&buf[a * 60], 0, 60); // Clear setting value
memcpy(&buf[a * 60], newSettings[a].settingValue, newSettings[a].settingLength); // Set setting value
}
++a;
}
Erase_User_Flash_Memory();
HAL_FLASH_Unlock();
Address = FLASH_USER_START_ADDR;
a = 0;
while (Address < FLASH_USER_END_ADDR)
{
if (HAL_FLASH_Program(FLASH_TYPEPROGRAM_WORD, Address, buf[a++]) == HAL_OK)
{
Address = Address + 4;
}
else
{
/* Error occurred while writing data in Flash memory.
User can add here some code to deal with this error */
while (1)
{
/* Make LED1 blink (100ms on, 2s off) to indicate error in Write operation */
BSP_LED_On(LED1);
HAL_Delay(100);
BSP_LED_Off(LED1);
HAL_Delay(2000);
}
}
}
/* Lock the Flash to disable the flash control register access (recommended
to protect the FLASH memory against possible unwanted operation) *********/
HAL_FLASH_Lock();
}
I started playing around with cleaning and invalidating the data cache. At least the 2 values are not NULL anymore, however, they are still the old values. All other values are the new, saved values. Do a reset, and all values are correct.
Anybody ever had some similar problem? Or maybe an idea of what I can try to get rid of this problem?
I have a function which opens a file from an SD card, uses the file size to set the size of a buffer, writes a block of information to that buffer, then does something with that information, as shown in this code:
char filename = "filename.txt";
uint16_t duration;
uint16_t pixel;
int q = 0;
int w = 0;
bool largefile;
File f;
int readuntil;
long large_buffer;
f = SD.open(filename);
if(f.size() > 3072) {
w = 3072;
} else {
w = f.size();
}
uint8_t buffer[w];
while(f.available()) {
f.read(buffer, sizeof(buffer));
while(q < sizeof(buffer)) {
doStuffWithInformation(buffer[q++]);
}
q=0;
}
f.close;
This works great with smaller file sizes, but anything over the hard limit buffer size of 3072 (which I arrived at empirically, its just the amount of memory that can be safely committed to this function), runs into a problem. Larger files read fine until they hit the last loop of while(f.available()), where they read the end of the file, but then continue reading the buffer, the tail end of which is filled with data from the last loop, that wasn't overwritten by the latest f.read(). How can I make sure that the last loop of the while(f.available()) function only works with the information that was written to the buffer during the current loop? My only idea right now is to solve for factors of the file size, and set the buffer size as the largest factor less than 3072, but this seems intensive to run every time this function is called. Is there an elegant solution staring me in the face?
Your program is not behaving correctly because f.read() is not guaranteed to read the whole buffer. Moreover, it is bound to happen when you read the last chunk of the file, unless the file size is a factor of buffer size (3072 in your case).
While Arduino specification (https://www.arduino.cc/en/Reference/FileRead) doesn't say so, SD.read function returns the number of bytes read. See code of the library here: https://github.com/arduino-libraries/SD/blob/master/src/utility/SdFile.cpp, int16_t SdFile::read(void* buf, uint16_t nbyte)
Knowing that, you should change your loop as following (while also rewriting it as a for loop for better readability and removing q definition above):
while(f.available()) {
uint16_t sz = f.read(buffer, sizeof(buffer));
for (uint16_t q = 0; q < sz; ++q) {
doStuffWithInformation(buffer[q]);
}
}
On a side note, now, when you have this logic in place, it would make sense for you to do away with variable length array and use a fixed buffer of size 512 - the standard sector size on the SD card. Most likely, it will yield the same performance in regards to read, and slightly better performance in regards to sizeof, which will becomes a compile-time constant rather than a run-time calculation. This also makes your program simpler. This makes for following code:
f = SD.open(filename);
...
uint8_t buffer[512];
I am trying to send (ideally) a 2d buffer from one process to another, over a Message Queue, but i am attempting to do it first with a 1d buffer.
The functions called to initialization the queue are the following:
HANDLE MsgQueueCommunicator::InitMessageQueue_data(bool IsRead,wchar16_t* wQueueName)
{
MSGQUEUEOPTIONS msgopts;
msgopts.dwSize = sizeof(MSGQUEUEOPTIONS);
msgopts.dwFlags = MSGQUEUE_ALLOW_BROKEN;//0;
msgopts.dwMaxMessages = 0;
msgopts.cbMaxMessage = sizeof(data[20]);
msgopts.bReadAccess = IsRead;
HANDLE hq = CreateMsgQueue(wQueueName, &msgopts);
if ( hq == NULL )
{
return NULL;
}
return hq;
}
Queue initialization in process 1:
HANDLE hMsgQueueR = MsgQueueCommunicator::getInstance()->InitMessageQueue_data(true, L"CommDataStreaming");
Queue initialization in process 2:
HANDLE s_hMsgQueue_Communication = MsgQueueCommunicator::getInstance()->InitMessageQueue_data(false,L"CommDataStreaming");
To write to the queue, i call the following functions:
BOOL MsgQueueCommunicator::Write_Array_To_Queue(HANDLE hq,double data[20])
{
return WriteMsgQueue(hq,(LPVOID)&data, sizeof(data),INFINITE,0);
}
MsgQueueCommunicator::getInstance()->Write_Array_To_Queue(s_hMsgQueue_Communication, usb_data);
Where usb_data is a 1d double array.
To read from the queue, i call the following functions:
BOOL MsgQueueCommunicator::Read_Array_From_Msg_Queue(HANDLE hq,double data[20])
{
DWORD dwBytesRead;
DWORD dwFlags;
return ReadMsgQueue(hq, (LPVOID)&data, sizeof(data), &dwBytesRead, INFINITE, &dwFlags);
}
MsgQueueCommunicator::getInstance()->Read_Array_From_Msg_Queue(hMsgQueueR, usb_data);
Where usb_data is again a 1d double array.
Now, when i check the values that are placed into usb_data[20] before it is written to the queue, i can see that they are non-zero integers. However, when i read the array from the queue and check its values, they are zero. Im not sure what is causing this issue. I've used message queues to send single values, strings, and structs, so i figured i would be able to follow the same procedure to send over an array, but this does not seem to be the case, unless i am overlooking something.
My question is, can i send arrays/buffers over a message queue, and if yes, have I set it up properly?
Note:This is being developed in a windows embedded compact 7 environment and VS2008.
There are several problems with the code provided.
1) Wrong parameter values - you do not need to take an address of the data buffer since the variable is already a pointer to the beginning of the memory that contains the elements. So change (LPVOID)&data to (LPVOID)data.
2) Wrong size - the sizeof operator will return 4 since that is the size of the pointer. In your case you would need to pass 160 as the size (20 * sizeof(double)).
As for variable size writes - this gets a bit more complicated since you need to know how much data to read at the other end. What you can do is use lets say first/first two/first four bytes of the buffer to contain size and then proceed with the data. Then you can have a function that accepts a double array of variable length and writes it. For example:
BOOL Write_Array_To_Queue(HANDLE hq,double data[], unsigned int count)
{
size_t buffer_size = sizeof(count) + count * sizeof(double);
BYTE* buffer = new BYTE[buffer_size];
memcpy(buffer, &count, sizeof(count));
memcpy(buffer + sizeof(count), &data, sizeof(double) * count);
return WriteMsgQueue(hq,(LPVOID)buffer, buffer_size,INFINITE,0);
}
or
BOOL Write_Array_To_Queue(HANDLE hq,double data[], unsigned int count)
{
return WriteMsgQueue(hq,(LPVOID)&count, sizeof(count),INFINITE,0) && WriteMsgQueue(hq,(LPVOID)data, sizeof(double) * count,INFINITE,0);
}
and then in the receiving side you would first read out an unsigned int and then read as much data as denoted by the read value.
I'm writing a C++ program that sends and receives images using Boost.Asio.
When compiling I don't get errors, but when executing and having sent an image the program that receives the image crashes giving the following error message (in Visual Studio 2012, Windows 7 32bit):
Debug Assertion Failed:
Program: […]\DataSender.exe
File: f:\dd\vctools\crt_bld\self_x86\crt\src\dbgdel.cpp
Line: 52
Expression: _BLOCK_TYPE_IS_VALID(pHead->nBlockUse)
I read packages the size of 4096 bytes into a pointer to a char array while there are still incoming bytes to read. In the final looping—if there are less than 4096 bytes to read—I delete the pointer and create a pointer to a char array the size of the remaining bytes. Until here it still works.
But when I try to delete the char pointer array again at the end of the loop (in order to create a new char pointer array with standard size 4096 for the next incoming images), the program crashes.
Here is my code's excerpt in question:
char* buffer = new char[4096];
[…]
int remainingBytes = imageSize;
[…]
// read data
while( remainingBytes > 0 )
{
boost::system::error_code error;
// use smaller buffer if remaining bytes don't fill the tcp package
// fully
if( remainingBytes < 4096 )
{
delete[] buffer; // this one doesn't give an error
bufferSize = remainingBytes;
char* buffer = new char[bufferSize];
}
// read from socket into buffer
size_t receivedBytes = socket.read_some(
boost::asio::buffer(buffer, bufferSize), error);
remainingBytes -= receivedBytes;
// count total length
totalReceivedBytes += receivedBytes;
// add current buffer to totalBuffer
for( int i = 0; i < bufferSize; i++)
{
totalBuffer.push_back(buffer[i]);
}
// if smaller buffer has been used delete it and
// create usual tcp buffer again
if( receivedBytes < 4096 )
{
delete[] buffer; // here the error occurs
bufferSize = 4096;
char* buffer = new char[bufferSize];
}
}
I ran the same code also on a Debian GNU/Linux 7.2 64bit machine, which returned the following error, at the same position in code:
*** glibc detected *** ./datasender: double free or corruption (!prev): 0x0000000002503970 ***
I assume I'm doing something wrong when deallocating the char pointer array but I haven't figured it out yet.
Can someone point me in the right direction?
You're actually deleting twice the buffer when remainingBytes and receivedBytes are less than 4096.
Indeed, you're deleting buffer once, then allocate memory into a local buffer, not the outer one.
Then, when you delete buffer in the second if block, you're deleting a second time the same buffer. The allocation you've made in the if scopes are memory leaks. These aren't the same variables.
When you do
char* buffer = new char[bufferSize];
in your if scopes, you're creating a new variable, not allocating memory into the outer buffer variable. Thus, you're leaking, and not allocating memory into the buffer you just deleted.
Without looking further, you should remove the char* in front of buffer in both if blocks and then continue debugging.
I would use std::vector instead:
#include <vector>
//...
std::vector<char> buffer(remainingBytes);
bufferSize = remainingBytes;
//...
while( remainingBytes > 0 )
{
boost::system::error_code error;
// use smaller buffer if remaining bytes don't fill the tcp package
// fully
if( remainingBytes < 4096 )
{
buffer.resize(remainingBytes);
bufferSize = remainingBytes;
}
// read from socket into buffer
size_t receivedBytes = socket.read_some(
boost::asio::buffer(&buffer[0], bufferSize), error);
remainingBytes -= receivedBytes;
// count total length
totalReceivedBytes += receivedBytes;
// add current buffer to totalBuffer
totalBuffer.insert(totalBuffer.end(), buffer.begin(),
buffer.begin() + receivedBytes);
// if smaller buffer has been used delete it and
// create usual tcp buffer again
if( receivedBytes < 4096 )
{
buffer.resize(4096);
bufferSize = 4096;
}
}
There will be no memory leaks.
Also, I think your code has a bug in that you are supposed to copy only the number of received bytes (the return value of the read_some() function). Instead you assumed that bufferSize characters were returned.