I am working on an Arduino with a WiFi shield. Currently, I am making a web request on the shield. I am trying to use malloc() to write the response to heap. Right now, when printing out _buffer, I get the response with all the headers followed by a bunch of garbage as pictured below:
char *NetConn::capture()
{
if (_client.available())
{
if (_buffer) // RELEASE ANY DATA HELD IN THE BUFFER IF THE BUFFER EXISTS
{
free(_buffer);
} // KEEP TRACK OF HOW LARGE THE BUFFER CURRENTLY IS TO PREPARE FOR ANY OVERFLOW
_buffer = (char *)malloc(_client.available() * sizeof(char)); // INITIALIZE THE BUFFER
Serial.println(_client.available()); // _client.available() returns 550 (size of payload)
char c;
int index = 0;
while (_client.available())
{
c = _client.read();
_buffer[index] = c;
index++;
}
int size = sizeof(_buffer) / sizeof(char);
Serial.println(size); // This is writing 2 for some reason
}
return _buffer;
}
I should mention that _client is an instance of an Arduino WiFi client.
I'm very very new to low level memory management and am trying to get a grasp of the whole memory management thing. I'm used to working with the comforts of high-level languages; However I am trying to branch out. Please help me understand what is going on here with an explanation as I am very new. Thank you!
Related
I'm using esp8266 with my arduino Mega and controlling it over serial with AT commands( i know that it is not the best way of working with ESP but I didn't know how to do it other way)...So my problem is that I have my POST request formulated and send over to my webpage ....It works fine at first try ,but after that, in every loop try, it fails...Please can someone check my code and see if you can find some error that can cause this?
This is my function for sending data:
void SendData(){
cmd = "AT+CIPSTART=\"TCP\",\"";
cmd+= server;
cmd+="\",80";
Serial1.println(cmd);
Serial.println(cmd);
delay(1000);
if(Serial1.find("OK"))
{
Serial.println("\r\nReady to send data!");
}
String retazec="cz="+cas_zaznamu; ////DATA from sensors etc.
retazec=retazec+"&tep="+t;
retazec=retazec+"&vlhv="+vv;
retazec=retazec+"&vlhp="+vp;
retazec=retazec+"&zav="+za;
retazec=retazec+"&kur="+ku;
retazec=retazec+"&vet="+ve;
retazec=retazec+"&pz="+datum_zavlaha;
int retazec_len=retazec.length();
retazec.toCharArray(retaz,70);
cmd = "POST /arduino.php"; ////POST request
cmd+=" HTTP/1.1\r\n";
cmd+="Host: myhost.com\r\n";
cmd+="Content-Type: application/x-www-form-urlencoded\r\n";
cmd+="Connection:Close\r\n";
cmd+="Content-Length: ";
cmd+=retazec_len;
cmd+="\r\n\r\n";
int size=cmd.length()+retazec_len+2;
Serial1.print("AT+CIPSEND=");
Serial1.println(size);
Serial.print("AT+CIPSEND=");
Serial.println(size);
delay(2000);
if (Serial1.find(">")){
Serial.println("Sending data...");
} else {
Serial1.println("AT+CIPCLOSE");
Serial.println("COnnection closed");
return;
}
Serial.print(cmd);
Serial1.print(cmd);
for(int i=0;i<=retazec_len;i++){
Serial.print(retaz[i]);
Serial1.print(retaz[i]);
}
Serial1.print("\r\n");
if(Serial1.find("OK"))
{
Serial.println("Succesfuly send!");
}
ReadString(); /// emptying the buffer by reading serial1
delay(5000);
Serial.println("-----end");
}
Also I have different function for GET request ,very similar to the one above , and that one worked multiple times in a row(dont want to say every time bcs it was running only for few minutes).
Please please any suggestions will be deeply appreciated. :)
My educated guess:The String class in combination with at commandsTry to get rid of the String class in webbased scenarios. Reason: You want to have a stable environment running for a long time. String class uses heap memory to dynamicly build and destroy the underlying structures it needs for e.g. append strings (... = .. + .. + ..). As there is no so called garbagge collection (and would have not enough memory anyway) the relativeley small memory is cut into small units, in which (at a certain length) the String does not fit anymore -> crash, reset => lost POST or GET request.Use fixed global char array(s) to build your messages, these are compiled into the flash and stop the "memory bleeding". Enclosed an example (although I dont understand - from your language - the name or the content of variables and with no type given I had to guess)
char retazec[256] = {'\0'}; /* Defined globally - Used for retazec functions max 255 chars */
char cmd[512] = {'\0'}; /* Defined globally - Used for POST/GET functions max 511 chars */
char numBuffer[16] = {'\0'}; /* Defined globally - Used for uint32_t or smaller number conversions to char */
unsigned long timeStamp = 0;
unsigned long delayTime = 2000;
setup(){...}
//The parts you gave "translated"into char manuipulation
const char* cmdStart[] = "POST /arduino.php HTTP/1.1\r\n Host: myhost.com\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection:Close\r\nContent-Length: ";
const char* cmdEnd[] = "\r\n\r\n";
strcpy (retazec, "cz=");
itoa(cas_zaznamu, numBuffer, 10); // convert int DATA from sensors etc.
// If its a float use this command
// dtostrf(floatvar, StringLengthIncDecimalPoint, numVarsAfterDecimal, charbuf);
// dtostrf(cas_zaznamu, 3, 2, numBuffer); // -> xx.xx
strcat(retazec, numBuffer);
strcat(retazec, "&tep=");
itoa(t, numBuffer, 10);
// or dtostrf(t, 4, 2, numBuffer); // -> xxx.xx
strcat(retazec, numBuffer);
strcat(retazec, "&vlhv=");
....
// ... till datum_zavlaha;
int retazec_len=strlen(retazec); // gives you the length
// not needed we have a char array allready retazec.toCharArray(retaz,70);
strcpy(cmd, cmdStart);
iota (retazec_len, numBuffer);
strcat(cmd, numBuffer);
strcat(cmd, cmdEnd);
int size=strlen(cmd)+retazec_len+2;
if (millis() - timeStamp > delayTime) {
Serial1.print("AT+CIPSEND=");
Serial1.println(size);
Serial.print("AT+CIPSEND=");
Serial.println(size);
timeStamp = millis();
}
// NO NEVER delay(2000);
if (Serial1.find(">")){
Serial.println("Sending data...");
} else {
Serial1.println("AT+CIPCLOSE");
Serial.println("COnnection closed");
return;
}
The second issue is delay (it stops processing dead in its tracks) So the cpu waits in your case 2 sec before restarting - not a clever idea when using communication protocols. I implemented a non-blocking delay - if you need it somewhere else as I said you only gave fragments of code
EDIT after OPs feedback
String class as a whole should be avoided in dynamic communication scenarios so in all parts of the programm.Should I use one big string (small s = char array) thats just coping what the original code does but instead of String with char array. So imho the structure has just to be adapted to char array without big changes (exept expicitly converting numbers to char) Btw the same takes place when you add a variable to String
uint8_t aNumber = 12;
String myBigString = "A number is " + aNumber;
does internally in the String library the same as
uint8_t aNumber = 12;
char numBuffer [16] = {'\0'} ;
char myBigString [32] = {'\0'} ;
strcpy (myBigString, "A number is ");
iota (aNumber, numBuffer, 10);
strcat (myBigString, numBuffer);
The difference is - you write the lines instead of lazely using a bad library but with full control over memory and the advantage to compile it to flash, saving memory. Const char are actually called via pointer - so for later there are functions to get only certain parts of an array handy to create variable messages out of one big array.After a code analysis on github there is one other big problem: The use of blocking delay():
loop() {
delay(1000);
// lot of code
delay(3000);
}
replace with a non blocking delay
unsigned long timeStamp = 0;
setup(){}
loop() {
if (millis() - timeStamp > 4000) {
// lot of code
timeStamp = millis(); // reset the timer
}
}
and as written before get rid of all Strings (e.g)
///////////PREMENNE PRIJIMANÝCH ÚDAJOV///////
String mode;
String inzavlaha;
String inkurenie;
String invetranie;
String invlhkost_p;
String invlhkost_v;
String incas;
String intrvanie;
String inopakovanie;
String inteplota;
and as a final tio.If you want worldwide help, start to write all of your program in English (variables and comments) it makes it easier to help.
I am writing a program to receive SSID and password via BlueTooth in an ESP32 (Arduino Framework). The function BTSerialRcvBuffer() waits for BlueTooth, when it receives a string it returns the base address and size of the string via a variable of type struct Buffer_return. The function returns the SSID but not the password. I don't know why? Do I have to allocate memory for Var.rtn_addr or allocating memory for variable buff1 and buff2 enough?
#include <Arduino.h>
#include <stdlib.h>
#include <BluetoothSerial.h>
#include <WiFi.h>
#define btrcv_buffer_size 256
BluetoothSerial SerialBT;
typedef struct
{
char *rtn_addr;
int buff_len;
} Buffer_return;
Buffer_return* BTSerialRcvBuffer() {
static int i = 0;
static char rcv_buffer[ btrcv_buffer_size ];
static Buffer_return Var;
memset(rcv_buffer,0, btrcv_buffer_size);
while (!SerialBT.available());
delayMicroseconds(500);
while(SerialBT.available()) {
rcv_buffer[i] = SerialBT.read();
i++;
}
rcv_buffer[i-1] = '\0';
rcv_buffer[i-2] = '\0';
SerialBT.flush();
Var.rtn_addr = rcv_buffer; //<------------Do I have to allocate memory for Var.rtn_addr?
Var.buff_len = i-1;
return &Var;
}
void WiFiConfig() {
//WiFi.printDiag(Serial);
Serial.println("Enter SSID");
Buffer_return *buff1 = BTSerialRcvBuffer();
char *ssid = (char*) malloc((buff1->buff_len) * sizeof(char));
strcpy(ssid,buff1->rtn_addr);
Serial.println(ssid);
Serial.println("Enter Password");
Buffer_return *buff2 = BTSerialRcvBuffer();
char *pass = (char*) malloc((buff2->buff_len) * sizeof(char));
strcpy(pass,buff2->rtn_addr);
Serial.println(pass);
//Serial.println(buff2->buff_len);
free(ssid)
free(pass);
while (WiFi.status() != WL_CONNECTED) {
delay(1000);
Serial.println("Establishing connection to WiFi..");
Serial.printf("Connection status: %d\n", WiFi.status());
}
}
void setup() {
Serial.begin(115200);
//WiFi.disconnect(true);
SerialBT.begin("ESP32_INO"); //Bluetooth device name
WiFi.mode(WIFI_STA);
Serial.println("The device started, now you can pair it with bluetooth!");
WiFiConfig();
Serial.println("Connected to network");
Serial.println(WiFi.macAddress());
Serial.println(WiFi.localIP());
}
void loop() {
}
Output:
Enter SSID
Airtel_5G <----- prints fine!
Enter Password
<----- Problem!
Establishing connection to WiFi..
Connection status: 6
Establishing connection to WiFi..
Connection status: 6
Establishing connection to WiFi..
Connection status: 6
The code uses the the following sequence to copy input data to a buffer.
Buffer_return *buff1 = BTSerialRcvBuffer();
char *ssid = (char*) malloc((buff1->buff_len) * sizeof(char));
strcpy(ssid,buff1->rtn_addr);
Recall the strings are NUL terminated, so that allocation must include extra byte!. Simple update to malloc call:
char *ssid = (char*) malloc((buff1->buff_len+1) * sizeof(char));
Per input from #lundin, malloc is not recommedned for Arduino. Better to use automatic allocation.
See also: https://arduino.stackexchange.com/questions/682/is-using-malloc-and-free-a-really-bad-idea-on-arduino
char ssid[buff2->buff_len+1] ;
strcpy(ssid, buff2->rtn_addr) ;
Update 1: error in BTSerialRcvBuffer
The BTSerialRcvBuffer uses static for many variables, including i. Recall that static variables are initialized once (at program startup). Suggesting removing 'static' from i - to fix the initialization, as there is no need to make it static.
Also, it is not clear why are the last 2 positions of rcv_buffer reset to zero ?
There's lots of bad practice and slow function calls in your code. Keep in mind that this is a 8 bit MCU, so it is dreadfully slow. Some things to fix:
It isn't necessary to zero-out the rx buffer each time. Just keep track of how large a portion of it that contains valid data. The memset call on 256 bytes is very expensive.
It is common practice to double-buffer rx buffers so that one buffer can be used for reception and another for decoding, simultaneously. You don't use interrupts so this is perhaps less of an issue. 256 bytes is a lot of RAM anyway, so double buffering might require a better MCU if you need to store that much data. I'll use a double buffer example below anyway just to show how it can be done.
The delayMicroseconds(500); does nothing meaningful except hanging up your program for 500ms. Remove that.
Bug: you don't check for buffer overrun during reception.
#define BT_RXBUF_SIZE 256
const char* BTSerialReceive (size_t* size_rec)
{
static char buf1 [BT_RXBUF_SIZE];
static char buf2 [BT_RXBUF_SIZE];
static char* buf = buf1;
buf = (buf == buf1) ? buf2 : buf1; // swap rx buffers
while (!SerialBT.available())
;
size_t i=0;
for(; SerialBT.available() && i<BT_RXBUF_SIZE; i++)
{
buf[i] = SerialBT.read();
}
buf[i] = '\0';
SerialBT.flush();
*size_rec = i;
return buf;
}
The pointer swap with double buffers eliminates the need for memcpy/strcpy which are slow and expensive. If you used UART interrupts you would have to use a design like this, for re-entrancy reasons.
Another thing that you must absolutely avoid is malloc. It is slow and senseless, see this. When working with embedded systems you must always use fixed-length buffers and deterministic amounts of memory. Getting rid of malloc means you can get rid of the whole heap segment in the linker script, which will free up lots of valuable RAM.
Is there an internal size limit of TIdtcpserver buffer? How come whatever method I use, It reaches a limit of 65535?
I have encountered this buffer issue of TidTCPSever these days. My code is very basic: preset the size of buffer array, extract server byte from InputBuffer, and copy buffer array to workspace. Here is the code
TByteDynArray buffer; // decliared in private
void __fastcall TmodWifiCom::IdServerExecute(TIdContext *AContext)
{
long readLength;
int c, s;
byte b;
DataH->FDataReceivedBytes=0;
AContext->Connection->IOHandler->CheckForDataOnSource(10);
while (!AContext->Connection->IOHandler->InputBufferIsEmpty()) {
// get hint of size of buffer
s = AContext->Connection->IOHandler->InputBuffer->Size;
buffer.set_length(s);
AContext->Connection->IOHandler->InputBuffer->ExtractToBytes(buffer,-1,false);
readLength = buffer.Length;
for (long i = 0; i < readLength; i++) {
b = buffer[i];
DataH->FDataBuffer[DataH->FDataReceivedBytes++]=b; // copy buffer bytes to workspace
}
// process workspace
}
}
The code appears to work fine, readLength and s are equal. FDataBuffer appears to recieve every bytes. However, as TidTCPSever reaches a limit deque fails.
// private of head file in other class
frameQue_Type frameQue0;
deque<frameQue_Type> frameQue;
// cpp file in other class
frameQue.push_back(frameQue0);
...
frameQue0 = DataH->frameQue.pop_front(); // [ERROR STOPS HERE]
The error message was: access volation #0048D893
I don't understand:
TidTCPSever and deque are in different classes
struct values in deque seem fine
Error occured as soon as buffer size reaches 65535 bytes
Am I using the buffer right?
Currently, I am working on real time interface with Visual Studio C++.
I faced problem is, when buffer is running for data store, that time .exe is not responding at the point data store in buffer. I collect data as 130Hz from motion sensor. I have tried to increase virtual memory of computer, but problem was not solved.
Code Structure:
int main(){
int no_data = 0;
float x_abs;
float y_abs;
int sensorID = 0;
while (1){
// Define Buffer
char before_trial_output_data[][8 * 4][128] = { { { 0, }, }, };
// Collect Real Time Data
x_abs = abs(inchtocm * record[sensorID].y);
y_abs = abs(inchtocm * record[sensorID].x);
//Save in buffer
sprintf(before_trial_output_data[no_data][sensorID], "%d %8.3f %8.3f\n",no_data,x_abs,y_abs);
//Increment point
no_data++;
// Break While loop, Press ESc key
if (GetAsyncKeyState(VK_ESCAPE)){
break;
}
}
//Data Save in File
printf("\nSaving results to 'RecordData.txt'..\n");
FILE *fp3 = fopen("RecordData.dat", "w");
for (i = 0; i<no_data-1; i++)
fprintf(fp3, output_data[i][sensorID]);
fclose(fp3);
printf("Complete...\n");
}
The code you posted doesn't show how you allocate more memory for your before_trial_output_data buffer when needed. Do you want me to guess? I guess you are using some flavor of realloc(), which needs to allocate ever-increasing amount of memory, fragmenting your heap terribly.
However, in order for you to save that data to a file later on, it doesn't need to be in continuous memory, so some kind of list will work way better than an array.
Also, there is no provision in your "pseudo" code for a 130Hz reading; it processes records as fast as possible, and my guess is - much faster.
Is your prinf() call also a "pseudo code"? Otherwise you are looking for trouble by having mismatch of the % format specifications and number and type of parameters passed in.
This is the function I am using to unzip buffer.
string unzipBuffer(size_t decryptedLength, unsigned char * decryptedData)
{
z_stream stream;
stream.zalloc = Z_NULL;
stream.zfree = Z_NULL;
stream.avail_in = decryptedLength;
stream.next_in = (Bytef *)decryptedData;
stream.total_out = 0;
stream.avail_out = 0;
size_t dataLength = decryptedLength* 1.5;
char c[dataLength];
if (inflateInit2(&stream, 47) == Z_OK)
{
int status = Z_OK;
while (status == Z_OK)
{
if (stream.total_out >= dataLength)
{
dataLength += decryptedLength * 0.5;
}
stream.next_out = (Bytef *)c + stream.total_out;
stream.avail_out = (uint)(dataLength - stream.total_out);
status = inflate (&stream, Z_SYNC_FLUSH);
}
if (inflateEnd(&stream) == Z_OK)
{
if (status == Z_STREAM_END)
{
dataLength = stream.total_out;
}
}
}
std::string decryptedContentStr(c, c + dataLength);
return decryptedContentStr;
}
And it was working fine until today when I realized that it crashes with large data buffer (Ex: decryptedLength: 342792) on this line:
status = inflate (&stream, Z_SYNC_FLUSH);
after one or two iterations. Can anyone help me please?
If your code generally works correctly, but fails for large data sets, then this could be due to a stack overflow as indicated by #StillLearning in his comment.
A usual (default) stack size is 1 MB. When your decryptedLength is 342,792, then you try to allocate 514,188 byte in the following line:
char c[dataLength];
Together with other allocations in your code (and finally in the inflate() function), this might already be too much. To overcome this problem, you should allocate the memory dynamically:
char* c = new char[dataLength];
If you so this, then please do not forget to release the allocated memory at the end of your unzipBuffer() function:
delete[] c;
If you forget to delete the allocated memory, then you will have a memory leak.
In case this doesn't (fully) solve your problem, you should do it anyway, because for even larger data sets your code will break for sure due to the limited size of the stack.
In case you need to "reallocate" your dynamically allocated buffer in your while() loop, then please take a look at this Q&A. Basically you need to use a combination of new, std::copy, and delete[]. However, it would be more appropriate if your exchange your char array with a std::vector<char> or even std::vector<Bytef>. Then you would be able enlarge your buffer easily by using the resize() function. You can directly access the buffer of a vector by using &my_vector[0] in order to assign it to stream.next_out.
c is not going to get bigger just because you increase datalength. You are probably overwriting past the end of c because your initial guess of 1.5 times the compressed size was wrong, causing the fault.
(It might be a stack overflow as suggested in another answer here, but I think that 8 MB stack allocations are common nowadays.)