ESP32 restarts when trying to write to NVS - c++

I am trying to make a save system for my ESP32 project, and I have the following code:
void write_string_nvs(char *memorySlot, String key, String value)
{
nvs_handle my_handle;
esp_err_t err = nvs_open(memorySlot, NVS_READWRITE, &my_handle);
if (err == ESP_OK)
{
int kL = key.length();
int vL = value.length();
char keyA[kL + 1];
key.toCharArray(keyA, kL + 1);
char valueA[vL + 1];
value.toCharArray(valueA, vL + 1);
Serial.println("Storing \"" + String(keyA) + "\"(" + String(kL) + ")>\"" + String(valueA) + "\"(" + String(vL) + ") in NVS.");
esp_err_t err = nvs_set_blob(my_handle, keyA, &valueA, vL);
if (err == ESP_OK)
{
err = nvs_commit(my_handle);
if (err == ESP_OK)
Serial.println("Correctly saved \"" + key + "\" in " + String(memorySlot));
else
Serial.println("write_string_nvs::commit -> Could not save \"" + key + "\" in " + String(memorySlot) + ": " + esp_err_toString(err, true));
}
else
Serial.println("write_string_nvs::nvs_set_blob -> Could not save \"" + key + "\" in " + String(memorySlot) + ": " + esp_err_toString(err, true) + "");
nvs_close(my_handle);
}
else
Serial.println("Could not initialize " + String(memorySlot) + " NVS slot: " + esp_err_toString(err, true) + "");
}
I call it the following way, from a serial command:
...
String params[3];
split(serialRead, ' ', params);
String s = params[0];
String k = params[1];
String v = params[2];
bool error = false;
if (s.length() <= 0) {
error = true;
Serial.println("Please, specify an storage name");
}
if (k.length() <= 0) {
error = true;
Serial.println("Please, specify a key");
}
if (v.length() <= 0) {
error = true;
Serial.println("Please, specify a value");
}
if (!error) {
String slotName = "";
if (startsWithIgnoreCase(s, "main")) {
slotName = "storage";
}
if (startsWithIgnoreCase(s, "wifi")) {
slotName = "wifi";
}
if (slotName.length() > 1) {
Serial.println("Writing \"" + v + "\"" + " at \"\"" + k + "\" in " + slotName);
char slot[slotName.length()];
slotName.toCharArray(slot, slotName.length());
write_string_nvs(slot, k, v);
} else
Serial.println("Specified invalid slot");
}
By doing this I am trying to make a command parser to store values and read them afterwards, with the following commands: storage write <wifi/main> <key> <value> and storage read <wifi/main> <key>.
But the problem comes when I try to type the write command, and the code executes, the ESP32 Serial returns:
assertion "heap != NULL && "realloc() pointer is outside heap areas"" failed: file "/Users/ficeto/Desktop/ESP32/ESP32/esp-idf-public/components/heap/heap_caps.c", line 285, function: heap_caps_realloc
abort() was called at PC 0x40152903 on core 1
Backtrace: 0x40091ca4:0x3ffce0c0 0x40091ed5:0x3ffce0e0 0x40152903:0x3ffce100 0x400847a9:0x3ffce130 0x4008483d:0x3ffce150 0x4008b2e9:0x3ffce170 0x4000bedd:0x3ffce190 0x400dd4e2:0x3ffce1b0 0x400dd544:0x3ffce1d0 0x400dd6a6:0x3ffce1f0 0x400dd6d1:0x3ffce210 0x400d1b06:0x3ffce240 0x400d5939:0x3ffce260 0x400de489:0x3ffce7d0 0x40094135:0x3ffce7f0
Rebooting...
I don't know what to do, I have tried some different write and read codes, but I can't find any that stores the values correctly. The read command works, but obviously, it doesn't return anything, because the memory is empty. Here's the read command, in case you want to take a look at it:
String read_string_nvs(char *memorySlot, String key)
{
nvs_handle my_handle;
esp_err_t err = nvs_open(memorySlot, NVS_READWRITE, &my_handle);
String espErrStr = esp_err_toString(err, true);
char *value;
if (err == ESP_OK || startsWithIgnoreCase(espErrStr, "ESP_OK"))
{
size_t string_size;
int kL = key.length();
char wifi_slot[kL + 1];
key.toCharArray(wifi_slot, kL + 1);
esp_err_t err = nvs_get_str(my_handle, wifi_slot, NULL, &string_size);
value = (char *)malloc(string_size);
err = nvs_get_str(my_handle, wifi_slot, value, &string_size);
nvs_close(my_handle);
return String(value);
}
else
Serial.println("Could not open memory (\"" + espErrStr + "\")");
return espErrStr;
}
I've been with this issue for some weeks, and I really don't know what to do, maybe the system is not good for what I want, or I may be doing something wrong.
For developing I am using VSCode with PlatformIO.
Please, take a look and it and if you could tell me what's wrong or what to do, I'd be really pleased.
Thanks in advance.

I am busy with the same problem (I am going to use 4Mb of the flash as a nvs partition) and I have found a some clue: https://www.esp32.com/viewtopic.php?t=6815
It seems that problem is with the RAM-size - the system needs a RAM to create the nvs-pages-map and if it's not enough for this task - it calls the system abort.
P.S. I have decoded my firmware.elf into firmware.lst with the addresses and the assembler code and the backtrace is so:
app_main -> initArduino -> nvs_flash_init -> nvs_flash_init_partition -> nvs_flash_init_custom -> ZN3nvs_storage_init_Ejj ->ZN3nvs_Storage_populateBlobIndicesIntrusiveList -> _Znwj -> _cxa_allocate_exception -> terminatev -> cxabiv111_terminateEPFvvE - here the system aborts
To decode the .elf into .lst - just copy the firmware.elf into the folder with the xtensa-esp32-elf-objdump.exe (it is probably here .platformio\packages\toolchain-xtensa32\bin) and run in the command prompt - xtensa-esp32-elf-objdump.exe -S -l -d firmware.elf > [YourFileName].lst

These lines are problematic:
char slot[slotName.length()];
slotName.toCharArray(slot, slotName.length());
write_string_nvs(slot, k, v);
slotName.length() will return the number of characters in slotName. slot is a C string, which needs a null terminating character at the end (\0), so it needs to be declared with one byte more than the number of characters in the string. The declaration you have is too short.
You can side-step the problem by rewriting these lines as:
write_string_nvs(slotName.c_str(), k, v);
String already stores its contents as a C string internally, so the c_str() method just gives you a pointer to the buffer it manages. Be careful with this, that pointer won't be valid after the String object becomes invalid, so if the String is a variable in a function or code block, its c_str() will stop being valid when that you leave that function or code block.
Since this is some kind of heap or memory allocation issue it's possible the bug is outside of the code you shared. I would review all the code looking for instances of where you convert a String to a C character array and try using the c_str() idiom instead.
This is a pretty common problem that bites a lot of programmers.
It's also possible the problem is in your write_string_nvs() implementation.

Related

QDir mkpath returns true, but directory not created

I'm having a weird issue with creating paths using Qt on Linux. I've written a standalone test program that creates paths and tests for their existence. This works as expected and creates the directory.
/* make path */
QString p("/usr2/archive/S1234ABC/5/6");
QDir d;
if (d.mkpath(p)) qDebug() << "mkpath() returned true";
else qDebug() << "mkpath() returned false";
QDir d2;
if (d2.exists(p)) qDebug() << "exists() returned true";
else qDebug() << "exists() returned false";
I made that test example into a more robust function, in another project. But it isn't working... mkpath() and exists() return true, but the paths don't exist on the hard disk.
bool nidb::MakePath(QString p, QString &msg) {
if ((p == "") || (p == ".") || (p == "..") || (p == "/") || (p.contains("//")) || (p == "/root") || (p == "/home")) {
msg = "Path is not valid [" + p + "]";
return false;
}
WriteLog("MakePath() called with path ["+p+"]");
QDir path;
if (path.mkpath(p)) {
WriteLog("MakePath() mkpath returned true [" + p + "]");
if (path.exists()) {
WriteLog("MakePath() Path exists [" + p + "]");
msg = QString("Destination path [" + p + "] created");
}
else {
WriteLog("MakePath() Path does not exist [" + p + "]");
msg = QString("Unable to create destination path [" + p + "]");
return false;
}
}
else {
msg = QString("MakePath() mkpath returned false [" + p + "]");
return false;
}
return true;
}
The output from my program:
[2019/06/04 13:19:37][26034] MakePath() called with path [/usr2/archive/S0836VYL/6/5/dicom]
[2019/06/04 13:19:37][26034] MakePath() mkpath returned true [/usr2/archive/S0836VYL/6/5/dicom]
[2019/06/04 13:19:37][26034] MakePath() Path exists [/usr2/archive/S0836VYL/6/5/dicom]
and the output from the command line...
[onrc#ado2dev /]$ cd /usr2/archive/S0836VYL/6/5/dicom
-bash: cd: /usr2/archive/S0836VYL/6/5/dicom: No such file or directory
[onrc#ado2dev /]$
What am I missing??
Try to use this :
QString p("/usr2/archive/S1234ABC/5/6");
QDir d(p);
if(!d.exists() && !d.mkpath(p)) qDebug() << "Error: can't create folder '"<< p <<"'.";
else qDebug() << "Folder '"<< p <<"' exists or created successfully".
Hope it helps you.
All right, this is one for the record books...
The issue was a null terminator appended to the end of the S1234ABC string before being inserted into the database. That string was later used to create the above paths. That S1234ABC string was created using the following code:
QString prefix = "S";
QChar C1, C2, C3, etc... (randomly generated characters)
QString newID = prefix + C1 + C2 + etc...
This created a QString with a \0 on the end of it. Qt stored this value in the MySQL database, which I then pulled back into Qt and tried to make a path with it. Since it's a null terminated string, it appears normal on phpMyAdmin, in xterm, and the log files. Except... in puTTY in Windows, where I saw the weird path it was trying to create:
/usr2/archive/S0836VYL\u0000/10/3/dicom
Thankfully puTTY displayed the actual unicode value instead of ignoring it. Thank you putty! I never, ever would've figured that out...
Recreating the S1234ABC string using QStrings for each character instead of QChar fixed the issue. Now I have regular old strings in the database, and normal paths.

CommandExecutor not registering command with multiple args?

I have a plugin that works with different commands throughout different classes. However, in this case when I have an argument length of 2, the command just doesn't seem to register.
public class FinalFrontierAdminCmds implements CommandExecutor{
FinalFrontier get;
#Override
public boolean onCommand(CommandSender sender, Command cmd, String cmdLabel,
String[] args) {
Player p = (Player) sender;
if (cmd.getName().equalsIgnoreCase("ff")){
// Check if only /ff is typed
if (args.length == 0){
p.sendMessage("This will display help menu");
return true;
}
if (args.length == 2){
if (args[0].equalsIgnoreCase("create")){
if(args[1] != null){
get.getConfig().set("Maps." + args[1] + ".world", p.getLocation().getWorld());
get.getConfig().set("Maps." + args[1] + ".x", p.getLocation().getBlockX());
get.getConfig().set("Maps." + args[1] + ".y", p.getLocation().getBlockY());
get.getConfig().set("Maps." + args[1] + ".z", p.getLocation().getBlockZ());
get.getConfig().set("Maps." + args[1] + ".isSet", false);
p.sendMessage(get.ffMsg + "You have succesfully created map " + ChatColor.GREEN + args[1] + ChatColor.YELLOW + "!");
return true;
}
}
}
return true;
}
return false;
}
}
Main class with onEnable and onDisable:
public class FinalFrontier extends JavaPlugin{
String ffMsg = ChatColor.GREEN + "[" + ChatColor.YELLOW + "Final Frontier" + ChatColor.GREEN + "]" + ChatColor.YELLOW + ": ";
public void onEnable(){
getConfig().addDefault("Maps.", "");
this.getCommand("ff").setExecutor(new FinalFrontierAdminCmds());
saveConfig();
}
public void onDisable(){
saveConfig();
}
}
This is the first time this happens to me. I usually do it like this, unless something has changed? Thanks
The line getConfig().addDefault("Maps.", ""); throws an IllegalArgumentException error because you are trying to set an empty path (the empty path being the non-existent string after the "."). If you remove the period it will create that section correctly.
I didn't see any code in your command executor class that initializes the get variable. I would add the constructor (if you haven't already done so),
public FinalFrontierAdminCmds(FinalFrontier plugin) {
this.get = plugin;
}
and update the instantiation of the object in your onEnable() method to reflect this change (this.getCommand("ff").setExecutor(new FinalFrontierAdminCmds(this))), otherwise your command executor class will throw an NPE when trying to add the values.
Last but not least I would also save the world's name (p.getLocation().getWorld().getName()) and not pass the world object itself to the set method, otherwise the config file will look something like this:
Maps:
example:
world: !!org.bukkit.craftbukkit.v1_8_R3.CraftWorld
PVP: true
ambientSpawnLimit: 15
animalSpawnLimit: 15
autoSave: true
difficulty: PEACEFUL
environment: NORMAL
fullTime: 1100
keepSpawnInMemory: true
monsterSpawnLimit: 70
thunderDuration: 32301
thundering: false
time: 1100
waterAnimalSpawnLimit: 5
weatherDuration: 56712
//more values down here
With these fixes your code should work as expected.

Unable to divide the latest freebase dump

I've dividing the freebase dump into smaller compressed files using a ruby script, then with freebase-rdf-2014-10-26-00-00 it stopped working, ending prematurely. Today I wrote a java program that acts the same way as ruby. Are there changes to freebase dump regarding compression?
package splitfreebase;
import java.io.*;
import java.util.zip.GZIPInputStream;
/**
*
* #author ron
*/
public class SplitFreebase {
/**
* #param args the command line arguments
*/
public static void main(String[] args) {
try {
BufferedReader in = new BufferedReader(
new InputStreamReader(
new GZIPInputStream(
new FileInputStream(args[0]))));
BufferedWriter out = null;
long cnt = 0;
int file_cnt = 0;
while (in.ready()) {
if ((cnt % 10000000) == 0) {
String name = args[0].substring(0, args[0].length() - 3)
+ "_" + file_cnt + ".gz";
if (out != null) {
out.close();
}
out = new BufferedWriter(
new OutputStreamWriter(
new GZIPOutputStream(
new FileOutputStream(name))));
file_cnt++;
}
if (out != null) {
out.write(in.readLine());
out.newLine();
}
cnt++;
}
if (out != null) {
out.close();
}
in.close();
} catch (FileNotFoundException ex) {
Logger.getLogger(SplitFreebase.class.getName()).log(Level.SEVERE, null, ex);
} catch (IOException ex) {
Logger.getLogger(SplitFreebase.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
This works pretty well. Java failed and ruby did too in the exact same way. I divided the problem substituting gunzip STDIN and it works now.
Freebase file:
-rw-r--r-- 1 ron users 29498770878 Nov 2 18:13 freebase-rdf-2014-11-02-00-00.gz
$ gunzip -c freebase_s19.gz | ./free_sp_stdin.rb howdy_pardner
Here is the script. I'm not a ruby programmer, may not be elegant.
require 'zlib'
file_cnt = 0;
error = ""
wgz = nil
if ARGV[1] != nil
file_cnt = ARGV[1].to_i;
end
base_name = ARGV[0]
STDIN.each_with_index do |line, idx|
#puts idx.to_s+" "+line
if((idx % 100000000) == 0)
if(idx != 0)
wgz.close
end
wgz = Zlib::GzipWriter.open(base_name + file_cnt.to_s + ".gz")
puts base_name + file_cnt.to_s + ".gz"
file_cnt+=1
end
if((idx % 1000000) == 0)
print "."
end
error = line
wgz << line
end
wgz.close
puts error

`fgetpos` Not Returning the Correct Position

Update: To get around the problem below, I have done
if (ftell(m_pFile) != m_strLine.size())
fseek(m_pFile, m_strLine.size(), SEEK_SET);
fpos_t position;
fgetpos(m_pFile, &position);
this then returns the correct position for my file. However, I would still like to understand why this is occurring?
I want to get the position in a text file. For most files I have been reading the first line, storing the position, doing some other stuff and returning to the position afterwards...
m_pFile = Utils::OpenFile(m_strBaseDir + "\\" + Source + "\\" + m_strFile, "r");
m_strLine = Utils::ReadLine(m_pFile);
bEOF = feof(m_pFile) != 0;
if (bEOF)
{
Utils::CompilerError(m_ErrorCallback,
(boost::format("File '%1%' is empty.") % m_strFile).str());
return false;
}
// Open.
pFileCode = Utils::OpenFile(strGenCode + "\\" + m_strFile, options.c_str());
m_strLine = Utils::Trim(m_strLine);
Utils::WriteLine(pFileCode, m_strLine);
// Store location and start passes.
unsigned int nLineCount = 1;
fpos_t position;
fgetpos(m_pFile, &position);
m_strLine = Utils::ReadLine(m_pFile);
...
fsetpos(m_pFile, &position);
m_strLine = Utils::ReadLine(m_pFile);
With all files provided to me the storage of the fgetpos and fsetpos works correctly. The problem is with a file that I have created which looks like
which is almost identical to the supplied files. The problem is that for the file above fgetpos(m_pFile, &position); is not returning the correct position (I am aware that the fpos_t position is implementation specific). After the first ReadLine I get a position of 58 (edited from 60) so that when I attempt to read the second line with
fsetpos(m_pFile, &position);
m_strLine = Utils::ReadLine(m_pFile);
I get
on 700
instead of
Selection: Function ADJEXCL
Why is fgetpos not returning the position of the end of the first line?
_Note. The Utils.ReadLine method is:
std::string Utils::ReadLine(FILE* file)
{
if (file == NULL)
return NULL;
char buffer[MAX_READLINE];
if (fgets(buffer, MAX_READLINE, file) != NULL)
{
if (buffer != NULL)
{
std::string str(buffer);
Utils::TrimNewLineChar(str);
return str;
}
}
std::string str(buffer);
str.clear();
return str;
}
with
void Utils::TrimNewLineChar(std::string& s)
{
if (!s.empty() && s[s.length() - 1] == '\n')
s.erase(s.length() - 1);
}
Edit. Following the debugging suggestions in the comments I have added the following code
m_pFile = Utils::OpenFile(m_strBaseDir + "\\" + Source + "\\" + m_strFile, "r");
m_strLine = Utils::ReadLine(m_pFile);
// Here m-strLine = " Logic Definition Report Chart Version: New Version 700" (64 chars).
long vv = ftell(m_pFile); // Here vv = 58!?
fpos_t pos;
vv = ftell(m_pFile);
fgetpos(m_pFile, &pos); // pos = 58.
fsetpos(m_pFile, &pos);
m_strLine = Utils::ReadLine(m_pFile);
Sorry, but your Utils functions have clearly been written by an incompetent. Some issues are just a matter of style. For trimming:
void Utils::TrimNewLineChar(std::string& s)
{
if (!s.empty() && *s.rbegin() == '\n')
s.resize(s.size() - 1); // resize, not erase
}
or in C++11
void Utils::TrimNewLineChar(std::string& s)
{
if (!s.empty() && s.back() == '\n')
s.pop_back();
}
ReadLine is even worse, replace it with:
std::string Utils::ReadLine(FILE* file)
{
std::string str;
char buffer[MAX_READLINE];
if (file != NULL && fgets(buffer, MAX_READLINE, file) != NULL)
{
// it is guaranteed that buffer != NULL, since it is an automatic array
str.assign(buffer);
Utils::TrimNewLineChar(str);
}
// copying buffer into str is useless here
return str;
}
That last str(buffer) in the original worries me especially. If fgets reaches a newline, fills the buffer, or reaches end of file, you're guaranteed to get a properly terminated string in your buffer. If some other I/O error occurs? Who knows? It might be undefined behavior.
Best not to rely on the value of buffer when fgets fails.

How to effectively use if and else for a filtering construct?

To parse function parameters I get from JavaScript, I need to perform a lot of checks. For example a function might expect an object as parameter that looks like this in JavaScript.
{
Fullscreen: [ 'bool', false ],
Size: [ 'Vector2u', 800, 600 ],
Title: [ 'string', 'Hello World' ],
// more properties...
}
In C++ I parse this by looping over all keys and check them. If one of those checks fail, an error message should be printed and this key value pair should be skipped. This is how my implementation looks at the moment. I hope you doesn't get distracted from some of the engine specific calls.
ModuleSettings *module = (ModuleSettings*)HelperScript::Unwrap(args.Data());
if(args.Length() < 1 || !args[0]->IsObject())
return v8::Undefined();
v8::Handle<v8::Object> object = args[0]->ToObject();
auto stg = module->Global->Get<Settings>("settings");
v8::Handle<v8::Array> keys = object->GetPropertyNames();
for(unsigned int i = 0; i < keys->Length(); ++i)
{
string key = *v8::String::Utf8Value(keys->Get(i));
if(!object->Get(v8::String::New(key.c_str()))->IsArray())
{
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
}
v8::Handle<v8::Array> values = v8::Handle<v8::Array>::Cast(object->Get(v8::String::New(key.c_str())));
if(!values->Has(0) || !values->Get(0)->IsString())
{
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
}
string type = *v8::String::Utf8Value(values->Get(0));
if(type == "bool")
{
if(!values->Has(1) || !values->Get(1)->IsBoolean())
{
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
}
stg->Set<bool>(key, values->Get(1)->BooleanValue());
}
else if(type == "Vector2u")
{
if(!values->Has(1) || !values->Has(2) || !values->Get(1)->IsUint32(), !values->Get(2)->IsUint32())
{
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
}
stg->Set<Vector2u>(key, Vector2u(values->Get(1)->Uint32Value(), values->Get(2)->Uint32Value()));
}
else if(type == "string")
{
if(!values->Has(1) || !values->Get(1)->IsString())
{
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
}
stg->Set<string>(key, *v8::String::Utf8Value(values->Get(1)));
}
}
As you can see, I defined when happens when a check fails at every filter.
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
continue;
I would like to only write that once, but I can only come up with a way using goto which I would like to prevent. Is there a better option to restructure the if else construct?
I think I'd start with a set of small classes to do the verification step for each type:
auto v_string = [](v8::Handle<v8::Array> const &v) {
return v->Has(1) && v->Get(1)->IsString();
}
auto v_Vector2u = [](v8::Handle<v8::Array> const &v) {
return v->Has(1) && v->Has(2) &&
v->Get(1)->IsUint32() && v->Get(2)->IsUint32();
}
// ...
Then I'd create a map from the name of a type to the verifier for that type:
std::map<std::string, decltyp(v_string)> verifiers;
verifiers["string"] = v_string;
verifiers["Vector2u"] = v_Vector2u;
// ...
Then to verify a type, you'd use something like this:
// find the verifier for this type:
auto verifier = verifiers.find(type);
// If we can't find a verifier, reject the data:
if (verifier == verifiers.end())
HelperDebug::Fail("script", "unknown type: " + type);
// found the verifier -- verify the data:
if (!verifier->second(values))
HelperDebug::Fail("script", "could not parse (" + key + ") setting");
A pretty usual way for such situations is to use a macro. That is #defined before or in the funciton and #undef-ed at function end. As you need continue there, the other ways are pretty much hosed.
goto would also be a solution but for this particular sample I'd not go with it.
A lighter way is to put at least the fail call into a function or lambda, what still leaves you with the continue part.
I'd probably make a macro that takes the filter expression as argument, leaving code like.
if(type == "bool")
{
ENSURE(values->Has(1) && values->Get(1)->IsBoolean());
stg->Set<bool>(key, values->Get(1)->BooleanValue());
}
...