Buffer gets overwritten - c++

I'm facing this issue on an ESP8266 (Arduino like board), but this problem is regarding c/c++, so I'm asking this here.
I have not that much experience with native languages like c/c++ and I'm facing a strange issue, which drives me crazy. So I'm using an Wemos D1 mini (ESP8266) which uses a class calles ConfigManager to read a configuration file from eeprom. The config file is formatted as json, so I'm using ArduinoJson to parse the content. I have declared a StaticJsonBuffer and pointer to a JsonObject in the header, like you can see in the code example:
//FILE: ConfigManager.h
#ifndef ConfigManager_H
#define ConfigManager_H
#include <Arduino.h>
#include <ArduinoJson.h>
#include <FS.h>
#include "Logger.h"
class ConfigManager {
public:
Settings *settings;
ConfigManager();
void read_from_eeprom();
private:
File configFile;
JsonObject *json;
StaticJsonBuffer<200> jsonBuffer;
void open_file(const char *permission);
void read_json();
void recreate_file();
void create_json();
void check_success();
void populate_settings();
void clean_up();
};
#endif
When the function read_from_eeprom is invoked, it opens the file and invokes the functionread_json:
void ConfigManager::read_json() {
size_t size = configFile.size();
Log.verbose("[ConfigManager] Config file size: %d", size);
std::unique_ptr<char[]> buf(new char[size]);
configFile.readBytes(buf.get(), size);
Log.verbose("[ConfigManager] File content: %s", buf.get());
Log.verbose("[ConfigManager] Parsing json");
json = &jsonBuffer.parseObject(buf.get());
Log.notice("[ConfigManager] Json is:");
json->printTo(Serial);
Serial.println();
}
Which is followed by a call to check_success()
void ConfigManager::check_success() {
Log.notice("[ConfigManager] Json is:");
json->printTo(Serial);
Serial.println();
bool should_recreate = true;
if (json->success()) {
Log.notice("[ConfigManager] Parsed json successfully");
auto version = json->get<const char*>("version");
if (version) {
if (strcmp(version, Settings::current_version) == 0) {
Log.notice("[ConfigManager] Config version is up2date");
should_recreate = false;
} else {
Log.warning("[ConfigManager] Config version outdated");
}
} else {
Log.warning("[ConfigManager] Invalid config file");
}
} else {
Log.warning("[ConfigManager] Config file is not valid json");
}
if (should_recreate) {
Log.notice("[ConfigManager] Recreating config file");
recreate_file();
create_json();
}
Log.notice("JSON IS: ");
json->prettyPrintTo(Serial);
Log.notice("[ConfigManager] Sucessfully read json");
}
So what I noticed is that the file content is fine. E.g. {"version":"0.2","led_count":"64"}.
Then the json is parsed, which succeeds and logs the json object, which is again {"version":"0.2","led_count":"64"}.
Afterwards the function returns, and calls check_success, which again prints the content of the json object to the log, but this time it seems that something has overwritten the JsonBuffer, which causes the json object to be corrupted. This time the logged content is {"v␂":"0.2","led_count":"64"} (with some strange unicorn characters that change as the source code changes). I'm trying to figure out whats going on for many hours now, but I'm stuck. Can someone please point me in the right direction to solve this problem? Thank you!
The full Log can be found HERE, as well as ConfigManager.h and ConfigManager.cpp

*I'd prefer write that in comments, because I don't have arduino and can't verify that my advice 100% helpful. But I can't use comments with "my reputation" :). So please don't press "minus button" if my answer didn't help... *
According to that it seems you need to keep original json string while you using json buffer.
Keep the JSON string in memory long enough
The library never make memory duplication. This has an important implication on string
values, it means that the library will return pointer to chunks of the
string.
For instance, let’s imagine that you parse ["hello","world"], like
this:
char[] json = "[\"hello\",\"world\"]";
StaticJsonBuffer<32> buffer;
JsonArray& array = buffer.parseArray(json);
const char* first = array[0];
const char* second = array[1];
In that
case, both first and second are pointers to the content of the
original string json. So this will only work if json is still in
memory.
So, it make sense try make std::unique_ptr buf a class member (same as StaticJsonBuffer) and check how it works.
BTW, IMO std::vector will be more suitable there... And I'm not sure that unique_ptr deletes arrays properly.

Related

Handling of const char* on ESP32

I'm working on making some Spotify API calls on an ESP32. I'm fairly new to C++ and while I seem to got it working how I wanted it to, I would like to know if it is the right way/best practice or if I was just lucky. The whole thing with chars and pointers is still quite confusing for me, no matter how much I read into it.
I'm calling the Spotify API, get a json response and parse that with the ArduinoJson library. The library returns all keys and values as const char*
The library I use to display it on a screen takes const char* as well. I got it working before with converting it to String, returning the String with the getTitle() function and converting it back to display it on screen. After I read that Strings are inefficient and best to avoid, I try to cut out the converting steps.
void getTitle()
{
// I cut out the HTTP request and stuff
DynamicJsonDocument doc(1024);
DeserializationError error = deserializeJson(doc, http.getStream(), );
JsonObject item = doc["item"];
title = item["name"]; //This is a const char*
}
const char* title = nullptr;
void loop(void) {
getTitle();
u8g2.clearBuffer();
u8g2.setDrawColor(1);
u8g2.setFont(u8g2_font_6x12_tf);
u8g2.drawStr(1, 10, title);
u8g2.sendBuffer();
}
Is it okay to do it like that?
This is not fine.
When seeing something like this, you should immediately become suspicious.
This is because in getTitle, you are asking a local object (item) for a pointer-- but you use the pointer later, when the item object no longer exists.
That means your pointer might be meaningless once you need it-- it might no longer reference your data, but some arbitrary other bytes instead (or even lead to crashes).
This problem is independent of what exact library you use, and you can often find relevant, more specific information by searching your library documentation for "lifetime" or "object ownership".
FIX
Make sure that item (and also DynamicJsonDocument, because the documentation tells you so!) both still exist when you use the data, e.g. like this:
void setTitle(const char *title)
{
u8g2.clearBuffer();
u8g2.setDrawColor(1);
u8g2.setFont(u8g2_font_6x12_tf);
u8g2.drawStr(1, 10, title);
u8g2.sendBuffer();
}
void updateTitle()
{
DynamicJsonDocument doc(1024);
DeserializationError error = deserializeJson(doc, http.getStream(), );
JsonObject item = doc["item"];
setTitle(item["name"]);
}
See also: https://arduinojson.org/v6/how-to/reuse-a-json-document/#the-best-way-to-use-arduinojson
Edit: If you want to keep parsing/display update decoupled
You could keep the JSON document "alive" for when the parsed data is needed:
/* "static" visibility, so that other c/cpp files ("translation units") can't
* mess mess with our JSON doc directly
*/
static DynamicJsonDocument doc(1024);
static const char *title;
void parseJson()
{
[...]
// super important to avoid leaking memory!!
doc.clear();
DeserializationError error = deserializeJson(doc, http.getStream(), );
// TODO: robustness/error handling (e.g. inbound JSON is missing "item")
title = doc["item"]["name"];
}
// may be nullptr when called before valid JSON was parsed
const char* getTitle()
{
return title;
}

C++: function to append file, called within other functions

So, I am trying to log information about the status of the c++ project code in a text file. The program terminates unexpectedly, so I need to append the file as I go rather than storing info in an array along the way. I wanted to call the function to write to the file from within other functions, eventually in the other c++ files as well.
The code is a huge project that has many files and the "main()" technically exists in a separate file from all of the functions that are called throughout the function of the code (therefore not a useful file for me). My plan was to open the file in the setup() function, and then call the function within other functions along the way. Just in case I did not explain the setup of the code well enough, here is the link to the file I am trying to add to: https://github.com/cstracq2/ardupilot/blob/master/ArduCopter/ArduCopter.cpp
I have seen other notes on what may help, but I am not that familiar with c++ and I don't know what most of it means. From what I saw, this is one of the ways I tried, and it is failing to compile.
#include "<existing header>.h"
#include <fstream>
#include <iostream>
void log_data( ofstream &datafile, int value);
void <>::function1()
{ ....<stuff that was already there>
log_data( datafile, <value> );
}
void <>::function2()
{ ....<stuff that was already there>
log_data( datafile, <value> );
}
void setup()
{ ....<stuff that was already there>
ofstream datafile;
datafile.open("data_log_file.txt");
}
void log_data( ofstream &datafile, int value)
{
data_file << value << endl;
}
If there is any advice that you could give me, I would really appreciate it.
In your case I would suggest to use the Singleton Pattern. Here is an example of how you could do it:
class Logger
{
std::ifstream logF;
static Logger *s_instance;
Logger(std::string &path)
{
logF.open(path, std::ios_base::in);
}
public:
void log_data(int val)
{
logF << val << std::endl;
}
static void create_instance(std::string &path)
{
s_instance = new Logger(path);
}
static Logger *instance()
{
return s_instance;
}
};
Now you can just include the header with the class def and call something like:
Logger::instance()->log_data(<value>);
And do not forget to init the class before calling the static method (somewhere in main for instance):
Logger::create_instance(<path>);
Of course, you can just make it easier by hard-coding a value for your path, but if the path changes you'll have to re-compile everything.
Or just use something already implemented like log4cpp
Ah yes now that you mentioned the use of datafile in other function I see the error: The variable datafile is a local variable inside the setup function.
It should either be a member variable or possible a global variable.

ESP8266WebServer setting a value inside a class

I'm having a bit of trouble with the ESP8266WebServer. My WebServer{} class is wrapped around the ESP8266WebServer object and looks like this:
Header file:
#include <WiFiClient.h>
#ifndef WebServer_h
#define WebServer_h
#include "Arduino.h"
class WebServer {
public:
WebServer();
void begin();
void handleClient();
void finishedProcessingData(String clientReply);
String queryString;
private:
// page/url handlers
friend void handleSomeData();
};
#endif
Cpp file:
#include <ESP8266WiFi.h>
#include <WiFiClient.h>
#include <ESP8266WebServer.h>
#include "Arduino.h"
#include "WebServer.h"
ESP8266WebServer server(80);
int aNumberHere = 0;
String queryString = "";
WebServer::WebServer(){
}
void handleSomeData(){
aNumberHere++;
queryString = "";
// this loop appends all the queries fro the query string back into a query string
// (not sure if there is an easier way to grab this directly from the server api)
int totalArgs = server.args();
for (int counter = 0; counter < totalArgs; counter++){
queryString += server.argName(counter) +"="+ server.arg(counter);
if(counter < (totalArgs - 1)){
queryString += "&";
}
}
Serial.println(queryString);
Serial.println(aNumberHere);
}
void WebServer::handleClient(){
server.handleClient();
}
void WebServer::begin(){
server.on("/data.html", handleSomeData);
server.begin();
}
void WebServer::finishedProcessingData(String clientReply){
// empty the string so it isn't cached (just in case)
Serial.print("Sending reply back to client: ");
Serial.println(clientReply);
queryString = "";
server.send(200, "text/plain", clientReply);
}
The idea is to grab a query string from an http request, do some processing, then return the response.
How it is called from outside is:
WebServer webServer;
String processingResult;
void setup(){
webServer.begin();
}
void loop(){
delay(10);
webServer.handleClient();
// check if the query string has stuff in it, if it doesn't then WebServer.handleSomeData() never fired, thus no request yet
if(webServer.queryString != ""){
// do stuff that results in a string being returned
processingResult = handWavyMagic();
// then respond back to client
webServer.finishedProcessingData(processingResult);
}
}
My issue is that from my outside loop (which is in my main sketch), "webServer.queryString" is always equal to an empty string. If I print out "queryString" from within handleSomeData(), I can see the printed result from within "handleSomeData()" (which is correct), but it somehow can't seem to set it 'outside' that method. I did check to see that it is able to access and modify a variable (which is why "aNumberHere" is printed) and there seem to be no issues; "aNumberHere" gets incremented as expected every time I go to the corresponding url.
My only suspect is that "queryString" is a String object which for some reason can't be set, while "aNumberHere" is a primitive int which is fine. Is this correct or is something else going on here?
I have also tried making "queryString" a static variable, but with no luck -either I did it wrong or it just doesn't work that way either.
I am new to friend functions (well, new to c/c++ in general actually -any advice is welcome), though I read that it should be able to access the classes private properties just fine, so I'm not sure how to fix this.
Any ideas?
I know, this is a bit late for the OP to help with his problem, but maybe other readers will find this useful.
To avoid splitting parts of the logic/data inside and outside the class it would be more elegant to have everything inside. Using callbacks to non-static methods of a class instance is a bit tricky (I learned it the hard way for my current project), but here is the alternative:
void WebServer::begin()
{
// instead of server.on("/data.html", handleSomeData);
server.on("/data.html", std::bind(&WebServer::handleSomeData, this));
server.begin();
}
void WebServer::handleSomeData()
{
// do whatever you need
}
This uses std:bind() to bind the instance method and its this pointer to the callback. As a result, everything is contained inside the web server instance, which is a cleaner approach.
There is 2 different variables called queryString :
the global one declared in WebServer.cpp
the member of WebServer declared in WebServer.h
In the callback handleSomeData you set the global one, but in the loop and finishedProcessingData you access to the member of WebServer.
To make the code works, you could remove the member of WebServer and use the global one (as you did for aNumberHere) like this :
extern String queryString;
void loop(){
delay(10);
webServer.handleClient();
// check if the query string has stuff in it, if it doesn't then WebServer.handleSomeData() never fired, thus no request yet
if(queryString != ""){
// do stuff that results in a string being returned
processingResult = handWavyMagic();
// then respond back to client
webServer.finishedProcessingData(processingResult);
queryString = "";
}
}

How can I get more details about errors generated during protobuf parsing? (C++)

I am new to protobuf (C++) and my code fails during parse of my messages. How can I get more details about the errors that occurred?
Example
The following snippet illustrates the problem:
const bool ok=my_message.ParseFromCodedStream(&stream);
if(ok){
std::cout<< "message parsed. evidence:\n"<< my_message.DebugString();
}
else{
std::cerr<< "error parsing protobuf\n";
//HOW CAN I GET A REASON FOR THE FAILURE HERE?
}
If you look inside protobuf code, you will find it's using its own logging system - based on macros. By default all these messages goes to stderr, but you can capture them in your program with SetLogHandler():
typedef void LogHandler(LogLevel level, const char* filename, int line,
const std::string& message);
The possible solution is to make your own errno-like mechanism (sorry for C++11-ishness):
typedef LogMessage std::tuple<LogLevel, std::string, int, std::string>; // C++11
typedef LogStack std::list<LogMessage>;
namespace {
LogStack stack;
bool my_errno;
} // namespace
void MyLogHandler(LogLevel level, const char* filename, int line,
const std::string& message) {
stack.push_back({level, filename, line, message}); // C++11.
my_errno = true;
}
protobuf::SetLogHandler(MyLogHandler);
bool GetError(LogStack* my_stack) {
if (my_errno && my_stack) {
// Dump collected logs.
my_stack->assign(stack.begin(), stack.end());
}
stack.clear();
bool old_errno = my_errno;
my_errno = false;
return old_errno;
}
And use it in your code:
...
else {
std::cerr<< "error parsing protobuf" << std::endl;
LogStack my_stack;
if (GetError(&my_stack) {
// Handle your errors here.
}
}
The main drawback of my sample code - it doesn't work well with multiple threads. But that can be fixed on your own.
Sometimes error information will be printed to the console, but that's it. There's no way to get extra error info through the API.
That said, there are only two kinds of errors anyway:
A required field was missing. (Information should be printed to the console in this case.)
The data is corrupt. It was not generated by a valid protobuf implementation at all -- it's not even a different type of protobuf, it's simply not a protobuf.
If you are seeing the latter case, you need to compare your data on the sending and receiving side and figure out why it's different. Remember that the data you feed to the protobuf parser not only must be the same bytes, but it must end at the same place -- the protobuf parser does not know where the message ends except by receiving EOF. This means that if you are writing multiple messages to a stream, you need to write the size before the data, and make sure to read only that many bytes on the receiving end before passing on to the protobuf parser.

Trash characters when using buffers in c++

I have a DLL that I need to handle in C++. I'm using WxWidgets (standard compilation, but I also tried Unicode on/off) and NetBeans. I also tried dealing with this without WxWidgets (windows.h) and had same problems.
Here is how I access the DLL functions using WxWidgets:
// -------------------- POINTERS TO FUNCTIONS
typedef bool(*TYPE_DLL_SetLicense)(char*, char*);
typedef bool(*TYPE_DLL_PingConnection)(char*);
typedef char*(*TYPE_DLL_ERR_DESCRIPTION)(void);
class DLL_Library
{
public:
// pointers to functions inside dll
TYPE_DLL_SetLicense DLL_SetLicense; //initialize - will wor fine as it returns only true/false (buffer only provide data)
TYPE_DLL_PingConnection DLL_PingConnection; //ping to serwer. Will return trahs, becouse it uses buffer to provide data ang get answear back
TYPE_DLL_ERR_DESCRIPTION DLL_ERR_DESCRIPTION; //error description. No buffer, no trouble. Returns correct string.
wxDynamicLibrary dynLib2;
int initialize(void)
{
//patch to dll
wxString path = wxStandardPaths::Get().GetExecutablePath().BeforeLast('\\') + _("\\DLL_dll\\DLLMOK.dll");
if(!wxFile::Exists(path)) return -1;
//load dll
if(!dynLib2.Load(path)) return -2;
//Assign functions in dll to variable
DLL_SetLicense=(TYPE_DLL_SetLicense) dynLib2.GetSymbol(wxT("DLL_SetLicense"));
DLL_PingConnection=(TYPE_DLL_PingConnection) dynLib2.GetSymbol(wxT("DLL_PingConnection"));
DLL_ERR_DESCRIPTION=(TYPE_DLL_ERR_DESCRIPTION) dynLib2.GetSymbol(wxT("DLL_ERROR_DESCRIPTION"));
return 0;
}
};
And here is the function I run. It should return and XML content, that I try to save to the file.
//DLL_PingConnection
//result ping to be save in file
wxFile file_ping_xml;
plik_ping_xml.Open(wxT("C:\\dll\\ping.xml"),wxFile::write);
char buffor_ping_xml[2000];
//I run the function here
bool is_ping = DLL_PingConnection(buffor_ping_xml);
if(is_ping)
{
tex_box->AppendText(wxT("DLL_PingConnection True\n"));
//we save result to file
bool is_write_ping_ok = file_ping_xml.Write(buffor_ping_xml,2000);
if (is_write_ping_ok){tex_box->AppendText(wxT("Save to file is ok ok\n"));}
else {tex_box->AppendText(wxT("Save to file failed :( \n"));}
}
else
{
tex_box->AppendText(wxT("DLL_PingConnection False\n"));
}
std::cout << "Error description: " << DLL_ERR_DESCRIPTION() << "\n"; //will work fine both in saving to file, and in streaming to screen.
The problem is that inside the file instead of good content I get rubbish like this:
NOTE that this only happens in functions that use buffers like:
char buffer[2000] //buffer will contain for example file xml
function do_sth_with_xml(buffer) //buffer containing xml will (should) be overwriten with xml results of the function - in our case DLL_PingCONNECTION should save in buffer xml with connection data
Documentation say that the DLL operates on Windows-1250. File ping.xml I have set to windows ANSI, but I don't think problem lies here.
EDIT: I have written problem without WxWidgets (I load DLL using windows.h) - same problems. Here is the code: Getting trash data in char* while using it as buffer in function . Please help :(
This
DLL_PingConnection=(TYPE_DLL_PingConnection)
shouldn't it be
DLL_PingConnection=(TYPE_DLL_PingConnection) dynLib2.GetSymbol(wxT("DLL_PingConnection"));
?
seems otherwise you will not get a valid pointer to the function in the DLL.
as a general rule you should check return values, especially from a DLL
you load dynamically since it happens that you sometimes get another version
of the DLL which may have a function with same name but other signature or
where is missing entirely.
You named a function
DLL_PingConnection=(TYPE_DLL_PingConnection) dynLib2.GetSymbol(....
and call it with
OSOZ.OSOZ_PingConnection(buffor_ping_xml);
you typedef a function
typedef bool(*TYPE_DLL_PingConnection)(char*);
you create a variable
char buffor_ping_xml[2000];
in your typedef it is char* and your buffor_ping_xml is char
how can that work ?
try
char *buffor_ping_xml = new char[2000];
/* or */
wchar_t *buffor_ping_xml = new wchar_t[2000];
/* or */
wxChar *buffor_ping_xml = new wxchar[2000];
bool is_ping = DLL_PingConnection(buffor_ping_xml);
wxString mystring = wxString::FromUTF8(buffor_ping_xml);
write mystring to file.
To Do:
look in your wxwidgets\libs folder for your libs
are there libwxmsw29ud_* with a 'u' in the name (after version number here 29)?
If not You can not use unicode
If yes next steps
for all different test char *, wchar_t *, wxChar * give the files different name.
for example file_ping_xml.Open(wxT("C:\dll\ping_w_t_FromUTF8.xml"), ...
for wchar_t * in combination with
wxString mystring = wxString::FromUTF8(buffor_ping_xml);
also in combination with
wxString mystring(buffor_ping_xml);
Then check out the look like, of the files in a browser .
To test you can go to your wxWidgets sample folder . Compile in the folder C:\wxWidgets\samples\docview\docview.cpp . Open with docview.exe a unicode file . How does it look.
Unicode download file
Unicode-related compilation settings
You should define wxUSE_UNICODE to 1 to compile your program in Unicode mode. This currently works for wxMSW, wxGTK, wxMac and wxX11. If you compile your program in ANSI mode you can still define wxUSE_WCHAR_T to get some limited support for wchar_t type.
Here is answear: Getting trash data in char* while using it as buffer in function.
Thanks everyone - expecially for patience.