I am working on an embedded device (microcontroller), and I want to save objects to permanent storage (an EEPROM). Most of the serialization solutions I can find, use the file-system in some way, but my target has no file-system.
Therefore my question is, how can I serialize an object to a byte-array so I can save that byte-array to an EEPROM afterwards?
Here is an example of what i am trying to do:
class Person{
//Constructor, getters and setters are omitted
void save(){
char buffer[sizeof(Person)];
serialize(buffer);
EEPROM::Save(buffer, sizeof(Person));
}
void load(){
char buffer[sizeof(Person)];
EEPROM::Load(buffer, sizeof(Person));
deserialize(buffer);
}
void serialize(char* result){
//?????
}
Person deserialize(char* buffer){
//??????
}
private:
char* name;
int age;
float weight;
};
It's likely that your code for save and load will be reasonably generic and would work best in a separate 'manager' class, leaving each data class only with the responsibility of rendering itself as re-loadable:
// Interface class
class Serializable
{
public:
virtual size_t serialize_size() const = 0;
virtual void serialize(char* dataOut) const = 0;
virtual void deserialize(const char* dataIn) = 0;
};
// Load / save manager
class EEPromManager
{
public:
void save( const Serializable& s )
{
char * data;
size_t data_len;
reserve_memory( data, data_len, s );
s.serialize( data );
EEPROM::Save( data , data_len );
delete [] data;
}
void load( Serializable& s )
{
char * data;
size_t data_len;
reserve_memory( data, data_len, s );
EEPROM::Load( data, data_len );
s.deserialize( data );
delete [] data;
}
private:
char* reserve_memory( char*& data, size_t& data_len, const Serializable& s )
{
return new char[ s.serialize_size() ];
}
};
Each class you intend to serialize / de-serialize should inherit from an interface which mandates the virtual interface for these functions. Note that you'll need to do your own memory management here. I've given a simple example but you'd probably want something a bit more robust.
Then each function should sequentially serialize all attributes of the class (chaining bases classes and calling serialize on aggregate objects if needed.)
class Person : public Serializable
{
public:
virtual size_t serialize_size() const
{
return SerializablePOD<char*>::serialize_size(name) +
SerializablePOD<int>::serialize_size(age) +
SerializablePOD<float>::serialize_size(weight);
}
virtual void serialize(char* dataOut) const
{
dataOut = SerializablePOD<char*>::serialize(dataOut, name);
dataOut = SerializablePOD<int>::serialize(dataOut, age);
dataOut = SerializablePOD<float>::serialize(dataOut, weight);
}
virtual void deserialize(const char* dataIn)
{
dataIn = SerializablePOD<char*>::deserialize(dataIn, name);
dataIn = SerializablePOD<int>::deserialize(dataIn, age);
dataIn = SerializablePOD<float>::deserialize(dataIn, weight);
}
private:
char* name;
int age;
float weight;
};
You'll benefit from generic code to serialize / de-serialize each separate type so you don't keep having code to write the length of strings etc. I.e. a serialize / de-serialize for each POD type:
template <typename POD>
class SerializablePOD
{
public:
static size_t serialize_size(POD str)
{
return sizeof(POD);
}
static char* serialize( char* target, POD value )
{
return memcpy( target, &value, serialize_size(value) );
}
static const char* deserialize( const char* source, POD& target )
{
memcpy( &target, source, serialize_size(target) );
return source + serialize_size(target);
}
};
template<>
size_t SerializablePOD<char*>::serialize_size(char* str)
{
return sizeof(size_t) + strlen(str);
}
template<>
const char* SerializablePOD<char*>::deserialize( const char* source, char*& target )
{
size_t length;
memcpy( &length, source, sizeof(size_t) );
memcpy( &target, source + sizeof(size_t), length );
return source + sizeof(size_t) + length;
}
Incidentally, you might also need to consider what will happen if you change the schema of an object in a software upgrade. Your saved objects would potentially become corrupted on reloading, unless you code round this using - for example - a class version identifier.
Final thought: At a micro level, what you're doing is in many ways similar to the way POD data is serialised for network transmission, so it may be that you can take advantage of libraries to do that - even if you don't have access to an operating system.
To save a string to binary, usually we save its length and then its content. To save other primitive data, we can simply store their binary form. So in your case, all you need to store is:
Length to name
char array of name
age
weight
So the code to serial is:
size_t buffer_size = sizeof(int) + strlen(name) + sizeof(age) + sizeof(weight);
char *buffer = new char[buffer_size];
*(int*)p = strlen(name); p += sizeof(int);
memcpy(p, name, strlen(name)); p += strlen(name);
*(int*)p = age; p += sizeof(int);
*(float*)p = weight;
EEPROM::Save(buffer, buffer_size);
delete[] buffer;
And to read a string from binary buffer, you read its length first, and then copy its data.
Related
I am looking on a way to use unique_ptr to allocate a structure that contains an array of char with a number of bytes that set dynamically to support different types of message.
Assuming:
struct MyMessage
{
uint32_t id;
uint32_t data_size;
char data[4];
};
How can I convert send_message() below to use a smart pointer?
void send_message(void* data, const size_t data_size)
{
const auto message_size = sizeof(MyMessage) - 4 + data_size;
const auto msg = reinterpret_cast<MyMessage*>(new char[message_size]);
msg->id = 3;
msg->data_size = data_size;
memcpy(msg->data, data, data_size);
// Sending the message
// ...
delete[] msg;
}
My attempt to use smart point using the code below does not compile:
const auto message_size = sizeof(MyMessage) - 4 + data_size;
const auto msg = std::unique_ptr<MyMessage*>(new char[message_size]);
Below a complete working example:
#include <iostream>
#include <iterator>
#include <memory>
using namespace std;
struct MyMessage
{
uint32_t id;
uint32_t data_size;
char data[4];
};
void send_message(void* data, const size_t data_size)
{
const auto message_size = sizeof(MyMessage) - 4 + data_size;
const auto msg = reinterpret_cast<MyMessage*>(new char[message_size]);
if (msg == nullptr)
{
throw std::domain_error("Not enough memory to allocate space for the message to sent");
}
msg->id = 3;
msg->data_size = data_size;
memcpy(msg->data, data, data_size);
// Sending the message
// ...
delete[] msg;
}
struct MyData
{
int page_id;
char point_name[8];
};
void main()
{
try
{
MyData data{};
data.page_id = 7;
strcpy_s(data.point_name, sizeof(data.point_name), "ab332");
send_message(&data, sizeof(data));
}
catch (std::exception& e)
{
std::cout << "Error: " << e.what() << std::endl;
}
}
The data type that you pass to delete[] needs to match what new[] returns. In your example, you are new[]ing a char[] array, but are then delete[]ing a MyMessage object instead. That will not work.
The simple fix would be to change this line:
delete[] msg;
To this instead:
delete[] reinterpret_cast<char*>(msg);
However, You should use a smart pointer to manage the memory deletion for you. But, the pointer that you give to std::unique_ptr needs to match the template parameter that you specify. In your example, you are declaring a std::unique_ptr whose template parameter is MyMessage*, so the constructor is expecting a MyMessage**, but you are passing it a char* instead.
Try this instead:
// if this struct is being sent externally, consider
// setting its alignment to 1 byte, and setting the
// size of the data[] member to 1 instead of 4...
struct MyMessage
{
uint32_t id;
uint32_t data_size;
char data[4];
};
void send_message(void* data, const size_t data_size)
{
const auto message_size = offsetof(MyMessage, data) + data_size;
std::unique_ptr<char[]> buffer = std::make_unique<char[]>(message_size);
MyMessage *msg = reinterpret_cast<MyMessage*>(buffer.get());
msg->id = 3;
msg->data_size = data_size;
std::memcpy(msg->data, data, data_size);
// Sending the message
// ...
}
Or this:
using MyMessage_ptr = std::unique_ptr<MyMessage, void(*)(MyMessage*)>;
void send_message(void* data, const size_t data_size)
{
const auto message_size = offsetof(MyMessage, data) + data_size;
MyMessage_ptr msg(
reinterpret_cast<MyMessage*>(new char[message_size]),
[](MyMessage *m){ delete[] reinterpret_cast<char*>(m); }
);
msg->id = 3;
msg->data_size = data_size;
std::memcpy(msg->data, data, data_size);
// Sending the message
// ...
}
This should work, but it is still not clear if accessing msg->data out of bounds is legal (but at least it is not worst than in your original code):
const auto message_size = sizeof(MyMessage) - ( data_size < 4 ? 0 : data_size - 4 );
auto rawmsg = std::make_unique<char[]>( message_size );
auto msg = new (rawmsg.get()) MyMessage;
I am trying to implement a custom data object for XML strings, so that if it exists in the clipboard I can parse that accordingly. The code for the XMLDataObject is as follows:
class XMLDataFormat : public wxDataFormat
{
public:
XMLDataFormat() : wxDataFormat(wxT("XMLDataFormat")) {}
};
class XMLDataObject : public wxDataObjectSimple
{
public:
XMLDataObject(const wxString& xmlstring = wxEmptyString) : wxDataObjectSimple(), m_XMLString(xmlstring)
{
SetFormat(XMLDataFormat());
}
size_t GetLength() const { return m_XMLString.Len() + 1; }
wxString GetXML() const { return m_XMLString; }
void SetXML(const wxString& xml) { m_XMLString = xml; }
// Must provide overloads to avoid hiding them (and warnings about it)
size_t GetDataSize() const
{
return sizeof(void*); //or return GetLength()
}
bool GetDataHere(void *buf) const
{
char* c = _strdup(m_XMLString.c_str());
buf = (void*)c;
return true;
}
bool SetData(size_t len, const void* buf)
{
char* c = (char*)buf;
std::string stdString(c, len);
m_XMLString << stdString;
return true;
}
private:
wxString m_XMLString;
};
I send the data to clipboard (when user clicks copy) in the following fashion:
wxDataObjectComposite* dataobj = new wxDataObjectComposite();
dataobj->Add(new XMLDataObject("XML"));
if (wxTheClipboard->Open()) wxTheClipboard->SetData(dataobj);
wxTheClipboard->Close();
To get the data from the clipboard:
if (wxTheClipboard->Open()) {
XMLDataObject xmlObj;
wxTheClipboard->GetData(xmlObj);
if (xmlObj.GetLength() != 0) wxMessageBox(xmlObj.GetXML());
}
wxTheClipboard->Close();
When user clicks paste, I get weird characters instead of the text "XML". I realized that in the functions bool GetDataHere(void *buf) const and bool SetData(size_t len, const void* buf) the address of buf is different. I am not sure but maybe that is how it should be since Clipboard owns the data or behind the scenes wxWidgets is doing something.
By the way, I am using VS 2015 on Windows 10 and using wxWidgets 3.1.0.
Any suggestions is appreciated.
The following code so far has worked correctly:
class XMLDataFormat : public wxDataFormat
{
public:
XMLDataFormat() : wxDataFormat(wxT("XMLDataFormat")) {}
};
class XMLDataObject : public wxDataObjectSimple
{
public:
XMLDataObject(const wxString& xmlstring = wxEmptyString) : wxDataObjectSimple(), m_XMLString(xmlstring)
{
SetFormat(XMLDataFormat());
}
size_t GetLength() const { return m_XMLString.Len() + 1; }
wxString GetXML() const { return m_XMLString; }
void SetXML(const wxString& xml) { m_XMLString = xml; }
size_t GetDataSize() const
{
return GetLength();
}
bool GetDataHere(void *buf) const
{
//char* c = _strdup(m_XMLString.c_str()); not needed as per suggestion
memcpy(buf, m_XMLString.c_str(), m_XMLString.Len()+1);
return true;
}
bool SetData(size_t len, const void* buf)
{
//char* c = new char[len + 1]; not needed as per suggestion
//memcpy(c, buf, len); not needed as per suggestion
m_XMLString = wxString::FromUTF8((const char*)buf); //changed as per suggestion
//delete c;
return true;
}
virtual size_t GetDataSize(const wxDataFormat&) const
{
return GetDataSize();
}
virtual bool GetDataHere(const wxDataFormat&, void *buf) const
{
return GetDataHere(buf);
}
virtual bool SetData(const wxDataFormat&, size_t len, const void *buf)
{
return SetData(len, buf);
}
private:
wxString m_XMLString;
};
Any suggestions as to improve it is again appreciated.
I have an ostream and data has been written to it. Now I want that data in the form of a char array. Is there a way to get the char buffer and its size without copying all of the bytes? I mean, I know I can use ostringstream and call str().c_str() on it but that produces a temporary copy.
I guess this is what you're looking for - a stream buffer that returns a pointer to its buffer:
#include <iostream>
#include <vector>
#include <string>
class raw_buffer : public std::streambuf
{
public:
raw_buffer(std::ostream& os, int buf_size = 256);
int_type overflow(int_type c) override;
std::streamsize showmanyc() override;
std::streamsize xsputn(const char_type*, std::streamsize) override;
int sync() override;
bool flush();
std::string const& str() const;
private:
std::ostream& os_;
std::vector<char> buffer;
std::string aux;
};
Now str() is simple. It returns a pointer to the underlying buffer of the auxillary buffer:
std::string const& raw_buffer::str() const
{
return aux;
}
The rest of the functions are the usual implementations for a stream buffer. showmanyc() should return the size of the auxiliary buffer (aux is just a running total of the entire buffer, buffer on the other hand is the size specified at construction).
For example, here is overflow(), which should update both buffers at same time but still treat buffer as the primary buffer:
raw_buffer::int_type raw_buffer::overflow(raw_buffer::int_type c) override
{
if (os_ && !traits_type::eq_int_type(c, traits_type::eof()))
{
aux += *this->pptr() = traits_type::to_char_type(c);
this->pbump(1);
if (flush())
{
this->pbump(-(this->pptr() - this->pbase()));
this->setp(this->buffer.data(),
this->buffer.data() + this->buffer.size());
return c;
}
}
return traits_type::eof();
}
flush() is used to copy the contents of buffer to the stream (os_), and sync() should be overrided to call flush() too.
xsputn also needs to be overrided to write to aux as well:
std::streamsize raw_buffer::xsputn(const raw_buffer::char_type* str, std::streamsize count) override
{
for (int i = 0; i < count; ++i)
{
if (traits_type::eq_int_type(this->sputc(str[i]), traits_type::eof()))
return i;
else
aux += str[i];
}
return count;
}
Now we can put this together with a customized stream:
class raw_ostream : private virtual raw_buffer
, public std::ostream
{
public:
raw_ostream(std::ostream& os) : raw_buffer(os)
, std::ostream(this)
{ }
std::string const& str() const
{
return this->raw_buffer::str();
}
std::streamsize count()
{
return this->str().size();
}
};
It can be used like this:
int main()
{
raw_ostream rostr(std::cout);
rostr << "Hello, World " << 123 << true << false;
auto& buf = rostr.str();
std::cout << buf;
}
I'd like serialize QVector into char* array. I do this by the following code:
QVector<int> in;
...
QByteArray bytes;
QDataStream stream(&bytes, QIODevice::WriteOnly);
stream << in;
std::copy(bytes.constData(), bytes.constData() + bytes.size(), out);
I guarantee that out is large enough. Due to the fact that this code is called extremely often I would like to avoid this unnecessary std::copy operation and make either QByteArray or QDataStream work on preallocated user memory pointed by out. Is that possible? Any bight ideas?
UPDATE: QByteArray::fromRawData() doesn't match the needs cause it does not allow to change char* buffer it was created on, in other words, QByteArray performs deep copy on first modification of such created instance.
As they say. This ensures that the raw data array itself will never be modified by QByteArray.
SOLUTION: The solution proposed by #skyhisi does perfectly match my needs. The complete code is the following.
SimpleBuffer.hpp
#pragma once
#include <QtCore/QIODevice>
class SimpleBuffer : public QIODevice {
Q_OBJECT
Q_DISABLE_COPY(SimpleBuffer)
public:
SimpleBuffer(char* const begin, const char* const end) :
_begin(begin),
_end(end){}
virtual bool atEnd() const {
return _end == _begin;
}
virtual bool isSequential() const {
return true;
}
protected:
virtual qint64 readData(char*, qint64) {
return -1;
}
virtual qint64 writeData(const char* const data, const qint64 maxSize) {
const qint64 space = _end - _begin;
const qint64 toWrite = qMin(maxSize, space);
memcpy(_begin, data, size_t(toWrite));
_begin += toWrite;
return toWrite;
}
private:
char* _begin;
const char* const _end;
};
main.cpp
#include "SimpleBuffer.hpp"
#include <QtCore/QVector>
#include <QtCore/QDataStream>
#include <QtCore/QByteArray>
int main(int, char**) {
QVector<int> src;
src << 3 << 7 << 13 << 42 << 100500;
const size_t dataSize = sizeof(quint32) + src.size() * sizeof(int);
char* const data = new char[dataSize];
// prepare stream and write out the src vector
{
SimpleBuffer simpleBuffer(data, data + dataSize);
simpleBuffer.open(QIODevice::WriteOnly);
QDataStream os(&simpleBuffer);
os << src;
}
// read vector with QByteArray
QVector<int> dst;
{
const QByteArray byteArray = QByteArray::fromRawData((char*)data, dataSize);
QDataStream is(byteArray);
is >> dst;
}
delete [] data;
// check we've read exactly what we wrote
Q_ASSERT(src == dst);
return 0;
}
I think you may need to implement a QIODevice, you could make a very simple sequential device quite easily. Here's one I've quickly thrown together, I haven't checked it works (feel free to get it working and edit the post).
class SimpleBuffer : public QIODevice
{
Q_OBJECT
public:
SimpleBuffer(char* begin, char* end):mBegin(begin),mEnd(end){}
virtual bool atEnd() const {return mEnd == mBegin; }
virtual bool isSequential() const { return true; }
protected:
virtual qint64 readData(char*, qint64) { return -1; }
virtual qint64 writeData(const char* data, qint64 maxSize)
{
const qint64 space = mEnd - mBegin;
const qint64 toWrite = qMin(maxSize, space);
memcpy(mBegin, data, size_t(toWrite));
mBegin += toWrite;
return toWrite;
}
private:
char* mBegin;
char* mEnd;
Q_DISABLE_COPY(SimpleBuffer)
};
Maybe fromRawData works:
QByteArray QByteArray::fromRawData ( const char * data, int size ) [static]
Using it something like :
char* out=new char[enoughbytes]; // preallocate at suitable scope
QVector<int> in;
QByteArray ba=QByteArray::fromRawData(out,enoughbytes);
QDataStream stream(&ba,QIODevice::WriteOnly);
stream << in;
Note that QDataStream adds some of it's own data at the start of the data (not much though), so remember to preallocate a bit more for that, as well as for whatever additional data QVector serializes.
Why not use QBuffer?
QByteArray myBuffer;
myBuffer.reserve(10000); // no re-allocation
QBuffer buffer(&myBuffer);
buffer.open(QIODevice::WriteOnly);
QDataStream out(&buffer);
out << QApplication::palette();
A function returns a pointer and a length (via the arguments) from an unknown DLL.
Result = SpamnEggs( &pBytes, &nBytes )
The pointer points to a valid memory address at which are nBytes sequential bytes.
These bytes contain valid ascci values for text. There is no null termination!
I am tasked with "ovelaying" a string type of some sort in as few simple operations in generic C++ code (without complex libraries or using byte) before output:
cout << sresult
Added:
without copying the bytes as this is a large buffer that must be traversed.
Prototype:
int SpamnEggs( void* pBytes, void* nBytes );
becomes
int SpamnEggs( char** pBytes, int* nBytes );
Many thanks all. Great answers and all very valid.
You can copy the raw memory and add the string terminating character yourself:
char* newStr = new char[size + 1];
memcpy ( newStr, source, size );
newStr[size] = "\0";
cout << newStr;
Without copying memory, you can create a class that holds the pointer and length as members and overload the stream operator to print only length characters:
class MyString
{
void* _pBuf;
int _length;
public:
MyString(void* pBuf, int length)
{
_pBuf = pBuf;
_length = length;
}
friend ostream& operator <<(ostream &os,const MyString& obj);
};
ostream& operator <<(ostream &os,const MyString& obj)
{
const char* pChar = (const char*)obj._pBuf;
for ( int i = 0 ; i < obj._length ; i++ )
{
os << pChar[i];
}
return os;
}
Example usage:
char* x = "blablabla";
int length = 3;
MyString str(x,length);
cout << str;
You can just construct a std::string from the pointer and a length.
std::string sResult(pBytes, nBytes);
std::cout << sResult;
(assuming pBytes is a char* pointer, otherwise you need a small cast).
How about something like this (untested) code:
class my_pointer_string
{
friend std::ostream &operator<<(std::ostream &, const my_pointer_string &);
public:
my_pointer_string(void *ptr, size_t len)
: m_pointer(ptr), m_length(len)
{ }
private:
void *m_pointer;
size_t m_length;
};
std::ostream &operator<<(std::ostream &os, const my_pointer_string &str)
{
char *string = reinterpret_cast<char *>(str.m_pointer);
for (size_t i = 0; i < str.m_length; i++)
os << *string++;
return os;
}
What you would have to do is
a) Create some class that encapsulates the char pointer and the size.
b) Write a << operator for that class to output its content to a stream.
EDIT: In contrast to the response by Bo Persson this would not imply copying the source data.