I want to write a std::map to a file and read it back. I'm looking for a rather simple and minimalist way to do it, without boost. I found that it is doable with vector like here Reading and writing a std::vector into a file correctly with iterators
I found this question as it relates to what I want to do, except I'm looking for the binary alternative.
reading a file of key-value pairs in to a std::map
For types with no dynamic memory (actually, pointers) involved
template<size_t N>
struct Serial
{
char bin[N];
friend ostream& operator<<(ostream& os, const Serial& s)
{
for(auto c : bin)
os << c;
return os;
}
friend istream& operator>>(istream& is, Serial& s)
{
for(auto& c : bin)
is >> c;
return is;
}
};
struct Key
{
static constexpr size_t size = sizeof(Key);
Key(const Serial<size>& s) { memcpy(this, s.bin, size); }
Serial<size>& serialize() { return *reinterpret_cast<Serial<size>*>(this); }
};
struct Value
{
static constexpr size_t size = sizeof(Value);
Key(const Serial<size>& s) { memcpy(this, s.bin, size); }
Serial<size>& serialize() { return *reinterpret_cast<Serial<size>*>(this); }
};
void write(ostream& os, const std::map<Key, Value>& m)
{
for(const auto& p : m)
os << p.first.serialize() << p.second.serialize();
}
void read(istream& is, std::map<Key, Value>& m)
{
Serial<Key::size> k;
Serial<Value::size> v;
while(is >> k >> v)
m[k] = v;
}
For types with dynamic memory (pointers) involved, the solution will be then entirely dependent on how they work, no magical solution can be provided.
Have you considered JSON?
Welcome to the messy, confusing, inconsistent world of serialization. Hope you enjoy the ride!!
This is an age-old problem: how to write a modestly complex data structure to some text or binary format, and then be able to later read it back. There are a couple of different ways to do this. However, you said you wanted to serialize to a binary format, so I would recommend using MessagePack.
There's a C++11 library for working with the MessagePack format called msgpack11 that's also rather lightweight, which would seem to fit your requirements. Here's an example:
std::map<A, B> my_map;
// To save my_map:
msgpack11::MsgPack msgpack{my_map};
std::string binary_data = msgpack.dump();
// Now you can save binary_data to a file.
// To get the map back:
string error_string;
auto msgpack = msgpack11::MsgPack::parse(binary_data, error_string);
std::map<A, B> my_map;
// Now you need to manually read back the data.
For binary writing you should use write method of ostream
ostream& ostream::write (const char* s, streamsize n);
See documentation here: http://en.cppreference.com/w/cpp/io/basic_ostream
You can't write map to file driectly, you should write it's represntation , developed by you. You would need to write each key/value pair individually or buffer them is a data block and write it into file.This really isn't much more complicated than a for loop, though. If map contains classes that aren't trivially constructed and destroyed, you should implement a method that allows to serialize class' binary data.
Binary implementations will necessarily be non-portable (for the resultant file). If that is not a concern then consider defining a custom allocator that uses a memory mapped file. You would then declare your std:map using that allocator as one of the template arguments. You could use that map directly, or using range insertion to save an existing map to a file. If the key or value require allocators (e.g. strings) you would have to do declare versions of those types using the memory mapped allocator in the template declaration and define assignment operators from the key/value type to the new types.
You can find some allocator implementations and further discussion by searching for "memory mapped file stl allocator". Also see: Memory mapped file storage in stl vector
void BinSerialize(ostream &out, int32_t x);
void BinSerialize(ostream &out, int16_t x);
void BinSerialize(ostream &out, int8_t x);
void BinSerialize(ostream &out, const string &s)
{
BinSerialize(out, (int32_t)s.size());
out.write(size.c_str(), s.size()));
}
temmplate<class KeyT, ValueT>
void BinSerialize(ostream &out, const std::map<KeyT, ValueT> &m)
{
BinSerialize(out, (int32_t)m.size());
for (auto& item : m)
{
BinSerialize(out, item.first);
BinSerialize(out, item.second);
}
}
void BinDeserialize(istream &input, int32& x);
void BinDeserialize(istream &input, int16& x);
void BinDeserialize(istream &input, int8& x);
void BinDeserialize(istream &input, string &s)
{
int32_t size;
BinDerialize(out, size);
s.resize(size);
out.read(size.c_str(), size);
}
temmplate<class KeyT, class ValueT>
void BinDeserialize(istream &input, std::map<KeyT, ValueT> &m)
{
int32_t size;
m.clear();
BinDeserialize(out, size);
for (int32_t i=0; i<size; ++i)
{
std::pair<KeyT, ValueT> item;
BinDeserialize(out, item.first);
BinDeserialize(out, item.second);
m.insert(item);
}
}
This is quickly written. It is possible to improve it with templates, to cover all basic types and all STL containers.
Also it would be nice to keep in mind about the endian.
It is better avoid use of overloaded operators in this case. But if you if to do it it is best to define class which will wrapp STL stream and will have own set of overloaded >> << operators. Take a look on Qt QDataStream.
Related
I am wondering if it is possible to have a std::unordered_map use a fixed size array as a key. For example, here is a simple cache that holds strings as the value but needs a uint8_t[] as the key:
using UserKeyV1 = uint8_t[16];
using UserKeyV2 = uint8_t[32];
template <typename T>
class StringCache
{
public:
bool addString(const T &userKey, const std::string &value)
{
auto [it, result] = m_cache.try_emplace(userKey, value);
// ^^^^^^^ this line won't compile: array initializer must be an initializer list or string literal
}
private:
struct EqualToFn {
bool operator()(const T &left, const T &right) const {
return std::memcmp(&left[0],
&right[0],
sizeof(T)) == 0;
}
};
struct HashFn {
size_t operator()(const T &k) const {
return std::_Hash_impl::hash(k);
}
};
std::unordered_map<T, std::string, HashFn, EqualToFn> cache_;
}
And in use would be something like:
StringCache<UserKey1> cache1;
uint8_t uniqueKey[16]; // this key was provided by 3rd party lib
cache1.addString(uniqueKey, strUsername)
This won't compile due to the error listed above but I'm not sure why. I created the custom hasher and equality functions for the array so that it knew how to handle such a key. I could probably solve this with std::array and copy the key to it first but wanted to avoid that if possible as it would involve a copy and this code will be potentially called 1000s of times a second.
Is what I am trying to achieve possible or do I just use std::array as a key?
it works fine with std::array
#include <cstdlib>
#include <iostream>
#include <unordered_map>
#include <cstring>
#include <array>
using UserKeyV1 = std::array<unsigned char,16>;
template <typename T>
class StringCache
{
public:
bool addString(const T &userKey, const std::string &value)
{
auto [it, result] = m_cache.try_emplace(userKey, value);
// ^^^^^^^ this line won't compile: array initializer must be an initializer list or string literal
return result;
}
private:
struct EqualToFn {
bool operator()(const T &left, const T &right) const {
return std::memcmp(&left[0],
&right[0],
sizeof(T)) == 0;
}
};
struct HashFn {
size_t operator()(const T &k) const {
return std::_Hash_impl::hash(k);
}
};
std::unordered_map<T, std::string, HashFn, EqualToFn> m_cache;
};
int main()
{
StringCache<UserKeyV1> cache1;
//uint8_t uniqueKey[16]; // this key was provided by 3rd party lib
UserKeyV1 uniqueKey;
std::string strUsername = "username";
cache1.addString(uniqueKey, strUsername);
}
I think if you want to avoid copy you can just use std::string_view class, the code becomes simpler.
#include <unordered_map>
#include <string_view>
using UserKeyV1 = std::string_view;
template <typename T>
class StringCache
{
public:
bool addString(const T &userKey, const std::string &value)
{
auto [it, result] = m_cache.try_emplace(userKey, value);
return result;
}
std::unordered_map<T, std::string> m_cache;
};
int main()
{
StringCache<UserKeyV1> cache1;
uint8_t uniqueKey[16]; // this key was provided by 3rd party lib
std::string strUsername = "username";
cache1.addString(std::string_view{(const char*)uniqueKey, sizeof(uniqueKey)}, strUsername);
}
for c++20 you can probably use std::span, or to implement your own wrapper for earlier standards.
I think the best way of doing this (without copying the value as you requested) would be to calculate an hash using the uint8_t[16] array and use the hash as key. You can choose one among the many hashing algorithms already existing (xxhash, md5, sha32, crc32 etc...).
The best match with your case would be crc32.
Any of these usually takes a const void* and a size_t to calculate the memory hash, you can pass your key address and size without copying it as you requested.
An example would be to use crc32 hash (basically an uint32_t) and to use it as key for the unordered_map.
So you can still keep your class, by adding one line in your example.
StringCache<uint32_t> cache1;
uint8_t uniqueKey[16];
cache1.addString(GetArrayHash(uniqueKey), strUsername);
You can't use this solution if you need to get back the uint8_t[16] from your cache. Based on your class code you posted there are no uses of the actual key value.
Note: an unordered_map with uint32_t as key will use an identify function as hash algorithm, in short by using crc32 hashing you are probably not losing efficiency but this is not guaranteed by the standard since it does not define exactly which algorithm to use to hash an array of uint8_t (but the msvc and gcc implementations I have been able to test use crc32).
using this approach you can definitely remove your HashFn and EqualToFn to your StringCache class.
template <typename T>
class StringCache
{
public:
bool addString(const T &userKey, const std::string &value)
{
auto [it, result] = m_cache.try_emplace(userKey, value);
}
private:
std::unordered_map<T, std::string> cache_;
}
the object like this:
#include <vector>
using namespace std;
class A
{
public:
vector<int> i;
//save other variable
};
void save(const A & a)
{
//dosomething
}
int main()
{
A a;
save(a);
return 0;
}
how to save it to a binary file?
in other word, how to write the save function ?
ofstream and copy should suffice:
void save(const A & a)
{
std::ofstream out_file("path/to/output/file", std::ios::binary);
std::copy(a.i.begin(), a.i.end(), std::ostream_iterator<int>(out_file));
}
BTW you should probably avoid using namespace std;.
As pointed out in comments (thanks #Daniel Langr) this will work only for types with working implementation of operator<< for ofstream or its base classes
There are libraries that can help you with storing to files, but you can do it yourself too.
If you want the data in binary format in the file, then you can use the ostream::write() function and for saving a container, at least the size should be stored.
So a very specific function would be
void save(const A & a)
{
std::ofstream out_file("path/to/output/file", std::ios::binary);
size_t sz = a.i.size();
out_file.write(reinterpret_cast<const char*>(&sz), sizeof(size_t));
for (const auto& i: a.i) out_file.write(reinterpret_cast<const char*>(&i), sizeof(int));
}
It would be better to pass the stream to the function and have the class save function call the container save function which in turns save each entry like this:
void save(std::ofstream& out, int i)
{
out.write(reinterpret_cast<const char*>(&i), sizeof(int));
}
template<typename E> void save(std::ofstream& out, const std::vector<E>& v)
{
size_t sz = v.size();
out.write(reinterpret_cast<const char*>(&sz), sizeof(size_t));
for (const auto& i: v) save(out, i);
}
void save(std::ofstream& out, const A & a)
{
save(out, a.i);
}
This way, containers with entries of any type an be saved as long as there is a save function.
I have a class which holds a single wstring.
I have 2 printing functions one for ostream and one for wostream.
I must use them like this:
MyClass<char> s;
s.push_back( 'H' );
s.push_back( 'i' );
//Printing with ostream
s.print( std::cout );
MyClass<wchar_t> w;
w.push_back( L'C' );
w.push_back( L'+' );
//Printing with wostream
w.print( std::wcout );
My problem is, i cannot make a function to work with ostream, here is my class:
template<typename T>
MyClass{
private:
std::wstring *data;
public:
void push_back(const T& t) {
data->push_back(t);
}
//Does not work
void print(std::ostream& out) const{
out<<*data; //Results in a compile error
}
//Does work verry well
void print(std::wostream& out) const {
out << *data;
}
};
How can i achive this?
P.S.: I know this is nonsense, but in fact this is in a school exam which i had last year. So i cannot change the standards. This is the task.
The problem is that you're storing a std::wstring regardless of whether the template type T is char or wchar_t. Instead, you want to use the template std::basic_string with the appropriate type:
std::basic_string<T> *data;
std::string is just a typedef for std::basic_string<char>, and std::wstring is just a typedef for std::basic_string<wchar_t>.
Similarly, std::ostream is just a typedef for std::basic_ostream<char>, and std::wostream is just a typedef for std::basic_ostream<wchar_t>, so you could have a single print function that takes a std::basic_ostream templated on the appropriate type.
Store the correct type of string
std::basic_string<T> data;
and overload for just the correct basic_ostream specialisation, not for both ostream and wostream:
void print(std::basic_ostream<T> & out) const {
out << data;
}
(Also, storing the string itself rather than an uninitialised pointer will give better results).
Sorry, new to C++, converting from C, and have struggled to find a good way to do this...
//Fragment follows
const char *List1[]={"Choice1", "Not a good choice", "choice3"}; //rom-able
const char *List2[]={"Hello", "Experts", "Can", "You", "Help?"};
class ListIF{
private:
int index;
char *list;
public:
void GetValString(char *tgt,int len);//get parameter value as string max len chars
void SetIndex(int n){index = n;};
int GetIndex(void){return index;};
};
//end Fragment
The problem is how to write the constructor so that I can "encapsulate" the lists inside the class, without getting heap bloat (embedded target). And then how to write the gettor so that we can see list[index] within the class.
I am going daft trying to do something that seems obvious, so I am missing something?
In C++, prefer using std::string over const char*. It will solve most of your problems you face with const char*.
For an array of strings, use std::vector<std::string>. It will solve most of your problems you face with const char *[].
You can even initialize the std::vector with multiple strings as,
std::vector<std::string> List1(adder<std::string>("Choice1")("Not a good choice")("choice3"));
std::vector<std::string> List2(adder<std::string>("Hello")("Experts")("Can")("You")("Help?"));
Where adder<> is a class template defined as:
template<typename T>
struct adder
{
std::vector<T> items;
adder(const T &item) { items.push_back(item); }
adder& operator()(const T & item) { items.push_back(item); return *this; }
operator std::vector<T>&() { return items ; }
};
Sample running code here : http://www.ideone.com/GLEZr
/** Wrapper for C style arrays; does not take ownership of the array */
template <typename T>
class static_array
{
T *array;
size_t nelems;
public:
template <size_t N>
static_array(T (&a)[N]) : array(a), nelems(N) {}
T &operator[](size_t i) { return array[i]; }
T const &operator[](size_t i) const { return array[i]; }
size_t size() const { return nelems; }
};
typedef static_array<char const *> static_cstr_array;
Construct as static_cstr_array array1(List1). The setter is operator[], i.e.
array1[1] = "foo!";
You can add any method that you want to this class.
(I chose the name static_array because, as far as the class is concerned, the underlying array must be static: it should not grow, shrink or move due to realloc or otherwise. It doesn't mean the array must have static linkage.)
Not sure what you want your functions to be but one way to wrap the arrays would be:
EDIT : changed to incorporate Larsmans suggestion (on the chance that your compiler can't handle his answer).
class ListIF
{
private:
std::vector<const char*> m_list;//stores ptrs to the ROM
public:
ListIF(char const **list, size_t n) : m_list(list, list+n) {}
const char* get( int pos )
{
return m_list[pos];
}
};
I have a std::vector<int> and I want serialize it. For this purpose I am trying to use a std::stringstream
vector<int> v;
v.resize(10);
for (int i=0;i<10;i++)
v[i]=i;
stringstream ss (stringstream::in | stringstream::out |stringstream::binary);
However when I copy the vector to the stringstream this copy it as character
ostream_iterator<int> it(ss);
copy(v.begin(),v.end(),it);
the value that inserted to buffer(_Strbuf) is "123456789"
I sucssesed to write a workaround solution
for (int i=1;i<10;i++)
ss.write((char*)&p[i],sizeof(int));
I want to do it something like first way by using std function like copy
thanks Herzl
Actually, this is your workaround but it may be used with std::copy() algorithm.
template<class T>
struct serialize
{
serialize(const T & i_value) : value(i_value) {}
T value;
};
template<class T>
ostream& operator <<(ostream &os, const serialize<T> & obj)
{
os.write((char*)&obj.value,sizeof(T));
return os;
}
Usage
ostream_iterator<serialize<int> > it(ss);
copy(v.begin(),v.end(),it);
I know this is not an answer to your problem, but if you are not limited to the STL you could try (boost serialization) or google protocol buffers
Boost even has build-in support for de-/serializing STL containers (http://www.boost.org/doc/libs/1_45_0/libs/serialization/doc/tutorial.html#stl)
To use std::copy with ostream::write, you'd need to write your own output iterator that knows how to correctly serialize the type. That said, I'm not sure what you expect to gain from this approach, but here's a first pass at that for an example:
struct ostream_write_int
: std::iterator<std::output_iterator_tag, int, void, void, void>
{
std::ostream *s;
ostream_write_int(std::ostream &s) : s (&s) {}
ostream_write_int& operator++() { return *this; }
ostream_write_int& operator++(int) { return *this; }
ostream_write_int& operator*() { return *this; }
void operator=(int x) {
s->write(reinterpret_cast<char*>(&x), sizeof(x));
}
};
This could be templated only if you defer the serialization logic to some other function (as the formatted stream iterators do to operator<<).
Like Fred, I don't see the point of this, what you are effectively trying to do is:
ss.rdbuf()->sputn(reinterpret_cast<char*>(&v[0]), sizeof(int) * v.size());