I have a std::vector<int> and I want serialize it. For this purpose I am trying to use a std::stringstream
vector<int> v;
v.resize(10);
for (int i=0;i<10;i++)
v[i]=i;
stringstream ss (stringstream::in | stringstream::out |stringstream::binary);
However when I copy the vector to the stringstream this copy it as character
ostream_iterator<int> it(ss);
copy(v.begin(),v.end(),it);
the value that inserted to buffer(_Strbuf) is "123456789"
I sucssesed to write a workaround solution
for (int i=1;i<10;i++)
ss.write((char*)&p[i],sizeof(int));
I want to do it something like first way by using std function like copy
thanks Herzl
Actually, this is your workaround but it may be used with std::copy() algorithm.
template<class T>
struct serialize
{
serialize(const T & i_value) : value(i_value) {}
T value;
};
template<class T>
ostream& operator <<(ostream &os, const serialize<T> & obj)
{
os.write((char*)&obj.value,sizeof(T));
return os;
}
Usage
ostream_iterator<serialize<int> > it(ss);
copy(v.begin(),v.end(),it);
I know this is not an answer to your problem, but if you are not limited to the STL you could try (boost serialization) or google protocol buffers
Boost even has build-in support for de-/serializing STL containers (http://www.boost.org/doc/libs/1_45_0/libs/serialization/doc/tutorial.html#stl)
To use std::copy with ostream::write, you'd need to write your own output iterator that knows how to correctly serialize the type. That said, I'm not sure what you expect to gain from this approach, but here's a first pass at that for an example:
struct ostream_write_int
: std::iterator<std::output_iterator_tag, int, void, void, void>
{
std::ostream *s;
ostream_write_int(std::ostream &s) : s (&s) {}
ostream_write_int& operator++() { return *this; }
ostream_write_int& operator++(int) { return *this; }
ostream_write_int& operator*() { return *this; }
void operator=(int x) {
s->write(reinterpret_cast<char*>(&x), sizeof(x));
}
};
This could be templated only if you defer the serialization logic to some other function (as the formatted stream iterators do to operator<<).
Like Fred, I don't see the point of this, what you are effectively trying to do is:
ss.rdbuf()->sputn(reinterpret_cast<char*>(&v[0]), sizeof(int) * v.size());
Related
Short version
Can I reinterpret_cast a std::vector<void*>* to a std::vector<double*>*?
What about with other STL containers?
Long version
I have a function to recast a vector of void pointers to a datatype specified by a template argument:
template <typename T>
std::vector<T*> recastPtrs(std::vector<void*> const& x) {
std::vector<T*> y(x.size());
std::transform(x.begin(), x.end(), y.begin(),
[](void *a) { return static_cast<T*>(a); } );
return y;
}
But I was thinking that copying the vector contents isn't really necessary, since we're really just reinterpreting what's being pointed to.
After some tinkering, I came up with this:
template <typename T>
std::vector<T*> recastPtrs(std::vector<void*>&& x) {
auto xPtr = reinterpret_cast<std::vector<T*>*>(&x);
return std::vector<T*>(std::move(*xPtr));
}
So my questions are:
Is it safe to reinterpret_cast an entire vector like this?
What if it was a different kind of container (like a std::list or std::map)? To be clear, I mean casting a std::list<void*> to std::list<T*>, not casting between STL container types.
I'm still trying to wrap my head around move semantics. Am I doing it right?
And one follow-up question: What would be the best way to generate a const version without code duplication? i.e. to define
std::vector<T const*> recastPtrs(std::vector<void const*> const&);
std::vector<T const*> recastPtrs(std::vector<void const*>&&);
MWE
#include <vector>
#include <algorithm>
#include <iostream>
template <typename T>
std::vector<T*> recastPtrs(std::vector<void*> const& x) {
std::vector<T*> y(x.size());
std::transform(x.begin(), x.end(), y.begin(),
[](void *a) { return static_cast<T*>(a); } );
return y;
}
template <typename T>
std::vector<T*> recastPtrs(std::vector<void*>&& x) {
auto xPtr = reinterpret_cast<std::vector<T*>*>(&x);
return std::vector<T*>(std::move(*xPtr));
}
template <typename T>
void printVectorAddr(std::vector<T> const& vec) {
std::cout<<" vector object at "<<&vec<<", data()="<<vec.data()<<std::endl;
}
int main(void) {
std::cout<<"Original void pointers"<<std::endl;
std::vector<void*> voidPtrs(100);
printVectorAddr(voidPtrs);
std::cout<<"Elementwise static_cast"<<std::endl;
auto dblPtrs = recastPtrs<double>(voidPtrs);
printVectorAddr(dblPtrs);
std::cout<<"reintepret_cast entire vector, then move ctor"<<std::endl;
auto dblPtrs2 = recastPtrs<double>(std::move(voidPtrs));
printVectorAddr(dblPtrs2);
}
Example output:
Original void pointers
vector object at 0x7ffe230b1cb0, data()=0x21de030
Elementwise static_cast
vector object at 0x7ffe230b1cd0, data()=0x21de360
reintepret_cast entire vector, then move ctor
vector object at 0x7ffe230b1cf0, data()=0x21de030
Note that the reinterpret_cast version reuses the underlying data structure.
Previously-asked questions that didn't seem relevant
These are the questions that come up when I tried to search this:
reinterpret_cast vector of class A to vector of class B
reinterpret_cast vector of derived class to vector of base class
reinterpret_cast-ing vector of one type to a vector of another type which is of the same type
And the answer to these was a unanimous NO, with reference to the strict aliasing rule. But I figure that doesn't apply to my case, since the vector being recast is an rvalue, so there's no opportunity for aliasing.
Why I'm trying to do this
I'm interfacing with a MATLAB library that gives me data pointers as void* along with a variable indicating the datatype. I have one function that validates the inputs and collects these pointers into a vector:
void parseInputs(int argc, mxArray* inputs[], std::vector<void*> &dataPtrs, mxClassID &numericType);
I can't templatize this part since the type is not known until runtime. On the other side, I have numeric routines to operate on vectors of a known datatype:
template <typename T>
void processData(std::vector<T*> const& dataPtrs);
So I'm just trying to connect one to the other:
void processData(std::vector<void*>&& voidPtrs, mxClassID numericType) {
switch (numericType) {
case mxDOUBLE_CLASS:
processData(recastPtrs<double>(std::move(voidPtrs)));
break;
case mxSINGLE_CLASS:
processData(recastPtrs<float>(std::move(voidPtrs)));
break;
default:
assert(0 && "Unsupported datatype");
break;
}
}
Given the comment that you're receiving the void * from a C library (something like malloc), it seems like we can probably narrow the problem down quite a bit.
In particular, I'd guess you're really dealing with something that's more like an array_view than a vector. That is, you want something that lets you access some data cleanly. You might change individual items in that collection, but you'll never change the collection as a whole (e.g., you won't try to do a push_back that could need to expand the memory allocation).
For such a case, you can pretty easily create a wrapper of your own that gives you vector-like access to the data--defines an iterator type, has a begin() and end() (and if you want, the others like rbegin()/rend(), cbegin()/cend() and crbegin()/crend()), as well as an at() that does range-checked indexing, and so on.
So a fairly minimal version could look something like this:
#pragma once
#include <cstddef>
#include <stdexcept>
#include <cstdlib>
#include <iterator>
template <class T> // note: no allocator, since we don't do allocation
class array_view {
T *data;
std::size_t size_;
public:
array_view(void *data, std::size_t size_) : data(reinterpret_cast<T *>(data)), size_(size_) {}
T &operator[](std::size_t index) { return data[index]; }
T &at(std::size_t index) {
if (index > size_) throw std::out_of_range("Index out of range");
return data[index];
}
std::size_t size() const { return size_; }
typedef T *iterator;
typedef T const &const_iterator;
typedef T value_type;
typedef T &reference;
iterator begin() { return data; }
iterator end() { return data + size_; }
const_iterator cbegin() { return data; }
const_iterator cend() { return data + size_; }
class reverse_iterator {
T *it;
public:
reverse_iterator(T *it) : it(it) {}
using iterator_category = std::random_access_iterator_tag;
using difference_type = std::ptrdiff_t;
using value_type = T;
using pointer = T *;
using reference = T &;
reverse_iterator &operator++() {
--it;
return *this;
}
reverse_iterator &operator--() {
++it;
return *this;
}
reverse_iterator operator+(size_t size) const {
return reverse_iterator(it - size);
}
reverse_iterator operator-(size_t size) const {
return reverse_iterator(it + size);
}
difference_type operator-(reverse_iterator const &r) const {
return it - r.it;
}
bool operator==(reverse_iterator const &r) const { return it == r.it; }
bool operator!=(reverse_iterator const &r) const { return it != r.it; }
bool operator<(reverse_iterator const &r) const { return std::less<T*>(r.it, it); }
bool operator>(reverse_iterator const &r) const { return std::less<T*>(it, r.it); }
T &operator *() { return *(it-1); }
};
reverse_iterator rbegin() { return data + size_; }
reverse_iterator rend() { return data; }
};
I've tried to show enough that it should be fairly apparent how to add most of the missing functionality (e.g., crbegin()/crend()), but I haven't worked really hard at including everything here, since much of what's left is more repetitive and tedious than educational.
This is enough to use the array_view in most of the typical vector-like ways. For example:
#include "array_view"
#include <iostream>
#include <iterator>
int main() {
void *raw = malloc(16 * sizeof(int));
array_view<int> data(raw, 16);
std::cout << "Range based:\n";
for (auto & i : data)
i = rand();
for (auto const &i : data)
std::cout << i << '\n';
std::cout << "\niterator-based, reverse:\n";
auto end = data.rend();
for (auto d = data.rbegin(); d != end; ++d)
std::cout << *d << '\n';
std::cout << "Forward, counted:\n";
for (int i=0; i<data.size(); i++) {
data[i] += 10;
std::cout << data[i] << '\n';
}
}
Note that this doesn't attempt to deal with copy/move construction at all, nor with destruction. At least as I've formulated it, the array_view is a non-owning view into some existing data. It's up to you (or at least something outside of the array_view) to destroy the data when appropriate. Since we're not destroying the data, we can use the compiler-generated copy and move constructors without any problem. We won't get a double-delete from doing a shallow copy of the pointer, because we don't do any delete when the array_view is destroyed.
No, you cannot do anything like this in Standard C++.
The strict aliasing rule says that to access an object of type T, you must use an expression of type T; with a very short list of exceptions to that.
Accessing a double * via a void * expression is not such an exception; let alone a vector of each. Nor is it an exception if you accessed the object of type T via an rvalue.
I want to write a std::map to a file and read it back. I'm looking for a rather simple and minimalist way to do it, without boost. I found that it is doable with vector like here Reading and writing a std::vector into a file correctly with iterators
I found this question as it relates to what I want to do, except I'm looking for the binary alternative.
reading a file of key-value pairs in to a std::map
For types with no dynamic memory (actually, pointers) involved
template<size_t N>
struct Serial
{
char bin[N];
friend ostream& operator<<(ostream& os, const Serial& s)
{
for(auto c : bin)
os << c;
return os;
}
friend istream& operator>>(istream& is, Serial& s)
{
for(auto& c : bin)
is >> c;
return is;
}
};
struct Key
{
static constexpr size_t size = sizeof(Key);
Key(const Serial<size>& s) { memcpy(this, s.bin, size); }
Serial<size>& serialize() { return *reinterpret_cast<Serial<size>*>(this); }
};
struct Value
{
static constexpr size_t size = sizeof(Value);
Key(const Serial<size>& s) { memcpy(this, s.bin, size); }
Serial<size>& serialize() { return *reinterpret_cast<Serial<size>*>(this); }
};
void write(ostream& os, const std::map<Key, Value>& m)
{
for(const auto& p : m)
os << p.first.serialize() << p.second.serialize();
}
void read(istream& is, std::map<Key, Value>& m)
{
Serial<Key::size> k;
Serial<Value::size> v;
while(is >> k >> v)
m[k] = v;
}
For types with dynamic memory (pointers) involved, the solution will be then entirely dependent on how they work, no magical solution can be provided.
Have you considered JSON?
Welcome to the messy, confusing, inconsistent world of serialization. Hope you enjoy the ride!!
This is an age-old problem: how to write a modestly complex data structure to some text or binary format, and then be able to later read it back. There are a couple of different ways to do this. However, you said you wanted to serialize to a binary format, so I would recommend using MessagePack.
There's a C++11 library for working with the MessagePack format called msgpack11 that's also rather lightweight, which would seem to fit your requirements. Here's an example:
std::map<A, B> my_map;
// To save my_map:
msgpack11::MsgPack msgpack{my_map};
std::string binary_data = msgpack.dump();
// Now you can save binary_data to a file.
// To get the map back:
string error_string;
auto msgpack = msgpack11::MsgPack::parse(binary_data, error_string);
std::map<A, B> my_map;
// Now you need to manually read back the data.
For binary writing you should use write method of ostream
ostream& ostream::write (const char* s, streamsize n);
See documentation here: http://en.cppreference.com/w/cpp/io/basic_ostream
You can't write map to file driectly, you should write it's represntation , developed by you. You would need to write each key/value pair individually or buffer them is a data block and write it into file.This really isn't much more complicated than a for loop, though. If map contains classes that aren't trivially constructed and destroyed, you should implement a method that allows to serialize class' binary data.
Binary implementations will necessarily be non-portable (for the resultant file). If that is not a concern then consider defining a custom allocator that uses a memory mapped file. You would then declare your std:map using that allocator as one of the template arguments. You could use that map directly, or using range insertion to save an existing map to a file. If the key or value require allocators (e.g. strings) you would have to do declare versions of those types using the memory mapped allocator in the template declaration and define assignment operators from the key/value type to the new types.
You can find some allocator implementations and further discussion by searching for "memory mapped file stl allocator". Also see: Memory mapped file storage in stl vector
void BinSerialize(ostream &out, int32_t x);
void BinSerialize(ostream &out, int16_t x);
void BinSerialize(ostream &out, int8_t x);
void BinSerialize(ostream &out, const string &s)
{
BinSerialize(out, (int32_t)s.size());
out.write(size.c_str(), s.size()));
}
temmplate<class KeyT, ValueT>
void BinSerialize(ostream &out, const std::map<KeyT, ValueT> &m)
{
BinSerialize(out, (int32_t)m.size());
for (auto& item : m)
{
BinSerialize(out, item.first);
BinSerialize(out, item.second);
}
}
void BinDeserialize(istream &input, int32& x);
void BinDeserialize(istream &input, int16& x);
void BinDeserialize(istream &input, int8& x);
void BinDeserialize(istream &input, string &s)
{
int32_t size;
BinDerialize(out, size);
s.resize(size);
out.read(size.c_str(), size);
}
temmplate<class KeyT, class ValueT>
void BinDeserialize(istream &input, std::map<KeyT, ValueT> &m)
{
int32_t size;
m.clear();
BinDeserialize(out, size);
for (int32_t i=0; i<size; ++i)
{
std::pair<KeyT, ValueT> item;
BinDeserialize(out, item.first);
BinDeserialize(out, item.second);
m.insert(item);
}
}
This is quickly written. It is possible to improve it with templates, to cover all basic types and all STL containers.
Also it would be nice to keep in mind about the endian.
It is better avoid use of overloaded operators in this case. But if you if to do it it is best to define class which will wrapp STL stream and will have own set of overloaded >> << operators. Take a look on Qt QDataStream.
I have some functions that I use to convert a 2D variant SAFEARRAY into various STL containers, kinda like so (illustrative only)
template<typename T>
std::set<T> SetFromSafeArray(VARIANT srcArray)
{
CComSafeArray<T> srcComArray(srcArray.parray);
std::set<T> destContainer;
for (ULONG i=0;i<srcComArray.GetCount();++i)
destContainer.insert(srcComArray.GetAt(i));
return destContainer;
}
I feel it's not a very c++-ish way of going about it and it means there's a separate function for each STL container I convert to.
My idea was to write a wrapper and custom iterator for CComSafeArrays so I could just do...
std::copy(srcComArray.begin(), srcComArray.end(), destContainer.begin());
but having never written an iterator before and being a novice I really don't know if it will be easy.
Is a custom CComSafeArray iterator my best, standard c++ like, option (in which case I'm sure I can find a good tutorial on writing an iterator)? Or is there some other way of going about it?
Boost is not an option.
TIA
My idea was to write a wrapper and custom iterator for CComSafeArrays
This is very good idea for creating iterator, but you don't need a wrapper around CComSafeArray<T>, only iterator is needed.
so I could just do...
std::copy(srcComArray.begin(), srcComArray.end(), destContainer.begin());
But instead of doing your way, you can do this:
SomeContainer<T> destContainer(begin(srcComArray), end(srcComArray));
Because almost every STL container has constructor from range (pair of iterators).
Assuming you have written iterator over CComSafeArray<T> - functions begin/end will be like these:
template <typename T>
CComSafeArrayIterator<T> begin(CComSafeArray<T>& container)
{
return CComSafeArrayIterator<T>(container, 0);
}
template <typename T>
CComSafeArrayIterator<T> end(CComSafeArray<T>& container)
{
return CComSafeArrayIterator<T>(container, container.getSize());
}
Notice that begin() is zero position, end() is getSize() position.
And writing an iterator is not rocket science. Just a few functions.
The most important is to know what you need to iterate. In your case: the container reference(pointer) and the current position. Iterating is just moving the position. Accessing is via container and position. Comparing is via comparing position.
template <typename T>
class CComSafeArrayIterator {
public:
CComSafeArrayIterator(CComSafeArray<T>& container, ULONG position)
: container(&container),
position(position)
{}
// prefix ++it
CComSafeArrayIterator& operator ++() { ++position; return *this; }
// postfix it++
CComSafeArrayIterator operator ++(int) {
CComSafeArrayIterator prev = *this;
++position;
return prev;
}
// access by ->: it-> !! ony if getAt return reference not value
const T* operator -> () const {
return &(container->getAt(position));
}
// access by *: *it
const T& operator * () const {
return container->getAt(position);
}
// comparing
friend bool operator == (const CComSafeArrayIterator<T>& l,
const CComSafeArrayIterator<T>& r)
{
return l.position == r.position;
}
friend bool operator != (const CComSafeArrayIterator<T>& l,
const CComSafeArrayIterator<T>& r)
{
return l.position != r.position;
}
private:
// this is what you need
CComSafeArray<T>* container;
ULONG position;
};
std::vector, std::list and std::deque have std::back_inserter, and std::set has std::inserter.
For std::stack and std::priority_queue I would assume the equivalent inserter would be a push() but I can't seem to find the correct function to call.
My intent is to be able to use the following function with the correct insert iterator:
#include <string>
#include <queue>
#include <iterator>
template<typename outiter>
void foo(outiter oitr)
{
static const std::string s1 ("abcdefghji");
static const std::string s2 ("1234567890");
*oitr++ = s1;
*oitr++ = s2;
}
int main()
{
std::priority_queue<std::string> spq;
std::stack<std::string> stk;
foo(std::inserter(spq));
foo(std::inserter(stk));
return 0;
}
The other alternative (simpler) is just to use the underlying data structure (std::stack is usually implemented using std::deque) and accept that you have to use e.g. push_back() instead of push(). Saves having to code your own iterator, and doesn't particularly affect the clarity of the code. std::stack isn't your only choice for modelling the stack concept.
You can always go your own way and implement an iterator yourself. I haven't verified this code but it should work. Emphasis on "I haven't verified."
template <class Container>
class push_insert_iterator:
public iterator<output_iterator_tag,void,void,void,void>
{
protected:
Container* container;
public:
typedef Container container_type;
explicit push_insert_iterator(Container& x) : container(&x) {}
push_insert_iterator<Container>& operator= (typename Container::const_reference value){
container->push(value); return *this; }
push_insert_iterator<Container>& operator* (){ return *this; }
push_insert_iterator<Container>& operator++ (){ return *this; }
push_insert_iterator<Container> operator++ (int){ return *this; }
};
I'd also add in the following function to help use it:
template<typename Container>
push_insert_iterator<Container> push_inserter(Container container){
return push_insert_iterator<Container>(container);
}
I want to find in a vector of Object pointers for a matching object. Here's a sample code to illustrate my problem:
class A {
public:
A(string a):_a(a) {}
bool operator==(const A& p) {
return p._a == _a;
}
private:
string _a;
};
vector<A*> va;
va.push_back(new A("one"));
va.push_back(new A("two"));
va.push_back(new A("three"));
find(va.begin(), va.end(), new A("two"));
I want to find the second item pushed into the vector. But since vector is defined as a pointers collection, C++ does not use my overloaded operator, but uses implicit pointer comparison. What is the preferred C++-way of solutiono in this situation?
Use find_if with a functor:
template <typename T>
struct pointer_values_equal
{
const T* to_find;
bool operator()(const T* other) const
{
return *to_find == *other;
}
};
// usage:
void test(const vector<A*>& va)
{
A* to_find = new A("two");
pointer_values_equal<A> eq = { to_find };
find_if(va.begin(), va.end(), eq);
// don't forget to delete A!
}
Note: your operator== for A ought to be const, or, better still, write it as a non-member friend function.
Either use std::find_if and provide a suitable predicate yourself, see other answers for an example of this.
Or as an alternative have a look at boost::ptr_vector, which provides transparent reference access to elements which are really stored as pointers (as an extra bonus, memory management is handled for you as well)
Try using find_if instead. It has a parameter for a predicate where you can decide exactly how to check wheter you found the right element.
http://www.sgi.com/tech/stl/find_if.html
You could also use Boost::Lambda:
using namespace boost::lambda;
find_if(va.begin(), va.end(), *_1 == A("two"));
Of course, you should prefer to use shared_ptrs so you don't have to remember to delete!