I'm using Boost.bimap for implementing a LRU cache with some complex key containing a string.
The problem is that the key is copied every-time I'm invoking find(). I would like to avoid this un-necessary copy (and in general: make as few as possible copies of the string, maybe via templates?)
A minimal test-case (with a gist version):
#include <string>
#include <iostream>
#include <boost/bimap.hpp>
#include <boost/bimap/list_of.hpp>
#include <boost/bimap/set_of.hpp>
class Test
{
public:
struct ComplexKey
{
std::string text;
int dummy;
ComplexKey(const std::string &text, int dummy) : text(text), dummy(dummy) {}
~ComplexKey()
{
std::cout << "~ComplexKey " << (void*)this << " " << text << std::endl;
}
bool operator<(const ComplexKey &rhs) const
{
return tie(text, dummy) < tie(rhs.text, rhs.dummy);
}
};
typedef boost::bimaps::bimap<
boost::bimaps::set_of<ComplexKey>,
boost::bimaps::list_of<std::string>
> container_type;
container_type cache;
void run()
{
getValue("foo", 123); // 3 COPIES OF text
getValue("bar", 456); // 3 COPIES OF text
getValue("foo", 123); // 2 COPIES OF text
}
std::string getValue(const std::string &text, int dummy)
{
const ComplexKey key(text, dummy); // COPY #1 OF text
auto it = cache.left.find(key); // COPY #2 OF text (BECAUSE key IS COPIED)
if (it != cache.left.end())
{
return it->second;
}
else
{
auto value = std::to_string(text.size()) + "." + std::to_string(dummy); // WHATEVER...
cache.insert(typename container_type::value_type(key, value)); // COPY #3 OF text
return value;
}
}
};
Bimap is not using the std:map directly (I'm saying this because you are using set_of) but rather creates views via different container adaptors and in the process the keys are copied. It's not possible to significantly improve performance the way you've defined bimap in the question.
To obtain better performance from bimap with respect to the ComplexKey you will rather have to store a pointer to the ComplexKey (preferably raw; no ownership of the keys implied by the bimap) and provide your own sorters. Pointers are cheap to copy, the downside is that you'll have to manage the keys lifetime in parallel to the map.
Do something along the lines of,
std::vector<unique_ptr<ComplexKey>> ComplexKeyOwningContainer;
typedef boost::bimaps::bimap<
boost::bimaps::set_of<ComplexKey*, ComplexKeyPtrSorter>,
boost::bimaps::list_of<std::string>
> container_type;
Note that if you are after performance you also need to be careful about std::string which is the other side of your bimap. They can be quite expensive, temporaries are frequent, and they will suffer from same issues as the ComplexKey key.
Related
I'm trying to insert a pointer object to a map through emplace() but it does not work.
I've created a simple representation of the problem below. I'm trying to insert to newFooList pointer object type Foo*.
I can't seem to find a way to create a type for FooMap* in std::map<int, FooMap*> m_fooMapList. Should it be done with the new on the second field of the map?
#include <iostream>
#include <utility>
#include <stdint.h>
#include <cstdlib>
#include <map>
class Foo
{
private:
int m_foobar;
public:
Foo(int value)
{
m_foobar = value;
}
void setfoobar(int value);
int getfoobar();
};
class FooMap
{
private:
std::map<int, Foo*> m_newFoo;
public:
FooMap() = default;
};
class FooMapList
{
private:
std::map<int, FooMap*> m_fooMapList;
public:
FooMapList() = default;
void insertFoo(Foo* newFooObj);
};
int Foo::getfoobar(void)
{
return(m_foobar);
}
void FooMapList::insertFoo(Foo* newFooObj)
{
if(m_fooMapList.empty())
{
std::cout << "m_fooMapList is empty" << std::endl ;
}
//m_fooMapList.emplace( newFooObj->getfoobar(), newFooObj );
// Need to find a way to insert newFooObj to m_fooMapList
m_fooMapList.second = new FooMap;
}
int main() {
FooMapList newFooList;
for (auto i=1; i<=5; i++)
{
Foo *newFoo = new Foo(i);
newFoo->getfoobar();
newFooList.insertFoo(newFoo);
}
return 0;
}
On g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
$ g++ -std=c++11 -Wall map_of_map.cpp
map_of_map.cpp: In member function ‘void FooMapList::insertFoo(Foo*)’:
map_of_map.cpp:51:18: error: ‘class std::map<int, FooMap*>’ has no member named ‘second’
m_fooMapList.second = new FooMap;
m_fooMapList is defined as
std::map<int, FooMap*> m_fooMapList;
So to insert into it, you need an int and a pointer to FooMap:
m_fooMapList.emplace(newFooObj->getfoobar(), new FooMap);
Having said that, you should use C++ value semantics and rely less on raw pointers:
std::map<int, FooMap> m_fooMapList; // no pointers
m_fooMapList.emplace(newFooObj->getfoobar(), {}); // construct objects in-place
That is, instances of FooMap can reside directly in the map itself.
That way you get better performance and avoid memory leaks.
It's also worth looking into smart pointers (e.g. unique_ptr) if you really want to work with pointers.
I am not sure if you need a map structure where the values are pointers to another map. The FooMapList class could be simple
std::map<int, FooMap> m_fooMapList;
On the other hand, the entire play with the row pointer will bring you nothing but a pain on the neck.
In case the use of
std::map<int, FooMap*> m_fooMapList; and std::map<int, Foo*> are neccesarry, I would go for smartpointers.
Following is an example code with replacing row pointers with std::unique_ptr and shows how to insert map of Foo s to map in-place. See live here
#include <iostream>
#include <utility>
#include <map>
#include <memory>
class Foo
{
private:
int m_foobar;
public:
Foo(int value): m_foobar(value) {}
void setfoobar(int value) noexcept { m_foobar = value; }
int getfoobar() const noexcept { return m_foobar; }
};
class FooMap
{
private:
std::map<int, std::unique_ptr<Foo>> m_newFoo;
// ^^^^^^^^^^^^^^^^^^^^
public:
FooMap() = default;
#if 0 // optional
// copy disabled
FooMap(const FooMap&) = delete;
FooMap& operator=(const FooMap&) = delete;
// move enabled
FooMap(FooMap&&) = default;
FooMap& operator=(FooMap&&) = default;
#endif
// provide a helper function to insert new Foo to the map of Foo s
void insertFoo(std::unique_ptr<Foo> newFooObj)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
{
std::cout << "inserting to FooMap..." << std::endl;
m_newFoo.emplace(newFooObj->getfoobar(), std::move(newFooObj)); // construct in place
}
};
class FooMapList
{
private:
std::map<int, std::unique_ptr<FooMap>> m_fooMapList;
// ^^^^^^^^^^^^^^^^^^^^^^^
public:
FooMapList() = default;
void insertFooMap(std::unique_ptr<Foo> newFooObj)
// ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
{
if (m_fooMapList.empty())
{
std::cout << "m_fooMapList is empty" << std::endl;
}
// create FooMap and insert Foo to it.
FooMap fooMap;
const auto key = newFooObj->getfoobar();
fooMap.insertFoo(std::move(newFooObj));
// finally insert the FooMap to m_fooMapList
std::cout << "inserting to fooMapList..." << std::endl;
m_fooMapList.emplace(key, std::make_unique<FooMap>(std::move(fooMap))); // construct in place
}
};
int main()
{
FooMapList newFooList;
for (auto i = 1; i <= 5; i++)
{
auto newFoo = std::make_unique<Foo>(i);
std::cout << newFoo->getfoobar() << std::endl;
newFooList.insertFooMap(std::move(newFoo));
}
return 0;
}
Output:
1
m_fooMapList is empty
inserting to FooMap...
inserting to fooMapList...
2
inserting to FooMap...
inserting to fooMapList...
3
inserting to FooMap...
inserting to fooMapList...
4
inserting to FooMap...
inserting to fooMapList...
5
inserting to FooMap...
inserting to fooMapList...
You can throw away your do-nothing map classes, stop using pointers, and just
#include <iostream>
#include <utility>
#include <stdint.h>
#include <cstdlib>
#include <map>
class Foo
{
private:
int m_foobar;
public:
Foo(int value) : m_foobar(value) { }
void setfoobar(int value) { m_foobar = value; }
int getfoobar() const { return m_foobar; }
// or more simply
// int foobar;
};
using FooMap = std::map<int, Foo>;
using FooMapMap = std::map<int, FooMap>;
int main() {
FooMapMap foos;
for (auto i=1; i<=5; i++)
{
foos[i][i] = Foo(i);
}
return 0;
}
Do note that the inner map is totally pointless at this stage, as they only ever have one entry
Except if you have a really good reason to do so, avoid to obfuscate things a la Java like this and try to leverage the STL anyway you can. To that end you can use type aliases
using FooMap = std::map<int, Foo*>; // Maybe use a smart pointer instead here?
using FooMapList = std::map<int, FooMap>; // Maybe List is not an appropriate name for a map
Now, you have a Foo element you just created and want to insert it in your map list, to do so you need a way to select in which map in the list you want to insert it. I will assume you'll insert in the first map in the list:
auto FooMap::emplace(int key, Foo* value)
{
return m_newFoo.emplace(key, value);
}
void FooMapList::insertFoo(Foo* newFooObj)
{
// If the map for `getfoobar` does not exist yet, operator[] will create it
auto& mapPtr = m_fooMapList[newFooObj->getfoobar()];
if (nullptr == mapPtr)
mapPtr = new FooMap();
mapPtr->emplace(
newFooObj->getfoobar(),
newFooObj
);
}
Note that I didn't handle memory cleanup. I suggest you try to use smart pointers when applicable (std::unique_ptr and std::shared_ptr)
I have considered valid points from each of the answers to remove pointers and remove useless double level map representations. But the real world abstraction is a much complex problem which involves thousands of objects on the fly which needs to be created and destroyed dynamically. Using pointers seemed a valid approach, but JeJo' approach seemed much better.
I tried to re-use his attempt but with object pointers and the below seems to work. With the below insert functions
In the FooMap class the function would be
void FooMap::insertFoo(Foo* newFooObj)
{
m_newFoo.emplace(newFooObj->getfoobar(), newFooObj);
}
const std::map<int, Foo*> FooMap::getList()
{
return m_newFoo;
}
and in FooMapList it would be
void FooMapList::insertFooList(Foo* newFooObj)
{
std::map <int, FooMap*>::iterator iter;
FooMap *localFooMap = NULL;
iter = m_fooMapList.find( newFooObj->getfoobar() );
if( iter == m_fooMapList.end() )
{
localFooMap = new FooMap;
localFooMap->insertFoo(newFooObj);
m_fooMapList.emplace(newFooObj->getfoobar(), localFooMap );
}
else
{
localFooMap = iter->second;
localFooMap->insertFoo(newFooObj);
m_fooMapList.emplace(newFooObj->getfoobar(), localFooMap );
}
}
const std::map<int, FooMap*> FooMapList::getList()
{
return m_fooMapList;
}
I would appreciate feedback for this approach too. I will be adding the calls to destructor to clean up the created objects
When writing a custom serializer for msgpack_c one also needs to implement object_with_zone.
The documentation how to implement this is very sparse ( https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_adaptor ).
In what circumstances is this method called?
You can create a msgpack::object from C++ types.
See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_object#conversion
When you call msgpack::object constructor with zone like msgpack::object(mc, z);, object_with_zone<T>::operator() is called internally.
If you don't want to create msgpack::object from C++ types, you don't need to define object_with_zone specialization. Packing, unpacking, and converting to C++ types from msgpack::object don't require it.
Here is an example:
#include <iostream>
#include <msgpack.hpp>
class my_class {
public:
my_class(std::string const& name, int age):name_(name), age_(age) {}
std::string const& get_name() const { return name_; }
int get_age() const { return age_; }
private:
std::string name_;
int age_;
};
// User defined class template specialization
namespace msgpack {
MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS) {
namespace adaptor {
template <>
struct object_with_zone<my_class> {
void operator()(msgpack::object::with_zone& o, my_class const& v) const {
std::cout << "object_with_zone<my_class> is called" << std::endl;
o.type = type::ARRAY;
o.via.array.size = 2;
o.via.array.ptr = static_cast<msgpack::object*>(
o.zone.allocate_align(sizeof(msgpack::object) * o.via.array.size, MSGPACK_ZONE_ALIGNOF(msgpack::object)));
o.via.array.ptr[0] = msgpack::object(v.get_name(), o.zone);
o.via.array.ptr[1] = msgpack::object(v.get_age(), o.zone);
}
};
} // namespace adaptor
} // MSGPACK_API_VERSION_NAMESPACE(MSGPACK_DEFAULT_API_NS)
} // namespace msgpack
int main() {
my_class mc("John", 42);
msgpack::zone z;
auto obj = msgpack::object(mc, z);
std::cout << obj << std::endl;
}
Output:
object_with_zone<my_class> is called
["John", 42]
Running demo: https://wandbox.org/permlink/dNmZX1FpUL3w8D5m
updated
Additional question. Why would i want to use the zone ?
Answer:
A zone is used internally when you unpack MessagePack formatted byte stream. You get msgpack::object_handle. msgpack::object_handle has a zone and a msgpack::object. See https://github.com/msgpack/msgpack-c/wiki/v2_0_cpp_object#what-is-msgpackobject.
The reason of using msgpack::zone is for performance. If msgpack::object is STR, BIN, or EXT, the msgpack::object need to allocate a memory dynamically. The msgpack::object can have a handle of the memory by itself but it is inefficient. The destructor of msgpack::object need to deallocate memory, if the msgpack::object allocate memory. msgpack::object is a composite data structure. That means the destructor cannot be inlined.
One of the goal of msgpack-c is efficient unpacking. So msgpack-c uses msgpack::zone.
It is unpacking story. msgpack::zone is also used when msgpack::object is created from C++ types. I'm not sure when users want to do, is is up to users.
I'm currently working on a small project which requires loading messages from a file. The messages are stored sequentially in the file and files can become huge, so loading the entire file content into memory is unrewarding.
Therefore we decided to implement a FileReader class that is capable of moving to specific elements in the file quickly and load them on request. Commonly used something along the following lines
SpecificMessage m;
FileReader fr;
fr.open("file.bin");
fr.moveTo(120); // Move to Message #120
fr.read(&m); // Try deserializing as SpecificMessage
The FileReader per se works great. Therefore we thought about adding STL compliant iterator support as well: A random access iterator that provides read-only references to specific messages. Used in the following way
for (auto iter = fr.begin<SpecificMessage>(); iter != fr.end<SpecificMessage>(); ++iter) {
// ...
}
Remark: the above assumes that the file only contains messages of type SpecificMessage. We've been using boost::iterator_facade to simplify the implementation.
Now my question boils down to: how to implement the iterator correctly? Since FileReader does not actually hold a sequence of messages internally, but loads them on request.
What we've tried so far:
Storing the message as an iterator member
This approach stores the message in the iterator instance. Which works great for simple use-cases but fails for more complex uses. E.g. std::reverse_iterator has a dereference operation that looks like this
reference operator*() const
{ // return designated value
_RanIt _Tmp = current;
return (*--_Tmp);
}
This breaks our approach as a reference to a message from a temporary iterator is returned.
Making the reference type equal the value type
#DDrmmr in the comments suggested making the reference type equal the value type, so that a copy of the internally stored object is returned. However, I think this is not valid for the reverse iterator which implements the -> operator as
pointer operator->() const {
return (&**this);
}
which derefs itself, calls the *operator which then returns a copy of a temporary and finally returns the address of this temporary.
Storing the message externally
Alternatively I though about storing the message externally:
SpecificMessage m;
auto iter = fr.begin<SpecificMessage>(&m);
// ...
which also seems to be flawed for
auto iter2 = iter + 2
which will have both iter2 and iter point to the same content.
As I hinted in my other answer, you could consider using memory mapped files. In the comment you asked:
As far as memory mapped files is concerned, this seems not what I want to have, as how would you provide an iterator over SpecificMessages for them?
Well, if your SpecificMessage is a POD type, you could just iterate over the raw memory directly. If not, you could have a deserialization helper (as you already have) and use Boost transform_iterator to do the deserialization on demand.
Note that we can make the memory mapped file managed, effectively meaning that you can just use it as a regular heap, and you can store all standard containers. This includes node-based containers (map<>, e.g.), dynamic-size containers (e.g. vector<>) in addition to the fixed-size containers (array<>) - and any combinations of those.
Here's a demo that takes a simple SpecificMessage that contains a string, and (de)derializes it directly into shared memory:
using blob_t = shm::vector<uint8_t>;
using shared_blobs = shm::vector<blob_t>;
The part that interests you would be the consuming part:
bip::managed_mapped_file mmf(bip::open_only, DBASE_FNAME);
shared_blobs* table = mmf.find_or_construct<shared_blobs>("blob_table")(mmf.get_segment_manager());
using It = boost::transform_iterator<LazyLoader<SpecificMessage>, shared_blobs::const_reverse_iterator>;
// for fun, let's reverse the blobs
for (It first(table->rbegin()), last(table->rend()); first < last; first+=13)
std::cout << "blob: '" << first->contents << "'\n";
// any kind of random access is okay, though:
auto random = rand() % table->size();
SpecificMessage msg;
load(table->at(random), msg);
std::cout << "Random blob #" << random << ": '" << msg.contents << "'\n";
So this prints each 13th message, in reverse order, followed by a random blob.
Full Demo
The sample online uses the lines of the sources as "messages".
Live On Coliru
#include <boost/interprocess/file_mapping.hpp>
#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/container/scoped_allocator.hpp>
#include <boost/interprocess/containers/vector.hpp>
#include <iostream>
#include <boost/iterator/transform_iterator.hpp>
#include <boost/range/iterator_range.hpp>
static char const* DBASE_FNAME = "database.map";
namespace bip = boost::interprocess;
namespace shm {
using segment_manager = bip::managed_mapped_file::segment_manager;
template <typename T> using allocator = boost::container::scoped_allocator_adaptor<bip::allocator<T, segment_manager> >;
template <typename T> using vector = bip::vector<T, allocator<T> >;
}
using blob_t = shm::vector<uint8_t>;
using shared_blobs = shm::vector<blob_t>;
struct SpecificMessage {
// for demonstration purposes, just a string; could be anything serialized
std::string contents;
// trivial save/load serialization code:
template <typename Blob>
friend bool save(Blob& blob, SpecificMessage const& msg) {
blob.assign(msg.contents.begin(), msg.contents.end());
return true;
}
template <typename Blob>
friend bool load(Blob const& blob, SpecificMessage& msg) {
msg.contents.assign(blob.begin(), blob.end());
return true;
}
};
template <typename Message> struct LazyLoader {
using type = Message;
Message operator()(blob_t const& blob) const {
Message result;
if (!load(blob, result)) throw std::bad_cast(); // TODO custom excepion
return result;
}
};
///////
// for demo, create some database contents
void create_database_file() {
bip::file_mapping::remove(DBASE_FNAME);
bip::managed_mapped_file mmf(bip::open_or_create, DBASE_FNAME, 1ul<<20); // Even sparse file size is limited on Coliru
shared_blobs* table = mmf.find_or_construct<shared_blobs>("blob_table")(mmf.get_segment_manager());
std::ifstream ifs("main.cpp");
std::string line;
while (std::getline(ifs, line)) {
table->emplace_back();
save(table->back(), SpecificMessage { line });
}
std::cout << "Created blob table consisting of " << table->size() << " blobs\n";
}
///////
void display_random_messages() {
bip::managed_mapped_file mmf(bip::open_only, DBASE_FNAME);
shared_blobs* table = mmf.find_or_construct<shared_blobs>("blob_table")(mmf.get_segment_manager());
using It = boost::transform_iterator<LazyLoader<SpecificMessage>, shared_blobs::const_reverse_iterator>;
// for fun, let's reverse the blobs
for (It first(table->rbegin()), last(table->rend()); first < last; first+=13)
std::cout << "blob: '" << first->contents << "'\n";
// any kind of random access is okay, though:
auto random = rand() % table->size();
SpecificMessage msg;
load(table->at(random), msg);
std::cout << "Random blob #" << random << ": '" << msg.contents << "'\n";
}
int main()
{
#ifndef CONSUMER_ONLY
create_database_file();
#endif
srand(time(NULL));
display_random_messages();
}
You are having issues because your iterator does not conform to the forward iterator requirements. Specifically:
*i must be an lvalue reference to value_type or const value_type ([forward.iterators]/1.3)
*i cannot be a reference to an object stored in the iterator itself, due to the requirement that two iterators are equal if and only if they are bound to the same object ([forward.iterators]/6)
Yes, these requirements are a huge pain in the butt, and yes, that means that things like std::vector<bool>::iterator are not random access iterators even though some standard library implementations incorrectly claim that they are.
EDIT: The following suggested solution is horribly broken, in that dereferencing a temporary iterator returns a reference to an object that may not live until the reference is used. For example, after auto& foo = *(i + 1); the object referenced by foo may have been released. The implementation of reverse_iterator referenced in the OP will cause the same problem.
I'd suggest that you split your design into two classes: FileCache that holds the file resources and a cache of loaded messages, and FileCache::iterator that holds a message number and lazily retrieves it from the FileCache when dereferenced. The implementation could be something as simple as storing a container of weak_ptr<Message> in FileCache and a shared_ptr<Message> in the iterator: Simple demo
I have to admit I may not fully understand the trouble you have with holding the current MESSAGE as a member of Iter. I would associate each iterator with the FileReader it should read from and implement it as a lightweight encapsulation of a read index for FileReader::(read|moveTo). The most important method to overwtite is boost::iterator_facade<...>::advance(...) which modifies the current index and tries to pull a new MESSAGE from the FileReader If this fails it flags the the iterator as invalid and dereferencing will fail.
template<class MESSAGE,int STEP>
class message_iterator;
template<class MESSAGE>
class FileReader {
public:
typedef message_iterator<MESSAGE, 1> const_iterator;
typedef message_iterator<MESSAGE,-1> const_reverse_iterator;
FileReader();
bool open(const std::string & rName);
bool moveTo(int n);
bool read(MESSAGE &m);
// get the total count of messages in the file
// helps us to find end() and rbegin()
int getMessageCount();
const_iterator begin() {
return const_iterator(this,0);
}
const_iterator end() {
return const_iterator(this,getMessageCount());
}
const_reverse_iterator rbegin() {
return const_reverse_iterator(this,getMessageCount()-1);
}
const_reverse_iterator rend() {
return const_reverse_iterator(this,-1);
}
};
// declaration of message_iterator moving over MESSAGE
// STEP is used to specify STEP size and direction (e.g -1 == reverse)
template<class MESSAGE,int STEP=1>
class message_iterator
: public boost::iterator_facade<
message_iterator<MESSAGE>
, const MESSAGE
, boost::random_access_traversal_tag
>
{
typedef boost::iterator_facade<
message_iterator<MESSAGE>
, const MESSAGE
, boost::random_access_traversal_tag
> super;
public:
// constructor associates an iterator with its FileReader and a given position
explicit message_iterator(FileReader<MESSAGE> * p=NULL,int n=0): _filereader(p),_idx(n),_valid(false) {
advance(0);
}
bool equal(const message_iterator & i) const {
return i._filereader == _filereader && i._idx == _idx;
}
void increment() {
advance(+1);
}
void decrement() {
advance(-1);
}
// overwrite with central functionality. Move to a given relative
// postion and check wether the position can be read. If move/read
// fails we flag the iterator as incalid.
void advance(int n) {
_idx += n*STEP;
if(_filereader!=NULL) {
if( _filereader->moveTo( _idx ) && _filereader->read(_m)) {
_valid = true;
return;
}
}
_valid = false;
}
// Return a ref to the currently cached MESSAGE. Throw
// an acception if positioning at this location in advance(...) failes.
typename super::reference dereference() const {
if(!_valid) {
throw std::runtime_error("access to invalid pos");
}
return _m;
}
private:
FileReader<MESSAGE> * _filereader;
int _idx;
bool _valid;
MESSAGE _m;
};
Boost PropertyMap
You could avoid writing the bulk of the code using Boost PropertyMap:
Live On Coliru
#include <boost/property_map/property_map.hpp>
#include <boost/property_map/function_property_map.hpp>
using namespace boost;
struct SpecificMessage {
// add some data
int index; // just for demo
};
template <typename Message>
struct MyLazyReader {
typedef Message type;
std::string fname;
MyLazyReader(std::string fname) : fname(fname) {}
Message operator()(size_t index) const {
Message m;
// FileReader fr;
// fr.open(fname);
// fr.moveTo(index); // Move to Message
// fr.read(&m); // Try deserializing as SpecificMessage
m.index = index; // just for demo
return m;
}
};
#include <iostream>
int main() {
auto lazy_access = make_function_property_map<size_t>(MyLazyReader<SpecificMessage>("file.bin"));
for (int i=0; i<10; ++i)
std::cout << lazy_access[rand()%256].index << "\n";
}
Sample output is
103
198
105
115
81
255
74
236
41
205
Using Memory Mapped Files
You could store a map of index -> BLOB objects in a shared vector<array<byte, N>>, flat_map<size_t, std::vector<uint8_t> > or similar.
So, now you only have to deserialize from myshared_map[index].data() (begin() and end() in case the BLOB size varies)
I need to make a thread-safe map, where I mean that each value must be independently mutexed. For example, I need to be able to get map["abc"] and map["vf"] at the same time from 2 different threads.
My idea is to make two maps: one for data and one for mutex for every key:
class cache
{
private:
....
std::map<std::string, std::string> mainCache;
std::map<std::string, std::unique_ptr<std::mutex> > mutexCache;
std::mutex gMutex;
.....
public:
std::string get(std::string key);
};
std::string cache::get(std::string key){
std::mutex *m;
gMutex.lock();
if (mutexCache.count(key) == 0){
mutexCache.insert(new std::unique_ptr<std::mutex>);
}
m = mutexCache[key];
gMutex.unlock();
}
I find that I can't create map from string to mutex, because there is no copy constructor in std::mutex and I must use std::unique_ptr; but when I compile this I get:
/home/user/test/cache.cpp:7: error: no matching function for call to 'std::map<std::basic_string<char>, std::unique_ptr<std::mutex> >::insert(std::unique_ptr<std::mutex>*)'
mutexCache.insert(new std::unique_ptr<std::mutex>);
^
How do I solve this problem?
TL;DR: just use operator [] like std::map<std::string, std::mutex> map; map[filename];
Why do you need to use an std::unique_ptr in the first place?
I had the same problem when I had to create an std::map of std::mutex objects. The issue is that std::mutex is neither copyable nor movable, so I needed to construct it "in place".
I couldn't just use emplace because it doesn't work directly for default-constructed values. There is an option to use std::piecewise_construct like that:
map.emplace(std::piecewise_construct, std::make_tuple(key), std::make_tuple());
but it's IMO complicated and less readable.
My solution is much simpler - just use the operator[] - it will create the value using its default constructor and return a reference to it. Or it will just find and return a reference to the already existing item without creating a new one.
std::map<std::string, std::mutex> map;
std::mutex& GetMutexForFile(const std::string& filename)
{
return map[filename]; // constructs it inside the map if doesn't exist
}
Replace mutexCache.insert(new std::unique_ptr<std::mutex>) with:
mutexCache.emplace(key, new std::mutex);
In C++14, you should say:
mutexCache.emplace(key, std::make_unique<std::mutex>());
The overall code is very noisy and inelegant, though. It should probably look like this:
std::string cache::get(std::string key)
{
std::mutex * inner_mutex;
{
std::lock_guard<std::mutex> g_lk(gMutex);
auto it = mutexCache.find(key);
if (it == mutexCache.end())
{
it = mutexCache.emplace(key, std::make_unique<std::mutex>()).first;
}
inner_mutex = it->second.get();
}
{
std::lock_guard<std::mutex> c_lk(*inner_mutex);
return mainCache[key];
}
}
If you have access to c++17, you can use std::map::try_emplace instead of using pointers and it should work just fine for non-copyable and non-movable types!
Your mutexes actually don't protect values. They are released before returning from get, and then other thread can get referrence to the same string second time. Oh, but your cache returns copies of strings, not references. So, there is no point in protecting each string with own mutex.
If you want to protect cache class from concurrent access only gMutex is sufficient. Code should be
class cache
{
private:
std::map<std::string, std::string> mainCache;
std::mutex gMutex;
public:
std::string get(const std::string & key);
void set(const std::string & key, const std::string & value);
};
std::string cache::get(const std::string & key) {
std::lock_guard<std::mutex> g_lk(gMutex);
return mainCache[key];
}
void cache::set(const std::string & key, const std::string & value) {
std::lock_guard<std::mutex> g_lk(gMutex);
mainCache[key] = value;
}
If you want to provide a way for many threads to work concurrently with string instances inside your map and protect them from concurrent access things become more tricky. First, you need to know when thread finished to work with string and release the lock. Otherwise once accessed value becomes locked forever and no other thread can access it.
As possible solution you can use some class like
#include <iostream>
#include <string>
#include <map>
#include <mutex>
#include <memory>
template<class T>
class SharedObject {
private:
T obj;
std::mutex m;
public:
SharedObject() = default;
SharedObject(const T & object): obj(object) {}
SharedObject(T && object): obj(std::move(object)) {}
template<class F>
void access(F && f) {
std::lock_guard<std::mutex> lock(m);
f(obj);
}
};
class ThreadSafeCache
{
private:
std::map<std::string, std::shared_ptr<SharedObject<std::string>>> mainCache;
std::mutex gMutex;
public:
std::shared_ptr<SharedObject<std::string>> & get(const std::string & key) {
std::lock_guard<std::mutex> g_lk(gMutex);
return mainCache[key];
}
void set(const std::string & key, const std::string & value) {
std::shared_ptr<SharedObject<std::string>> obj;
bool alreadyAssigned = false;
{
std::lock_guard<std::mutex> g_lk(gMutex);
auto it = mainCache.find(key);
if (it != mainCache.end()) {
obj = (*it).second;
}
else {
obj = mainCache.emplace(key, std::make_shared<SharedObject<std::string>>(value)).first->second;
alreadyAssigned = true;
}
}
// we can't be sure someone not doing some long transaction with this object,
// so we can't do access under gMutex, because it locks all accesses to all other elements of cache
if (!alreadyAssigned) obj->access([&value] (std::string& s) { s = value; });
}
};
// in some thread
void foo(ThreadSafeCache & c) {
auto & sharedString = c.get("abc");
sharedString->access([&] (std::string& s) {
// code that use string goes here
std::cout << s;
// c.get("abc")->access([](auto & s) { std::cout << s; }); // deadlock
});
}
int main()
{
ThreadSafeCache c;
c.set("abc", "val");
foo(c);
return 0;
}
Of course, real implementation of these classes should have more methods providing more complex semantic, take const-ness into account and so on. But I hope main idea is clear.
EDIT:
Note: shared_ptr to SharedObject should be used because you can't delete mutex while lock is held, so there is no way of safe deleting map entries if value type is SharedObject itself.
Is there any practical way to get objects to work with maps? (I don't really know much about maps so sorry if this is a bad question). I'm interested in using a map for an inventory system that will contain item objects. An item object has a name, description, and money value. The key would be the item's name, and the linking variable would be the the quantity of items I have for that particular item.
And if a map doesn't work, does anyone have a good alternative to this type of system? I need something keeping track of the quantity of each type of item I have.
The C++ standard library template map is just a storage container so it can definitely be used with objects. The map will take your object as its templated argument parameter.
A map would work well for your inventory system. Use something like:
#include <pair>
#include <map>
#include <string>
#include <iostream>
class Item {
public:
Item(void) {}
~Item(void) {}
Item(std::string new_name) {
my_name=new_name;
}
void setName(std::string new_name) {
my_name= new_name;
}
std::string getName(void) {
return my_name;
}
private:
std::string my_name;
};
class Item_Manager {
public:
Item_Manager(void) {}
~Item_Manager(void) {}
void addItem(Item * my_item, int num_items) {
my_item_counts.insert( std::pair<std::string,int>(Item.getName(),num_items) );
}
int getNumItems(std::string my_item_name) {
return my_item_counters[my_item_name];
}
private:
std::map<std::string, int> my_item_counts;
};
main () {
Item * test_item = new Item("chips");
Item * test_item2 = new Item("gum");
Item_Manager * my_manager = new Item_Manager();
my_manager->addItem(test_item, 5);
my_manager->addItem(test_item2,10);
std::cout << "I have " << my_manager->getNumItems(test_item->getName())
<< " " << test_item->getName() << " and "
<< my_manager->getNumItems(test_item2->getName())
<< " " << test_item2->getName() << std::endl;
delete test_item;
delete test_item2;
delete my_manager;
}
Here's a reference on the stdlib map and its functions:
http://www.cplusplus.com/reference/stl/map/
Look at the function pages for examples of how to iterate through/index a map, etc.
If you're talking about std::map, it's a template which can work with any type of object, as long as a way to compare objects for ordering is provided. Since your key (the name) is a string, it will work right out of the box with std::map
struct Item
{
std::string description;
int value;
};
int main()
{
// associate item names with an item/quantity pair
std::map<std::string, std::pair<Item, int> > itemmap;
}
I need something keeping track of the quantity of each type of item I have.
How about std::vector<std::pair<Item, int> >?