i need to place a STL map in shared memory. Also have multiple process access that map. Any pointers to how it is done ?
I have checked this link. But need a more simpler method.
Map in Shared memory
For this to work you need to use a custom allocator that will allocate from the shared memory region, so that the map nodes are all in shared memory, and so that the pointer type of the allocator is not just a raw pointer but can refer to the shared memory region when it is mapped to different addresses in different processes.
You also need your std::map implementation to correctly use the allocator's pointer type everywhere it needs to use a pointer, and this isn't guaranteed by the standard.
The simplest way to do this currently is to use Boost.Interprocess which provides a nice API for shared memory and also provides allocators and containers that work correctly with it.
Related
I want to create a custom allocator for a multimap that will allocate the elements in shared memory.I came across boost.interprocess but found it quite complicated to implement.Is there any other workaround ?
I will not give here any implementation, rather to give you some directions.
If your shared memory abstraction or region, for example start at adress void* shMemAddr and if you decide that your stl container to use shared memory,
what needs to be done is to make container allocate memory starting at shMemAddr and further, until there is available memory to allocate in your shared pool. You can implement that using any allocation strategy, for example using malloc or placement new. Further, to be available for your container to use your allocator you need to provide your allocator as template argument, for multimap it would be multimap::allocator_type
class Alloc = allocator > as fourth template argument, after less as compare function,and, for example, if you store in your multimap pairs of int,double as key,value pairs, it would likely be something like this
multimap<int,double,less<int>,CustomAlloc<pair<int,double>>>
Now, your CustomAlloc allocator need to satisfies concept of Allocators which encapsulate specific lowlevel memory management, especially, if shared memory is resource to be allocate in, you need to arrange proper allocation of memory in a multithreaded enviornment. That means that, first, you need some structure for evidence of used memory. It can be some chained data structure, for example, and implementations like that is pretty common, so you need to keep invariants of that structure consistent. What that means is if your structure for book keeping of used(or free) memory need to be updated after succesfull allocation or deallocation it needs to be done atomicaly, so thread which possibly try to allocate memory see only structure in states before CustomAllocator allocation job is started or after allocation job is finished. For example, your first choice to do that could be using mutex to protect data, avoid races and keep invariants. This is just directions, and considering write your own allocators is not very hard, I hope this will help as good starting point.
In the Boost.Interprocess documentation Where is this being allocated? it is stated that Boost.Interprocess containers are placed in shared memory using two mechanisms at the same time:
Boost.Interprocess construct<>, find_or_construct<>... functions. These functions place a C++ object in the shared
memory. But this places only the object, not the memory that this
object may allocate dynamically.
Shared memory allocators. These allow allocating shared memory portions so that containers can allocate dynamically fragments
of memory to store newly inserted elements.
What is the use case to have a boost.vector where internal memory lives in the current process, but using a shared memory allocator so that elements are placed in shared memory ?
If I want to share this structure to another process :
struct Shared
{
vector<string> m_names;
vector<char> m_data;
};
I guess I want the vectors to be accessible to the other process so that it can iterate on them, right ?
find_or_construct and friends are for your own direct allocations.
The allocators are to be passed to library types to do their internal allocations in similar fashion. Otherwise, only the "control structure" (e.g. 16 bytes for a typical std::string) would be in the shared memory, instead of all the related data allocated by the standard library container internally.
Well, you cannot access the vector as such from the other process but you can access the elements (so in your example the strings) e.g. via a pointer
I am working in a project which have two different processes.
The first process is a cache base on a std::map or std::set which allocate all the data in a share memory region.
The second process is a producer/consumer which will have access to the share memory, so whenever it needs some data it will ask through an unix pipe to the cache process the starting address of the shared memory which contain the requested data.
So far, I came up with two approaches, first is changing the allocation function for std::set to always allocate in the shared memory, or maybe in a easier approach storing as the value of the map a pointer to that shared region:
map<key, pointer to share region>
Any idea? :D
Thanks!!
In theory, you can use a custom allocator for std::set or std::map to do this. Of course, you'll have to ensure that any contents that might dynamically allocate also use the same custom allocator.
The real problem is that the mapped addresses of the shared memory might not be the same. It's often possible to work around this by using mmap and specifying the address, but the address range must be free in both processes. I've done this under Solaris, which always allocates (or allocated) static and heap at the bottom of the address space, and stack at the top, leaving a big hole in the middle, but even there, I don't think there was any guarantee, and other systems have different policies. Still, if the processes aren't too big otherwise, you may be able to find a solution empirically. (I'd recommend making the address and the size a configuration parameter.)
Alternatively, in theory, the allocator defines a pointer type, which the container should use; you should be able to define a pointer type which works with just the offset into the shared memory. I've no experience with this, however, and I fear that it could be very tricky, since the reference type will still be a true reference (and thus a pointer under the hood), and you cannot change this.
I'm trying to use an mmap-like segment to allocate objects on stl containers, for that I'm using boost::interprocess which provides with memory mappings, allocators and anonymous memory mapping support.
A bit like this
My problem is that the anonymous_shared_memory function here returns something that looks half mapped file and half shared memory(makes sense with mmap :) ) and although both styles work with interprocess allocators this one looks like its missing a segment_manager which does the actual chunk allocation.
As it returns a high-level mapped_region already mapped in the process but with no manager and no way that I can see to hook in a segment_manager.
A mapped_region is a low to mid-level object, and literally represents just the memory. Managed shared memory, however
is an advanced class that combines a shared memory object and a mapped region that covers all the shared memory object,
so it is the managed memory that possess the segment_manager.
Given that you want to use anonymous_shared_memory, first you'd get the memory_region as per the example, then you would use placement new to put a segment_manager at the beginning of it. Its constructor takes the size of the memory segment that it is being constructed in. I do not know if this includes the size of the manager, although I suspect it is included.
GIVEN that you have a fixed area of memory already allocated that you would like to use, what C or C++ libraries will allow you to store a dynamic structure (e.g. a hash) in that memory?
i.e. the hash library must not contain any calls to malloc or new, but must take a parameter that tells it the location and size of the memory it is permitted to use.
(bonus if the library uses offsets rather than pointers internally, in case the shared memory is mapped to different address spaces in each process that uses it)
You can write your own custom allocators for STL containers.
Dr.Dobb's: What Are Allocators Good For?
SO: Compelling examples of custom C++ STL allocators?
It's trivial to adapt a simple linear probing hash table to use a block of memory - just set its table(s) to point at the allocated memory when you create it, and don't implement anything to allocate more memory to let the table grow.