In the Boost.Interprocess documentation Where is this being allocated? it is stated that Boost.Interprocess containers are placed in shared memory using two mechanisms at the same time:
Boost.Interprocess construct<>, find_or_construct<>... functions. These functions place a C++ object in the shared
memory. But this places only the object, not the memory that this
object may allocate dynamically.
Shared memory allocators. These allow allocating shared memory portions so that containers can allocate dynamically fragments
of memory to store newly inserted elements.
What is the use case to have a boost.vector where internal memory lives in the current process, but using a shared memory allocator so that elements are placed in shared memory ?
If I want to share this structure to another process :
struct Shared
{
vector<string> m_names;
vector<char> m_data;
};
I guess I want the vectors to be accessible to the other process so that it can iterate on them, right ?
find_or_construct and friends are for your own direct allocations.
The allocators are to be passed to library types to do their internal allocations in similar fashion. Otherwise, only the "control structure" (e.g. 16 bytes for a typical std::string) would be in the shared memory, instead of all the related data allocated by the standard library container internally.
Well, you cannot access the vector as such from the other process but you can access the elements (so in your example the strings) e.g. via a pointer
Related
i need to place a STL map in shared memory. Also have multiple process access that map. Any pointers to how it is done ?
I have checked this link. But need a more simpler method.
Map in Shared memory
For this to work you need to use a custom allocator that will allocate from the shared memory region, so that the map nodes are all in shared memory, and so that the pointer type of the allocator is not just a raw pointer but can refer to the shared memory region when it is mapped to different addresses in different processes.
You also need your std::map implementation to correctly use the allocator's pointer type everywhere it needs to use a pointer, and this isn't guaranteed by the standard.
The simplest way to do this currently is to use Boost.Interprocess which provides a nice API for shared memory and also provides allocators and containers that work correctly with it.
I have a process that internally makes "std::system" call. (I believe system call spawns a child process). In the system call, I am executing different application as
void generate()
{
std::system("./childProcess.exe);
}
The above "generate function" would be called by multiple threads.
Now, I need to share a "complex object" from Parent process to child process, i.e., childProcess.exe.
I tried with Boost::interprocess::shared_memory but in vain.
The complex object i mentioned earlier internally dynamically allocates memory many a times. I believe those dynamically allocated memories are not associated with my shared memory segment. Please correct me if otherwise
My class is like this
Complex compClass
{
int cnt;
subclass *subobj;
}
compClass::compClass()
{
subobj=new subclass;
subobj->func;
}
subclass::func()
{
YClass *y = new YClass;
}
and so on, it internally has many such memory allocations.
When I create a shared memory segment of object type "Comp class" in parent process, and open the shared memory segment in child process, i am able to access "cnt" variable in child process. But, I am not able to access subobject in child process.
I believe this is because subobject is dynamically allocated and we have different dynamic memory allocated in child processes and they are not associated with shared memory segment.
I found even for std::string, boost comes up with boost::interprocess::string as string internally makes "new" call.
Please suggest the best IPC mechanism to share this "Complex object" between multiple processes.
If you want to use that class member arrangement in shared memory, you'll have to replace the calls to new with ones that retrieve memory from a pool also in shared memory: you'll need to find or write a suitable pool allocator for that - the Standard doesn't provide one, but boost provides something pretty close using offsets instead of pointers (which is more flexible because insisting on loading shared memory at specific absolute virtual addresses isn't particularly reliable or scalable) - see here
Alternatives are to use threads, to serialise the data to shared memory and deserialise it (e.g. to a stringstream then copying the .str() data to shared memory), and to store the subclass object directly in Complex instead of having a pointer.
I am working in a project which have two different processes.
The first process is a cache base on a std::map or std::set which allocate all the data in a share memory region.
The second process is a producer/consumer which will have access to the share memory, so whenever it needs some data it will ask through an unix pipe to the cache process the starting address of the shared memory which contain the requested data.
So far, I came up with two approaches, first is changing the allocation function for std::set to always allocate in the shared memory, or maybe in a easier approach storing as the value of the map a pointer to that shared region:
map<key, pointer to share region>
Any idea? :D
Thanks!!
In theory, you can use a custom allocator for std::set or std::map to do this. Of course, you'll have to ensure that any contents that might dynamically allocate also use the same custom allocator.
The real problem is that the mapped addresses of the shared memory might not be the same. It's often possible to work around this by using mmap and specifying the address, but the address range must be free in both processes. I've done this under Solaris, which always allocates (or allocated) static and heap at the bottom of the address space, and stack at the top, leaving a big hole in the middle, but even there, I don't think there was any guarantee, and other systems have different policies. Still, if the processes aren't too big otherwise, you may be able to find a solution empirically. (I'd recommend making the address and the size a configuration parameter.)
Alternatively, in theory, the allocator defines a pointer type, which the container should use; you should be able to define a pointer type which works with just the offset into the shared memory. I've no experience with this, however, and I fear that it could be very tricky, since the reference type will still be a true reference (and thus a pointer under the hood), and you cannot change this.
I'm trying to use an mmap-like segment to allocate objects on stl containers, for that I'm using boost::interprocess which provides with memory mappings, allocators and anonymous memory mapping support.
A bit like this
My problem is that the anonymous_shared_memory function here returns something that looks half mapped file and half shared memory(makes sense with mmap :) ) and although both styles work with interprocess allocators this one looks like its missing a segment_manager which does the actual chunk allocation.
As it returns a high-level mapped_region already mapped in the process but with no manager and no way that I can see to hook in a segment_manager.
A mapped_region is a low to mid-level object, and literally represents just the memory. Managed shared memory, however
is an advanced class that combines a shared memory object and a mapped region that covers all the shared memory object,
so it is the managed memory that possess the segment_manager.
Given that you want to use anonymous_shared_memory, first you'd get the memory_region as per the example, then you would use placement new to put a segment_manager at the beginning of it. Its constructor takes the size of the memory segment that it is being constructed in. I do not know if this includes the size of the manager, although I suspect it is included.
GIVEN that you have a fixed area of memory already allocated that you would like to use, what C or C++ libraries will allow you to store a dynamic structure (e.g. a hash) in that memory?
i.e. the hash library must not contain any calls to malloc or new, but must take a parameter that tells it the location and size of the memory it is permitted to use.
(bonus if the library uses offsets rather than pointers internally, in case the shared memory is mapped to different address spaces in each process that uses it)
You can write your own custom allocators for STL containers.
Dr.Dobb's: What Are Allocators Good For?
SO: Compelling examples of custom C++ STL allocators?
It's trivial to adapt a simple linear probing hash table to use a block of memory - just set its table(s) to point at the allocated memory when you create it, and don't implement anything to allocate more memory to let the table grow.