I hope any expert using the boost managed shared memory can help me. I´m trying to write the memory to a file. I can not figure it out with the boost examples. ¿Can anyone provide me some examples?
Thanks.
If you truly need this, I see roughly 2 approaches, at a glance:
Copy it
use serialization/deserialization
or just copy by constructing clones in a different segment manager (obviously linked to a memory mapped file, this time)
Use a Managed External Buffer. A managed buffer is basically your segment manager on top of some transparent memory buffer. You can decide whether it exists in local process address space, shared memory, or indeed a memory-mapped file.
This is the supported method to use the same segment manager + segment data layout in both.
If you're really desperate you COULD try to bitwise copy the complete shared memory object into a file of equal size and simply open it. This might work IFF the managed_mapped_file implementation has exactly the same or compatible segement management structure, headers and layout. That's a long call though, and even if it appears to work, it's at best undocumented, and therefore likely invokes undefined behaviour.
Perhaps you are looking for mapped_file: http://www.boost.org/doc/libs/1_63_0/libs/iostreams/doc/classes/mapped_file.html
It is a memory-mapping API for files, and you can open the same file in multiple processes.
Related
I'm using shared memory based on POSIX API on a linux machine to communicate between multiple MPI process. I have a working solution but I want to know how to make efficient use of the shared memory space for large data.
I have a machine with 64GB shared memory limit and it might happen that I have to write data > 64GB into this space, but these are smaller chunks of 1-2GB.
What I want to know is this:
How can I really delete the memory occupied by my 1-2GB chunk as soon as it's purpose is served and I don't want that data anymore?
I'm using shm_unlink() but it doesn't seem to clear space in /dev/shm/
Please help!
From the sum_unlink description in http://pubs.opengroup.org/onlinepubs/009695399/functions/shm_unlink.html
it seems that the memory removal may be actually postponed, read below:
If one or more references to the shared memory object exist when the object is unlinked, the name shall be removed before shm_unlink() returns, but the removal of the memory object contents shall be postponed until all open and map references to the shared memory object have been removed.
I hope this help.
I am trying to copy file from one location to other location using memcpy.
And I am getting a crash every time when the file path which I provide as source is non existing.
Is this a correct behaviour, if yes then how could I prevent the crash.
memcpy's purpose is not to copy files, but chunks of memory from one place to another. Depending on platform, I'm sure there are better ways to copy files between directories.
No, no, no! What in the name of holy memory are you doing? memcpy is strictly used for memory transfer. For example, say you have a bunch of vertices in a graphics application that you want to give to the vertex buffer without actually passing a pointer. That way, you copy the entire array of bytes that represent your vertices and from thereafter - they are a problem of the underlying API. That's the intended usage. Also, be careful how you use it. You specify the amount of bytes that are to be copied. Make sure it is exact. At the lowest levels, there is no such thing as types, if you give it too much, it'll wreak havoc - tear into bytes that "belong" to other variables. If you're lucky, such code will kill your application. If you're not - it'll continue working.
For file reading/writing/copying, you have to consult standard C/C++ file input/output (or some more refined methods courtesy of the underlying platform, as one of the answers suggested). There's a lot of resources online, so I won't reinvent the wheel here. Hope it gave you a better idea of what memcpy is for.
I'm looking for MemoryMappedViewAccessor С++ analog. It exists?
I want to send data from C++ app to .net app using shared memory. And I need random access. Is it possible?
Thank you!
There is no equivalent in C++. Mostly because none is needed, you can simply access the shared memory with a pointer. MemoryMappedViewAccessor is required to serialize garbage collected objects to the view.
Which makes shared memory pretty inefficient from managed code. Do make sure that a pipe or a socket doesn't solve your problem first. They usually do, dealing with concurrency is also much easier with them. Shared memory requires a raft of named mutexes to arbitrate access.
I am looking for a way to store several gb's of data in memory. The data is loaded into a tree structure. I want to be able to access this data through my main function, but I'm not interested in reloading the data into the tree every time I run the program. What is the best way to do this? Should I create a separate program for loading the data and then call it from the main function, or are there better alternatives?
thanks
Mads
I'd say the best alternative would be using a database - which would be then your "separate program for loading the data".
If you are using a POSIX compliant system, then take a look into mmap.
I think Windows has another function to memory map a file.
You could probably solve this using shared memory, to have one process that it long-lived build the tree and expose the address for it, and then other processes that start up can get hold of that same memory for querying. Note that you will need to make sure the tree is up to being read by multiple simultaneous processes, in that case. If the reads are really just pure reads, then that should be easy enough.
You should look into a technique called a Memory mapped file.
I think the best solution is to configure a cache server and put data there.
Look into Ehcache:
Ehcache is an open source, standards-based cache used to boost
performance, offload the database and simplify scalability. Ehcache is
robust, proven and full-featured and this has made it the most
widely-used Java-based cache.
It's written in Java, but should support any language you choose:
The Cache Server has two apis: RESTful resource oriented, and SOAP.
Both support clients in any programming language.
You must be running a 64 bit system to use more than 4 GB's of memory. If you build the tree and set it as a global variable, you can access the tree and data from any function in the program. I suggest you perhaps try an alternative method that requires less memory consumption. If you post what type of program, and what type of tree you're doing, I can perhaps give you some help in finding an alternative method.
Since you don't want to keep reloading the data...file storage and databases are out of question, but several gigs of memory seem like such a hefty price.
Also note that on Windows systems, you can access the memory of another program using ReadProcessMemory(), all you need is a pointer to use for the location of the memory.
You may alternatively implement the data loader as an executable program and the main program as a dll loaded and unloaded on demand. That way you can keep the data in the memory and be able to modify the processing code w/o reloading all the data or doing cross-process memory sharing.
Also, if you can operate on the raw data from the disk w/o making any preprocessing of it (e.g. putting it in a tree, manipulating pointers to its internals), you may want to memory-map the data and avoid loading unused portions of it.
Our app depends on an external, 3rd party-supplied configuration (including custom driving/decision making functions) loadable as .so file.
Independently, it cooperates with external CGI modules using a chunk of shared memory, where almost all of its volatile state is kept, so that the external modules can read it and modify it where applicable.
The problem is the CGI modules require a lot of the permanent config data from the .so as well, and the main app performs a whole lot of entirely unnecessary copying between the two memory areas to make the data available. The idea is to make the whole Shared Object to load into Shared Memory, and make it directly available to the CGI. The problem is: how?
dlopen and dlsym don't provide any facilities for assigning where to load the SO file.
we tried shmat(). It seems to work only until some external CGI actually tries to access the shared memory. Then the area pointed to appears just as private as if it was never shared. Maybe we're doing something wrong?
loading the .so in each script that needs it is out of question. The sheer size of the structure, connected with frequency of calls (some of the scripts are called once a second to generate live updates), and this being an embedded app make it no-go.
simply memcpy()'ing the .so into shm is not good either - some structures and all functions are interconnected through pointers.
The first thing to bear in mind when using shared memory is that the same physical memory may well be mapped into the two processes virtual address space as different addresses. This means that if pointers are used anywhere in your data structures, they are going to cause problems. Everything must work off an index or an offset to work correctly. To use shared memory, you will have to purge all the pointers from your code.
When loading a .so file, only one copy of the .so file code is loaded (hence the term shared object).
fork may also be your friend here. Most modern operating systems implement copy-on-write semantics. This means that when you fork, your data segments are only copied into separate physical memory when one process writes to the given data segment.
I suppose the easiest option would be to use memory mapped file, what Neil has proposed already. If this option does not fill well, alternative is to could be to define dedicated allocator. Here is a good paper about it: Creating STL Containers in Shared Memory
There is also excellent Ion Gaztañaga's Boost.Interprocess library with shared_memory_object and related features. Ion has proposed the solution to the C++ standardization committee for future TR: Memory Mapped Files And Shared Memory For C++
what may indicate it's worth solution to consider.
Placing actual C++ objects in shared memory is very, very difficult, as you have found. I would strongly recommend you don't go that way - putting data that needs to be shared in shared memory or a memory mapped file is much simpler and likely to be much more robust.
You need to implement object's Serialization
Serialization function will convert your object into bytes, then you can write bytes in SharedMemory and have your CGI module to deserialize bytes back to object.