I would like to pass a pointer (I am putting a file with data in memory with mmap) to processes spawned using fork + exec, but I am stuck on how to pass a pointer to the exec() spawned process?
UPDATE1:
Thanks for your inputs, I do use shared memory creating it with mmap with MAP_INHERIT flag:
Each mapped file and shared memory region created with the mmap() function
is unmapped by a successful call to any of the exec functions, except those
regions mapped with the MAP_INHERIT option. Regions mapped with the
MAP_INHERIT option remain mapped in the new process image.
source: http://www.uwm.edu/cgi-bin/IMT/wwwman?topic=exec(2)&msection=
UPDATE2:
This is homework excercise, but I think I must stop thinking about pointers and think about the IPC itself. I guess I will go with trying to mmap the same file in child process.
Short code example much appreciated.
Thanks in advance for your help.
If you use shared memory, you can't pass the pointer. The pointer will contain the virtual address, which is different from one process to another. You have to exchange offset values, based on the start of the shared memory area.
If you don't use shared memory, you can't exchange pointers of any kind: The other process won't be able to access the memory of your process.
This can't work. The new process should mmap the file itself as well.
The spawned process should probably open a pipe back to the parent process and ask for the data it needs to map the shared memory segment.
Alternatively you can use boost::interprocess to create a shared memory segment for you and actually pass around the address (it can do the mapping). You're on your own reading that documentation though: http://www.boost.org/doc/libs/1_38_0/doc/html/interprocess.html
Consider passing the offset to the memory within the file to the child process. If the offset is zero, then don't bother, but if you need to pass a 'pointer' to part way through the file, then convert that to an offset from the start address, and pass that to the child. The child can then get to the data by adding the offset to the address it obtains for the mapped file.
Just pass as text in command-line argument, or in environment variable.
This is a big area, and you have a lot to choose from.
The key finding those solution is to search for something like Linux inter processor communication or maybe Linux IPC.
A intro into IPC can also be found in books like, Advance Linux Programming (ISBN: 0-7357-1043-0)
Related
I'm trying to understand something about HGLOBALs, because I just found out that what I thought is simply wrong.
In app A I GlobalAlloc() data (with GMEM_SHARE|GMEM_MOVABLE) and place the string "Test" in it. Now, what can I give to another application to get to that data?
I though (wrongfully!) that HGLOBALs are valid in all the processes, which is obviously wrong, because HGLOBAL is a HANDLE to the global data, and not a pointer to the global data (that's where I said "OHHHH!").
So how can I pass the HGLOBAL to another application?
Notice: I want to pass just a "pointer" to the data, not the data itself, like in the clipboard.
Thanks a lot! :-)
(This is just a very long comment as others have already explained that Win32 takes different approach to memory sharing.)
I would say that you are reading into books (or tutorials) on Windows programming which are quite old and obsolete as Win16 is virtually dead for quite some time.
16-bit Windows (3.x) didn't have the concept of memory isolation (or virtual /flat/ address space) that 32-bit (and later) Windows versions provide. Memory there used to be divided into local (to the process) and global sections, both living in the same global address space. Descriptors like HGLOBAL were used to allow memory blocks to be moved around in physical memory and still accessed correctly despite their new location in the address space (after proper fixation with LocalLock()/GlobalLock()). Win32 uses pointers instead since physical memory pages can be moved without affecting their location in the virtual address space. It still provides all of the Global* and Local* API functions for compatibility reasons but they should not be used anymore and usual heap management should be used instead (e.g. malloc() in C or the new operator in C++). Also several different kind of pointers existed on Win16 in order to reflect on the several different addressing modes available on x86 - near (same segment), far (segment:offset) and huge (normalised segment:offset). You can still see things like FARPTR in legacy Win16 code that got ported to Win32 but they are defined to be empty strings as in flat mode only near pointers are used.
Read the documentation. With the introduction of 32-bit processing, GlobalAlloc() does not actually allocate global memory anymore.
To share a memory block with another process, you could allocate the block with GlobalAlloc() and put it on the clipboard, then have the other process retreive it. Or you can allocate a block of shared memory using CreateFileMapping() and MapViewOfFile() instead.
Each process "thinks" that it owns the full memory space available on the computer. No process can "see" the memory space of another process. As such, normally, nothing a process stores can be seen by another process.
Because it can be necessary to pass information between processess, certain mechanisms exists to provide this functionality.
One approach is message passing; one process issues a message to another, for example over a pipe, or a socket, or by a Windows message.
Another is shared memory, where a given block of memory is made available to two or more processes, such that whatever one process writes can be seen by the others.
Don't be confused with GMEM_SHARE flag. It does not work the way you possibly supposed. From MSDN:
The following values are obsolete, but are provided for compatibility
with 16-bit Windows. They are ignored.
GMEM_SHARE
GMEM_SHARE flag explained by Raymond Chen:
In 16-bit Windows, the GMEM_SHARE flag controlled whether the memory
should outlive the process that allocated it.
To share memory with another process/application you instead should take a look at File Mappings: Memory-mapped files and how they work.
I am working on implementing a database server in C that will handle requests from multiple clients. I am using fork() to handle connections for individual clients.
The server stores data in the heap, which consists of a root pointer to hash tables of dynamically allocated records. The records are structs that have pointers to various data-types. I would like for the processes to be able to share this data so that, when a client makes a change to the heap, the changes will be visible for the other clients.
I have learned that fork() uses COW (Copy On Write), and my understanding is that it copies the heap (and stack) memory of the parent process when the child tries to modify the data in memory.
I have found out that I can use the shm library to share memory.
Would the code below be a valid way to share heap memory (in shared_string)? If a child were to use similar code (i.e. starting from //start), would other children be able to read/write to it while the child is running and after it's dead?
key_t key;
int shmid;
key = ftok("/tmp",'R');
shmid = shmget(key, 1024, 0644 | IPC_CREAT);
//start
char * string;
string = malloc(sizeof(char) * 10);
strcpy(string, "a string");
char * shared_string;
shared_string = shmat(shmid, string, 0);
strcpy(shared_string, string);
Here are some of my thoughts/concerns regarding this:
I'm thinking about sharing the root pointer of the database. I'm not sure if that would work or if I have to mark all allocated memory as shared.
I'm not sure if the parent / other children are able to access memory allocated by a child.
I'm not sure if a child's allocated memory stays on the heap after it is killed, or if that memory is released.
First of all, fork is completely inappropriate for what you're trying to achieve. Even if you can make it work, it's a horrible hack. In general, fork only works for very simplistic programs anyway, and I would go so far as to say that fork should never be used except followed quickly by exec, but that's aside from the point here. You really should be using threads.
With that said, the only way to have memory that's shared between the parent and child after fork, and where the same pointers are valid in both, is to mmap (or shmat, but that's a lot fuglier) a file or anonymous map with MAP_SHARED prior to the fork. You cannot create new shared memory like this after fork because there's no guarantee that it will get mapped at the same address range in both.
Just don't use fork. It's not the right tool for the job.
I think you are basically looking to do what is done by Redis (and probably others).
They describe it in http://redis.io/topics/persistence (search for "copy-on-write").
threads defeat the purpose
classic shared memory (shm, mapped memory) also defeats the purpose
The primary benefit to using this method is avoidance of locking, which can be a pain to get right.
As far as I understand it the idea of using COW is to:
fork when you want to write, not in advance
the child (re)writes the data to disk, then immediately exits
the parent keeps on doing its work, and detects (SIGCHLD) when the child exited.
If while doing its work the parent ends up making changes to the hash, the kernel
will execute a copy for the affected blocks (right terminology?).
A "dirty flag" is used to track if a new fork is needed to execute a new write.
Things to watch out for:
Make sure only one outstanding child
Transactional safety: write to a temp file first, then move it over so that you always have a complete copy, maybe keeping the previous around if the move is not atomic.
test if you will have issues with other resources that get duplicated (file descriptors, global destructors in c++)
You may want to take gander at the redis code as well
I'm thinking about sharing the root pointer of the database. I'm not sure if that would work or if I have to mark all allocated memory as shared.
Each process will have its own private memory range. Copy-on-write is a kernel-space optimization that is transparent to user space.
As others have said, SHM or mmap'd files are the only way to share memory between separate processes.
If you must you fork, the shared memory seems to be the 'only' choice.
Actually, I think in your scene, the thread is more suitable.
If you don't want to be multi-threaded. Here is another choice,you can only use one-process & one-thread mode, like redis
With this mode,you don't need worry about something like lock and if you want to scale, just design a route policy,as route with the hash value of the key
As you have discovered, if you want to share memory between separate processes (from fork or otherwise), you need to use shared memory, either the SYSV shm library or mmap with MAP_SHARED. Unfortunately, these are coarse-grained tools, suitable only for dealing with a small number of large blocks, and not suitable for fine-grained memory management as you would do with malloc/free.
In order to have useful shared memory between processes, you need to build a heap on top of shm or mmap. You can do that with my small shm_malloc library, which allows you to use calls to shm_malloc and shm_free exactly as you would use malloc/free.
I'm very new to Memory Mapped Files, and I'm a little lost on something.
I know that if I had a file, I could load it and access it from various processes at once using MMaps.
But in my situation, I'm creating a DLL attached to Process A, and that DLL has been given a pointer to a cSurface which Process A has prepared. I need to share that cSurface's data with Process B. I really don't want to have to call up a blank MMap and copy my Process A's surface into it, only to copy it out again in process B.
Is it possible to map my surface as if it were a file so the MMap already points to the surface data when it's created (as it would were I loading SomeTextFile.txt)?
My plan, in theory, would be to receive a pointer to the surface in Proc A, tell windows to share that surface's memory with a given name, and use Mutexes to coordinate access - the idea being that both processes read the same physical copy of the surface with no cumbersome copying.
Is that possible?
Yes, it's all there in the documentation of CreateFileMapping.
You can give your memory mapped file a name. If another process open a mmap with the same name it will point to the same memory. It the handle in CreateFileMapping is put to INVALID_HANDLE_VALUE it keeps it purely in memory. Check the documentation of CreateFileMapping
Thanks for all your comments.
I did some further research and found the answer.
You can use Memory Mapped Files to share either a file, or a blank memory space. If you want to share data already initialised in memory prior to setting up the map, you have to create a blank map and subsequently copy your data into it.
Can any one look into this and suggest me with an API.
We have APIs for a process which can create and/or attach a shared memory to its own process. But I don't find an API to attach a shared memory to one process by other process(for e.g., process A should call one API(like shmat()) to attach the shared memory to process B).
Shared memory doesn't belong to any particular process (unless you create it with a private IPC_PRIVATE key). It belongs to the system.
So, when you use shmget with a non-private key (and the IPC_CREAT flag), you will either create a shared memory block or attach to an existing one.
You need a way for both processes to use the same IPC key and this is often done by using ftok which uses a file specification and an identifier to give you an IPC key for use in the shmget call (and other IPC type calls, such as msgget or semget).
For example, in the programs pax1 and pax2, you may have a code segment like:
int getMyShMem (void) {
key_t mykey = ftok ("/var/pax.cfg", 0); // only one shm block so use id of 0
if (mykey == (key_t)-1) // no go.
return -1;
return shmget (mykey, 1024, IPC_CREAT); // get (or make) a 1K block.
}
By having both processes use the same file specification and ID, they'll get the same shared memory block.
You can use different IDs to give you distinct shared memory blocks all based on the same file (you may, for example, want one for a configuration shared memory block and another for storing shared state).
And, given that it's your configuration file the IPC key is based on, the chances of other programs using it is minuscule (I think it may be zero but I'm not 100% sure).
You can't forcefully inject shared memory into a process from outside that process (well, you may be able to but it would be both dangerous and require all sorts of root-level permissions). That would break the protected process model and turn you system into something about as secure as MS-DOS :-)
Let's see, allow one process to force a shared memory segment on to another? What is the receiver going to do with it? How will it know it now has mapped this block in - what is expected of it.
You're thinking about the problem the wrong way - simply hoisting a block of memory on to a second process is not going to allow you to do what you want. You need to notify the second process also that it has now mapped this block and so can start doing stuff with it. I suggest you take a step back and really look at your design and what you are doing. My recommended approach would be
A connects to B via some other IPC (say socket)
A informs B that it should attach with the details (name etc.)
B then attaches - and now B is aware of it and can start doing stuff with it. (say for example once the attach completes, B confirms to A, and then they can start talking over the shared memory block).
As for wrapping shared memory in a nice library - consider boost::interprocess.
You are asking to attach the process memory of other process, right?
Just open(2) the file /proc/<pid>/mem and use it. Check the /proc/<pid>/map for the list of usable address in the file.
I currently try to implement some interprocess communication using the Windows CreateFileMapping mechanism. I know that I need to create a file mapping object with CreateFileMapping first and then create a pointer to the actual data with MapViewOfFile. The example then puts data into the mapfile by using CopyMemory.
In my application I have an image buffer (1 MB large) which I want to send to another process. So now I inquire a pointer to the image and then copy the whole image buffer into the mapfile. But I wonder if this is really necessary. Isn't it possible to just copy an actual pointer in the shared memory which points to the image buffer data? I tried a bit but didn't succeed.
Different processes have different address spaces. If you pass a valid pointer in one process to another process, it will probably point to random data in the second process. So you will have to copy all the data.
I strongly recommend you use Boost::interprocess. It has lots of goodies to manage this kind of stuff & even includes some special Windows-only functions in case you need to interoperate w/ other processes that use particular Win32 features.
The most important thing is to use offset pointers rather than regular pointers. Offset pointers are basically relative pointers (they store the difference between where the pointer is and where the thing pointed to is). This means that even if the two pointers are mapped to different address spaces, as long as the mappings are identical in structure then you are fine.
I've used all kinds of complicated data structures with offset smart pointers and it worked like a charm.
Shared Memory doesn't mean sending and receiving of Data. Its a memory created for number of processes without violation. For that you have to follow some mechanisms like locks so that the data will not corrupt.
In process 1 :
CreateFileMapping() : It will create the Shared Memory Block, with the name provided in last parameter, if it is not already present and returns back a handle (you may call it a pointer), if successful.
MapViewOfFile() : It maps (includes) this shared block in the process address space and returns a handle (again u can say a pointer).
With this pointer returned by MapViewOfFile() only you can access that shared block.
In process 2 :
OpenFileMapping() : If the shared memory block is successfully created by CreateFileMapping(), you can use it with the same name (name used to create the shared memory block).
UnmapViewOfFile() : It will unmap (you can remove the shared memory block from that process address space). When you are done using the shared memory (i.e. access, modification etc) call this function .
Closehandle() : finally to detach the shared memory block from process , call this with argument,handle returned by OpenFileMapping() or CreateFileMapping().
Though these functions look simple, the behaviour is tricky if the flags are not selected properly.
If you wish to read or write shared memory, specify PAGE_EXECUTE_READWRITE in CreateFileMapping().
Whenever you wish to access shared memory after creating it successfully, use FILE_MAP_ALL_ACCESS in MapViewOfFile().
It is better to specify FALSE (do not inherit handle from parent process) in OpenFileMapping() as it will avoid confusion.
You CAN get shared memory to use the same address over 2 processes for Windows. It's achieveable with several techniques.
Using MapViewOfFileEx, here's the significant experpt from MSDN.
If a suggested mapping address is
supplied, the file is mapped at the
specified address (rounded down to the
nearest 64K-boundary) if there is
enough address space at the specified
address. If there is not enough
address space, the function fails.
Typically, the suggested address is
used to specify that a file should be
mapped at the same address in multiple
processes. This requires the region of
address space to be available in all
involved processes. No other memory
allocation can take place in the
region that is used for mapping,
including the use of the VirtualAlloc
or VirtualAllocEx function to reserve
memory.
If the lpBaseAddress parameter
specifies a base offset, the function
succeeds if the specified memory
region is not already in use by the
calling process. The system does not
ensure that the same memory region is
available for the memory mapped file
in other 32-bit processes.
Another related technique is to use a DLL with a section marked Read + Write + Shared. In this case, the OS will pretty much do the MapViewOfFileEx call for you and for any other process which loads the DLL.
You may have to mark your DLL to a FIXED load address, not relocateable etc.. naturally.
You can use Marshalling of pointers.
If it's possible, it would be best to have the image data loaded/generated directly into the shared memory area. This eliminates the memory copy and puts it directly where it needs to be. When it's ready you can signal the other process, giving it the offset into your shared memory where the data begins.