I've started playing with mmap. I'm trying to create an example workspace that will be then extended to the real case.
This is what I want to achieve:
PROCESS 1:
mmap a file (actually a device, but it's okay to generate an example with a text file)
PROCESS 2: (not foked from process 1; just an independent process)
read the memory mapped by process 1
change some bits
write it to a new file
I've read several examples and documentations, but I still didn't find how to achieve this. What I'm missing is:
how can process 2 access the memory mapped by process 1, without knowing anything about the opened file?
how can I put the mmap content in a new file? I suppose I have to ftruncate a new file, mmap this file and memcpy the content of process 1 memory map to process 2 memory map (then msync)
Side info, I have a message queue opened between the two processes, so they can share some messages if needed (ex. the memory address/size, ...).
Any hints?
Thanks in advance!
MIX
This answer considers you are trying to do this stuff on linux/unix.
how can process 2 access the memory mapped by process 1, without knowing anything about the opened file?
Process 1 passes to mmap[1] the flag MAP_SHARED.
You can:
A) Share the file descriptor using unix domain sockets[2].
B) Send
the name of the file using the queues you mentioned at the end of
your message.
Process 2 opens mmap with the flag MAP_SHARED. Modifications to the mmaped memory in Process 1 will be visible for Process 2. If you need fine control of when the changes from process 1 are shown to process 2 you should control it with msync[3]
how can I put the mmap content in a new file? I suppose I have to
ftruncate a new file, mmap this file and memcpy the content of process
1 memory map to process 2 memory map (then msync)
Why just don't write the mmaped memory as regular memory with write?
[1]http://man7.org/linux/man-pages/man2/mmap.2.html
[2]Portable way to pass file descriptor between different processes
[3]http://man7.org/linux/man-pages/man2/msync.2.html
Related
Sorry but I didn't find clear answer to my question,
I know that each file has its own seek_ptr, let's suppose the main process opened connection to file_A then before doing anything called fork()
Then forked process reads 2 chars, which is correct?
will seek_ptr be equal to 2 for both files?
seek_ptr be equal to 2 for the child process and still 0 for main process?
Only if the answer is 1:
How can I open 2 files in notepad and each file has its indicator/cursor in different locations?
In Unix, (pid, fd) acts as a pointer into the kernel's table of open file descriptions. When a process is forked, the child process will have a different PID, call it pid2. So (pid2, fd) is a different key from (pid, fd). However, these two pointers actually point to the same open file description: fork does not fork the open file descriptions themselves. Therefore, they share a single offset. If one process seeks, it affects the other process as well. If one process reads, it affects the other process as well.
However, either process is free to call close to dissociate fd from the existing open file description, then call open to create a new open file description which may refer to the same file. After this is done, the two processes will have different open file descriptions, and seeking in one does not affect the other.
Each successful call to open always creates a new open file description.
I have a memory-mapped file on disk in my application. My application creates multiple threads and all of them are memory mapping the same file at the same time or different time. In one of the threads, I am calling unmapViewOfFile followed by SetFilePointer and SetEndOfFile. SetEndOfFile call fails with error ERROR_USER_MAPPED_FILE. This links suggests this is a known problem that it requires to unmap all views of a mapped file. How to check if a file is still mapped in another thread/process and how to unmap it from all the threads/processes?
Problem- Multiple processes want to update a file simultaneously.I do not want to use file locking functionality as highly loaded environment a process may block for a while which i don't want. I want something like all process send data to queue or some shared place or something else and one master process will keep on taking data from there and write to the file.So that no process will get block.
One possibility using socket programming.All the processes will send data to to single port and master keep on listening this single port and store data to file.But what if master got down for few seconds.if it happen than i may write to some file based on timestamp and than later sync.But i am putting this on hold and looking for some other solution.(No data lose)
Another possibility may be tacking lock for the particular segment of the file on which the process want to write.Basically each process will write a line.I am not sure how good it will be for high loaded system.
Please suggest some solution for this problem.
Have a 0mq instance handle the writes (as you initially proposed for the socket) and have the workers connect to it and add their writes to the queue (example in many languages).
Each process can write to own file (pid.temp) and periodically rename file (pid-0.data, pid-1.data, ...) for master process that can grab all this files.
You may not need to construct something like this. If you do not want to get processes blocked just use the LOCK_NB flag of perl flock. Periodically try to flock. If not succeeds continue the processing and the values can stored in an array. If file locked, write the data to it from the array.
fopen/fwrite and multi-threading?
Some multi-threading programs open the same file, each thread create a file pointer to that the file.
There is one thread created by a paricular program that will update the file at some random time, whilst other threads, created by a different program, will simply read the contents of the file.
I guess this create a racing/data-inconsistence problem there if the writing thread change contents in the file whilst other threads try to read the contents.
The problem here is the thread that update the file should compiled into a different exe program than the the program that creates threads that read the contents of the file, so within-program level thread control become impossible.
My solution is create a very small "flag" file on the harddisk to indicates 3 status of the file:
1) writing-thread is updating the contents of the file;
2) reading-thread are reading the contents of the file;
3) Neither 1) or 2);
Using this flag file to block threads whenever necessary.
Are there some more-compact/neat solution to this problem?
It might be easier to use a process-global "named" semaphore that all the processes know about. Plus then you could use thread/process-blocking semaphore mechanisms instead of spin-looping on file-open-close and file contents...
I am currently developing a modular framework using shared memory in C & C++.
The goal is to have independent programs in both C and C++, talk to each other through shared memory.
E.g. one program is responsible for reading a GPS and another responsible for processing the data from several sensors.
A master program will start all the slave programs
(currently i am using fp = popen(./slave1/slave1,"r"); to do this) and then make shared memory segments that each slave can connect to.
The thought behind this is that if a slave dies, it can be revived by the master and reconnect to the same shared memory segment.
Slaves can also be exchanged during runtime (e.g. switch one GPS with another).
The problem is that I spawn the slave via popen, and pass the shared memory ID to the slave. Via the pipe the slave transmits back the size needed.
After this is done i want to reroute the slave's pipe to terminal to display debug messages and not pass through the master.
Suggestions are greatly appreciated, as well as other solutions to the issue.
The key is to have some form of communication prior to setting up the shared memory.
I suggest to use another mean to communicate. Named pipe are the way I think. Rerouting standard out/err will be tricky at best.
I suggest to use boost.interprocess to handle IPC. And be attentive to synchronization :)
my2c
You may want to look into the SCM_RIGHTS transfer mode of unix domain sockets - this lets you pass a file descriptor across a local socket. This can be used to pass stderr and the like to your slave processes.
You can also pass shared memory segments as a file descriptor as well (at least on Linux). Create a file with a random name in /dev/shm and unlink it immediately. Now ftruncate to the desired size and mmap with MAP_SHARED. Now you have a shared memory segment tied to a file descriptor. One of the nice things about this approach is the shared memory segment is automatically destroyed if all processes holding it terminate.
Putting this together, you can pass your slaves one end of a unix domain socketpair(). Just leave the fd open over exec, and pass the fd number on the command line. Pass whatever configuration information you want as normal data over the socket pair, then hand over a file descriptor pointing to your shared memory segment.
Pipes are not reroutable -- they go where they go when they were created. What you need to do is have the slave close the pipe when its done with it, and then reopen its stdout elsewhere. If you always want output to the terminal, you can use freopen("/dev/tty", "w", stdout), but then it will always go to the terminal -- you can't redirect it anywhere else.
To address the specific issue, send debug messages to stderr, rather than stdout.