I seem to be having an issue with boost::interprocess::file_lock
I have process 1 that is essentially
boost::interprocess::file_lock test_lock("testfile.csv");
test_lock.lock();
sleep(1000);
test_lock.unlock();
When I run a second process while the first process is sleeping, I find that I am still able to read testfile.csv. What's worse, I can even overwrite it.
Am I misinterpreting how file_lock works? I was under the impression that calling .lock() gives it exclusive lock over the file and prevents any other process from read/modifying the file.
file_lock is not for locking a file. It is a mutual exclusion object that uses a file as its backing technology. The contents of the file are basically irrelevant; what is relevant is that all instances of a file_lock pointing to that file will respect the locking characteristics of the lock.
As with any mutex-type object, the lock itself is there to protect or otherwise meter access to some other resource.
It has nothing to do with filesystem protection of files.
Reference
To ensure portability the kind of lock you are expecting doesn't exist in boost. There is no kernel level enforceable lock on the Unix/Linux/OSX OS's that boost supports.
see:
http://www.boost.org/doc/libs/1_51_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.file_lock
Because of this the boost interprocess lock is an advisory or cooperative lock. You can accomplish what you are trying to do with boost::interprocess::file_lock but, you must also use the interprocess::file_lock in other processes that might try to read/write the file. Simply try to acquire the lock in your other process before accessing the file.
This is how interprocess::file_lock was designed to be used. You can do it some OS specific way that enforces kernel level locking if you are on some OS's, but if you use the boost::interprocess::file_lock, your code will be portable.
Its not a filesystem lock.
Simply locking in one process is not going to prevent another process from accessing the file. Think of it "like" a mutex. To have your second process honor the file_lock, you need to acquire the lock in the 2nd process as well. Then you will see that proc2 will block, waiting for proc1 to release the lock.
Related
I need to use read-write lock, but I would also like to be sure that the lock is recursive in both read and write mode, so that calling function that uses it on the same thread doesn't lock.
I found nothing on MSDN docs. Maybe it's hidden somewhere and I didn't find it?
SRWLocks are not recursive, from the docs:
An SRW lock is the size of a pointer. The advantage is that it is fast to update the lock state. The disadvantage is that very little state information can be stored, so SRW locks cannot be acquired recursively. In addition, a thread that owns an SRW lock in shared mode cannot upgrade its ownership of the lock to exclusive mode.
I have files residing in different directories as follows:
\201603\30
file_1
file_2
\201603\31
file_3
file_4
file_5
There are 2 writers & 2 readers and they are all threads in my program.
Lock will be applied on directory not individual files (e.g. \201603\30 or \201603\31).
I would like to use boost library to make my program portable.
I have searched and been introduced to boost::interprocess::file_lock. I was just wondering that do I really need an 'interprocess' lock here, and is that lock sharable between readers? Do you have any suggestions other than boost::interprocess::file_lock?
Since all lock users in are in your program, what do you need interprocess locking for? file_lock puts the mutex into a file which lets the kernel manage locking/unlocking. This is only useful if multiple processes need to respect the lock.
It sounds like you need nothing more than a regular read-write lock, which Boost calls shared_lock. See this answer for an example on how to use one: https://stackoverflow.com/a/989816/321772
In summary, a shared lock lets multiple threads acquire the lock for reading, but only allows one thread to upgrade the lock to a writer (excludes all other readers/writers).
As the other responder mentioned, because only one process is using the files you don't need to lock on the filing system, a std::shared_mutex or even a plain old std::mutex should be plenty.
If however you were needing to lock efficiently across multiple processes, and it is now 2017 or later, you might consider using one of the shared_fs_mutex algorithms in the post-peer-review rewrite of proposed Boost.AFIO v2: https://ned14.github.io/boost.afio/namespaceboost_1_1afio_1_1v2_1_1algorithm_1_1shared__fs__mutex.html. The reason I caveat the year being 2017 is that AFIO v2 doesn't even have a test suite yet, it is highly alpha code and I cannot recommend anyone use it yet.
Does Windows offer any kind of mutex that can be placed in a memory mapped file and used across multiple processes?
Ideally it must be completely self contained such that it can survive by itself in the file, even across a reboot.
Also, no resources should be leaked if I simply remove the file manually while no processes are running.
If possible the solution should also offer the accompanying 'condition' concept which should also be an object that can sit in a shared memory mapped file.
In short, I need something similar to a PTHREADS mutex with the SHARED attribute.
As far as I understand, simply using a PTHREADS mutex is not possible because the SHARED attribute is unsupported in the Windows port of PTHREADS.
To share a synchronization object, give it a name and use the same name in each process when you Create the object.
The following synchronization objects can be shared between process that way :
Mutex
Semaphore
Event
Critical sections cannot be shared, but are faster.
Testing or waiting on those objects is done with the wait family of functions, often WaitForMultipleObjects.
Use the file as its own mutex: Use the LockFileEx function and have everybody agree to lock byte 0 of the file when they want to claim the mutex.
That's not possible. The mutex object itself lives in kernel space to protect it from user code messing with its state. The handle you acquired to it is only valid for the process that acquired it. Technically you could use DuplicateHandle() and put the returned handle in the mmf, but only if you have a handle to the other process that accesses the memory section. That's fairly brittle.
This is why you can specify a name for the mutex in the CreateMutex() function. The other process gets to it by using the same name in the OpenMutex call.
Could you help me to understand how to use mutexes in multithread Linux application, where:
during data writing it is need to lock variable on write and read
during data reading from the variable it is need to lock it on write.
So it is possible to read simultaneously, but writing opertion is a single opertaion in the same time. During writing, all other operation should wait before it finishes.
You're asking about something that is a bit higher level than mutexes. A mutex is a simple, low-level device. When you lock a thread with a mutex, the CPU is either executing code in the thread that obtained the lock or it is executing some other process entirely. In other words, the mutex has locked out all other threads that belong to the same (heavyweight) process.
You are asking about a read-write lock. Read-write locks use mutexes underneath the hood. The POSIX functions that deal with read-write locks start with pthread_rwlock_. Since you are on a Linux machine, just type man pthread and look for the section marked "READ/WRITE LOCK ROUTINES".
You need a reader/writer lock to allow multiple readers/single writer.
Boost.Thread has one of these (boost::shared_mutex), if you have no other preferred threading library. This uses PThreads primitives under the covers, and will probably save you time in wrapping the raw APIs yourself.
I would not recommend implementing this yourself - it's easy to get something that appears to work, but under load either crashes or kills performance or (worst of all) silently modifies your data in a way it should not be, so you get bad results.
A simple boost::mutex can also be used here as noted by #Als, but won't allow multiple concurrent reads. That is simpler to implement, and may be sufficient for your needs, depending on your read/write access profile.
You will need to use mutexes, if you have global or static objects which are being accessed(read and written to) from different threads.
Here's the thing: I have two applications, written in C++ and running on two machines with different OS (one Linux and one Windows). One of this process is in charge of updating an XML file on a NAS (Network Attached Storage) while the other one reads this file.
Is it possible to synchronize these two processes in order to avoid reading of the file at the same time it's being modified?
You could create a lock file on the server that is created before you do a write, wait then write and delete on completion., Have the read process check for the token before reading the file.
Edit: To address the comments, you can implement a double-checked locking type pattern. Have both reader and writer have a locking file and double check before you do work, something like:
Reader: Check for write lock file, create read lock file, check for write lock file, if exists delete read file and abort.
Writer: Check for read lock file, create write lock file, check for read lock file, if exists delete write lock file and abort.
This will stop your processes trampling on each other but a potential race condition may occur in that the you could potentially have both processes check, create then recheck simultaneously though this will not cause the data to be read in an inconsistent state but will cause both read and write processes to abort for your specified delay
Thank you all for your answers.
At last we managed to resolve our problem, not by using locking commands of the OS (because we were not sure they would propagate correctly to the OS of the NAS head), but by creating lock directories instead of lock files. Directory creation is an atomic operation, and returns an error value if the folder already exists. Therefore, we don't have to check the lock existence before acquiring it, both operations are made in a single step.
OK you need some form of locking mechanism to control accesses.
Most *nix File systems provide this. I suspect it is also available on Windows File System (as this mechanism is used by perl) but it may have another name.
Take a look at the flock().
This is a file locking mechanism. It is an advisory lock so it does not actually lock the file and prevent usage but it provides a mechanism for marking the file. If both applications use the mechanism then you can control accesses to the file.
flock() provides both shared locks (or READ Lock) and exclusive locks (or WRITE Lock). flock will block your thread (in a non busy way) until the file has been unlocked by the user (it also provides NON blocking checks so you can do other things while waiting).
Check out flock in section 2 of the man pages.
int flock(int fd, int operation);
Flock() applies or removes an advisory lock on the file associated with the file
descriptor fd. A lock is applied by specifying an operation parameter that is
one of LOCK_SH or LOCK_EX with the optional addition of LOCK_NB. To unlock an
existing lock operation should be LOCK_UN.
If the files reside on an NFS share you can use fcntl(2) to lock the file. Check question D10 in the Linux NFS FAQ. I have very little experience with windows APIs but from what I've heard they have good POSIX support so you should be able to use fcntl as long as they support POSIX.1-2001.
If you are accessing the files using different protocols (i.e. AFS or SMB) maybe you could set up a simple synchronization server that manages locks via an IPC interface?
Would it be possible to switch from files to a database?
This type of concurency is something that DBMSs manage very well. It need no be expensive or difficult to install. MySql, Postgress or JavaDB would all handle this elegantly at little or no cost.
Failing the database option I would have the writing process write to a "hidden" file name like ".updateinprogress.xml" and rename the file when the update is complete. On most systems "mv" or "ren" is an atomic operation so the reading process either picks up hte old file or the newer file but never a half written one.