Windows supports LockFile function for applying exclusive locks on a file, and the LockFileEx functions for applying an exclusive locks and shared read for all processes.
Is it possible to have ProcessA apply an exclusive lock, and then allow Only ProcessA And ProcessB to read the file, while at the time, no other process - outside those two processes - would have read access on the locked file?
According to the Doc：LockFileEx function
Exclusive locks cannot overlap an existing locked region of a file.
Shared locks can overlap a locked region provided locks held on that
region are shared locks. A shared lock can overlap an exclusive lock
if both locks were created using the same file handle. When a shared
lock overlaps an exclusive lock, the only possible access is a read by
the owner of the locks.
In my opinion, Exclusive lock doesn't allow read and write operations.Shared lock allows only read operation.
If the file is set Exclusive lock with LockFileEx function, all other processes will be denied both read and write access.
If the file is set Shared lockwith LockFileEx function, all processes can read the locked region.
If the file is set shared lock and exclusive lock at the same time, the only possible access is a read by the owner of the locks.
Related
How is C++ mutex implemented under the hood ? Does it only use Decker's, Peterson's or other algorithms to enforce mutual exclusion or does it also use hardware support such as interrupt ad compare-and-swap (CAS).
Is it possible to implement an entire multi-threading library consisting of mutex and condition variable without any hardware support ?
Standard requires that there are atomic operations. Not necessarily CAS, but at least exchange. std::atomic_flag is required to be true atomic, although CAS is superfluous for it, simple exchange is satisfactory.
Note that any algorithm such as Decker's, Peterson's or other would at least require atomic load and atomic store, and they will not work if load and store are non-atomic. Also non-atomic types do not enforce memory ordering, which is implied by those algorithms.
It is not really required that std::mutex will be based on operating system calls, and that there are operating system calls at all.
In theory, std::mutex could be spinning only mutex.
It could also be blocking only mutex.
In practice a good implementation would first try to handle blocking with atomic, but when it is resorted to wait, it would do OS wait.
It asks the operating system to do it.
On Linux that'll probably be using pthreads (How does pthread implemented in linux kernel 3.2?).
On Windows it's with the Windows API. You can try asking Microsoft about that one.
The standard library implementation won't be making direct hardware access. This is what operating systems are for.
On Linux, when the mutex is uncontended, the lock and unlock happen in user-space only using atomic instructions, such as compare-and-swap. In a contended case a call into the kernel is required to wait on the mutex till it's been unlocked (on lock) or to wake up waiters (on unlock). Locking user-space mutexes does not disable interrupts.
Here is the source code for low-level mutex primitives in glibc, the comments are instructive:
/* Low-level locks use a combination of atomic operations (to acquire and
release lock ownership) and futex operations (to block until the state
of a lock changes). A lock can be in one of three states:
0: not acquired,
1: acquired with no waiters; no other threads are blocked or about to block
for changes to the lock state,
>1: acquired, possibly with waiters; there may be other threads blocked or
about to block for changes to the lock state.
We expect that the common case is an uncontended lock, so we just need
to transition the lock between states 0 and 1; releasing the lock does
not need to wake any other blocked threads. If the lock is contended
and a thread decides to block using a futex operation, then this thread
needs to first change the state to >1; if this state is observed during
lock release, the releasing thread will wake one of the potentially
blocked threads.
Much of this code takes a 'private' parameter. This may be:
LLL_PRIVATE: lock only shared within a process
LLL_SHARED: lock may be shared across processes.
Condition variables contain an optimization for broadcasts that requeues
waiting threads on a lock's futex. Therefore, there is a special
variant of the locks (whose name contains "cond") that makes sure to
always set the lock state to >1 and not just 1.
Robust locks set the lock to the id of the owner. This allows detection
of the case where the owner exits without releasing the lock. Flags are
OR'd with the owner id to record additional information about lock state.
Therefore the states of robust locks are:
0: not acquired
id: acquired (by user identified by id & FUTEX_TID_MASK)
The following flags may be set in the robust lock value:
FUTEX_WAITERS - possibly has waiters
FUTEX_OWNER_DIED - owning user has exited without releasing the futex. */
And pthread_mutex_lock code.
Mutex implementation requires multiple threads to agree on a value of a shared variable - the mutex, whether it is locked or unlocked. This is also known as consensus problem and it is not possible to solve it using only atomic loads and stores, a compare-and-swap is required.
I used boost::interprocess to create a boost::multi_index data structure in shared memory. There are many client processes that will access this data structure. When accessing, I will lock the data structure. The problem I encountered is Once the client process is accessing the data structure and crashes without releasing the occupied lock, then all other client processes cannot access the data structure. I use boost::interprocess::named_mutex, I Know that boost::interprocess::file_lock can be automatically released when the process crashes, but because he has a lot of restrictions, so I have no use, I don't know if there is any good way to solve this problem, thank you!
Do not place a mutex in shared memory. The boost documentation for named_mutex says:
https://www.boost.org/doc/libs/1_70_0/doc/html/boost/interprocess/named_mutex.html
A mutex with a global name, so it can be found from different processes. This mutex can't be placed in shared memory, and each process should have it's own named_mutex.
The whole point of using a named mutex is that multiple processes can create their own local mutex objects using the same name and they will share an underlying mutex that they can sync on. If a given process locks the mutex and then crashes, the underlying shared mutex will be released automatically by the OS, allowing another process to lock it (depending on OS, the underlying mutex API may report that the mutex had been unlocked abnormally).
I guess you can try to access the mutex with timed_lock and, if you get a timeout, forcibly delete the mutex with remove.
The boost documentation for upgradable and shared locks says that when shared locks are held only one other thread may acquire an upgradable lock. So that if other threads try to acquire an upgradable lock when shared lock(s) are held along with the upgradable lock they will block.
Is there some deadlock possibility that I am missing when more than one thread acquire upgradable locks along with one (or more than one shared lock)? Or is this just a logical requirement (so a "should not do this" sort of thing)?
Note that I am not talking about exclusively locked states. Only an upgradeable locked state. If an upgradable lock is held along with other shared locks it is in essence a READ lock. Then why can't two upgradable locks be held together?
Is there some deadlock possibility that I am missing when more than one thread acquire upgradable locks
TL;DR Yes, there is.
along with one (or more than one shared lock)
This doesn't really affect the deadlock possibility. Allowing shared locks while upgradeable lock exists is simply a feature of upgradeable locks, compared to an exclusive lock.
Let us first consider what upgradeable locks can be used for. We shall imagine the following situation:
Several writer threads must check a condition (a read operation), then depending on that, modify the state
Checking the condition is expensive
The condition is satisfied rarely.
Other threads also read the state.
Now, let us consider that we only have reader (shared) / writer (exclusive) locks, with no upgradeable lock:
Writer takes an exclusive lock, and starts checking the condition
Other threads must block while the expensive check operation is running - even if they only need to read.
It may be considered a disadvantage that the read portion of the check - write cycle blocks even reading threads. So, let us consider an alternative:
Writer takes a shared lock, and starts checking the condition
Other threads may also take shared locks
Writer has checked the condition, and releases the read lock
The condition was satisfied, and writer now tries to take an exclusive lock to proceed
Between 3. and 4. more than one writer may have finished checking the state - since they can check concurrently - and are now racing to take the exclusive lock. Only one can win, and others must block. While they are blocking, the winner is modifying the state.
In this situation, the writers waiting to grab the exclusive lock can not assume that the condition that they checked is valid any more! Another writer may have grabbed the lock before them, and the state has now been modified. They could ignore this, which might lead to undefined behaviour, depending on what the relation with the condition and the modification is. Or, they could check the condition again when they get the exclusive lock, which reverts us back to the first approach, except with potentially redundant checks that were useless because of the race. Either way, this approach is worse than the first one.
A solution for the situation above is the privileged read (upgradeable) lock:
Writer takes a privileged lock, and starts checking the condition
Other threads may take shared locks
Writer has checked the condition which was satisfied, and upgrades to exclusive lock (must block until other locks have been released)
Let us consider a situation where multiple writers have been granted a privileged lock. They get to check the expensive condition concurrently, which is nice, but they still have a race to upgrade the lock. This time, the race results in a dead lock, because each writer hold the read lock, and wait for all read locks to be released before they can upgrade.
If the upgradeable lock is exclusive in relation to other upgradeable locks, then this dead lock cannot occur, and the race doesn't exist between the expensive check and the modification, but reader threads may still operate while the writer is checking.
I seem to be having an issue with boost::interprocess::file_lock
I have process 1 that is essentially
boost::interprocess::file_lock test_lock("testfile.csv");
test_lock.lock();
sleep(1000);
test_lock.unlock();
When I run a second process while the first process is sleeping, I find that I am still able to read testfile.csv. What's worse, I can even overwrite it.
Am I misinterpreting how file_lock works? I was under the impression that calling .lock() gives it exclusive lock over the file and prevents any other process from read/modifying the file.
file_lock is not for locking a file. It is a mutual exclusion object that uses a file as its backing technology. The contents of the file are basically irrelevant; what is relevant is that all instances of a file_lock pointing to that file will respect the locking characteristics of the lock.
As with any mutex-type object, the lock itself is there to protect or otherwise meter access to some other resource.
It has nothing to do with filesystem protection of files.
Reference
To ensure portability the kind of lock you are expecting doesn't exist in boost. There is no kernel level enforceable lock on the Unix/Linux/OSX OS's that boost supports.
see:
http://www.boost.org/doc/libs/1_51_0/doc/html/interprocess/synchronization_mechanisms.html#interprocess.synchronization_mechanisms.file_lock
Because of this the boost interprocess lock is an advisory or cooperative lock. You can accomplish what you are trying to do with boost::interprocess::file_lock but, you must also use the interprocess::file_lock in other processes that might try to read/write the file. Simply try to acquire the lock in your other process before accessing the file.
This is how interprocess::file_lock was designed to be used. You can do it some OS specific way that enforces kernel level locking if you are on some OS's, but if you use the boost::interprocess::file_lock, your code will be portable.
Its not a filesystem lock.
Simply locking in one process is not going to prevent another process from accessing the file. Think of it "like" a mutex. To have your second process honor the file_lock, you need to acquire the lock in the 2nd process as well. Then you will see that proc2 will block, waiting for proc1 to release the lock.
Here's the thing: I have two applications, written in C++ and running on two machines with different OS (one Linux and one Windows). One of this process is in charge of updating an XML file on a NAS (Network Attached Storage) while the other one reads this file.
Is it possible to synchronize these two processes in order to avoid reading of the file at the same time it's being modified?
You could create a lock file on the server that is created before you do a write, wait then write and delete on completion., Have the read process check for the token before reading the file.
Edit: To address the comments, you can implement a double-checked locking type pattern. Have both reader and writer have a locking file and double check before you do work, something like:
Reader: Check for write lock file, create read lock file, check for write lock file, if exists delete read file and abort.
Writer: Check for read lock file, create write lock file, check for read lock file, if exists delete write lock file and abort.
This will stop your processes trampling on each other but a potential race condition may occur in that the you could potentially have both processes check, create then recheck simultaneously though this will not cause the data to be read in an inconsistent state but will cause both read and write processes to abort for your specified delay
Thank you all for your answers.
At last we managed to resolve our problem, not by using locking commands of the OS (because we were not sure they would propagate correctly to the OS of the NAS head), but by creating lock directories instead of lock files. Directory creation is an atomic operation, and returns an error value if the folder already exists. Therefore, we don't have to check the lock existence before acquiring it, both operations are made in a single step.
OK you need some form of locking mechanism to control accesses.
Most *nix File systems provide this. I suspect it is also available on Windows File System (as this mechanism is used by perl) but it may have another name.
Take a look at the flock().
This is a file locking mechanism. It is an advisory lock so it does not actually lock the file and prevent usage but it provides a mechanism for marking the file. If both applications use the mechanism then you can control accesses to the file.
flock() provides both shared locks (or READ Lock) and exclusive locks (or WRITE Lock). flock will block your thread (in a non busy way) until the file has been unlocked by the user (it also provides NON blocking checks so you can do other things while waiting).
Check out flock in section 2 of the man pages.
int flock(int fd, int operation);
Flock() applies or removes an advisory lock on the file associated with the file
descriptor fd. A lock is applied by specifying an operation parameter that is
one of LOCK_SH or LOCK_EX with the optional addition of LOCK_NB. To unlock an
existing lock operation should be LOCK_UN.
If the files reside on an NFS share you can use fcntl(2) to lock the file. Check question D10 in the Linux NFS FAQ. I have very little experience with windows APIs but from what I've heard they have good POSIX support so you should be able to use fcntl as long as they support POSIX.1-2001.
If you are accessing the files using different protocols (i.e. AFS or SMB) maybe you could set up a simple synchronization server that manages locks via an IPC interface?
Would it be possible to switch from files to a database?
This type of concurency is something that DBMSs manage very well. It need no be expensive or difficult to install. MySql, Postgress or JavaDB would all handle this elegantly at little or no cost.
Failing the database option I would have the writing process write to a "hidden" file name like ".updateinprogress.xml" and rename the file when the update is complete. On most systems "mv" or "ren" is an atomic operation so the reading process either picks up hte old file or the newer file but never a half written one.